Here's what I think: The Doctor remembers every Doctor Who story ever told. Every episode, target book, comic strip and every game of companions and TARDISes you played as a kid. The universe he lives in has no record of it, because paradoxes and divergent dimensions and the time war have reset things. But the Doctor remembers and sometimes when he is sad it's because you've stopped being 8 years old and he can't run around the school playground with you anymore.
(And the comments proceed to kick around the Timey-Wimey Calvinball.)