Ask anyone for memories from childhood, and everyone will readily produce a handful of stories like this:
My neighbor's father died when I was four. I remember sitting with my friend in front of their house, watching people come and go. It was strange. No one said anything.
It's hard to distinguish memories from memories of memories. Indeed, there's little evidence that any of our adult memories really go way back to infancy; what seem like early memories may be nothing more than reconstructions of our older thoughts. For one thing, recollections from our first five years seem strangely isolated; if we ask what happened earlier that day, the answer almost always is, I can't remember that. Furthermore, many of those early memories involve incidents so significant that they probably occupied the child's mind repeatedly over a period of years. Most suspicious of all is the fact that such recollections are frequently described as seen through other, older eyes — with the narrator portrayed inside the scene, right near the center of the stage. Since we never actually see ourselves, these must be reconstructed memories, rehearsed and reformulated since infancy.
I suspect that this amnesia of infancy is no mere effect of decay over time but an inevitable result of growing out of infancy. A memory is not a separate entity, apart from how it works upon the mind. To remember an early experience, you must be able not only to retrieve some old records, but to reconstruct how your earlier mind reacted to them — and to do that, you would have to become an infant again. To outgrow infancy, you have to sacrifice your memories because they're written in an ancient script that your later selves can no longer read.
We reconstruct our recent memories as well, since they portray less what we saw than what we recognized. From every moment to the next, your mental state is shaped not only by signals from the outer world, but by agents activated by the memories these evoke. For example, when you see a chair, what makes it appear to you to be a chair — rather than an assortment of sticks and boards? It must evoke some memories. Only a part of your impression comes from agents activated directly by your vision; most of what your higher-level agencies experience comes from the memories those vision-agents activate. Usually, we have no conscious sense of this happening, and we never use words like memory or remembering when the process works quickly and quietly; instead, we speak of seeing or recognizing or knowing. This is because such processes leave too few traces for the rest of the mind to contemplate; accordingly, such processes are unconscious, because consciousness requires short-term memory. It is only when a recognition involves substantial time and effort that we speak of remembering.
Then what do we mean by memory? Our brains use many different ways to store the traces of our pasts. No single word can describe so much, unless it is used only in a general, informal sense.
Memories are processes that make some of our agents act in much the same ways they did at various times in the past.