{"title": "Replay, Repair and Consolidation", "book": "Advances in Neural Information Processing Systems", "page_first": 19, "page_last": 26, "abstract": null, "full_text": "Replay, Repair and Consolidation\n\nSzabolcs K\u00b4ali\n\nInstitute of Experimental Medicine\nHungarian Academy of Sciences\n\nBudapest 1450, Hungary\n\nkali@koki.hu\n\nPeter Dayan\n\nGatsby Computational Neuroscience Unit\n\nUniversity College London\n\n17 Queen Square, London WC1N 3AR, U.K.\n\ndayan@gatsby.ucl.ac.uk\n\nAbstract\n\nA standard view of memory consolidation is that episodes are stored tem-\nporarily in the hippocampus, and are transferred to the neocortex through\nreplay. Various recent experimental challenges to the idea of transfer,\nparticularly for human memory, are forcing its re-evaluation. However,\nalthough there is independent neurophysiological evidence for replay,\nshort of transfer, there are few theoretical ideas for what it might be\ndoing. We suggest and demonstrate two important computational roles\nassociated with neocortical indices.\n\n1 Introduction\n\nParticularly since the analysis of subject HM,1 the suggestion that human memories would\nconsolidate,2 has gripped experimental and theoretical communities. The idea is that stor-\nage of some sorts of knowledge (notably declarative information) involves a two-stage\nprocess, with memories moving from an initial, temporary, home (usually taken to be the\nhippocampus), which offers fast acting, but short-lived, plasticity, into a \ufb01nal, permanent\nresting place (usually the neocortex), whose learning and forgetting are much slower.\nVarious sources of evidence have been adduced in favor of this proposition. First, it has\nbeen suggested that for patients (or animal subjects) who have suffered insults to the hip-\npocampus, recent memories are more compromised than older ones, suggesting that they\nhave yet to be consolidated to cortex. 3, 4 Second, the same patients suffer from anterograde\namnesia (that is, they cannot lay down new memories), even though many neocortical areas\nare palpably functioning, and procedural storage (including aversive conditioning and skill\nlearning) works (more) normally. 5 Third, starting with the seminal work of Marr, 6 who\n(possibly by a mis-calculation7) suggested that the hippocampus was just large enough a\ndynamic RAM as to store one day\u2019s events, a variety of theoretical treatments has suggested\nthe possible characteristics and advantages of two-stage procedures. 8\u201310 This is widely re-\ngarded as reaching its apogee in the work of McClelland et al,11 who performed a careful\ncomputational analysis of fast and slow learning in connectionist networks. Fourth, and\nperhaps most compelling, an obvious substrate for replay to cortex is provided by the neu-\nrophysiologically observed12\u201314 reactivation during slow wave and REM sleep of patterns\nof (rat) hippocampal neuronal \ufb01ring observed during times when the subject is awake and\nbehaving, together with evidence of at least some coordination between hippocampal and\nneocortical states during this reactivation.15\nThe \ufb01rst and third of these evidentiary foundations are currently under active debate, spe-\ncially for episodic memories (ie autobiographical memories for happenings). Solid evi-\ndence that hippocampal damage really spares memories for distant events compared with\n\n\fthose for recent ones is extremely sparse, and the relevance of infra-human studies is put\ninto question by the orders-of-magnitude differences in the memory time-scales shown be-\ntween humans and animals.16 The modeling studies are also more ambiguous than they\nmight seem, since their most convincing focus is on the tribulations of catastrophic inter-\nference.17 That is, slow learning is necessary in systems with rich distributed or population\ncoding because changes in synaptic ef\ufb01cacies occasioned by incorporating new informa-\ntion can easily overwrite the neural substrate for the storage of old information (the hoary\nstability-plasticity dilemma18). This catastrophic interference can be avoided by re-storing\nold patterns (or something equivalent 10, 19) at the same time as storing new information.\nThus, according to these schemes, patterns are stored wholesale in the hippocampus when\nthey \ufb01rst appear, and are continually read back to cortex to cause plasticity along with the\nnew information. However, if the hippocampus is permanently required to prevent a catas-\ntrophe, then, \ufb01rst, there is no true consolidation: if neocortical plasticity is not inhibited\nby hippocampal damage,20 then its integrity is permanently required to prevent degrada-\ntion; and, second, what is the point of consolidation \u2013 couldn\u2019t the hippocampus suf\ufb01ce by\nitself? This is particularly compelling in the case of episodes, since they are intrinsically\nisolated events. We came to a realization of this through development of our own model for\nconsolidation,21 whose behavior convinced us of a \ufb02aw in our thinking. This second point\nlies exactly at the heart of the perspective espoused by Nadel and Moscovitch, 16 amongst\nothers. They regard the hippocampus as the \ufb01nal point of storage for all episodic memory,\nand permanently required for its recall. Of course, this idea equally well accounts for the\nsecond strand of evidence above about anterograde amnesia.\nIf the hippocampus stores patterns permanently, what could the point be of replay? Here,\nwe consider two roles, both associated with concerns about the pattern matching process at\nthe heart of retrieval from the hippocampus. One is a new take on catastrophic interference,\narguing that replay is necessary to keep the patterns stored in the hippocampus in register\nwith the evolving cortical representation, so that they can still be recalled (and interpreted)\ncorrectly even though the cortical code may have changed since they were stored. The other\ncomputational role for replay is a new take on indexing, arguing that the cortical patterns\nthat should lead to retrieval of a hippocampal memory are not only close syntactic relatives\nof the pattern that was originally stored, ie patterns whose actual neural code is similar,\nbut also patterns that are close semantic relatives, ie patterns that are closely related via the\nnetwork of semantic relationships that is stored in neocortex. In this scheme, the role of\nreplay is building an index to the memory, effectively a form of recognition model. 22\nWe \ufb01rst discuss brie\ufb02y our existing model of consolidation, 21 and its failings. Section 3\ntreats the repair of hippocampal indexing in the light of the vicissitudes of semantic change.\nSection 4 sketches our account of the semantic elaboration of the index.\n\n2 Semantic and Episodic Memory\n\nFigure 1 shows our existing account of the interaction between the neocortex and the hip-\npocampus in semantic and episodic memory. 21 The neocortex is separated into \u2018lower\u2019\nareas (\u0002\u0001\u0004\u0003\u0005\u0002\u0006\u0007\u0003\t\b\n\b\t\b ) which are connected via bi-directional, variable, weights \u000b\nwith an\n), and collectively act as a restricted Boltzmann\nentorhinal/parahippocampal (EP) area (\nmachine (RBM), trained in an unsupervised manner, using contrastive divergence. 23\nIt\nlearns a model of the statistical relationships amongst the inputs, so that it can produce\n\u000b\u0016\u0015 . The con-\nventional interpretation for this is as a model of semantic memory \u2013 the generic facts of\nthe world, stripped of information about the time and place and other circumstances under\nwhich they were learnt. However, the individual patterns on which the semantic learning\nis based are treated as episodic patterns, which should be recalled wholesale. One main\ncontribution of that work was to put episodic and semantic information into such particular\ncorrespondence.\n\nsamples from conditional probability distributions such as \r\u000f\u000e\n\n\u0011\u0010\u0013\u0012\n\n\u0002\u0001\u0004\u0003\u0005\u0002\u0006\u0007\u0014\n\n\f\n\fA\n\nWA\n\n\u0001\u0001\n\u0002\u0001\u0002\n\u0001\u0001\n\u0002\u0001\u0002\n\n\t\u0001\t\n\n\u0001\n\n\t\u0001\t\n\n\u0001\n\n\u0003\u0001\u0003\n\u0007\u0001\u0007\n\u0005\u0001\u0005\u0001\u0005\n\u0004\u0001\u0004\n\u0006\u0001\u0006\n\b\u0001\b\n\u0003\u0001\u0003\n\u0007\u0001\u0007\n\u0005\u0001\u0005\u0001\u0005\n\u0004\u0001\u0004\n\u0006\u0001\u0006\n\b\u0001\b\nx AA\n\nHC\n\nB\n\n\u001f\u0001\u001f\n \u0001 \n\u001f\u0001\u001f\n \u0001 \n\n!\u0001!\u0001!\n\"\u0001\"\u0001\"\n!\u0001!\u0001!\n\"\u0001\"\u0001\"\n\n#\u0001#\n$\u0001$\n#\u0001#\n$\u0001$\n\n%\u0001%\n&\u0001&\n%\u0001%\n&\u0001&\n\n'\u0001'\n(\u0001(\n'\u0001'\n(\u0001(\n\ny E/P\n\nWC\n\nC\n\n100\n\nd\ne\n\nl\nl\n\na\nc\ne\nr\n \nt\n\nn\ne\nc\nr\ne\nP\n\n80\n\n60\n\n40\n\n20\n\n0\n0\n\n100\n\n\u000b\u0001\u000b\n\f\u0001\f\n\u000b\u0001\u000b\n\f\u0001\f\n\n\u0001\r\u0001\r\n\u000e\u0001\u000e\u0001\u000e\n\r\u0001\r\u0001\r\n\u000e\u0001\u000e\u0001\u000e\n\n\u000f\u0001\u000f\n\u0010\u0001\u0010\n\u000f\u0001\u000f\n\u0010\u0001\u0010\n\n\u0011\u0001\u0011\n\u0012\u0001\u0012\n\u0011\u0001\u0011\n\u0012\u0001\u0012\nxBB\n\n\u0013\u0001\u0013\n\u0014\u0001\u0014\n\u0013\u0001\u0013\n\u0014\u0001\u0014\n\n\u0015\u0001\u0015\u0001\u0015\n\u0016\u0001\u0016\u0001\u0016\n\u0015\u0001\u0015\u0001\u0015\n\u0016\u0001\u0016\u0001\u0016\n\n\u001d\u0001\u001d\u0001\u001d\n\u001e\u0001\u001e\u0001\u001e\n\u001d\u0001\u001d\u0001\u001d\n\u001e\u0001\u001e\u0001\u001e\n\n\u0017\u0001\u0017\n\u001b\u0001\u001b\n\u0019\u0001\u0019\u0001\u0019\n\u0018\u0001\u0018\n\u001a\u0001\u001a\n\u001c\u0001\u001c\n\u0017\u0001\u0017\n\u001b\u0001\u001b\n\u0019\u0001\u0019\u0001\u0019\n\u0018\u0001\u0018\n\u001a\u0001\u001a\n\u001c\u0001\u001c\nxCC\n\nd\ne\n\nl\nl\n\na\nc\ne\nr\n \nt\nn\ne\nc\nr\ne\nP\n\n200\n\n400\n\n600\n\n800\n\nTime (thousand presentations)\n\none\u2212shot\nconsolidated\n\n20\n\n40\n\nTime (thousand presentations)\n\n60\n\n80\n\n100\n\n80\n\n60\n\n40\n\n20\n\n0\n0\n\nFigure 1: (A) Model architecture. All units in neocortical areas A, B, and C are connected to all\nunits in area E/P through bidirectional, symmetric weights, but connections between units in the\ninput layer are restricted to the same cortical area. Each neocortical area contains 100 binary units.\nThe hippocampus (HC) is not directly implemented, but it can in\ufb02uence and store the patterns in\nEP. All communication between the HC and the input areas is via area EP. (B) The consolidation of\nepisodic memories. Recall performance on speci\ufb01c (episodic) patterns as a function of time between\nthe initial presentation of the episodic pattern and testing (or, equivalently, time between training and\nlesion in hippocampals) in the simulations. (C) Extinction of an episode due to semantic training,\nin the isolated neocortical network trained to asymptotic performance on the episodic pattern (thin\nline), and directly after the removal of the hippocampus from the full network, for a pattern which\nhas been hippocampally \u201cconsolidated\u201d for 250,000 presentations (thick line).\n\nIn this previous model, the hippocampus acts as a fast-learning repository for the EP repre-\nsentation of patterns that have been (relatively recently) experienced, and plays two roles:\naiding recall and training the neocortex. The hippocampus improves recall by performing\npattern completion on the EP representations induced by partial or noisy inputs \n, thus \ufb01nd-\n. In turn, this, through neocortical semantic knowledge,\ning the nearest matching stored\nengenders recall of an appropriate \n. The hippocampus trains the neocortex in an off-line\n(sleep) mode, reporting the patterns that it has stored to the neocortex to give the latter\u2019s\nincremental plasticity the opportunity to absorb the new information. Given hippocam-\npal damage, patterns that have been repeatedly replayed to cortex by the hippocampus (ie\nolder patterns) have a greater chance of being recalled correctly through neocortical infer-\nence than patterns that were learned more recently, and are therefore still dependent for\ntheir recall on the integrity of the hippocampus.\nFigure 1B shows the basic consolidation phenomenon in this model. The upper (thin) curve\nshows how well on average the full model can recall whole items from a partial cue as a\nfunction of time since the item was stored; the lower (thick) curve shows the same in the\ncase that the hippocampal contribution is eliminated immediately before testing. This is\nthe standard inverted U-shaped curve of graded retrograde amnesia, with distant memo-\nries spared compared with recent ones. However, \ufb01gure 1C reveals what is really going\non. Both curves show how the neocortical network forgets particular episodic patterns as\na function of continued semantic training. Thick/thin lines are with/without prior consol-\nidation using the hippocampus. Consolidation clearly does not help the longevity of the\nmemory \u2013 if anything, it actually impedes it. This is essentially because the cortical code\nchanges slowly over presentations. Thus, \ufb01rst, the hippocampus is mandatorily required if\nmemories are to be preserved \u2013 the forgetting curve for the normals in \ufb01gure 1B is actually\n\n\f\n\fdominated by hippocampal forgetting. Second, the inverted U-shaped curve in \ufb01gure 1B\narises because testing happens immediately after hippocampal removal. The same curves\nplotted for successive times after removal would show catastrophic memory failure.\nMemories might turn out to be stabilized in the face of hippocampal damage in other\nways.21 For instance, cortical plasticity might be suppressed, if the hippocampus reports\nunfamiliarity as a plasticizing signal. This is somewhat unlikely, since various forms of\ncontinued plasticity remain active.3, 20 Alternatively, there might be synaptic stabilizing\nmechanisms in the cortex such that synapses come never to change. This is certainly pos-\nsible, but does not explain how recall can survive changes in the cortical code.\nIn sum, the model turns out to illustrate the key problem with standard theory of memory\ntransfer for episodes. We are thus forced to start from the possibility that the hippocampus\nmight indeed be a permanent repository, and reconsider the issue of replay and consoli-\ndation in the resulting light. In this new scheme, there is still a critical role for replay,\nbut one that is focused on the indexing relationship between neocortical and hippocampal\nrepresentations rather than on writing into cortex the contents of the hippocampus.\n\n3 Maintaining Access to Episodes\n\nConsider the fate of an episode that is stored in the hippocampus. In a hierarchical network\nwhere the hippocampus is directly connected only to the topmost areas, successful recall of\nsuch an episode depends on the correspondence between low- and high-level cortical areas\nembodied by the neocortical network. This dependence actually has two related compo-\nnents. First, the high-level neocortical representation of the recall cue needs to be effective\nin activating the correct hippocampal memory trace; second, the high-level representation\nactivated by hippocampal recall should effect the recall of the appropriate components of\nthe corresponding episode in lower level areas as well. These are both aspects of indexing.\nThe neocortical network is the substrate of neocortical learning, re\ufb02ecting, for instance,\nre\ufb01nement of the existing semantic representation, changes in input statistics, or acquisi-\ntion of a new semantic domain. Such plasticity may disrupt the recall of stored episodic\npatterns by changing the correspondence between the input areas and EP. Thus, if the brain\nis still to be able to recall hippocampally stored episodes, it either needs to maintain the\ncorrespondence between the low-level and EP representations of the episodes by restricting\nneocortical learning (achieved in the previous model by having the hippocampus replay its\nold episodic patterns along with the new semantic patterns governing continued neocortical\nplasticity), or it needs to update the connections between the hippocampus and EP such that\nthe hippocampally stored pattern continues to match the EP representation of the input pat-\ntern corresponding to the episode. The \ufb01rst of these possibilities may restrict the learning\nabilities of the neocortical network. However, replay can be used to allow the connections\ninto and out of the hippocampus to track the changing neocortical representational code.\nIn order to assess the effect of neocortical learning on the recall of previously stored\nepisodes, either in the presence or absence of replay, the following paradigm was em-\nployed. We started training the neocortical network by presenting to the input areas ran-\ndom combinations of valid patterns (20 independently generated random binary patterns\nfor each area). After a moderate amount of such general training (10,000 pattern presen-\ntations total), the EP representations of  particular input patterns were associated with\ncorresponding stored hippocampal traces, forming a set of stored episodes. The quality of\nrecall for these episodes was then monitored while general training continued. Figure 2A\nshows as a function of the length of general semantic training the percentage of correct re-\ncall for the episodes stored after 10,000 presentations. The main plot is an average over all\n episodes; the smaller plots show some individual episodes. Clearly, neocortical learning\ncomes to erase the route to recall, even though the episode remains perfectly stored in the\nhippocampus throughout.\n\n\fA\n\nB\n\nD\n\n100\n\nd\ne\n\nl\nl\n\na\nc\ne\nr\n \nt\n\nn\ne\nc\nr\ne\nP\n\n80\n\n60\n\n40\n\n20\n\n0\n0\n\n50\n\n100\n\n150\n\n200\n\nTime (thousand presentations)\n\nd\ne\nr\no\nt\ns\n \nm\no\nr\nf\n \ne\nc\nn\na\nt\ns\nD\n\ni\n\n20\n\n15\n\n10\n\n5\n\n0\n0\n\n100\n\n50\n\nd\ne\n\nl\nl\n\na\nc\ne\nr\n \nt\n\nn\ne\nc\nr\ne\nP\n\n200\n\n100\n\nC\n\n0\n0\n200\nTime (thousand presentations)\n\n100\n\n100\n\nd\ne\n\nl\nl\n\na\nc\ne\nr\n \nt\n\nn\ne\nc\nr\ne\nP\n\n80\n\n60\n\n40\n\n20\n\n0\n0\n\n50\n\n100\n\n150\n\n200\n\nTime (thousand presentations)\n\nFigure 2: How semantic training affects episodic recall for patterns stored after the \ufb01rst 10,000\npresentations (A) without replay and (D) with the correspondence between hippocampal and neo-\ncortical representations updated during off-line replay. The larger graphs are averages over all stored\nepisodes, while the smaller graphs are for individual episodes. Recall was assessed by presenting par-\ntial episodic patterns (the original activations replaced by random patterns in one of the input areas),\nperforming hippocampal pattern completion in EP if the distance from a stored EP representation\nwas less than 20, and then performing 20 full iterations of Gibbs sampling in the neocortical network\nwith the cue areas clamped. A resulting distance of less than 5 from the target pattern was considered\na match. (B) and (C) analyze the reasons why episodic recall breaks down in (A). (B) shows how the\nEP representation of stored episodes drifts away from the original stored patterns. (C) shows how\nwell recall works if it starts from the stored EP representation of the episode.\n\nFigure 2B,C indicate the reasons for this behavior. Figure 2B shows that semantic learning\nafter the storage of the episode causes the EP representation of the episode to move away\nfrom the version with which the stored hippocampal trace is associated. The magnitude\nof this change is such that, eventually, even the full original episode may fail to activate\nthe corresponding hippocampal memory trace. The effect of representational change on\nhippocampally directed recall in the input areas is milder in our case, as seen in Figure 2C;\nprovided that the correct hippocampal trace does get activated, the full episode can be\nsuccessfully recalled most of the time. However, this component accounts for the relatively\nslower initial rise of episodic recall in Figure 2A (compare with Figure 2D), as well as\nsome of the variability between patterns in Figure 2A (data not shown).\nIn the \u201creplay\u201d condition, the general training was interleaved with epochs of hippocam-\npally initiated replay, assumed to take place during sleep. Within these epochs, the memory\ntraces stored in the hippocampus get activated at random, which leads to the reactivation of\nthe associated EP pattern, which in turn reactivates the input areas according to the existing\nsemantic mapping. The resulting pattern may be different from the one that initially gave\nrise to the stored episode, due to subsequent changes in the neocortical connections. How-\never, assuming that the neocortical semantic representation has not changed fundamentally\nsince the last time that particular episode was replayed (or when it was established), the\ninput representation resulting from replay should be close to the current low level repre-\nsentation of that particular episode. Indeed, maintaining this representational proximity\nexactly sets the requirement for the frequency of replay of the episodes.\nAs in our previous model, we assume that the local connections within each neocortical\narea implement a local attractor structure, which, in the absence of feedforward activation,\nrestricts activity patterns within that area to those that correspond to valid input patterns.\nThese local attractors turn feedback activation which is close to a valid pattern (namely, the\noriginal episode) into an exact version of that pattern. Such an off-line reconstruction of\nthe low-level representation of stored episodes may then support a wide variety of memory\nprocesses (including the previous model\u2019s focus on gradually incorporating the information\ncarried by that episode into the neocortical knowledge base 11, 21). Here we focus on its\nuse for maintenance of the episodic index. To this end, starting from the reconstructed\n\n\fepisode, the semantic correspondence between the different levels is employed in the feed-\nforward direction in order to determine the up-to-date EP representation of the episode.\nThis EP pattern is then associated with the stored hippocampal episode which initiated the\nreplay, so that the hippocampal and input level representations of the episode are again in\nregister. Figure 2B demonstrates the ef\ufb01cacy of replay: the hippocampally stored episode\nnow remains tied to the (shifting) EP representation of the episode, and episodic replay\nstays at high levels despite substantial changes in the neocortical network.\n\n4 Index Extension\n\nAnother important potential role for replay is extending the semantic aspects of the in-\ndexing scheme. It should be possible to retrieve episodic memories on the basis of all\ninput patterns to which they are closely related through the network of cortical semantic\nknowledge. At present, this can happen only if the cortex produces similar EP codes for\nall those input patterns that are semantically related. However, requiring that all semantic\nproximity be coded by syntactic proximity in essentially one single layer, is far too strin-\ngent a requirement. Rather, we should expect that the bulk of semantic information lives\nin synapses that are invisible to this layer, ie connections within and between lower layers,\nand this information must also in\ufb02uence indexing.\nOne way to extend semantic indexing involves on-line sampling. That is, probabilistic\nupdating in the cortical semantic network starting from a given input pattern is the canonical\nway of exploring the semantic neighborhood of an input. One can imagine doing this in a\non-line manner, spurred by an input. Over sampling, the cortical pattern and its EP code\nchange together, providing the opportunity for a match to be made between the EP activity\nand the contents of episodic memory. These sampling dynamics would allow the recall of\nsemantically relevant episodes, even if their explicit code is rather distant.\nThe role for replay in this process is to allow the semantic index to be extended through\noff-line rather than on-line sampling starting from the episodic patterns stored in the hip-\npocampus. It is thus analogous to Sutton\u2019s 24 use of replay in his DYNA architecture, in\nwhich an internal model of a Markov decision process is used to erase inconsistencies\nin a learned value function, and also to the wake-sleep algorithm\u2019s 22 use of sleep sam-\npling to learn a recognition model. For the latter, off-line sampling ensures that inputs can\nbe mapped using a feedforward network, into codes associated with a generative model,\nrather than relying on sluggish statistical or dynamical methods for inverting the generative\nmodel, such as Gibbs sampling or its mean-\ufb01eld approximations. The main requirement\nis for a further plastic layer between EP and CA3 (presumably the perforant path) so that\nwhen replay based on an episode leads to a semantically, but not syntactically, related pat-\ntern, then the EP code for that pattern can induce hippocampal recall of the episode.\nFigure 3 illustrates this use of replay in a highly simpli\ufb01ed case (subject to the limita-\ntions of the RBM). Here, there are 3 modules of\nunits, each with\npossible patterns,\n\u0015\u0011\u0005\u0012\u0007\n\u0015\u0006\u0005\b\u0007\n\u0015\u0011\u0005\nand a semantic structure such that\n\u0002\u0006\n\u0004\u000e\r\u0010\u000f\n\u0004\u0013\r\u0015\u0014\n\r\u000f\u000e\n\r\u000f\u000e\n\u0015\u0017\u0005\u0018\u0007\nindependent of the choice\n\u000f ) and \u0002\u0010\n\u0002\u0001\n\u0002\u0006\n\u0019\b\u001a\n\u0004\u0013\r\u0015\u0016\n\r\u000f\u000e\n\u0007\f\u0007 EP units\nin \u0002\u0001 and \u0002\u0006\n. Figure 3A shows the covariance matrix of the activities of the\npossible input patterns (arranged lexicographically). The relatedness of the EP\nto the\nrepresentation of related patterns is clear in the rich structure of this matrix \u2013 this shows the\nextent of the explicit code learnt by the RBM. However, this code does not make indexing\nperfect. Imagine that\n\u001f have been stored as\n\u0002\u0010\n\u0002\u0001\nepisodic patterns. That is, their EP representations are stored in the hippocampus and are\navailable for recall and replay. We may expect to retrieve\nfrom its semantic relation\n\u001f . Figure 3B shows the explicit proximity (inverse square distance, see\n\u0003\u0005\u0002\u0010\n\u0002\u0001\n\u001c$#\ncaption) of the EP representations of the\n\u000f .\nAlthough\nis close, so are many other patterns that are not nearly so closely semantically\nrelated. For instance, \u001d\n\n\u0002\u0001\n\u000b (with wrap-around, so, eg, \n\ninput patterns to the EP representation of\n\n\u000f \u001f and\n\nare closer.\n\n\u0002\u0001\n\n\u0003\u0005\u0002\u0006\n\n\u0014\u0007\u0003\n\n\u0002\u0010\n\nand \u001d\n\n\u0002\u0001\n\b\n\t\f\u000b\n\n\u0003\u0005\u0002\u0006\n\n\u0005!\u001d\n\n\u0005\u001e\u001d\n\n\u000f\u000e\n\n\u0005%\u001d\n\n\u0002\u0006\n\n\u0002\u0001\n\n\u0002\u0006\n\n\u0002\u0001\n\n\u0001\u001b\u0003\n\n\u0002\u0006\n\n\u0003\u0005\u0002\u0006\n\n\u0003\u0005\u0002\u0010\n\n\u001f&\u001a\n\n\u0002\u0001\n\n\u0002\u0006\n\n\u0002\u0010\n\n\u001f&\u001a\n\n*\u0003+\n\n\u0001\f\u0003\n(')\n\n\u0003\n\n\u0006\n\u0004\n\u0012\n\u0004\n\u0014\n\u0012\n\u0004\n\b\n\n\u000b\n\u0014\n\u0012\n\u0004\n\u0012\n\u0004\n\b\n\u0007\n\u0006\n\n\u001c\n\u000f\n\u000f\n\u001c\n\u0014\n\u0016\n\u0003\n\"\n\u0003\n\"\n\u001c\n\u000f\n\u000f\n\u000f\n\u000f\n\u001c\n\u001c\n#\n\n\u0001\n\u000f\n\u0016\n\u000f\n\u000f\n\u0003\n\"\n\u0003\n\u000f\n\fB\n\n111\n\n131\n\n141\n\n122\n\n221\n\n321\n\n421\n\n121\n\n10\n\n20\n\n100 samples\n\n30\n\nfailures\n\n40\n\nE2\n\n50\n\n344\n\n60\n\n500 samples\n\n114 124\n\n334\n\n332\n\n444\n\n441\n\n20\n40\n60\n121\n\n20 40 60\n\n111\n\n221\n\n1.5\n1\n0.5\n0\n\u22120.5\n\nr\na\ne\nn\n\ni\nl\n\ny\nt\ni\n\ni\n\nm\nx\no\nr\np\n0\nE1\n\nA\n\n200\nC\n0\n100\n\n0\n100\n\n0\nD\n\n114\n\n111\n\n121\n\n211\n\n221\n\n223\n\n331\n\n441\n\n2000 samples\n\nlog scaled\nproximities\n\n324\n\n234\n\n334 344\n\n444\n\n434\n\n0\n\n10\n\n20\n\n30\n\n40\n\n50\n\n60\n\n0\n\n10\n\n20\n\n30\n\n40\n\n50\n\n60\n\n,\n\n\u000e\u001c\u000f\u0011\n\u0013\u0015\n\n\u2013\n\ndenote the\n\n\u000e\u0010\u000f\u0011\n\u0013\u0012\n\n\u000e\u0014\u000f\u0011\n\r\u0015\n\n\u000e\u0017\u0016\n\n\u0003\u0005\u0007\u0006\t\b\u000b\n\r\f\n\n\u0005\u000b\u0018\u0019\u0006\u001a\b\u000b\n\u001b\f\n\n\u001d ) of the EP representations (\n\n\u0001\u0003\u0002\nshows patterns that are not within Hamming distance of\n\nFigure 3: Index expansion. Plots relate to the 3-module network. Conventions:\n\u0001\u0004\u0002\n,\npossible input patterns or their EP representations.\n\u001d\u001c\u0016\n\u000e\u0017\u000f\u0011\n\r\u0012\netc. In (C), the entry for\nof any input\npattern. For this simulation, for reasons of simulation time, the input patterns were chosen to be\northogonal; the hidden unit representations were nevertheless highly non-orthogonal;\niterations of\nGibbs sampling were used during RBM learning. The weights associated with the network are not\nover-trained. A) The covariance matrix of the EP representation of the\npossible patterns. The\nbanding shows the semantic structure (see text), but, as seen in (B), only weakly. B) The proximities\n(\n#\r)+*\n\u000b \"!$#\r%'&(#\u0013)+*\u0005!\n). The numbers refer to the patterns as in the convention described. Despite\nis blank; see boxed\n\u000b\u0018-\nis\nthe covariance structure in (A), the syntactic representation of semantic closeness is weak:\n.\u0018/\n, for instance. Thus, episodic recall would be imperfect. Ratio of max-min\nnot closely related to\n\u0002-\u0018-\ntimes each)\nproximity (bar\n\u0018\u0005\u001f\u0004\u001e\n(right column).\nfrom the hippocampally replayed EP representations of\npossible input patterns,\nHere, we determine to which (if any \u2014 thus the \u2018failure\u2019 entry\nthe sampled activities of the visible units are closest, and plot histograms of the resulting frequencies.\nAfter only few iterations,\nand\ndelta-rule learning for the mapping from EP representations of the patterns in (C) to\nrespectively. Now, the remapped EP representations of semantically relevant inputs are vastly closer\nto their associated episodic memories. Ratios of max-min proximities are 14000 (\n).\n\n\u0003\u0005\n. D) Logarithmically scaled proximities following\n\n) is 4. C) Three stages of (unclamped) Gibbs sampling starting (\n\nstill dominate; after more, the semantically close patterns\n\n) for all the patterns to that for\n\n(left column) and\n\n(the entry for\n\ndominate for\n\n) and 7000 (\n\nand\n\n0\u00050\u0004\u0002\n\n) of the\n\n\u000b\u0018-\nand\n\n0\u0004\u0002\u0005\u0002\nand\n\nfor\n\n\u0001\u0003\u0002\n\n\u0001\u0004\u0002\n\n\u0018\u0003\u0018/\n\n\u0002\u0005\u0002\u0005\u0002\n\n\u000b\u0018-\n\nand\n\n#\u0017%\n\n\u0007\f\u0007\n\n\u0007\f\u0007\n\n\u0007\f\u0007\n\nrounds of Gibbs sampling starting (1\n(left) and\n\nFigure 3C shows the course of replay. The two columns show histograms of the patterns\nretrieved in the visible layer after\n\u000321\ntimes) from the hippocampal representation of\n(right). The network has\nlearnt much about the semantic relationships, although it is far from perfect (over-training\nseems to make it worse, for reasons we do not understand), and equally likely patterns are\nnot generated exactly equally often. 21 The \u0007 columns of these histograms show how many\nvalid inputs; this happens only rarely.\nsampled visible patterns are not close to one of the\nDuring replay, the EP representation of these semantically-related patterns is then available\nso that a model mapping EP to an appropriate input to the hippocampal pattern matching\nprocess can be learnt. Figure 3D shows how this affects the proximities for a model trained\n\u0014 ; now the semantic\nusing the delta rule. Again, left and right columns are for\nassociates of these patterns are mapped into inputs to the hippocampal pattern matching\nprocess that are far nearer (note the logarithmic scale) to the stored representations of\nand\n\u0014 , and so the episodes can be appropriately retrieved from their semantic cousins.\n\n\u000f and\n\n\u0001\u001b\u0003\n\n5 Discussion\n\nThe important, but narrow, issue of whether episodic memories can ever be recalled with-\nout the hippocampus has polarized theoretical ratiocination about memory replay, a phe-\n\n\n\u001e\n\n\u001f\n,\n\u000e\n,\n\u000e\n,\n\u001d\n\u001e\n,\n\u000e\n,\n\u001d\n,\n\u000e\n,\n\u001d\n,\n\u000e\n,\n\u001d\n\n\u0003\n\u000b\n\u0007\n\u000b\n\u0007\n\u001c\n\u000f\n\u001c\n\u0014\n\u001c\n\u001c\n\u001c\n\u000f\n\u001c\n\fnomenon for which there is increasing neurophysiological evidence. This polarization has\nhindered the \ufb01eld from studying the wider computational context of replay. In this paper,\nwe have considered two particular aspects of the consolidation of the indexing relationship\nbetween semantic memory (in the neocortex) and episodic memory (in the hippocampus).\nWe showed how replay could be used to maintain the index in the face of on-going neocor-\ntical plasticity, and to broaden it in the light of neocortical semantic knowledge that is not\ndirectly accessible through the explicit code in the upper layers of cortex. Unlike memory\nconsolidation, neither of these involves neocortical plasticity during replay. There may yet\nbe many other computations that can be accomplished through replay.\nBroadening the index poses an interesting, only incompletely answered, theoretical ques-\ntion about the metrics of memory. The semantic model can be seen as a sort of manifold\nin the space of all inputs; the episodes as particular points on the manifold; and retrieval\nas \ufb01nding the closest episodes to a presented cue, according to a distance function that\ninvolves mapping the cue to the manifold, and mapping between points on the manifold.\nDespite some theoretical suggestions,25 it is not clear how the semantic model speci\ufb01es\nthese distances. Our pragmatic solution was to replay the episodes and rely on the tran-\nsience of the Markov chain induced by Gibbs sampling to produce semantic cousins with\nwhich it should be related. It would be desirable to consider more systematic approaches.\nOur model involves interaction between a hippocampal store for episodes and a neocortical\nstore for semantics. However, the computational issues about indexing apply with the\nsame force if the episodes are actually stored separately elsewhere, such as in more frontal\nstructures (McClelland, personal communication). There are equal opportunities for these\nareas to induce replay, and thus improve the index. What now seems unlikely, despite our\nbest earlier efforts, is that the problems of indexing can be circumvented by storing the\nepisodes wholly within the semantic network. By itself, this solves nothing.\n\nAcknowledgements\n\nWe are very grateful to Jay McClelland for helpful discussions. Funding was from the\nHungarian Academy of Sciences and the Gatsby Charitable Foundation.\n\nReferences\n[1] W. Scoville and B. Milner, J Neurol Neurosurg Psychiatry 20, 11 (1957).\n[2] T. Ribot, Les maladies de la memoire, Appleton-Century-Crofts, New York, 1881.\n[3] L. R. Squire, Psychol Rev 99, 195 (1992).\n[4] L. R. Squire, R. E. Clark, and B. J. Knowlton, Hippocampus 11, 50 (2001).\n[5] A. R. Mayes and J. J. Downes, Memory 5, 3 (1997).\n[6] D. Marr, Philos Trans R Soc Lond B Biol Sci 262, 23 (1971).\n[7] D. J. Willshaw and J. T. Buckingham, Philos Trans R Soc Lond B Biol Sci 329, 205 (1990).\n[8] P. Alvarez and L. R. Squire, Proc Natl Acad Sci U S A 91, 7041 (1994).\n[9] J. M. Murre, Memory 5, 213 (1997).\n[10] R. M. French, Connection Science 9, 353 (1997).\n[11] J. L. McClelland, B. L. McNaughton, and R. C. O\u2019Reilly, Psychol Rev 102, 419 (1995).\n[12] M. A. Wilson and B. L. McNaughton, Science 265, 676 (1994).\n[13] W. E. Skaggs and B. L. McNaughton, Science 271, 1870 (1996).\n[14] K. Louie and M. A. Wilson, Neuron 29, 145 (2001).\n[15] A. G. Siapas and M. A. Wilson, Neuron 21, 1123 (1998).\n[16] L. Nadel and M. Moscovitch, Curr Opin Neurobiol 7, 217 (1997).\n[17] M. McCloskey and N. J. Cohen,\nAcademic Press, New York, 1989.\n\nin The psychology of learning and motivation, vol 24, edited by G. Bower, 109\u2013165,\n\n[18] G. A. Carpenter and S. Grossberg, Trends Neurosci 16, 131 (1993).\n[19] A. Robins, Connection Science 8, 259 (1996).\n[20] F. Vargha-Khadem et al., Science 277, 376 (1997).\n[21] S. K\u00b4ali and P. Dayan, in NIPS 13, edited by T. K. Leen, T. G. Dietterich, and V. Tresp, 24\u201330, MIT Press, Cambridge, 2001.\n[22] G. E. Hinton, P. Dayan, B. J. Frey, and R. M. Neal, Science 268, 1158 (1995).\n[23] G. E. Hinton, Neural Computation, 14 (2002).\n[24] R. S. Sutton, in Machine Learning: Proceedings of the Seventh International Conference, 216\u2013224, 1990.\n[25] L. K. Saul, in NIPS 9, edited by M. C. Mozer, M. I. Jordan, and T. Petsche, 267\u2013273, MIT Press, London, UK, 1997.\n\n\f", "award": [], "sourceid": 2251, "authors": [{"given_name": "Szabolcs", "family_name": "K\u00e1li", "institution": null}, {"given_name": "Peter", "family_name": "Dayan", "institution": null}]}