{"title": "Direct memory access using two cues: Finding the intersection of sets in a connectionist model", "book": "Advances in Neural Information Processing Systems", "page_first": 635, "page_last": 641, "abstract": null, "full_text": "Direct memory access using two cues: Finding \nthe intersection of sets in a connectionist model \n\nJanet Wiles, Michael S. Humphreys, John D. Bain and Simon Dennis \n\nDepartments of Psychology and Computer Science \nUniversity of Queensland QLD 4072 Australia \n\nemail: janet@psych.psy.uq.oz.au \n\nAbstract \n\nFor lack of alternative models, search and decision processes have provided the \ndominant paradigm for human memory access using two or more cues, despite \nevidence against search as an access process (Humphreys, Wiles & Bain, 1990). \nWe present an alternative process to search, based on calculating the intersection \nof sets of targets activated by two or more cues. Two methods of computing \nthe intersection are presented, one using information about the possible targets, \nthe other constraining the cue-target strengths in the memory matrix. Analysis \nusing orthogonal vectors to represent the cues and targets demonstrates the \ncompetence of both processes, and simulations using sparse distributed \nrepresentations demonstrate the performance of the latter process for tasks \ninvolving 2 and 3 cues. \n\n1 INTRODUCTION \nConsider a task in which a subject is asked to name a word that rhymes with oast. The \nsubject answers \"most\", (or post, host, toast, boast, ... ). Now the subject is asked to find \na word that means a mythical being that rhymes with oast. She or he pauses slighUy and \nreplies \"ghost\". \n\nThe difference between the first and second questions is that the first requires the use of \none cue to access memory. The second question requires the use of two cues - either \ncombining them before the access process, or combining the targets they access. There \nare many experimental paradigms in psychology in which a subject uses two or more \ncues to perform a task (Rubin & Wallace, 1989). One default assumption underlying \n\n635 \n\n\f636 Wiles, Humphreys, Bain, and Dennis \n\nmany explanations for the effective use of two cues relies on a search process through \nmemory. \n\nModels of human memory based on associative access (using connectionist models) have \nprovided an alternative paradigm to search processes for memory access using a single cue \n(Anderson, Silverstein, Ritz & Jones, 1977; McClelland & Rumelhart, 1986), and for \ntwo cues which have been studied together (Humphreys, Bain & Pike 1989). In some \nrespects, properties of these models correspond very closely to the characteristics of \nhuman memory (Rumelhart, 1989). In addition to the evidence against search processes \nfor memory access using a single cue, there is also experimental evidence against \nsequential search in some tasks requiring the combination of two cues, such as cued recall \nwith an extra-list cue, cued recall with a part-word cue, lexical access and semantic access \n(Humphreys, Wiles & Bain, 1990). Furthermore, in some of these tasks it appears that \nthe two cues have never jointly occurred with the target. In such a situation, the tensor \nproduct employed by Humphreys et. a1. to bind the two cues to the target cannot be \nemployed, nor can the co-occurrences of the two cues be encoded into the hidden layer of a \nthree-layer network. In this paper we present the computational foundation for an \nalternative process to search and decision, based on parallel (or direct) access for the \nintersection of sets of targets that are retrieved in response to cues that have not been \nstudied together. \nDefinition of an intersection in the cue-target paradigm: Given a set of cue-target pairs, \nand two (or more) access cues, then the intersection specified by the access cues is defined \nto be the set of targets which are associated with both cues. If the cue-target strengths are \nnot binary, then they are constrained to lie between 0 and 1, and targets in the intersection \nare weighted by the product of the cue-target strengths. A complementary definition for a \nunion process could be the set of targets associated with anyone or more of the access \ncues, weighted by the sum of the target strengths. \n\nIn the models that are described below, we assume that the access cues and targets are \nrepresented as vectors, the cue-target associations are represented in a memory matrix and \nthe set of targets retrieved in response to one or more cues is represented as a linear \ncombination, or blend, of target vectors associated with that cue or cues. Note that under \nthis definition, if there is more than one target in the intersection, then a second stage is \nrequired to select a unique target to output from the retrieved linear combination. We do \nnot address this second stage in this paper. \n\nA task requiring intersection: In the rhyming task described above, the rhyme and \nsemantic cues have extremely low separate probabilities of accessing the target, ghost, \nbut a very high joint probability. In this study we do not distinguish between the \nrepresentation of the semantic and part-word cues, although it would be required for a \nmore detailed model. Instead, we focus on the task of retrieving a target weakly associated \nwith two cues. We simulate this condition in a simple task using two cues, C1 and C2, \nand three targets, T1, T2 and T3. Each cue is strongly associated with one target, and \nweakly associated with a second target, as follows (strengths of association are shown \nabove the arrows): \n\n\fDirect Memory Access Using Two Cues \n\n637 \n\nThe intersection of the targets retrieved to the two cues, Cl and C2, is the target, T2, \nwith a strength of 0.01. Note that in this example, a model based on vector addition \nwould be insufficient to select target, T2, which is weakly associated with both cues, in \npreference to either target, Tl or TJ, which are strongly associated with one cue each. \n\n2 IMPLEMENTATIONS OF INTERSECTION PROCESSES \n\n2.1 LOCAL REPRESENTATIONS \n\nGiven a local representation for two sets of targets, their intersection can be computed by \nmultiplying the activations elicited by each cue. This method extends to sparse \nrepresentations with some noise from cross product terms, and has been used by Dolan \nand Dyer (1989) in their tensor model, and Touretzky and Hinton (1989) in the \nDistributed Connectionist Production System (for further discussion see Wiles, \nHumphreys, Bain & Dennis, 1990). However, multiplying activation strengths does not \nextend to fully distributed representations, since multiplication depends on the basis \nrepresentation (Le., the target patterns themselves) and the cross-product terms do not \nnecessarily cancel. One strong implication of this for implementing an intersection \nprocess, is that the choice of patterns is not critical in a linear process (such as vector \naddition) but can be critical in a non-linear process (which is necessary for computing \nintersections). An intersection process requires more information about the target patterns \nthemselves. \n\nIt is interesting to note that the inner product of the target sets (equivalent to the match \nprocess in Humphreys et. al.1s (1989) Matrix model) can be used to determine whether or \nnot the intersection of targets is empty, if the target vectors are orthogonal, although it \ncannot be used to find the particular vectors which are in the intersection. \n\n2.2 USING INFORMATION ABOUT TARGET VECfORS \n\nA local representation enables multiplication of activation strengths because there is \nimplicit knowledge about the allowable target vectors in the local representation itself. \nThe first method we describe for computing the intersection of fully distributed vectors \nuses information about the targets, explicitly represented in an auto-associative memory, \nto filter out cross-product terms: In separate operatiOns, each cue is used to access the \nmemory matrix and retrieve a composite target vector (the linear combination of \nassociated targets). A temporary matrix is formed from the outer product of these two \ncomposite vectors. This matrix will contain product terms between all the targets in the \nintersection set as well as noise in the form of cross-product terms. The cross-product \nterms can be filtered from the temporary matrix by using it as a retrieval cue for accessing \na three-dimensional auto-associator (a tensor of rank 3) over all the targets in the original \nmemory. \nIf the target vectors are orthonormal, then this process will produce a vector \nwhich contains no noise from cross-product terms, and is the linear combination of all \ntargets associated with both cues (see Box 1). \n\n\f638 Wiles, Humphreys, Bain, and Dennis \n\nBox 1. Creating a temporary matrix from the product of the target vectors, then filtering \nout the noise terms: Let the cues and targets be represented by vectors which are mutually \northonormal (Le., Ci.Ci = Ti.Ti = 1, Ci,Cj = Ti.Tj = 0, i, j = 1,2,3). The memory \nmatrix, M, is formed from cue-target pairs, weighted by their respective strengths, as \nfollows: \n\nwhere T' represents the transpose of T, and Cj T;' is the outer product of Cj and Tj \u2022 \nIn addition, let Z be a three-dimensional auto-associative memory (or tensor of rank 3) \ncreated over three orthogonal representations of each target (i.e., Tj is a column vector, T;' \nis a row vector which is the transpose of Tj , and Tj \" is the vector in a third direction \northogonal to both, where i=I,2,3), as follows: \n\nz = I\u00b7 T- T-' T\u00b7\" \n\nI \n\nI \n\nI \n\nI \n\nLet a two-dimensional temporary matrix, X, be formed by taking the outer product of \ntarget vectors retrieved to the access cues, as follows: \n\nX = \n= \n\n(Cl M) (C2 M)' \n(0.9Tl + 0.lT2) (0.1T2 + 0.9Tj )' \n\n= \n\n0.09Tl T2' + 0.81Tl Tj' + 0.01T2T2' + 0.09T2Tj' \n\nUsing the matrix X to access the auto-associator Z, will produce a vector from which all \nthe cross-product terms have been flltered, as follows: \n\nX Z = \n= \n\n(0.09Tl T2' + 0.81 TlT3' + 0.01T2T2' + 0.09T2Tj' ) (Ij Tj T;' T;\") \n\n(0.09TlT2') (Ii Ti T;' T;',) + (0.81Tl Tl) (Ii Tj T;' Ti\") \n+ (0.01 T2T2') ( Ii Tj T;' Ti \") + (0.09T2Tj') <l:i Tj T;' T;\" ) \n\nsince all other terms cancel. \n\nThis vector is the required intersection of the linear combination of target vectors \nassociated with both the input cues, Cl and C2 weighted by the product of the strengths \nof associations from the cues to the targets. \n\nA major advantage of the above process is that only matrix (or tensor) operations are used, \nwhich simplifies both the implementation and the analysis. The behaviour of the system \ncan be analysed either at the level of behaviours of patterns, or using a coordinate system \nbased on individual units, since in a linear system these two levels of description are \nisomorphic. In addition, the auto-associative target matrix could be created incrementally \nwhen the target vectors are first learnt by the system using the matrix memory. The \n\n\fDirect Memory Access Using Two Cues \n\n639 \n\ndisadvantages include the requirement for dynamic creation and short term storage of the \ntwo dimensional product-of-targets matrix, and the formation and much longer term \nstorage of the three dimensional auto-associative matrix. It is possible, however, that an \nauto-associator may be part of the output process. \n\n2.3 ADDITIVE APPROXIMATIONS TO MULTIPLICATIVE PROCESSES \nAn alternative approach to using the target auto-associator for computing the intersection, \nis to incorporate a non-linearity at the time of memory storage, rather than memory \naccess. The aim of this transform would be to change the cue-target strengths so that \nlinear addition of vectors could be used for computing the intersection. An operation that \nis equivalent to multiplication is the addition of logarithms. If the logarithm of each cue(cid:173)\ntarget strength was calculated and stored at the time of association, then an additive access \nprocess would retrieve the intersection of the inputs. More generally, it may be possible \nto use an operation that preserves the same order relations (in terms of strengths) as \nmultiplication. It is always possible to find a restricted range of association strengths \nsuch that the sum of a number of weak cue-target associations will produce a stronger \ntarget activation than the sum of a smaller number of strong cue-target associations. For \nexample, by scaling the target strengths to the range [(n-l)/n, 1] where n is the number \nof simultaneously available cues, vector addition can be made to approximate \nmultiplication of target strengths. \n\nThis method has the advantage of extending naturally to non-orthogonal vectors, and to \nthe combination of three or more cues, with performance limits determined solely by \ncross-talk between the vectors. Time taken is proportional to the number of cues, and \nnoise is proportional to the product of the set sizes and cross-correlation between the \nvectors. \n\n3 SIMULATIONS OF THE ADDITIVE PROCESS \n\nTwo simulations of the additive process using scaled target strengths were performed to \ndemonstrate the feasibility of the method producing a target weakly associated with two \ncues, in preference to targets with much higher probabilities of being produced in \nresponse to a single cue. As a work-around for the problem of how (and when) to \ndecompose the composite output vector, the target with the strongest correlation with the \ncomposite output was selected as the winner. To simulate the addition of some noise, \nnon-orthogonal vectors were used. \n\nThe first simulation involved two cues, C 1 and C2, and three targets, T1, T2 and T3, \nrepresented as randomly generated 100 dimensional vectors, 20% Is, the remainder Os. \nCue C1 was strongly associated with target T1 and weakly associated with target T2, cue \nC2 was strongly associated with target T3 and weakly associated with target T2. A trial \nconsisted of generating random cue and target vectors, forming a memory matrix from \ntheir outer products (multiplied by 0.9 for strong associates and 0.6 for weak associates; \nnote that these strengths have been scaled to the range, [0,1]), and then pre-multiplying \nthe memory matrix by the appropriate cue (i.e., either C1 or C2 or C1 + C2). \n\n\f640 Wiles, Humphreys, Bain, and Dennis \n\nThe memory matrix, M, was formed as shown in Box 1. Retrieval to a cue, Cl , was as \nfollows: C1 M = 0.9 C1 .Cz Tz' + 0.6 C1 .C1 T2 ' + 0.6 C1 .C2 T2 ' + 0.9 C1.C2 T/. In \nthis case, the cross product terms, Cl.C2, do not cancel since the vectors are not \northogonal, although their expected contribution to the output is small (expected \ncorrelation 0.04). The winning target vector was the one that had the strongest \ncorrelation (smallest normalized dot product) with the resulting output vector. The results \nare shown in Table 1. \n\nTable 1: Number of times each target was retrieved in 100 trials. \n\nc1 \nc2 \n\nc1+c2 \n\nt1 \n\n92 \no \n11 \n\nt2 \n\n8 \n9 \n80 \n\nt3 \n\no \n91 \n9 \n\nOver 100 trials, the results show that when either cue Cl or C2 was presented alone, the \ntarget with which it was most strongly paired was retrieved in over 90% of cases. Target \nT2 had very low probabilities of recall given either Cl or C2 (8% and 9% respectively), \nhowever, it was very likely to be recalled if both cues were presented (80%). \nThe first simulation demonstrated the multi-cue paradigm with the simple two-cue and \nthree-target case. In a second simulation, the system was tested for robustness in a \nsimilar case involving three cues, C 1 to C 3, and four targets, T 1 to T 4. The results \nshow that T4 had low probabilities of recall given either Cl , C2 or C3 (13%, 22% and \n18% respectively), medium probabilities of recall given a combination of two cues (36%, \n31 % and 28%), and was most likely to be recalled if all three cues were presented (44%). \nFor this task, when three cues are presented concurrently, in the ideal intersection only T4 \nshould be produced. The results show that it is produced more often than the other targets \n(44% compared with 22%, 18% and 16%), each of which is strongly associated with two \nout of the three cues, but there is considerably more noise than in the two-cue case. (See \nWiles, Humphreys, Bain & Dennis, 1990, for further details.) \n\n4 DISCUSSION \n\nThe simulation results demonstrated the effect of the initial scaling of the cue-target \nstrengths, and non-linear competition between the target outputs. It is important to note \nthe difference between the association strengths from cues to targets and the cued recall \nprobability of each target. In memory research, the association strengths have been \ntraditionally identified with the probability of recall. However, in a connectionist model \nthe association strengths are related to the weights in the network and the cued recall \nprobability is the probability of recall of a given target to a given cue. \n\nThis paper builds on the idea that direct access is the default access method for human \nmemory, and that all access processes are cue based. The immediate response from \nmemory is a blend of patterns, which provide a useful intermediate stage. Other processes \nmay act on the blend of patterns before a single target is selected for output in a \n\n\fDirect Memory Access Using Two Cues \n\n641 \n\nsuccessive stage. One such process that may act on the intermediate representation is an \nintersection process that operates over blends of targets. Such a process would provide an \nalternative to search as a computational technique in psychological paradigms that use \ntwo or more cues. We don't claim that we have described the way to implement such a \nprocess - much more is required to investigate these issues. The two methods presented \nhere have served to demonstrate that direct access intersection is a viable neural network \ntechnique. This demonstration means that more processing can be performed in the \nnetwork dynamics, rather than by the control structures that surround memory. \n\nAcknowledgements \n\nOur thanks to Anthony Bloesch, Michael Jordan, Julie Stewart, Michael Strasser and \nRoland Sussex for discussions and comments. This work was supported by grants from \nthe Australian Research Council, a National Research Fellowship to J. Wiles and an \nAustralian Postgraduate Research Award to S. Dennis. \n\nReferences \nAnderson, J.A., Silverstein, J.W., Ritz, S.A. and Jones, R.S. Distinctive features, \ncategorical perception, and probability learning: Some applications of a neural model. \nPsychological Review, 84,413-451, 1977. \n\nDolan, C. and Dyer, M.G. Parallel retrieval and application of conceptual knowledge. \nProceedings of the 1988 Connectionist Models Summer School, San Mateo, Ca: \nMorgan Kaufmann, 273-280, 1989. \nHumphreys, M.S., Bain, J.D. and Pike, R. Different ways to cue a coherent memory \nsystem: A theory for episodic, semantic and procedural tasks. Psychological Review, \n96:2, 208-233, 1989. \n\nHumphreys, M.S., Wiles, J. and Bain, J.D. Direct Access: Cues with separate histories. \nPaper presented at Attention and Performance 14, Ann Arbor, Michigan, July, 1990. \n\nMcClelland, J.L. and Rumelhart, D.E. A distributed model of memory. In McClelland, \nJ.L. and Rumelhart, D.E. (eds.) Parallel Distributed Processing: Explorations in the \nmicrostructure of cognition, 170-215, MIT Press, Cambridge, MA, 1986. \n\nRubin, D.C. and Wallace, W.T. Rhyme and reason: Analysis of dual retrieval cues. \nJournal of Experimental Psychology: Learning, Memory and Cognition, 15:4, 698-\n709, 1989. \n\nRumelhart, D.S. The architecture of mind: A connectionist approach. In Posner, M.1. \n(ed.) Foundations of Cognitive Science, 133-159, MIT Press, Cambridge, MA, 1989. \n\nTouretzky, D.S. and Hinton, G.E. A distributed connectionist production system. \nCognitive Science, 12, 423-466, 1988. \n\nWiles, J., Humphreys, M.S., Bain, J.D. and Dennis, S. Control processes and cue \ncombinations in a connectionist model of human memory. Department of Computer \nScience Technical Report, #186, University of Queensland, October 1990, 40pp. \n\n\f", "award": [], "sourceid": 434, "authors": [{"given_name": "Janet", "family_name": "Wiles", "institution": null}, {"given_name": "Michael", "family_name": "Humphreys", "institution": null}, {"given_name": "John", "family_name": "Bain", "institution": null}, {"given_name": "Simon", "family_name": "Dennis", "institution": null}]}