{"title": "Implications of Recursive Distributed Representations", "book": "Advances in Neural Information Processing Systems", "page_first": 527, "page_last": 536, "abstract": null, "full_text": "RECURSIVE DISTRIBUTED REPRESENTATIONS \n\nIMPLICATIONS OF \n\n527 \n\nJordan B.  Pollack \n\nLaboratory for A I Research \n\nOhio State University \nColumbus, OH -'3210 \n\nABSTRACT \n\nI  will  describe  my  recent  results  on  the  automatic  development  of fixed(cid:173)\nwidth recursive  distributed representations  of variable-sized  hierarchal data \nstructures.  One  implication  of this  wolk  is  that  certain  types  of AI-style \ndata-structures can now be represented in fixed-width analog vectors. Simple \ninferences  can  be  perfonned  using  the  type  of pattern  associations  that \nneural  networks excel  at  Another implication arises from  noting that these \nrepresentations  become  self-similar in  the  limit Once  this door to  chaos is \nopened.  many  interesting new  questions  about  the  representational  basis  of \nintelligence emerge, and can (and will) be discussed. \n\nINTRODUCTION \n\nA  major problem  for  any  cognitive  system  is the  capacity  for,  and  the  induction of the \npotentially  infinite  structures  implicated  in  faculties  such  as  human  language  and \nmemory. \nClassical cognitive architectures handle this problem  through finite  but recursive  sets of \nrules,  such  as  fonnal  grammars  (Chomsky,  1957).  Connectionist  architectures,  while \nyielding intriguing insights into fault-tolerance  and machine leaming, have, thus far,  not \nhandled such productive systems in an adequate fashion. \nSo, it is not surprising that one  of the  main  attacks  on  connectionism, especially  on its \napplication to  language processing models, has  been on the adequacy of such systems to \ndeal  with  apparently  rule-based  behaviors  (Pinker  &  Prince,  1988)  and  systematicity \n(Fodor & Pylyshyn, 1988). \nI  had  earlier  discussed  precisely  these  challenges  for  connectionism,  calling  them  the \ngenerative capacity problem for language, and the representational adequacy problem for \ndata structures (Pollack,  1987b).  These problems are  actually intimately related,  as  the \ncapacity  to  recognize  or generate  novel  language  relies  on the  ability  to  represent  the \nunderlying concept. \nRecently, I have developed an approach to the representation problem, at least for recur(cid:173)\nsive structures  like  sequences  and  trees.  Recursive  auto-associative  memory  (RAAM) \n(Pollack,  1988a).  automatically  develops  recursive  distributed  representations  of finite \ntraining sets of such structures, using Back-Propagation (Rumelhart et al.,  1986).  These \nrepresentations appear to occupy a novel position in the space of both classical and con(cid:173)\nnectionist symbolic representations. \nA  fixed-width  representation  of variable-sized  symbolic  trees  leads  immediately  to  the \nimplication  that  simple  fonns  of neural-netwolk  associative  memories  may  be  able  to \nperfonn inferences of a type that are  thought to require complex machinery such as  vari(cid:173)\nable binding and unification. \nBut when we take seriously the infinite part of the representational adequacy problem, we \nare  lead  into  a  strange  intellectual  area,  to  which  the  second  part  of  this  paper  is \naddressed. \n\n\f528 \n\nPollack \n\nBACKGROUND \n\nRECURSIVE AUTO-ASSOCIATIVE MEMORY \n\nA RAAM is composed of two mechanisms: a compressor, and a reconstructor, which are \nsimultaneously trained.  The job of the compressor is to encode a small set of fixed-width \npatterns  into  a  single  pattern  of the  same  width.  This  compression  can  be  recursively \napplied, from  the  bottom up, to  a fixed-valence  tree with distinguished labeled terminals \n(leaves),  resulting in a fixed-width pattern representing  the  entire  structure.  The job of \nthe  reconstructor is to  accurately  decode  this pattern into  its parts,  and  then  to  further \ndecode  the parts as  necessary, until the tenninal patterns are  found,  resulting in a recon(cid:173)\nstruction of the original tree. \nFor  binary  trees  with  k-bit  binary  patterns  as  the  leaves,  the  compressor  could  be  a \nsingle-layer feedforward network with 2k inputs and k outputs, along with additional con(cid:173)\ntrol  machinery.  The  reconstructor  could  be  a  single-layer  feedforward-network  with  k \ninputs and 2k outputs, along with a mechanism for testing whether a pattern is a tenninal. \nWe simultaneously train these two networks in an auto-associative framework as  follows. \nConsider the tree, \u00ab0 (A N\u00bb(Y (P (0 N\u00bb), as  one member of a training set of such trees, \nwhere the lexical categories are pre-encoded as k-bit vectors.  If the 2k-k-2k network is \nsuccessfully  trained (defined below)  with  the  following patterns (among other such pat(cid:173)\nterns in the training environment), the resultant compressor and reconstructor can reliably \nfonn representations for these binary trees. \n\ninput pattern \n\nhidden pattern \n\noutput pattern \n\n~  RAN(t) \n~  RDAN(t) \n~  RDN(t) \n~  RpDN(t) \n~  RVPDN(t) \n~  RDANVPDN<t) \n\nA+N \nO+RAN(t) \nD+N \nP+RDN(t) \nY+RpDN(t) \nRDAN(t)+RvPDN(t) \nThe (initially random)  values of the hidden  units, Rj(t), are part of the  training environ(cid:173)\nment, so it (and the representations) evolve along with the weights. I \nBecause  the training regime involves multiple compressions, but only single reconstruc(cid:173)\ntions, we rely on an induction that the reconstructor works. If a reconstructed pattern, say \nRpDN',  is sufficiently close to the original pattern, then its  parts can  be  reconstructed  as \nwell. \n\n~  A,+N, \n~  O~RAN(t)' \n~  O,+N, \n~  P,+RDN(t), \n~  Y,+RpDN(t), \n~  RDAN(t)I+RvPDN(t), \n\nAN  EXPERIMENT \n\nThe tree considered above was  one member of the first experiment done  on RAAM's.  I \nused a simple context-free parser to parse a set of lexical-category sequences into a set of \nbracketed binary trees: \n\n(0 (A (A (A N\u00bb\u00bb \n\u00ab0 N)(P (0 N\u00bb) \n\n(Y (0 N\u00bb \n\n(P (0 (A N\u00bb) \n\n\u00ab0 N) Y) \n\n1  This  ~moving target\" strategy is also used by (Elman,  1988) and (Dyer et aI.,  1988). \n\n\fImplications of Recursive Distributed Representations \n\n529 \n\n\u00abD N) (V (D (A N\u00bb\u00bb) \n\u00abD (A N\u00bb  (V (P (D N\u00bb\u00bb \n\nEach terminal pattern (D A N V &  P) was represented as a  l-bit-in-5 code padded with 5 \nzeros. A 20-10-20 RAAM devised the representations shown in figure  I. \n\noOOa\u00b7  ,\u00b7\u00b700 \nDODO'  .\u2022. - 0 \n0000' .\u2022. \u00b70 \no\u00b7  a  - .\u2022 .  00\u00b7 \n0\u00b70\u00b7 .  \u2022  \u00b700 D \n\u00b7  -Do \u00b70 ' 000 \n\u00b7 \u00b7oD\u00b7o\u00b7o\u00b7O \n\u00b7  \u00b70\u00b7 \u00b7o\u00b7oao \n\u00b7  -0' \u00b70 \u00b7 000 \n\n(P (0 N\u00bb \n(P (0 (A N\u00bb) \n(A N) \n\n\u2022 \n\n(A (A N\u00bb  .. 000\u00b7  .  D  a \n\n. \nA  A  AN' DODO\u00b7  \u2022  \u2022 0  -\n\u00ab0 N) V)  00\u00b700 D  000\u00b7 \n\npp \n\np \n\n\u00ab0 N)(V (0 (A N\u00bb\u00bb  o\u00b7  D \n\n\u2022 \n\n\u00ab0 (A N\u00bb (V (P (0 N\u00bb\u00bb \n\n\u2022 \n\n\u2022 \n\n. o\u00b7 \u00b7 . 0 \n\n\u2022  0  D \n\n\u2022 \n\n.  00  . \n\nFigure  I. \n\nRepresentations  of all  the  binary  trees  in  the  training  set.  devised  by  a \n20-10-20 RAAM.  manually clustered by phrase-type.  The squares represent \nvalues between 0 and 1 by area. \n\nI  labeled each tree and its  representation  by the  phrase  type in the  grammar,  and sorted \nthem  by  type.  The  RAAM,  without  baving  any  intrinsic  concepts  of phrase-type,  has \nclearly  developed  a  representation  with  similarity  between  members  of the  same  type. \nFor example,  the  third  feature  seems  to  be  clearly  distinguishing  sentences  from  non(cid:173)\nsentences, the fifth feature seems to be involved in separating adjective phrases from oth(cid:173)\ners,  while  the  tenth  feature  appears  to distinguish prepositional  and  noun  phrases  from \nothers.2 \nAt the same time, the representation must be keeping enough information about the sub(cid:173)\ntrees in order to allow the reconstructor to accurately  recover the  original structure.  So, \nknowledge about structural regularity flows into the wt:ights while constraints about con(cid:173)\ntext similarity guide the development of the representations. \n\nRECURSIVE DISTRIBUTED REPRESENTATIONS \n\nThese vectors are a very new kind of representation, a recursive, distributed represen(cid:173)\ntation, hinted at by Hinton's (1988) notion of a reduced description. \nThey combine aspects of several disparate representations.  Like feature-vectors, they are \nfixed-width,  similarity-based,  and  their content  is  easily  accessible.  Like symbols,  they \ncombine only in syntactically well-formed ways.  Like symbol-structures, they  have con(cid:173)\nstituency and compositionality.  And, like pointers. they refer to larger symbol structures \n\n2  In fact,  by these  metrics,  the  test case \u00abD N)(P (D N))) should really be classified as a sentence;  since  it was \nnot used in any other construction, there was no reason for the RAAM to believe otherwise. \n\n\f530 \n\nPollack \n\nwhich can be efficiently retrieved. \nBut.  unlike  feature-vectors.  they  compose.  Unlike  symbols.  they  can  be  compared. \nUnlike symbol structures. they are fixed in size.  And. unlike pointers. they have content. \nRecursive distributed representations  could. potentially. lead to a  reintegration of syntax \nand semantics at a very low level3\u2022 Rather than having meaning-free symbols which syn(cid:173)\ntactically  combine. and meanings which are  recursively  ascribed.  we  could functionally \ncompose symbols which bear their own meanings. \n\nIMPLICATIONS \n\nOne of the reasons for the  historical split between symbolic AI and fields such as  pattern \nrecognition  or neural  networks  is that  the  structured representations  AI  requires  do  not \neasily commingle with the representations offered by n-dimensional vectors. \nSince recursive distributed representations form  a  bridge from  structured representations \nto  n-dimensional  vectors.  they  will  allow  high-level  AI  tasks  to  be  accomplished  with \nneural networks. \n\nASSOCIATIVE INFERENCE \n\nThere are  many kinds of inferences which seem to be very easy  for  humans to perform. \nIn fact,  we must perform incredibly long chains of inferences in the act of understanding \nnatural language (Birnbaum. 1986). \nAnd yet, when we consider performing those inferences using standard techniques which \ninvolve  variable  binding  and  unification,  the  costs  seem prohibitive.  For humans. how(cid:173)\never. these inferences  seem  to  cost no  more  than  simple  associative priming  (Meyer & \nSchvaneveldt. 1971). \nSince RAAMS can devise representations of trees as analog patterns which can actually \nbe associated, they may lead to very fast neuro-Iogical inference engines. \nFor example. in a larger experiment. which was reported in (Pollack.  1988b). a 48-16-48 \nRAAM developed representations for a set of ternary trees. such as \n\n(THOUGHT PAT (KNEW JOHN (LOVED MARY JOHN\u00bb) \n\nwhich  corresponded  to  a  set  of  sentences  with  complex  constituent  structure.  This \nRAAM was  able  to  represent.  as points within a  16-dimensional hypercube.  all cases of \n(LOVED X  Y) where X and Y were chosen from the set {JOHN, MARY. PAT. MAN}. \nA  simple test of whether or not  associative  inference  were  possible.  then,  would be  to \nbuild  a  \"symmetric  love\"  network,  which  would  perform  the  simple  inference:  \"If \n(LOVED X Y) then (LOVED Y X)\". \nA netwoIk with  16 input and output units and 8 hidden units was successfully trained on \n12 of the  16 possible associations. and worked perfectly on the remaining 4. (Note that it \naccomplished  this task without any  explicit machinery  for  matching  and moving X  and \nY.) \nOne might think. that in order to chain simple inferences like this one we will need many \nhidden  layers.  But  there  has  recently  been  some  coincidental  work  showing  that  feed-\n\n3  The wrong distinction is the inverse of the  undifferentiated concept problem in science, such as  the fusing  of \nthe  notions of heat and temperature in the  17th century (Wiser & Carey.  1983). For example. a company which \nmanufactured workstations based on a hardware distinction between characters and  graphics had deep trouble \nwhen trying to build a modem window system ... \n\n\fImplications of Recursive Distributed Representations \n\n531 \n\nforward  networks  with  two  layers  of hidden  units  can  compute  arbitrary  mappings \n(Lapedes &  Farber.  1988a;  Lippman.  1987).  Therefore,  we can assume that the sequen(cid:173)\ntial application of associative-style inferences can be  speeded up, at least by retraining. to \na simple 3-cycle process. \n\nOPENING THE DOOR TO CHAOS \n\nThe Capacity of RAAM's \n\nAs discussed in the introduction. the question of infinite generative capacity is central.  10 \nthe  domain  of RAAM's  the  question  becomes:  Given a  finite  set  of trees  to  represent. \nhow can the system then represent an infinite number of related trees. \nFor the  syntactic-tree experiment reported above. the  20-10-20 RAAM was  ooly able  to \nrepresent  32  new  trees. The  48-16-48  RAAM  was  able  to  represent  many  more  than it \nwas trained on. but not yet an infinite number in the linguistics sense. \nI  do  not  yet  have  any  closed  analytical  forms  for  the  capacity  of a  recursive  auto(cid:173)\nassociative  memory.  Given  that  is  is  not  really  a  file-cabinet  or  content-addressable \nmemory, but a memory for a gestalt of rules for recursive pattern compression and recon(cid:173)\nstruction. capacity results such as  those of (Willshaw.  1981) and (Hopfield,  1982) do not \ndirectly  apply.  Binary  patterns  are  not  being  stored.  so  one  cannot  simply  count  how \nmany. \nI have considered. however. the capacity of such a memory in the limit, where the actual \nfunctions  and  analog  representations  are  not  bounded  by  single  linear transformations \nand sigmoids or by 32-bit floating point resolution. \n\nA plot of the bit-interspersal function.  The x and y axis represent the left and \nright subtrees. and the height represents the output of the function. \n\nFigure 2. \n\nConsider just  a  2-1-2  recursive  auto-associator.  It  is  really  a  reconstructible  mapping \nfrom  points  in the  unit  square  to  points  on the  unit line.  10 order to work.  the  function \nshould define  a  parametric  I-dimensional  curve  in  2-space.  perhaps  a  set  of connected \nsplines.4  As  more  and more  data points  need  to  be  encoded.  this parametric  curve  will \nbecome more convoluted to cover them . In the limit, it will no longer be  a  I-dimensional \ncurve. but a space-filling curve with a fractional dimension. \n\n4  (Saund,  1987) originally made  the connection between auto-association and dimensionality reduction.  If such \n\n\f532 \n\nPollack \n\nOne possible  functional  basis  for  this  ultimate  2-1-2  recursive  auto-associator  is  \"bit(cid:173)\ninterspersal,\"  where the  compression function would return a number, between 0  and 1, \nby interleaving the bits of the  binary-fractional representations of the left and  right sub(cid:173)\ntrees.  Figure 2 depicts this function,  not as a space-filling curve, but as a surface, where \nno  two  points project to  the  same  height.  The  surface  is  a  3-dimensional variant  of a \nrecognizable instance of Cantor dust called the devil's staircase. \nThus, it is my working hypothesis that alternative activation functions (i.e.  other than the \nusual  sigmoidal  or  threshold),  based  on  fractal  or  chaotic  mathematics,  is  the  critical \nmissing link between neural networlcs and infinite capacity systems. \n\nBetween AI and Chaos \n\nThe remainder of this paper is what is behind the door;  the result  of simultaneous con(cid:173)\nsideration of the fields  of AI, Neural Networks,  Fractals, and Olaos.s It is, in essence,  a \nproposal on where (I am planning) to look for fruitful interplay between these fields,  and \nwhat some interdisciplinary problems are which could be solved in this context. \nThere has already been some intrusion of interest in chaos in the physics-based study of \nneural networlcs  as dynamical systems. For example both (Hubennan &  Hogg,  1987) and \n(Kurten,  1987) show  how phase-transitions  occur in particular  neural-like  systems,  and \n(Lapedes  &  Farber,  1988b)  demonstrate  how  a  network  trained  to  predict  a  simple \niterated function would follow that function's bifurcations into chaos. \nHowever, these efforts are either noticing chaos, or working with it as a domain.  At the \nother end of the spectrum  are  those relying on chaos to explain such things as the emer(cid:173)\ngence of consciousness, or free  will (Braitenberg, 1984, p. 65). \nIn  between  these  extremes  lies  some  very  hard  problems  recognized  by  AI  which,  I \nbelieve, could benefit from a new viewpoint. \n\nSelf-Similarity and the Symbol-Grounding Problem \n\nThe  bifurcation  between  structure  and  form  which  leads  to  the  near  universality  of \ndiscrete symbolic structures with ascribed  meanings has  lead to  a yawning gap between \ncognitive and perceptual subareas of AI. \nThis  gulf  can  be  seen  between  such  fields  as  speech  recognition  and  language \ncomprehension,  early  versus  late  vision,  and  robotics  versus  planning.  The  low-level \ntasks require numeric, sensory re~resentations, while the  high-level ones require compo(cid:173)\nsitional symbolic representations. \nThe idea of infinitely regressing symbolic representations which bottom-out at perception \nhas been an unimplementable folk idea (\"Turtles all the way down\") in AI for quite some \ntime. \nThe reason  for its lack  of luster is that the  amount  of information in such a  structure  is \nconsidered combinatorially explosive. Unless, of course, one considers self-similarity to \nbe an information-limiting construction. \n\na  complete  2-1-2  RAAM  could be found.  it  would  give  a  unique  number to  every binary  tree  such that the \nnumber of a tree would be a invertible function of the numbers of its two subtrees . \n.5  Talking about 4  disciples is both difficult,  and dangerous. considering the current size of the chasm. and the \nmutual hostilities: AI thinks NN is just a spectre. NN thinks AI is dead, F  thinks it subsumes C, and C  thinks F \nis its just showbiz. \n6  It is no surprise then. that neural networks arc much more successful at the former tasks. \n\n\fImplications of Recursive Distributed Representations \n\n533 \n\nWhile  working  on  a ~w activation  function  for  RAAMS  which  would  magically  have \nthis property,  I  have  started  building  modular systems  of RAAMs,  following  Ballard's \n(1987) work on non-recursive auto-associators. \nWhen  viewing a RAAM as  a constrained system, one can see  that the  terminal patterns \nare  overconstrained  and the highest-level non-terminal patterns  are  unconstrained.  Only \nthose  non-terminals  which  are  further  compressed  have  a  reasonable  similarity  con(cid:173)\nstraint. One could imagine a cascade of RAAMs, where the highest non-terminal patterns \nof a  low-level  RAAM  (say,  for  encodings  of letters)  are  the  terminal  patterns  for  a \nmiddle-level RAAM (say, for words), whose non-terminal patterns are the terminals for a \nhigher-level RAAM (say, for sentences). \nIf all  the  representations  were  the  same  width,  then  there  must  be  natural  similarities \nbetween the structures at different conceptual scales. \n\nInduction Inference and Strange Automata \n\nThe problem of inductive inference1,  of developing a machine which can learn to recog(cid:173)\nnize or generate a language is a pretty hard problem, even for regular languages. \nIn  the  process  of extending  my  work  on  a recurrent  high-order  neural  network  called \nsequential cascaded nets (Pollack, 1987a), something strange occurred. \nIt is always possible to  completely map out any unknown finite-state machine by provid(cid:173)\ning each known state with every input token,  and keeping track of the states.  TIris is, in \nfact, what defines such a machine as finite. \nSince  a  recurrent  network  is  a  dynamical  system,  rather  than  an  automaton,  one  must \nchoose  a fuzz-factor  for comparing real  numbers.  For a particular network trained  on  a \ncontext-free  grammar, I  was  unable  to  map  it out.  Each  time  I reduced  the  fuzz-factor, \nthe machine doubled in size, much like Mandelbrot's coastline (Mandelbrot, 1982) \nTIris  suggests  a  bidirectional  analogy  between  finite  state  automata and dynamical  sys(cid:173)\ntems of the neural network sort8.  An automaton has an initial state, a set of states, a lexi(cid:173)\ncon, and and a function which produces a new state given an old state and input token.  A \nsubset of states  are  distinguished as  accepting  states.  A dynamical  system  has  an  initial \nstate,  and  an  equation  which  defines  its  evolution  over  time,  perhaps  in  response  to \nenvironment. \nSuch dynamical systems have  elements known  as  attractor states,  to  which  the  state  of \nthe  system  usually evolves.  Two such varieties, limit points and limit cycles, correspond \ndirectly to similar elements in finite-state automata, states with loops back to themselves, \nand  short boring  cycles  of states  (such  as  the  familiar  \"Please  Login.  Enter Password. \nBad Password. Please Login ..... ). \nBut there is an element in non-linear dynamical systems which does not have a correlate \nin  formal  automata  theory,  which  is  the  notion  of a  chaotic,  or strange,  attractor,  fiISt \nnoticed  in  work  on  weather  prediction  (Lorenz,  1963).  A  chaotic  attractor  does  not \nrepeat. \nThe implications for inductive inference is that while, formally, push-down automata and \nTuring  machines  are  necessary  for  recognizing  harder  classes  of languages,  such  as \ncontext-free  or context-sensitive,  respectively,  the  idiosyncratic  state-table  and  external \nmemory  of such  devices  make  them  impossible  to  induce.  On  the  other hand,  chaotic \ndynamical systems look much like automata, and should be about as  hard to  induce. The \n\n7  For a good survey see (Angluin & Smith,  1983). J.  Feldman recently posed this  as  a \"challenge\" problem for \nneural networks (c.f. Servan-Scrieber, Cleermans, & McClelland (this volume\u00bb. \n8  Wolfram (1984) has, of course, made the analogy between dynamical systems and cellular automata. \n\n\f534 \n\nPollack \n\ninfinite memory  is internal  to  the state  vector,  and  the finite-state-control  is built  into  a \nmore regular, but non-linear, function. \n\nFractal Energy Landscapes and Natural Kinds \n\nHopfield (1982) described an associative memory in which each of a finite set of binary \nvectors  to  be  stored  would  define  a  local  minima  in  some  energy  landscape.  The \nBoltzmann  Machine  (Ackley  et  al.,  1985)  uses  a  similar physical  analogy  along  with \nsimulated  annealing  to  seek  the  global  minimum  in such  landscapes  as  well.  Pineda \n(1987) has  a continuous version of such a memory, where  the  attract or states are  analog \nvectors. \nOne can think of these energy minimization process as a ball rolling down hills.  Given a \nsmooth  landscape,  that  ball  will  roll  into  a  local  minima.  On  the  other  hand,  if the \nlandscape were constructed by recursive similarity, or by a midpoint displacement tech(cid:173)\nnique, such as those  used in figures of fractal mountains, there will be an infinite number \nof local  minima,  which  will  be  detected  based  on  the  size  of the  ball.  N aillon  and \nTheeten's report (this volume), in which an exponential number of attractors are  used, is \nalong the proposed line. \nThe idea of high-dimensional feature  vectors  has  a long history in psychological studies \nof memory  and  representation,  and is known to  be  inadequate  from  that perspective  as \nwell as  from  the representational requirements of AI. But AI has no good empirical can(cid:173)\ndidates for a theory of mental representation either. \nSuch theories generally break down when dealing with novel instances of Natural Kinds, \nsuch as  birds, chairs, and games.  A robot with  necessary  and sufficient conditions, logi(cid:173)\ncal rules, or circumscribed regions in feature  space cannot deal with walking into a room, \nrecognizing and sitting on a hand-shaped chair. \nIf the chairs we know fonn the  large-scale local minima of an associative memory, then \nperhaps  the  chairs we don't know can also  be  found  as  local minima in the same space, \nalbeit on a smaller scale. Of course, all the chairs we know are only smaller-scale minima \nin our memory for furniture. \n\nFractal Compression and the Capacity of Memory \n\nConsider something  like  the  Mandelbrot  set  as  the  basis  for  a  reconstructive  memory. \nRather than storing all pictures, one merely  has  to store  the \"pointer\" to a picture,9  and, \nwith the help of a simple function and large computer, the picture can be retrieved.  Most \neveryone has seen glossy pictures of the colorful prototype shapes of yeasts and dragons \nthat infinitely appear as the location and scale are changed along the chaotic boundary. \nThe first  step  in this hypothetical  construction is to develop  a related set with the  addi(cid:173)\ntional property that it can be  inverted in the  following  sense:  Given a rough sketch of a \npicture likely to be in the set, return the best \"pointer\" to it 10 \nThe second step, perhaps using nonnal neural-netwOIk technology, is to build an inverti(cid:173)\nble DOn-linear mapping from  the prototypes in a application domain (like chess positions, \nhuman faces, sentences, schemata, etc .. )  to the largest-scale prototypes in the mathemati(cid:173)\ncal memory space. \n\n9  I.e. a point on the complex plane and the window size \n10  Related sets  might show  up  with  great frequency  using  iterated  systems,  like  Newton's  method  or  back(cid:173)\npropagation.  And  a  more  precise  notion  of inversion,  involving  both  representational  tolerance  and  scale.  is \nrequired. \n\n\fImplications of Recursive Distributed Representations \n\n535 \n\nTaken together, this hypothetical system turns out to be a look-up table for an infinite set \nof similar representations which incurs no memory cost for its contents. Only the pointers \nand the reconstruction function need to be stored. Such a basis for reconstructive storage \nwould render meaningless the recent attempts at  \"counting the bits\" of human memory \n(Hillis, 1988; Landauer,  1986). \nWhile these  two steps together sound  quite  fantastic,  it is  closely related  to  the RAAM \nidea  using  a  chaotIc  activation  function.  The  reconstructor  produces  contents  from \npointers, while the compressor retums pointers from  contents.  And the idea of a unifonn \nfractal  basis  for  memory  is  not  really  too  distant  from  the  idea of a  unifonn  basis  for \nvisual images, such as iterated fractal surfaces based on the collage theorem (Barnsley et \naI.,1985). \nA  moral  could  be  that  impressive  demonstrations  of compression,  such  as  the  bidirec(cid:173)\ntional mapping from  ideas to language, must be easy when one can discover the underly(cid:173)\ning regularity. \n\nCONCLUSION \n\nRecursive  auto-associatIve  memory  can  develop  fixed-width  recursive  distributed \nrepresentations  for  variable-sized  data-structures  such  as  symbolic  trees.  Given  such \nrepresentations,  one  implication  is  that  complex  inferences,  which  seemed  to  require \ncomplex infonnation handling strategies, can be accomplished with associations. \nA  second  implication  is  that  the  representations  must  become  self-similar  and  space(cid:173)\nfilling in the limit.  This implication, of fractal and chaotic structures in mental represen(cid:173)\ntations,  may lead  to  a  reconsideration  of many  fundamental  decisions  in computational \ncognitive science. \nDissonance for  cognitive scientists can be  induced by comparing the  infinite output of a \nfonnallanguage  generator (with anybody's rules), to the  boundary areas  of the Mandel(cid:173)\nbrot set with its simple underlying function.  Which is vaster? Which more natural? \nFor when one considers the relative success of fractal versus euclidean geometry at com(cid:173)\npactly  describing  natural  objects,  such  as  trees  and  coastlines,  one  must  wonder at  the \naccuracy of the pervasive description of naturally-occurring mental objects as features or \npropositions which bottom-out at meaningless tenns. \n\nReferences \nAckley,  D.  H.,  Hinton,  G.  E.  &  Sejnowski,  T.  J.  (1985).  A  learning  algorithm  for  Boltzmann  Machines. \n\nCognitive Science. 9,  147-169. \n\nAngluin, D. & Smith,  C.  H.  (1983).  Inductive  Inference: Theory and Methods.  Computing Surveys. 15, 237-\n\n269. \nBallard,  D.  H. \n\n(1987).  Modular  Learning  in  Neural  Networl<;~.  In  Proceedings  of the  Sixth  Nationd \n\nConference on Artificial Intelligence.  Seattle, 279-284. \n\nBamsley, M. F.,  Ervin,  V., Hardin, D. & Lancaster, J.  (1985).  Solution of an  inverse problem for fractals and \n\nother sets.  Proceedings of the National Academy of Science. 83. \n\nBirnbaum, L.  (1986).  Integrated processing in planning and understanding.  Research Report 489, New Haven: \n\nComputer Science Dept., Yale Univeristy. \n\nBraitenberg, V.  (1984).  Vehicles: Experiments in synthetic psychology.  Cambridge: MIT press. \nOtomsky, N.  (1957).  Syntactic structures.  The Hague: Mouton and Co .. \nDyer, M. G., Rowers,  M. &  Wang, Y. A.  (1988).  Weight Matrix = Pattern of Activation:  Encoding Semantic \nNetworks  as  Distributed  Representations  in  DUAL,  a  PDP  architecture.  UCLA-Artificial \nIntelligence-88-5, Los Angeles: Artificial Intelligence Laboratory, UCLA. \n\nElman.  J.  L.  (1988).  Finding Structure in  Time.  Report 8801.  San Diego:  Center for  Research in Language. \n\nUCSD. \n\n28,3-71. \n\nFodor, J.  &  Pylyshyn, A.  (1988).  Connectionism and Cognitive Architecture:  A Critical Analysis.  Cognition. \n\nHillis. W. D.  (1988).  Intelligence as emergent behavior; or. the songs of eden.  Daedelus.117,  175-190. \nHinton,  G.  (1988).  Representing  Part-Whole  hierarchies  in  connectionist  networks.  In  Proceedings  of the \n\nTenth Annual Conference of the Cognitive Science SOciety.  Montreal, 48-54. \n\n\f536 \n\nPollack \n\nHopfield, J. J.  (1982).  Neural Networks and physical systems with emergent collective computational abilities. \n\nProcudings of the National Academy of Sciences USA.  79, 2554-2558. \n\nHubennan,  B.  A.  &  Hogg,  T.  (1987).  Phase  Transitions  in  Artificial  Intelligence  Systems.  Artificial \n\nIntelligence. 33, 155-172. \n\nKurten,  K.  E.  (1987).  Phase  transitions  in  quasirandom  neural  networks.  In  Institute  of Electrical  and \n\nElectronics Engineers First International Conference on Neural Networks.  San Diego, n-197-20. \n\nLandauer,  T.  K.  (1986).  How  much  do  people  remember?  Some  estimates  on  the  quantity  of  learned \n\ninfonnation in long-term memory ..  Cognitive Science. 10, 477-494. \n\nLapedes, A. S. & Farber, R. M.  (1988).  How Neural Nets Work.  LAUR-88-418: Los Alamos. \nLapedes,  A.  S.  &  Farber,  R.  M.  (1988).  Nonlinear Signal  Processing  using  Neural Networks:  Prediction  and \n\nsystem modeling.  Biological Cybernetics. To appear. \n\nElectronics Engineers ASSP Magazine. April, 4-22. \n\nLippman,  R.  P.  (1987).  An  introduction  to  computing  with  neural  networks.  Institute  of Electrical  and \nLorenz, E. N.  (1963).  Detenninistic Nonperiodic flow.  Journal of Atmospheric Sciences. 20,  130-141. \nMandelbrot, B.  (1982).  The Fractal Geometry of Nature.  San Francisco: Freeman. \nMeyer,  D.  E.  &  Schvaneveldt,  R.  W.  (1971).  Facilitation  in  recognizing  pairs  of  words:  Evidence  of a \n\ndependence between retrieval operations.  Journal of Experimental Psychology. 90, 227-234. \n\nPineda,  F.  J.  (1987).  Generalization  of Back-Propagation  to  Recurrent  Neural  Networks.  Physical  Review \n\nLetters. 59, 2229-2232. \n\nPinker, S. & Prince, A.  (1988).  On Language and Connectionism:  Analysis of a parallel distributed processing \n\nmodel of language inquisition ..  Cognition. 28, 73-193. \n\nPollack, J. B.  (1987).  Cascaded Back Propagation on Dynamic Connectionist Networks.  In Proceedings oftM \n\nNinth Conference of the Cognitive Science Society.  Seattle, 391-404. \n\nPollack,  J.  B.  (1987).  On  Connectionist  Models  of  Natural  Language  Processing.  Ph.D.  Thesis,  Urbana: \nComputer  Science  Department,  University  of  Illinois.  (Available  as  MCCS-87-IOO,  Computing \nResearch Laboratory, Las Cruces, NM) \n\nPollack,  J.  B. \n\nPollack,  J.  B. \n\n(1988).  Recursive  Auto-Associative  Memory:  Devising  Compositional  Distributed \nRepresentations.  In  Proceedings of the  Tenth Annual Conference  of the Cognitive Science Society. \nMontreal, 33-39. \n\n(1988).  Recursive  Auto-Associative  Memory:  Devising  Compositional  Distributed \nRepresentations.  MCCS-88-124,  Las  Cruces:  Computing  Research  Laboratory, New  Mexico State \nUniversity. \n\nRumelhart,  D.  E.,  Hinton,  G.  &  Williams,  R.  ( 1986).  Learning  Internal  Representations  through  Error \nPropagation.  In  D.  E.  Rumelhart,  J.  L.  McClelland  &  the  PDP  research  Group,  (Eds.),  Parallel \nDistributed Processing: Experiments in the  Microstructure  of Cognition,  Vol.  l.  Cambridge:  MIT \nPress. \nSaund,  E.  (1987).  Dimensionality  Reduction  and  Constraint  in  Later  Vision.  In  Proceedings  of tM  Ninth \n\nAnnual Conference of the Cognitive Science Society.  Seattle, 908-915. \n\nWillshaw, D. J.  (1981).  Holography, Associative Memory,  and Inductive Generalization.  In G. E. Hinton & J. \nA.  Anderson,  (Eds.),  Parallel  models  of  associative  memory.  Hillsdale:  Lawrence  Erlbaum \nAssociates. \n\nWiser,  M.  &  Carey,  S.  (1983).  When heat and  temperature were  one.  In  D.  Gentner  &  A.  Stevens,  (Eds.), \n\nMental Models.  Hillsdale: Erlbaum. \n\nWolfram, S.  (1984).  Universality and Complexity in Cellular Automata.  Physica.1OD,  1-35. \n\n\f", "award": [], "sourceid": 114, "authors": [{"given_name": "Jordan", "family_name": "Pollack", "institution": null}]}