{"title": "Predicting Complex Behavior in Sparse Asymmetric Networks", "book": "Advances in Neural Information Processing Systems", "page_first": 556, "page_last": 563, "abstract": null, "full_text": "Predicting Complex Behavior \nin Sparse Asymmetric Networks \n\nAn A. Minai and William B. Levy \n\nDepartment of Neurosurgery \n\nBox 420. Health Sciences Center \n\nUniversity of Virginia \n\nCharlottesville. V A 22908 \n\nAbstract \n\nRecurrent  networks  of threshold  elements  have  been  studied  inten(cid:173)\nsively  as  associative  memories and pattern-recognition devices. While \nmost  research  has  concentrated  on  fully-connected  symmetric  net(cid:173)\nworks.  which  relax  to  stable  fixed  points.  asymmetric  networks  show \nricher dynamical behavior. and can be used as  sequence  generators or \nflexible  pattern-recognition  devices.  In  this  paper.  we  approach  the \nproblem of predicting  the  complex global behavior of a  class  of ran(cid:173)\ndom asymmetric networks in terms of network parameters.  These net(cid:173)\nworks can show fixed-point.  cyclical or effectively aperiodic  behavior. \ndepending  on  parameter  values.  and  our approach  can  be used  to  set \nparameters.  as  necessary.  to  obtain a  desired complexity of dynamics. \nThe  approach  also  provides  qualitative  insight  into  why  the  system \nbehaves as it does and suggests possible applications. \n\n1  INTRODUCTION \nRecurrent  neural  networks  of threshold elements  have  been  intensively  investigated  in \nrecent  years.  in  part  because  of their  interesting  dynamics.  Most  of the  interest  has \nfocused  on  networks  with  symmetric  connections.  which  always  relax  to  stable  fixed \npoints  (Hopfield.  1982)  and can be used as  associative memories  or pattern-recognition \ndevices.  Networks  with  asymmetric  connections.  however.  have  the  potential for  much \n\n556 \n\n\fPredicting  Complex Behavior in  Sparse Asymmetric Networks \n\n557 \n\nricher dynamic behavior and may be used for learning sequences (see, e.g., Amari, 1972; \nSompolinsky and Kanter,  1986). \nIn this paper, we introduce an approach for predicting the complex global behavior of an \ninteresting class of random sparse asymmetric networks  in terms of network parameters. \nThis  approach  can  be  used  to  set parameter  values,  as  necessary,  to  obtain  a  desired \nactivity level and qualitatively different varieties of dynamic behavior. \n\n2  NETWORK PARAMETERS AND EQUATIONS \nA network  consists  of n  identical  011  neurons  with  threshold  O.  The  fixed  pattern  of \nexcitatory connectivity  between neurons  is  generated prior to simulation by a Bernoulli \nprocess  with  a probability p  of connection  from  neuron j  to  neuron  i.  All  excitatory \nconnections have  the fixed  value w, and there is  a global inhibition that is  linear in the \nnumber of active  neurons.  If m (t ) is the number of active neurons at time t, K  the inhi(cid:173)\nbitory weight, y; (t) the net excitation and Z; (t) the firing status of neuron i  at t, and C;j  a \n011  variable  indicating  the  presence  or absence  of a  connection  from  j  to  i, then  the \nequations for i  are: \n\ny;(t) = \n\nw  ~ C;jZj(t-1) \n\nj -\n\nw  ~ C;jzj(t-I) +Km(t-I) \n\nl:t \n\n,I~m(t-I)~n \n\nZ;(t)={1 \n\nify;(t)2!O \no  otherwise  ' \n\n0<0<1 \n\n(1) \n\n(2) \n\n(3) \n\nIf m (t-I) = 0,  y; (t) = 0 'Vi.  Equation  (1)  is  a simple  variant of the  shunting  inhibition \nneuron  model  studied by  several researchers,  and the  network is  similar to the  one pro(cid:173)\nposed by Marr (Marr,  1971).  Note that (1) and (2) can be combined to write the neuron \nequations in a more familiar subtractive inhibition format Defining a ==  OK / (I-9)w , \n\nz; (t ) = 1 \n\nif Jt C;j Zj (t -1) - a ~ Zj (t -I) 2! 0 \no  otherwise \n\n3  NETWORK BEHAVIOR \nIn this paper, we study the evolution of total activity, m(t), as  the system relaxes.  From \nEquation (3), the firing condition for neuron i  at time t, given the activity m (t-l)=M  at \ntime  t-I, is:  e;(t).= ~C;jZj(t-l) 2! aM.  Thus, in order to fire  at time  t, neuron i  must \nhave at least r aMl  active inputs. This allows us to calculate the average firing probability \nof a neuron given the prior activity M  as: \n\nl:t \n\nP{lo! active  inputs 2!raMll=  ~  (AfJpk (l-p)M.....t  =p(M;n,p,a) \n\nk,{aM! \n\n(4) \n\nIf M  is large enough, we can use  a Gaussian approximation to  the 'binomial distribution \n\n\f558 \n\nMinai and Levy \n\nand a hyperbolic tangent approximation to the error function to get \n\np(M; n~.a) = ~ [t-eif[ ~]] = ~ [l-mn+#x]] \n\nwhere \n\nFinally, when M  is large enough to assume [aMl = aM, we get an even simpler form: \n\nx ==  [aM]  -Mp \n\"Mp(l-p) \n\np(M; n~.a)= t[l-tanh ~] \n\n(5) \n\n(6) \n\nwhere \n\nT = _1_ ~ 1tp(l-p) \n\n- a-p  2 '  \n\na.~p \n\nAssuming that neurons  fire  independently,  as  they  will  tend to do  in such  large,  sparse \nnetworks (Minai and Levy,  I992a,b), the network's activity at time t  is distributed as \n\nP {m (1 )=N  I m (t-I)=M} = (Z) p(M)H (1- p(M\u00bb\"-N \n\n(7) \n\nwhich leads to a stochastic return map for the activity: \n\nm (1) = n p(m (t-I\u00bb + O(..Jii) \n\n(8) \nIn Figure  1,  we  plot m (t) against m (1 -1) for  a  120  neuron  network  and two  different \nvalues  of  a..  The  vertical  bars  show  two  standard  deviations  on  either  side  of \nn p(m (t-I\u00bb.  It is clear that the network's activity falls within the range predicted by (8). \nAfter an initial transient period, the system either switches off permanently (correspond(cid:173)\ning to the zero activity fixed point) or gets trapped in an 0 ( ..Jii) region around the point \nm defined by m (t) = m (t -1). We call this the attracting region of the map.  The size and \nlocation of the  attracting  region  are  determined by a and  largely dictate  the  qualitative \ndynamic behavior of the network. \nAs  a.  ranges  from  0  to  1,  networks  show  three  kinds  of behavior:  fixed  points,  short \ncycles, and effectively aperiodic dynamics.  Before describing  these behaviors, however, \nwe introduce the notion of available neurons.  Let k;  be the number of input connections \nto neuron i  (the fan-in of n. Given m (t -1) = M, if k;  < [aMl, neuron;  cannot possibly \nmeet the firing criterion at time t. Such a neuron is said to be disabled by activity M. The \ngroup of neurons  not disabled are considered available neurons.  At any specific activity \nM, there is a unique set, Na (M), of available neurons in a given network, and only neu(cid:173)\nrons  from  this  set  can  be  active  at  the  next  time  step.  Clearly,  Na (M 1) !:: Na (M 2)  if \nM 1 ~ M 2. The average size of the available set at a given activity M  is \n\nna (M; n,p ,a) ==  n  [1  - P {k;  < [aMl} ] = n  t  (~)pA: (l-p )\"-A: \n\nA:=faMl \n\n(9) \n\n\fPredicting Complex Behavior in  Sparse Asymmetric Networks \n\n559 \n\n(a) Eft'edlytJ, Aperiodic B.Jllwior \n8 - 0.85, K - 0.016, w  - 0.4 =- a - 0.227 \n\no-~da&a(2000\") \n\n. . \n\n. . \n\n.' \n\n(11) IB&h AcdYit, C,cIe \n8-0.85, K -0.012, w  -0.4 =-a-o.l1 \n\n.' \n\n8-+~--------------------------~  .' \n\n8 \n\n8I(t) \n\n118  8 \n\na.(t) \n\nUO \n\nFigure  1: Predicted Distribution of m (t+l) given met), and Empirical  Data  (0)  for  Two \nNetworks A and B.  The vertical bars represent 4 standard deviations of the predicted dis(cid:173)\ntribution for each m (t).  Note that the empirical values fall in the predicted range. \n\n~~--------------------------------------------------------------~ \n\n8I(t) \n\n(a) Eft'edlyitly Aperiodic BebaYIor \n8-0.85,K -0.016, w  -0.4 =-a-0.221 \n\nU8 1000 \n\n8-+--------------------------------------________________________ ~ \nr-----------------------------, \n\n10000 \n\n(e) Low AcdYit, C,cIe \n8-0.85, K -0.0241, w -0.4 ... a-0.35 \n\na.(t) \n\n(11) Hlet- At:thit, C,cIe \n8-0.85, K -0.012, w  -0.4 ... 11-0.11 \n\n--+-----------------------------4 \n\n400  :zoo \n\nFigure  2:  Activity  time-series  for  three  kinds  of behavior shown  by  a  120  neuron  net(cid:173)\nwork. Graphs (a) and (b) correspond to the data shown in Figure 1. \n\n\f560 \n\nMinai and Levy \n\nIt can be  shown that na (M) ~ n p(M), so there  are  usually enough neurons  available  to \nachieve the average activity as per (8). \n\nWe now describe the three kinds of dynamic behavior exhibited by our networks. \n(1) \n\nFixed Point Behavior: If a is  very small, m  is close to n, inhibition is not strong \nenough to control activity and almost all neurons switch on permanently. If a is too \nlarge, iff is close to 0 and the stochastic dynamics eventually finds, and remains at, \nthe zero activity fixed point. \n\n(2)  Effectively Aperiodic Behavior: While deterministic, finite  state systems  such as \n\nour networks  cannot show  truly aperiodic  or chaotic  behavior,  the  time  to repeti(cid:173)\ntion  can  be  so  long  as  to  make  the  dynamics  effectively  aperiodic.  This  occurs \nwhen  the  attracting  region  is  at a  moderate  activity  level,  well  below  the  ceiling \ndefined by the number of available neurons. In such a situation, the network, start(cid:173)\ning  from  an  initial condition, successively visits a  very large  number of different \nstates, and the activity, m (t), yields  an effvectively aperiodic time-series of ampli(cid:173)\ntude 0  (-{,1), as shown in Figure 2(a). \n\n(3)  Cyclical Behavior: If the  attracting region is  at a  high activity  level,  most of the \navailable  neurons  must  fire  at  every  time  step  in  order  to  maintain  the  activity \npredicted by (8). This forces  network states to be very similar to each other, which, \nin turn,  leads  to  even more  similar successor states and the  network settles  into a \nrelatively  short  limit  cycle  of high  activity  (Figure  2(b\u00bb.  When  the  attracting \nregion is at an  activity level just above switch-off, the  network can  get into a low(cid:173)\nactivity limit cycle mediated by a very small group of high fan-in  neurons  (Figure \n2(c\u00bb.  This  effect,  however,  is  unstable  with  regard  to  initial  conditions  and  the \nvalue of a; it is expected to become less significant with increasing network size. \n\nu-.-------------------------------------------------. \n\n(a) \n\nMean-0.2B7 \nV 8riaDce - 0.0689 \n\n(II) \n\nMean-0.310 \nV8riance - 0.0003 \n\n\u2022 \n\n\u2022 \n\nus \n\nFlrin& Probabilit, \n\n0.75 \n\n1 \n\n\u2022 \n\nG.25 \n\nFIrinc Probabilit, \n\n8.75 \n\n1 \n\nFigure 3: Neuron firing probability histograms for two 120-neuron networks in the effec(cid:173)\ntively aperiodic  phase  (a ~ 0.227).  Graph (a) is  for  a network with random connectivity \ngenerated  through  a  Bernoulli  process  with p  = 0.2,  while  Graph  (b)  is  for  a  network \nwith a fixed fan-in of exactly 24, which corresponds to the mean fan-in for p  = 0.2. \n\n\fPredicting Complex Behavior  in  Sparse  Asymmetric Networks \n\n561 \n\nOne interesting issue that arises in the context of effectively aperiodic behavior is that of \nstate-space sampling within the O(f,i) constraint on activity.  We assess  this by looking \nat the histogram of individual neuron firing rates.  Figure 3(a) shows the histogram for a \n120 neuron network in the effectively aperiodic phase. Clearly, some subspaces are being \nsampled much  more than others  and the histogram is  very broad. This is  mainly due  to \ndifferences in the fan-in of individual neurons, and will diminish in larger networks.  Fig(cid:173)\nure 3(b) shows the neuron firing histogram for a  120 neuron network where each neuron \nhas a  fan-in  of 24.  The sampling is clearly much more \"ergodic\" and the dynamics  less \nbiased towards certain subspaces. \n\n0.75 \n\n0.2.5 \n\no-+----------------------~--------------------------------------------------~------------------------~ \n\no \n\n0.2.5 \n\n0.75 \n\nFigure 4: The complete set of non-zero activation values available to  two  identical neu(cid:173)\nrons i  and j  with fan-in 24 in a 12O-neuron network. \n\n\f562 \n\nMinai and Levy \n\n4  ACTIVATIONDYNAMICS \nWhile  our modeling  so  far  has  focused  on neural  firing,  it is  instructive  to  look at the \nunderlying  neuron  activation  values, Yi.  If m (t -1) = M , the  possible Yi (t) values  for  a \nneuron i  with fan-in ki  are given by the set \n\ny(M, ti) ={  wq :qKM I MAX(O,ki-n+M)SqSMIN(M, til}  M  > 0 \n\n(10) \n\nn \n\nwith  Y (0, ki ) ==  {O}.  Here  q  represents  the  number  of active  inputs  to  i, and  the  set \nYi  == ~ Y(M, ki) represents the set of all possible activation values for  the neuron. The \nnetwork's n -dimensional activation state, y(t) ==  [y It Y2, ... , Yn ], evolves upon the activa(cid:173)\ntion  space  Y 1 X  Y 2 X  \u2022\u2022\u2022  x Y n' which  is  an  extremely complex  but regular  object.  In \ncalled  a  Y -Y  plot - of the \nFigure  4,  we  plot  a  2-dimensional  subspace  projection  -\nactivation space for a 120-neuron network excluding the zero states. Both neurons shown \nhave  a  fan-in  of 24.  In actuality, only a small subset of the  activation space  is  sampled \ndue to the constraining effects of the dynamics and the improbability of most q  values. \n\n5  RELATING THE ACTMTY LEVEL TO ex \nFrom a practical standpoint, it would be useful to know how the average activity in a net(cid:173)\nwork is related to its a. parameter. This can be done using the hyperbolic tangent approxi(cid:173)\nmation of Equation (6). First, we define the activity level at time t  as  r (t) ==  n-1m (t), i.e., \nthe  proportion of active  neurons.  This  is a  macrostate  variable  in the  sense  of (Amari, \n1974). In the long term, the activity level becomes confined to a o (1/...Jn) region around \nthe  value corresponding to the  activity fixed  point  Thus, it is reasonable  to use r as  an \nestimate for the time-averaged activity level (r). To relate m  (and thus r) to a, we must \nsolve the fixed point equation m = n p(m ). Substituting this and the definition of r into \n\nl-r--------~~--------------------~ \n\n0.75 \n\n., \n\n.-r----~~----~----~------.-----~ \n\n0.04 \n\nII  ~ \n\n0.1 \n\n\u2022 \n\nFigure  5:  Predicted  and  empirical  activities  for  1000  neuron  networks  with  p  = 0.05. \nEach data point is averaged over 7 networks. \n\n\fPredicting  Complex Behavior in  Sparse Asymmetric  Networks \n\n563 \n\n(6) gives: \n\na(r) = p  + ~ 1tp (I-p) tanh-I (1  - 2r ) \n\n2nr \n\n(11) \n\nWhile a  can range  from 0  to  1,  the  approximation of (11)  breaks down at very high or \nvery small values of r. However, the range of its applicability gets wider as n  increases. \nFigure  5  shows  the  perfonnance  of (11)  in  predicting  the  average  activity  level  in  a \n1000-neuron network. Note that a = p  always leads to r = 0.5 by Equation (11). \n6 CONCLUSION \nWe have studied a general class of asymmetric networks and have developed a statistical \nmodel  to relate its dynamical behavior to its parameters. This behavior, which is  largely \ncharacterized by a composite parameter a, is richly varied.  Understanding such behavior \nprovides  insight into  the  complex  possibilities offered  by sparse  asymmetric  networks, \nespecially  with  regard to modeling  such brain regions  as  the  hippocampal  CA3  area in \nmammals.  The  complex  behavior of random  asymmetric  networks  has  been  discussed \nbefore by Parisi (Parisi,  1986), Niitzel  (Niitzel,  1991), and others. We show how to con(cid:173)\ntrol this complexity in our networks by setting parameters appropriately. \n\nAcknowledgements:  This  research  was  supported  by  NIMH  MH00622  and  NIMH \nMH48161  to WBL, and by the  Department of Neurosurgery,  University of Virginia, Dr. \nJohn A. Jane, Chairman. \n\nReferences \n\nS.  Amari  (1972).  Learning  Patterns  and Pattern  Sequences  by Self-Organizing  Nets  of \nThreshold Elements.  IEEE Trans. on Computers C-21, 1197-1206 \nS. Amari (1974).  A Method of Statistical Neurodynamics. Kybemetik 14,201-215 \nJ.1.  Hopfield  (1982).  Neural  Networks  and  Physical  Systems with Emergent Collective \nComputational Abilities. Proc. Nat. Acad.  Sci.  USA  79, 2554-2558. \nD.  Marr (1971).  Simple Memory:  A Theory for Archicortex.  Phil.  Trans.  R.  Soc.  Lond. \nB 262, 23-81. \nA.A.  Minai  and  W.B.  Levy  (1992a).  The  Dynamics  of Sparse  Random  Networks.  In \nReview. \nA.A.  Minai and W.B.  Levy (1992b).  Setting the  Activity Level in Sparse Random Net(cid:173)\nworks.  In Review. \n\nK.  Niitzel  (1991).  The  Length  of Attractors  in Asymmetric  Random  Neural  Networks \nwith Deterministic Dynamics.  J. Phys. A: Math.  Gen 24, LI51-157. \nG. Parisi (1982). Asymmetric Neural Networks and the Process of Learning.  J. Phys.  A: \nMath.  Gen.  19, L675-L680. \nH.  Sompolinsky and I. Kanter (1986), Temporal Association in Asymmetric Neural  Net(cid:173)\nworks.  Phys.  Rev. Lett. 57,2861-2864. \n\n\f", "award": [], "sourceid": 592, "authors": [{"given_name": "Ali", "family_name": "Minai", "institution": null}, {"given_name": "William", "family_name": "Levy", "institution": null}]}