{"title": "A Winner-Take-All Circuit with Controllable Soft Max Property", "book": "Advances in Neural Information Processing Systems", "page_first": 717, "page_last": 723, "abstract": null, "full_text": "A  Winner-Take-All  Circuit  with \nControllable  Soft  Max Property \n\nShih-Chii Lin \n\nInstitute for  Neuroinformatics,  ETHjUNIZ \n\nWinterthurstrasse 190,  CH-8057 Zurich \n\nSwitzerland \n\nshih@ini.phys.ethz.ch \n\nAbstract \n\nI describe a silicon network consisting of a group of excitatory neu(cid:173)\nrons  and a  global inhibitory neuron.  The output of the inhibitory \nneuron is normalized with respect to the input strengths.  This out(cid:173)\nput models the normalization property of the wide-field direction(cid:173)\nselective cells in the fly  visual system.  This normalizing property is \nalso useful  in any system where we  wish  the output signal to code \nonly the strength of the inputs, and not be dependent on the num(cid:173)\nber of inputs.  The circuitry in each neuron is equivalent to that in \nLazzaro's winner-take-all  (WTA)  circuit  with one additional tran(cid:173)\nsistor  and  a  voltage  reference.  Just  as  in  Lazzaro's  circuit,  the \noutputs of the excitatory neurons code the neuron with the largest \ninput.  The difference here is  that multiple winners can be chosen. \nBy  varying  the  voltage  reference  of the  neuron,  the  network  can \ntransition  between  a  soft-max  behavior  and  a  hard  WTA  behav(cid:173)\nior.  I show results from  a fabricated chip of 20  neurons in a  1.2J.Lm \nCMOS technology. \n\n1 \n\nIntroduction \n\nLazzaro  and  colleagues  (Lazzaro,  1988)  were  the  first  to  implement  a  hardware \nmodel of a  winner-take-all  (WTA)  network.  This network consists of N excitatory \ncells  that are inhibited by  a  global signal.  Improvements of this network with  ad(cid:173)\ndition  of positive  feedback  and  lateral  connections  have  been  described  (Morris, \n1998;  Indiveri,  1998).  The  dynamics  and  stability  properties  of networks  of cou(cid:173)\npled excitatory and inhibitory neurons have been analyzed by  many (Amari,  1982; \nGrossberg,  1988).  Grossberg described conditions under which these networks will \nexhibit WTA behavior.  Lazzaro's network computes a single winner as reflected by \nthe outputs of the  excitatory  cells.  Several  winners  can  be chosen  by  using  more \nlocalized inhibition. \nIn  this work,  I  describe two  variants of a similar architecture where the outputs of \nthe excitatory neurons  code  the  relative  input  strengths as  in  a  soft-max compu(cid:173)\ntation.  The  relative  values  of the outputs  depend  on the  number of inputs,  their \nrelative strengths and two parameter settings in the network.  The global inhibitory \n\n\f718 \n\n8.-c.  Liu \n\nFigure 1:  Network model of recurrent inhibitory network. \n\nsignal  can  also  be  used  as  an output.  This output saturates with  increasing num(cid:173)\nber of active  inputs,  and  the  saturation level  depends  on the  input  strengths and \nparameter settings.  This normalization property is similar to the normalization be(cid:173)\nhavior of the wide-field direction-selective cells in  the fly  visual system.  These cells \ncode the temporal frequency of the visual inputs and are largely independent of the \nstimulation size.  The circuitry in  each  neuron  in  the silicon  network is  equivalent \nto that  in  Lazzaro et.  al.'s  hard  WTA  network  with  an  additional  transistor  and \na  voltage reference.  By  varying  the  voltage  reference,  the  network  can  transition \nbetween  a  soft-max computation and  a  hard WTA  computation.  In the two  vari(cid:173)\nants,  the outputs of the  excitatory neurons either  code  the strength of the inputs \nor  are  normalized  with  respect  to  a  constant  bias  current.  Results  from  a  fabri(cid:173)\ncated network of 20  neurons in  a  1.2J.Lm  AMI  CMOS  show the different  regimes of \noperation. \n\n2  Network with Global Inhibition \n\nThe generic architecture of a recurrent network with excitatory neurons and a single \ninhibitory neuron is  shown in  Figure 1.  The excitatory neurons receive  an external \ninput, and they synapse onto a global inhibitory neuron.  The inhibitory neuron, in \nturn, inhibits the excitatory neurons.  The dynamics  of the network is  described as \nfollows: \n\ndYi \ndt =  -Yi + ei  - g(~ WjYj) \n\nN \n~ \n\n(1) \n\nwhere  Wj  is  the  weight  of the synapse between  the jth excitatory neuron  and the \ninhibitory neuron,  and Yj  is  the state of the jth neuron.  Under steady-state condi-\ntions,  Yi  = ei  - YT,  where YT  = g(L:~l WjYj)\u00b7 \nAssume a  linear relationship between YT  and Yj,  and letting Wj  =  W, \n\nj=l \n\n\"N \n_  ~  _  W  L..Jj=l ej \n1 + wN \n\nYT  - W ~ Yj  -\n\nN \n\nj=l \n\nAs  N  increases, YT = L:;l ej  \u2022  If all  inputs have the same level,  e,  then YT = e. \n\n\fA Winner-Take-All Circuit with Controllable Soft Max Property \n\n719 \n\nFigure 2:  First variant of the architecture.  Here we show the circuit for two excita(cid:173)\ntory neurons and the global  inhibition neuron,  M 4 \u2022  The circuit in each excitatory \nneuron  consists  of an  input  current  source,  h, and  transistors,  M1  to  M 3 .  The \ninhibitory  transistor  is  a  fixed  current  source,  lb .  The  inputs  to  the  inhibitory \ntransistor,  101  and  I~2 are normalized with respect to lb. \n\n3  First Variant  of Network with Fixed  Current  Source \n\nIn  Sections  3  and  4,  I  describe  two  variants  of the  architecture  shown  in  Figure \n1.  The two  variants differ  in  the  way  that the inhibition signal is  generated.  The \nfirst  network  in  Figure  2  shows  the  circuitry  for  two  excitatory  neurons  and  the \ninhibition neuron.  Each excitatory neuron is a linear threshold unit and consists of \nan input current, h, and transistors,  Ml,  M 2 ,  and  M 3 .  The state of the neuron is \nrepresented by  the current,  Ir1 .  The diode-connected transistor,  M 2 ,  introduces a \nrectifying nonlinearity into the system since Ir1  cannot be negative.  The inhibition \ncurrent, Ir, is sunk by M 1 ,  and is determined by the gate voltage, VT.  The inhibition \nneuron consists of a  current source,  Ib,  and VT  is  determined by the corresponding \ncurrent,  Ir1  and the corresponding transistor,  M3  in each neuron.  Notice that  IT \ncannot  be  greater  than  the  largest  input  to  the  network  and  the  inputs  to  this \nnetwork  can  only  be  excitatory.  The  input  currents  into  the  transistor,  M 4 ,  are \ndefined as 101  and 102  and are normalized with respect to the current source, h. In \nthe hard WTA  condition,  the output current of the winning neuron is equal to the \nbias  current, h. \nThis  network  exhibits  either  a  soft-maximum  behavior  or  a  hard  WTA  behavior \ndepending  on  the  value  of  an  external  bias,  Va.  The  inhibition  current,  IT,  is \nderived  as: \n\n(2) \n\nwhere  N  is  the  number  of  \"active\"  excitatory  neurons  (that  is,  neurons  whose \nIi  >  IT),  Ii  is  the  same  input  current  to  each  neuron,  and  Ia  =  Ioe\",vQ/uT.  In \nderiving the above equation, we  assumed that  K,  =  1.  The inhibition current, IT,  is \na  linear combination of the states of the neurons because Ir =  2:f Iri  x  Ial h\u00b7 \nFigure 3(a)  shows  the  response of the  common-node  voltage,  VT,  as  a  function  of \nthe number of inputs for  different  input  values  measured from  a  fabricated silicon \nnetwork of 20  neurons.  The input  current  to each  neuron is  provided  by  a  pFET \ntransistor that  is  driven  by  the  gate  voltage,  Yin.  All  input  currents  are equal in \nthis  figure.  The saturation  behavior  of the  network  as  a  function  of the  number \n\n\f720 \n\ns.-c.  Liu \n\n0.8,---~-~-~-~-~----, \n\nVin=3.9V \n\n-' . \n\n0.7 \n\n0.6 \n\n.II'~''''--.'' ......... \u2022\u2022 ' .. - - - ... ... ..... .. \n\nVin=4.3V \n\n5 \n\n10 \n\n20 \nNumber of inputs \n\n15 \n\n25 \n\n30 \n\n5 \n\n10 \n\n20 \nNumber of inputs \n\n15 \n\n25 \n\n30 \n\n(a) \n\n(b) \n\nFigure  3:  (a)  Common-node  voltage,  VT,  as  a  function  of  the  number  of input \nstimuli.  Va  =  O.8V.  (b)  Common-node  voltage,  VT,  as  a  function  of the  number \nof inputs  with  an input  voltage of 4.3V  and Vb  =  O.7V.  The curves  correspond  to \ndifferent  values of Va . \n\nof inputs  can  be  seen  in  the  different  traces  and  the saturation level  increases  as \nVin  decreases.  As  seen in Equation 2,  the point  at which  the response saturates is \ndependent  on  the ratio,  h / I a.  In  Figure  3(b),  I  show  how  the  curve saturates at \ndifferent  points for  different  values of Va  and a fixed  hand Vin. \nIn Figure 4,  I set all inputs to zero except for  two inputs, Vin1  and Vin2  that are set \nto the same  value.  I  measured 101  and  101  as  a  function  of Va  as shown  in  Figure \n4(a).  The four  curves  correspond to four  values of Vin.  Initially  both currents 101 \nand  102  are  equal  as  is  expected  in  the  soft-max  condition.  As  Va  increases,  the \nnetwork starts exhibiting a WTA behavior.  One of the output currents finally  goes \nto  zero  above  a  critical  value  of Va.  This  critical  value  increases  for  higher  input \ncurrents  because  of  transistor  backgate  effects.  In  Figure  4(b),  I  show  how  the \noutput  currents  respond  as  a  function  of the  differential  voltage  between  the  two \ninputs as shown in Figure 4.  Here,  I fixed  one input at 4.3V and swept the second \ninput differentially around it.  The different  curves correspond to different values of \nVa.  For a  low  value of Va,  the linear differential input range is  about  lOOmV.  This \nlinear range decreases as  Va  is  increased (corresponding to the WTA  condition). \n\n4  Second Variant  with Diode-Connected Inhibition \n\nTransistor \n\nIn  the second  variant shown  in  Figure  5,  the  current  source,  M4  is  replaced  by  a \ndiode-connected  transistor  and  the  output  currents,  10i'  follow  the  magnitude  of \nthe input currents.  The inhibition current,  Ir,  can be expressed as follows: \n\nwhere la  is defined in Section 3.  We sum Equation 3 over all neurons and assuming \nequal  inputs,  we  get  Ir  =  J'LJri x la.  This  equation  shows  that  the  feedback \nsignal  has  a  square  root  dependence  on  the  neuron  states.  As  we  will  see,  this \ncauses the feedback signal to saturate quickly with the number of inputs. \n\n(3) \n\n\fA  Winner-Take-All Circuit with Controllable Soft Max Property \n\n721 \n\n6 \n\n5 \n~4 \n~ , \n\n2 \n\nr-... \n\n\".\\'0\" \n\n2.5  Va=O 5~'~\\ I /~. \" \nVa=O 6V  Al\\ ',/'  Va=O.4V \n~ 2 \n-: 1.5 \n..s \n\nVa=O.7V \n\n,i \nl~ \n) ' ~  ,. \n,  <\\  ,,~ \n.. \\  \\> \n/ \n\n0.5 \n\n<  .,.~J  \\  \\ '\" \n\n0 \n\n0.1 \n\n0.2 \n\n0.3 \n\n-0.2 \n\n-0.1 \n\nVio2-Viol  (V) \n\n(a) \n\n(b) \n\nFigure 4:  (a)  Output  currents,  101  and  102 ,  as  a  function  of Va:  for  a  subthreshold \nbias current and Yin  =  4.0V to 4.3V.  (b)  Outputs,  101  and 102 ,  as a function of the \ndifferential input  voltage,  ~ Vin,  with  Yinl  =  4.3V. \n\nFigure 5:  Second variant of network.  The schematic shows two excitatory neurons \nwith diode-connected inhibition transistor. \n\nSubstituting lri =  Ii - IT  in  Equation 3,  we  solve for  Jr, \n\nIT =  -1a:N  + \n\n(Ia: N )2  + 41a: L Ii \n\nN \n\n(4) \n\nFrom measurements from  a fabricated circuit  with 20  neurons,  I show the depen(cid:173)\ndence of VT  (the natural logarithm of Jr)  on the number of inputs in  Figure 6(a). \nThe output saturates quickly with the number of inputs and the level of saturation \nincreases with increased input strengths.  All  the inputs have the same value. \nThe network can  also  act  as  a  WTA  by  changing Va:.  Again,  all  inputs  are set to \nzero except  for  two  inputs  whose  gate  voltages  are  both set  at  4.2V.  As  shown  in \nFigure 6(b), the output currents, 101  and 102 , are initially equal, and as Va:  increases \nabove 0.6V, the output currents split apart and eventually, 102  =  OA.  The final value \nof 101  depends on the maximum input  current.  This data shows  that the network \nacts as  a  WTA  circuit  when  Va:  > 0.73V.  If I set Vin2  =  4.25V instead, the output \ncurrents split  at a  lower value  of Va:. \n\n\f722 \n\ns.-c.  Liu \n\n0.45.--~-~-~-~-~------, \n\nVinl=4.2V. Vin2=4.25V \n\n5 \n\n4 \n$ \n~3 \n\n2 \n\n0.20'---:-2 -~4-~6:--8=---1-:':0:----:'12 \n\nNumber of inputs \n\n(a) \n\n0.6 \n\n0.7 \n\nVa (V) \n(b) \n\n0.8 \n\n0.9 \n\nFigure 6:  (a)  Common-node voltage,  VT,  as a function  of the number of inputs for \ninput  voltages,  3.9V,  4.06V,  and 4.3V  for  Va  = O.4V.  (b)  Outputs,  101  and  1 02 ,  as \na  function  of Va  for  Vinl  = 4.2V,  Vin2  = 4.25V for  the 2  curves with  asterisks and \nfor Vinl  = Vin2  = 4.2V for  the 2 curves with circles. \n\n5 \n\nInhibition \n\nThe  WTA  property  arises  in  both variants of this  network if the gain  parameter, \nVa,  is  increased so  that the diode-connected transistor,  M 2 ,  can  be ignored.  Both \nvariants then reduce to Lazzaro's network.  In the first variant, the feedback current \n(Ir) is a linear combination ofthe neuron states.  However, when the gain parameter \nis  increased  so  that  M2  can  be  ignored,  the  feedback  current  is  now  a  nonlinear \ncombination of the input states so the WTA behavior is exhibited by these reduced \nnetworks. \nUnder hard WTA conditions, if Ir is initially smaller than all the input currents, the \ncapacitances C at the nodes Vr1  and Vr2  are charged up  by the difference  between \nthe individual input current and IT,  i.e.,  d~t =  liCIT.  Since the inhibition current \nis  a  linear  combination  of  Iri  and  Iri  is  exponential  in  Vri ,  we  can  see  that  IT  is \na  sum of the exponentials of the  input  currents,  h  Hence the feedback  current  is \nnonlinear in the input currents.  Another way of viewing this condition in electronic \nterms is that in the soft  WTA condition,  the output node of each neuron is  a  soft(cid:173)\nimpedance node, or a low-gain node.  In the hard WTA case, the output node is now \na high-impedance node or a high-gain node.  Any input differences are immediately \namplified in the circuit. \n\n6  Discussion \n\nHahnloser (Hahnloser, 1998) recently implemented a silicon network of linear thresh(cid:173)\nold excitatory neurons that are coupled to a global inhibitory neuron.  The inhibitory \nsignal is  a  linear combination of the output states of the excitatory neurons.  This \nnetwork  does  not  exhibit  WTA  behavior  unless  the  excitatory  neurons  include  a \nself-excitatory term.  The inhibition  current in  his  network is  also  generated via  a \ndiode-connected  transistor.  The  circuitry  in  two  variants  described  here  is  more \ncompact  than the circuitry in  his  network. \n\nRecurrent networks with the architecture described in this paper have been proposed \nby Reichardt and colleagues (Reichardt, 1983) in modelling the aggregation property \n\n\fA  Winner-Take-All Circuit with Controllable Soft Max Property \n\n723 \n\nof the wide-field  direction-selective  cells in flies.  The synaptic inputs  are inhibited \nby  a  wide-field  cell  that pools  all  the synaptic inputs.  Similar networks  have  also \nbeen used to model cortical processing, for example, orientation selectivity (Douglas, \n1995). \n\nThe network implemented here can model the aggregation property of the direction(cid:173)\nselective  cells  in  the fly.  By  varying  a  voltage  reference,  the  network  implements \neither a  soft-max computation or a  hard WTA  computation.  This circuitry will  be \nuseful  in  hardware models  of cortical  processing or  motion  processing  in  inverte(cid:173)\nbrates. \n\nAcknowledgments \n\nI  thank Rodney  Douglas for  supporting this  work,  and the  MOSIS  foundation  for \nfabricating  this  circuit.  I  also  thank Tobias  Delbriick for  proofreading this  docu(cid:173)\nment.  This work was supported in part by the Swiss National Foundation Research \nSPP grant and the U.S.  Office of Naval Research. \n\nReferences \n\nAmari, S.,  and Arbib,  M.  A.,  \"Competition and cooperation in neural networks,\" \nNew  York, Springer-Verlag, 1982. \n\nGrossberg, W.,  \"Nonlinear neural networks:  Principles,  mechanisms,  and architec(cid:173)\ntures,\"  Neural  Networks,  1, 17-61, 1988. \nHanhloser,  R.,  \"About the  piecewise  analysis of networks of linear threshold  neu(cid:173)\nrons,\"  Neural  Networks,  11,691- 697,  1988. \nHahnloser,  R.,  \"Computation  in  recurrent  networks  of linear  threshold  neurons: \nTheory,  simulation  and  hardware  implementation,\"  Ph.D.  Thesis,  Swiss  Federal \nInstitute of Technology,  1998. \n\nLazzaro, J., Ryckebusch,  S.  Mahowald, M.A.,  and Mead.  C.,  \"Winner-take-all net(cid:173)\nworks of O(n)  complexity,\"  In Tourestzky,  D.  (ed),  Advances in Neural Information \nProcessing Systems 1,  San Mateo,  CA:  Morgan Kaufman Publishers,  pp.  703-711, \n1988. \n\nMorris,  T .G.,  Horiuchi,  T.  and  Deweerth,  S.P.,  \"Object-based selection  within  an \nanalog  VLSI  visual  attention system,\"  IEEE  Trans.  on  Circuits  and  Systems  II, \n45:12,  1564-1572, 1998. \n\nIndiveri, G.,  \"Winner-take-all networks with lateral excitation,\"  Neuromorphic  Sys(cid:173)\ntems  Engineering,  Editor,  Lande,  TS.,  367-380,  Kluwer  Academic,  Norwell,  MA, \n1998. \nReichardt,  W.,  Poggio, T.,  and Hausen, K.,  \"Figure-ground discrimination by  rel(cid:173)\native movement in the visual system of the fly,\"  BioI.  Cybern.,  46,  1-30,  1983. \nDouglas, RJ., Koch, C., Mahowald, M.,  Martin, KAC., and Suarez, HH.,  \"Recurrent \nexcitation in neocortical circuits,\"  Science,  269:5226,981-985, 1995. \n\n\f", "award": [], "sourceid": 1725, "authors": [{"given_name": "Shih-Chii", "family_name": "Liu", "institution": null}]}