{"title": "Can Simple Cells Learn Curves? A Hebbian Model in a Structured Environment", "book": "Advances in Neural Information Processing Systems", "page_first": 125, "page_last": 132, "abstract": null, "full_text": "Can Simple Cells Learn Curves?  A Hebbian Model in a Structured Environment \n\n125 \n\nCan Simple Cells Learn Curves?  A \n\nHebbian Model in a  Structured \n\nEnvironment \n\nWilliam R.  Softky \n\nDaniel M.  Kammen \n\nDivisions of Biology and Physics \n\nDivisions of Biology and  Engineering \n\n103-33 Caltech \n\nPasadena, CA 91125 \nbill@aurel.caltech.edu \n\n216-76  Caltech \n\nPasadena, CA 91125 \n\nkammen@aurel.cns.caltech.edu \n\nABSTRACT \n\nIn the mammalian visual cortex,  orientation-selective 'simple cells' \nwhich detect  straight lines  may be adapted  to detect  curved  lines \ninstead.  We  test  a  biologically  plausible,  Hebbian,  single-neuron \nmodel,  which learns oriented receptive fields  upon exposure  to un(cid:173)\nstructured (noise) input and maintains orientation selectivity upon \nexposure  to  edges  or  bars  of all  orientations  and  positions.  This \nmodel  can  also  learn  arc-shaped  receptive  fields  upon  exposure \nto an environment of only  circular  rings.  Thus,  new  experiments \nwhich try to induce an abnormal (curved) receptive field  may pro(cid:173)\nvide insight into the plasticity of simple cells.  The model suggests \nthat exposing  cells  to only  a  single  spatial  frequency  may induce \nmore  striking spatial frequency  and  orientation  dependent effects \nthan heretofore observed. \n\nIntroduction \n\n1 \nAlthough most mathematical theories of cortical function assume plasticity of indi(cid:173)\nvidual cells,  there is a  strong debate in the biological community between  \"instruc(cid:173)\ntional\"  (plastic)  and  \"selectional\"  (hard-wired)  models of orientation-selective cells \n\n\f126 \n\nSoftky and Kammen \n\n(which we  will  call  \"simple cells\")  in striate visual cortex.  Thus, a  theory of simple \ncell  learning which can make experimental predictions is desirable. \n\n1.1  Overview of Plasticity Experiments \n\nThe  most  illuminating  experiments  addressing  the  plasticity  of visual  cortex  are \ncollectively called  \"stripe-rearing.\"  Such experiments artificially  restrict the visual \nenvironment of animals (usually kittens) toa few straight, dark, parallel lines (e.g.  3 \nvertical stripes.)  In the many cases studied, examination of the visual cortex reveals \nthat animals which viewed such limited visual environments posses more simple cells \ntuned to the exposed orientation than tuned to other orientations.  (For comparison, \nthe  simple  cells  of animals  with  normal  visual  experience  are  equally  distributed \namong all orientations.)  But the observed changes in cell populations can be equally \nwell explained  by \"instructional\" and  \"selectional\"  hypotheses (Stryker  et  al.1978). \n\nAlthough many variations on stripe-rearing have  been tried  (different orientations \nfor  each  eye,  one  eye  closed,  etc.),  only  environments spanning  a  very  restricted \nsubset  (straight lines)  of the  natural environment  have  been studied  (Hirsch  et  al. \n1983,  Blakemore  et  al.  1978,  and  see  references  therein).  Conclusions  regarding \nplasticity have been based on changes in populations of simple cells,  rather than on \nchanges in  individual cells.  Statistical arguments based  on changes in  large  groups \nof cells  are  questionable,  since  the  well-documented  lateral  interactions  between \ncortical neurons may constrain population ratios,  e.g.  limit  the  fraction of neurons \nresponding to a  single orientation. \n\n1.2  New  Experimental Approach \n\nWe propose several experiments to alter the receptive field  (RF) of a single cell (see \nalso Fregnac et  al.  1988).  How might that be done?  The RF ofa simple cell has only \none characteristic spatial frequency (Jones &  Palmer 1987 and ref's therein).  To try \naltering the shape of that RF,  it  is necessary to present a  pattern which is different \nfrom  a  simple  bar  or  edge,  but  is  still  sufficiently  similar  in  spatial  frequency  to \nactivate the same population of retinal cells that detect the bar.  An arc-shaped RF \nsatisfies  this condition;  to generate  an arc-shaped  RF,  an environment  of circular \nrings (rather than bent bars) is necesary, since complete circles lack sharp end-effects \nwhich could  overexcite spatial opponent cells and  thus disturb learning. \nThis paper  proposes  a  very simple  Hebbian  model  of a  neuron,  and  examines  the \nresulting plasticity upon exposure to edge,  bar,  and arc-shaped  stimuli. \n\n2  Mathematical Model \nThe model applies a simple Hebbian learning rule to an array of about 400 synapses. \nThere  are  several important features  of this  model.  One  is  that  the  stimulus is  a \nvisual  environment  of structured  input  (bars,  edges,  or  circles)  rather  than  only \nstochastic (noise) input, as was used in the previous Hebb-Iearning models of Linsker \n(1986)  and Kammen &  Yuille (1988).  (For a review of Hebbian learning and neural \ndevelopment see  Kammen and  Yuille  1990).  Second,  the  input  is  Laplace  filtered \n\n\fCan Simple Cells Learn Curves?  A Hebbian Model in a Structured Environment \n\n127 \n\nto simulate the  retinal  processing stage;  and  third,  all  connections are  rectified  to \nbe excitatory, like  direct afferent input to simple cells. \n\n2.1  Overview \n\nWe  model  the  neuron  as  an  array of non-negative  synapses,  distributed  within  a \ncircular region.  To let  the neuron  \"see\"  a  single  pattern in  the visual  environment \n(see Figure  1,  end of text), the array is overlaid on a much larger positive array (the \nfiltered  image),  which represents the environment.  Each synapse value is multiplied \nby its corresponding input pixel,  and the sum of these products forms  the neuron's \n\"output.\"  If the output is  above a threshold  value, each synapse is changed slightly \nto make it more like its corresponding pixel (the synapse is  increased for  a  positive \npixel, and decreased for a zero pixel.)  If the output is low,  nothing is changed.  This \nprocess  implements  the  correlation-based  (\"Hebbian\")  learning  rule  for  synapse \nmodification.  To  ensure  maturation,  we  presented  roughly  one  million  training \nimages to each neuron.  Because there are many filtered  images,  only one  is  chosen \nat random for  each iteration,  and the neuron is  overlapped at some random spatial \noffset. \n\n2.2 \n\nInput Filtering Process \n\nThe visual environment is a collection of N  black-on-white pictures of a single shape \n(such as straight lines),  at fixed  contrast.  The environment seen  by  the neuron  is \na  set  of  N  filtered  images,  whose  non-negative  elements  are  produced  from  the \npictures by a  rectified,  Laplace-like,  center-surround  process similar  to that of the \nmammalian retina (Van Essen &,  Anderson 1988).  To determine the RF of a mature \narray of synapses,  the combined efficacy of all synapses is calculated for  each pixel, \nand  displayed  as  a  grey scale (white = excitatory,  black = inhibitory).  See  Figure \n2,  at end  of text,  for  several examples of mature RF's. \n\n2.3  Plasticity Under Visual Stimulation \n\nThe neuron's input synapses cover a  circle much smaller than the filtered  image.  A \nsingle exposure to the environment overlaps the synapse array at a random position \non  the  input  image  (chosen  randomly  from  the  training  set).  This  overlap  pairs \neach synapse with an input from a  filter  whose  center has like polarity (on or  off), \nso that each synapse  represents a  definite  polarity of retinal cell. \n\nA  typical  run  involves  perhaps  106  exposures.  There  is  no  time  variable,  so  that \nmotion and  temporal correlations between images are entirely absent.  During each \nexposure a  Hebb  rule  (section  2.4)  changes synaptic  weights  based  on current cell \noutput and  input values.  When  the neuron  is  exposed  to filtered  stochastic input \n(\"noise-rearing\"),  synapses are intitialized randomly.  When the neuron  is  exposed \nto structured environments, synapses are initialized with the orderly synapse arrays \nwhich result from noise-rearing.  (As in animals,  synapses may evolve in response to \nfiltered  random input before they are exposed  to the external environment.) \n\n\f128 \n\nSoftky and Kammen \n\n2.4  A  Choice of Hebb Rules  for Learning Plasticity \n\nHebb  postulated  (1949)  that  neurons  modify  their  synapses  according  to  the  fol(cid:173)\nlowing  rule:  the  synapse  will  increase  in  efficacy  if the  post-synaptic  and  presy(cid:173)\nnaptic excitations are  coincident.  There are many different formulae  which satisfy \nHebb's criterion;  this model explores some simple representative ones.  During each \nexposure  to  input,  the  synapses  are  adjusted  according  to  the  following  type  of \nhard-limited  Hebb  rule: \n\nout \n\nAnd  if (out -\n\nthresh) > 0  : \n\n(out - thresht X  ini  X  growth \nif ini > 0 and syni < 10 \n-(out - thresh)\"  X  decay \nif ini = 0 and syni > 0.5 \no otherwise \n\n(1) \n\n(2) \n\n(3) \n\n(4) \n\nThe constants growth  and  decay  are positive,  and  the exponent  n  is  at  least  one. \nBoth  types  of threshold  depend  on  the  neuron's  recent  output  history:  either  the \naverage  of the  previous  200  outputs,  or  one  half the  maximum  previous  output \n(decaying  by  .9995  each  exposure  until  a  new  m;~  exceeds  it).  This  Hebb  Rule \nassumes that the cell  can detect  the current input  value  before  its modification  by \na  synapse. \n\n2.5  Choice of Parameters \n\nThe constants growth and decay are not sensitive parameters.  We  found  that only \nthree  parameter  regimes  exist:  all synapses  saturate at maximum,  all  saturate at \nminimum, or some  at  maximum and  some  at  minimum.  Only the  latter  regime  is \nof interest, because only it contains structured RF's. \nMost  simulations  used  n  =  1,2,3  with  both  thresholds.  The  threshold  based  on \nmaximum output enhances  learning selectivity,  while  the  averaged  output  version \ncan  be  derived  from  a  principle  of \"excess  information\"  (See  Appendix).  Because \nsimple  cell  RF's  have  approximately  Gaussian  envelopes  (Jones  &  Palmer  1987), \nsome  simulations  were  done  with  Gaussian  envelopes  modulating  the  maximum \nsynapse values.  That modification made no difference  in the results observed. \n\n3  Results and  Discussion \nThe  production  of oriented  RFs  during  exposure  to  unstructured  input  confirms \nprevious  results  by  Linsker  (1986)  and  Yuille  et  al.  (1989),  but  with  some  im(cid:173)\nportant differences.  Like  those  models,  the  neurons  simulated  here  learn  oriented \nstripe-patterns as a  kind  of lowest-energy configuration under exposure to spatially \n\n\fCan Simple Cells Learn Curves?  A Hebbian Model in a Structured Environment \n\n129 \n\ncorrelated  inputs.  But unlike those  models,  we  do not  use:  inhibitory connections \nor synapses;  a  synaptic-density gradient; a  global conservation of synapse strength; \nor adjustable  free  parameters  which can yield  differently-shaped  RFs.  (In  Linsker \n1986  the ratio of \"on\"  to  \"off\"  synapses is  art  adjustable  parameter;  here, on and \noff pixels are represented equally.)  Also,  unlike previous models,  mature RF's could \nhave more than 3 lobes,  depending on  the ratio of filter size  to RF size (Figure  2). \n\nUnder exposure to images of bars at all orientations, the neuron developed a mature \nRF  matching  a  single  one  of them.  Under  exposure  to stripes  of nearly  a  single \norientation, development of a mature RF depended on the stripes' spatial frequency. \nIn  all  cases,  input  patterns  were  learned  much  more  quickly  and  strongly  when \ntheir  spatial  frequency  corresponded  to  the  frequency  of the  Laplace  filters.  For \ninput frequencies near the filter frequency,  the resulting RF had a spatial frequency \nintermediate  between  the  two.  Otherwise,  no  learning  occured  unless  the  input \nfrequency  was a  harmonic  of the filter  frequency,  in  which case  the filter  frequency \nwas  learned.  Thus,  this  model  predicts  that  enhanced  learning  might  take  place \nin  kittens  exposed  to stripes  of a  single  frequency,  if that  frequency  is  typical  of \nsimple-cell RF frequencies. \n\nUnder exposure  to arcs or circles (with diameter ~ 3  x  annular width),  the model \nconsistently  developed  RF's  which  matched a  portion of the  circle.  These  results \nsuggest  that  animals  which  see  only  circles  of a  certain  scale  during  the  critical \nperiod may develop curved RFs (Barrow 1987)  which differ qualitatively from those \nobserved  by such  experiments  as  Jones  &,  Palmer's  (1987),  who  report  seeing  no \ncurved  contours  in  their  point-by-point  mappings  of the  RFs  of  normally-reared \nkittens.  As  with  the stripes,  the circles'  annular width determines  the spatial fre(cid:173)\nquency of the retinal and  simple cells  which will  respond  best. \n\nSuch predictions must be treated with caution, because this paper does not simulate \nany  version  of the  competing  \"selectional\"  model.  It is  possible  that some  of the \neffects predicted here for  the \"instructional\"  Hebbian model could  also  be observed \nby a  \"selectional\"  system. \nTo experimentally  observe  such  effects  in  laboratory  animals,  many  other  known \nbiological  influences  (eye  acuity,  interneuron effects,  etc.)  must  be  accounted  for. \nWe consider such problems elsewhere  (Softky  &,  Kammen  in  preparation),  because \nthey are of secondary importance  to the striking and  robust  results of the model. \n\nIn summary, we  have a single-cell model which contains essential biological features \n(such  as all-excitatory  input  and  synapses,  and  no global  renormalizations).  This \nmodel developes mature, oriented receptive fields under exposure to stochastic input \nfor  a  wide  variety of Hebb  rules  and for  all  non-trivial  parameter  regimes studied, \nwith  no  apparent  limitations  on  the  number  of lobes  learned.  Under  exposure \nto  structured  input  characteristic  of normal  environments,  the  model  maintains \noriented RF's;  under exposure  to input of \"resonant\"  spatial frequency,  the model \ndevelops  RF's  which  reflect  any  novel  orientation,  spatial  frequency,  or  curvature \nof the stimuli.  This general,  rule-independent  response  to the spatial frequency  of \n\n\f130 \n\nSoftky and Kammen \n\na  stimulus - and  the  specific  mechanism for  generating abnormally curved  RF's -\nmay be  useful  in  deciding  experimentally  whether simple  cortical  cells  are  indeed \nmodifiable  by Hebbian mechanisms. \n\nThis model does not attempt to explain curve-detection in a  normal visual system. \nWe already  know  that  normal  simple  cells  are  not  tuned  for  curves,  and  there  are \ncredible  theories  of normal  curve-detection  (Dobbins  et  al.  1987.)  Rather,  this \nmodel proposes using stimuli tuned to the natural spatial frequency  of simple cells \nto induce a RF property which is  distinctly abnormal, in order to better understand \nthe rules by which normal visual properties emerge. \n\n4  Appendix - Choice of Thresholds for  the Hebb  Rule \nThe choice of the average output as a  threshold for  a  Hebb  rule  can be  interpreted \nas  follows.  Consider  a  developing  neuron  whose  output  is  the  sum of N  inputs, \neach  of which  has  independent  probability  distribution  of mean  a  and  standard \ndeviation  u.  We  can  calculate  the  information  content  in  that  sum,  whose  value \nhas probability distribution (from the central limit  theorem) of \n\nP(out) \n\noc \n\n( -(out - (out))2) \n\n2u2 \n\n\u2022 \n\nexp \n\n(5) \n\nThe Shannon information  (Shannon &  Weaver 1962)  carried  by the sequence  is \n\nH( event) \n\n-In P( event) . \n\nThe excess  information above the information carried  by the average is  thus \n\nH(out) - H( < out\u00bb \n(out - (out) )2 \n\n2u2 \n\noc \n\n(6) \n\n(7) \n\n(8) \n\nThus, a  Hebb  rule  using n = 2 and  thresh = (out)  is  equivalent to learning  based \non the excess information carried  in the output of an immature neuron. \nThe alternate threshold ( tmax) enhances selective learning for  the following reason. \nIf we  consider  the  whole ensemble of patterns and  shifts,  the output  characteristic \nwhich  best  distinguishes  a  matched  synapse  pattern  from  a  random  one  is  not  its \naverage output  (the  two averages  are  comparable for  the all-excitatory  case),  but \nits  maximum  output.  Thus,  if a  neuron  can  only  'remember'  one  characteristic \nnumber  to  serve  as  a  threshold,  then  a  number  which  changes  during  evolution \n(e.g.  the maximum output)  will  refine selectivity more than one which is  relatively \nconstant.  In  addition,  storing  a  maximum  rather  than  an  average  removes  the \nneed  to compute a  running average,  allowing unhindered  evolution even after  long \nperiods of no input. \n\n\fCan Simple Cells Learn Curves?  A Hebbian Model in a Structured Environment \n\n131 \n\nAcknowledgements \n\nD.K.  is a  Weizmann Postdoctoral Fellow and acknowledges support from the Weiz(cid:173)\nmann  Foundation,  the  James  S.  McDonnell  Foundation  and  a  NSF  Presidential \nYoung Investigator A ward  to Christof Koch. \n\nReferences \n\nBarrow,  H.  (1987)  \"Learning  Receptive  Fields.\"  First  I.E.E.E.  Conference  on \n\nNeural Networks,  IV, 115-121. \n\nBlakemore C.,  Movshon J.A.,  &  Van Sluyters R.C.  (1978)  \"Modification of the \nKitten's  Visual  Cortex  by  Exposure  to Spatially  Periodic  Patterns.\"  Exp.  Brain \nRes.,  31, 561-572. \n\nDobbins A.,  Zucker S.  &  Cynader M.  (1987)  \"Endstopped Neurons in the Visual \n\nCortex as a  Substrate for  Calculating Curvature.\"  Nature,  329, 438-44l. \n\nFregnac Y.,  Shultz D.,  Thorpe S.  &  Bienenstock E.  (1988)  \"A cellular analog of \n\nvisual cortical  plasticity.\"  Nature,  333, 367-370. \n\nHebb,  D.O.  (1949)  \"The Organization of Behavior:  A  Neuropsychological The(cid:173)\n\nory.\"  Wiley  &  Sons,  New  York. \n\nHirsch  H.,  Leventhal A.,  McCall  M.  &  Tieman  D.  (1983)  \"Effects  of Exposure \nto Lines  of One or  Two Orientations on  Different  Cell  Types in Striate Cortex of \nCat.\"  1.  Physiol.,  337, 241-255. \n\nJones J. &  Palmer L.  (1987)  \"The Two-Dimensional Spatial Structure of Simple \n\nReceptive Field  in Cat Striate Cortex.\"  1.  Neurophys.,  58,  1187-1232. \n\nKammen  D.M.  &  Yuille  A.  (1988)  \"Spontaneous  Symmetry-Breaking  Energy \n\nFunctions and  the  Emergence  of Orientation Selective  Cortical Cells.\"  Bioi.  Cy(cid:173)\nbern.,  59,  23-31. \n\nKammen  D.M.  &  Yuille  A.  (1990)  \"Self-Organizing  Networks of Neural  Units: \n\nHebbian Learning in Development and Biological Computing.\"  In:Advances in Con(cid:173)\ntrol Networks and Large Scale  Distributed Processing Models,  Ablex Publishing, New \nJersey. \n\nLinsker R.  (1986)  \"From basic  network  principles to neural architecture:  Emer(cid:173)\ngence of orientation-selective cells.\"  Proc.  Natl.  Acad.  Sci.  USA,  83, 8390-8394. \n\nShannon C.  &  Weaver W  (1962)  The  Mathematical  Theory  of Communication, \n\nUniv.  of Illinois Press,  Urbana. \n\nStryker M.,  Sherk  H.,  Leventhal A.  &  Hirsch  H.  (1978)  \"Physiological  Conse(cid:173)\nquences for  the Cat's Visual Cortex of Effectively Restricting Early Visual Experi(cid:173)\nence  with Oriented Contours.\"  1.  Neurophys.,  41, 896-909. \n\nVan  Essen  D.  &  Anderson  C.  (1988)  \"Information  Processing  Strategies  and \nto  Neural  and \n\nPathways  in  the  Primate  Retina  and  Visual  Cortex.\" \nElectronic  Networks,  Academic  Press,  Florida. \n\nIn:  Intro. \n\nYuille A.,  Kammen D.M.  &  Cohen D.  (1989)  \"Quadrature and the Development \nof Orientation Selective Cortical Cells  by Hebb Rules.\"  Bioi.  Cybern.,  61, 183-194. \n\n\f132 \n\nSoftky and Kammen \n\n1) \n\n3)  Filter retinal \n\nfield,  then  expose \nthe filtered image \nto the  neuron's \nsynapses. \n\nCalculate  neuron's \noutput, then adjust \nsynaptic weights \naccording to Hebb \nrule. \n\n2)  Place retinal field at a \n\nrandom location on the image. \n\nFigure 1:  Synapses Change Slightly During Each of a  Million Iterations \n\nFigure 2:  Learned  Receptive Fields.  Top row:  Random pixel input,  large  (1)  and \nsmall (r)  filter  sizes.  Bottom row:  Structured input, circular rings  (1)  and edges at \ndifferen t  orientations (r). \n\n\f", "award": [], "sourceid": 252, "authors": [{"given_name": "William", "family_name": "Softky", "institution": null}, {"given_name": "Daniel", "family_name": "Kammen", "institution": null}]}