{"title": "A Neural Model of Visual Contour Integration", "book": "Advances in Neural Information Processing Systems", "page_first": 69, "page_last": 75, "abstract": null, "full_text": "A  neural model of visual  contour \n\nintegration \n\nComputer Science,  Hong  Kong University of Science and Technology \n\nZhaoping Li \n\nClear Water Bay,  Hong Kong \nzhaoping~uxmail.ust.hkl \n\nAbstract \n\nWe  introduce  a  neurobiologically plausible model  of contour inte(cid:173)\ngration from visual inputs of individual oriented edges.  The model \nis  composed  of interacting  excitatory  neurons  and  inhibitory  in(cid:173)\nterneurons, receives visual inputs via oriented receptive fields  (RFs) \nlike those in VI.  The RF centers are distributed in space.  At each \nlocation,  a  finite  number  of cells  tuned  to  orientations  spanning \n1800  compose a  model hypercolumn.  Cortical interactions modify \nneural activities  produced by  visual  inputs,  selectively amplifying \nactivities for edge elements belonging to smooth input contours.  El(cid:173)\nements within one contour produce synchronized neural activities. \nWe  show  analytically  and  empirically  that  contour  enhancement \nand  neural  synchrony  increase  with  contour  length,  smoothness \nand closure, as  observed experimentally.  This model gives testable \npredictions, and in  addition, introduces a  feedback  mechanism  al(cid:173)\nlowing  higher  visual  centers  to  enhance,  suppress,  and  segment \ncontours. \n\n1.  Introduction \n\nThe  visual  system  must  group  local  elements  in  its  input  into  meaningful  global \nfeatures to infer the visual objects in the scene.  Sometimes local features group into \nregions, as  in  texture segmentation; at other times they group into contours which \nmay  represent  object  boundaries.  Although  much  is  known  about  the  processing \nsteps that extract local features such as  oriented input edges, it is still unclear how \nlocal  features  are  grouped  into  global  ones  more  meaningful  for  objects.  In  this \n\n1r  would  very  much  like  to  thank Jochen  Braun for  introducing me to  the topic,  and \nPeter Dayan for  many helpful  conversations and comments on the drafts.  This work  was \nsupported by the Hong Kong Research  Grant Council. \n\n\f70 \n\nZ.  Li \n\nstudy,  we  model  the neural mechanisms underlying the  grouping of edge elements \ninto contours -\n\ncontour integration. \n\nRecent psychophysical and physiological observations[14, 8]  demonstrate a  decrease \nin detection threshold of an edge element, by human observers or a primary cortical \ncell, if there are aligned neighboring edge elements.  Changes in neural responses by \nvisual stimuli presented outside their RFs have been observed physiologically[9,  8]. \nHuman observers easily identify a  smooth curve composed  of individual,  even dis(cid:173)\nconnected,  Gabor  \"edge\"  elements  distributed among many similar elements scat(cid:173)\ntered in the background[4].  Horizontal neural connections observed in  the primary \nvisual cortex[5],  and  the finding  that these connections  preferably link cells  tuned \nto similar  orienations[5],  provide  a  likely  neural  basis  underlying  the  primitive vi(cid:173)\nsual grouping phenomena such as contour integration.  These findings  suggest that \nsimple and local  neural interactions even in VI could contribute to grouping. \n\nHowever, it has been difficult  to model contour integration using only VI elements \nand operation.  Most existing models[15, 18] of contour integration lack well-founded \nbiological  bases.  More  neurally  based  models,  e.g.,  the  one  by  Grossberg  and \nMingolla[7],  require  operations  beyond  VI  or  biologically  questionable.  It is  thus \ndesirable to find  out whether contour enhancement can indeed  occur within VI  or \nhas  to be  attributed to  top-down  feedback.  We  introduce  a  VI  model  of contour \nintegration,  using  orientation selective  cells,  local  cortical  circuits,  and  horizontal \nconnections.  This model captures the essentials of the contour integration behavior. \nMore details of the model can be found  in a  longer paper[12]. \n\n2.  The Model \n\n2.1  Model outline \n\nK  neuron  pairs  at each  spatial location  i  model  a  hypercolumn  in  VI  (figure  1). \nEach  neuron  has  a  receptive  field  center  i  and  an optimal  orientation fJ  = k1r / K \nfor  k  =  1,2, ... K.  A  neuron  pair  consist  of  a  connected  excitatory  neuron  and \ninhibitory  neuron  which  are  denoted  by  indice  (ifJ)  for  their  receptive  field  center \nand preferred orientation, and are referred to as an edge segment.  An edge segment \nreceives the visual input via the excitatory cell, whose output quantifies the saliency \nof the edge segment and projects to higher visual  centers.  The inhibitory cells  are \ntreated as  interneurons.  When an input image contains an edge at i  oriented at fJo, \nthe edge  segment  ifJ  receives  input  Ii9  (X  </J(fJ  - fJo ),  where  </J(fJ)  =  e- 191 /(1I\"/8)  is  a \ncell's orientation tuning curve. \n\nVisual space, bypen:oluDlDS \nand neural edge segmenos \n\nexcitatory \n\n~~~ \n~ ~--~ ~ \n\n~:~j .. ~ \n\n~~~~ \n\nA hype<oolumn  A oeuraI edge \natlocalion i \n\nsegment i8 \n\nAn intercon \n\nneuron pair (ClI' \nedge segment ill \n\nEdge outputs to higher visual areas \n\nFigure 1:  Model visual space, hypercolumn, edge segments, neural elements, visual inputs, \nand neural connections.  The input space is  a discrete  hexagonal or  Manhatten grid. \n\n\fA Neural Model o/Visual Contour Integration \n\n71 \n\nModel  visual  input \n\nModel  output \n\n'~ 1\\-\n\n, \n\n, \n\n1 \\1 ' \\ - \\ '   -\n\n- -\n, .... ,...... \n\" \nr./,,,I\\  \\', \n............... \n\" \n,I  A  (I  -\n,  ... \n,1  -\n...  \\  \"'....'.,!' ...  ,' \n\nII.... \nI \n\n, \n\n-\n\\ \n\n, \n\n'..{...  I ,   I, \n\nOutput after  removing ed,ges \nof activities lower  than 1/\"1. \nof the most  active edge \n\n\"\\ \nI \n. ............. . \nI \n\\.. \n\nFigure 2:  Model  neural connections and performance for  contour enhancement  and noise \nreduction.  The top graph depicts connections  J;(J,j(J1  and W;(J,j(J1  respectively  between the \ncenter  (white)  horizontal edge and other edges.  The whiteness  and blackness of each edge \nis  proportional  to  the  connection  sthength  Ji(J,j(J1  and  W;(J,j(J1  respectively.  The  bottom \nrow plots visual input and the maximum model outputs.  The edge thickness in this row is \nproportional to the value of edge input or  activity.  The same format  applies to the other \nfigures  in this paper. \n\nLet  Xifl  and YiO  be the cell  membrane potentials  for  the excitatory and  inhibitory \ncells  respectively in the edge segment, then \n\n-axXiO  - 9y(YiO) + J 09x(XiO) +  L  J i Oj OI9x(XjOI) + Iio  + 10 \n-ayYio + 9x(XiO) +  L  WiOjOI9x(XjO') + Ie \n\njO'#iO \n\n(1) \n\n(2) \n\nYiO \n\njO'#iO \n\nwhere 9x(XiO)  and 9y(YiO) are the firing rates from the excitatory and inhibitory cells \nrespectively,  l/ax  and  l/ay are the membrane constants,  Jo  is  the self-excitatory \nconnection weight,  JiO ,jO'  and WiO,jOI  are synaptic weights between neurons, and 10 \nand  Ie  are the  background inputs  to the excitatory and  inhibitory  cells.  Without \nloss  of generality,  we  take ax  = a y  = 1,  9x(X)  as  threshold  linear  with  saturation \nand an unit gain 9~(X) = 1 in the linear range. \nThe synaptic connections  JiO,jO'  and Wi(},jOI  are local and translation and rotation \ninvariant  (Figure  (2)).  JiO,jO'  increases  with  the  smoothness  (small  curvature)  of \nthe  curve  that  best  connects  (if})  and  (jf}') , and edge  elements  inhibit  each  other \nvia  WiO,jOI  when  they  are alternative  choices  in  a  smooth  curve  route.  Given  an \ninput pattern lUI,  the network approaches a  dynamic state after several membrane \ntime constants.  As  in Figure  (2),  the neurons with relatively higher final  activities \nare those belonging to smooth curves in the input. \n\n2.2  Model analysis \n\nIgnoring neural connections between edge segments, the neuron in  edge segment if} \n\n\f72 \n\nhas input sensitivity \n\n1 + g~(Yo)g~(xo) - Jog~(xo) \n\ng~(Yo)g~(xo) \n\nZ.  Li \n\n(3) \n\n(4) \n\nwhere  gx(xo) and  gy(yo)  are roughly the  average neural activities  (omitting iO  for \nsimplicity).  Thus the edge activity increases with Ii9  and decreases with Ie  (in cases \nthat interest us,  g~(Yo)g~(xo) > Jog~(xo) -1) .  The resulting input-output function \ngiven  Ie,  gx(xo) vs.  Ii9 , corresponds well  with physiological data. \nBy  effectively increasing IiO  or  Ie,  the edge element  (jO')  can excite or  inhibit the \nelement  (iO)  with  excitatory-to-excitatory input  Ji9 ,j9,gx(Xj9 1 )  and  excitatory-to(cid:173)\ninhibitory  input  WiOj9' gx(Xj9')  respectively.  Contour  enhancement  is  so  (Fig.  2) \nachieved.  In the simplest example when  the visual  input has equally spaced equal \nstrength edges from  a  line and all other edge segments are silent,  we  can treat the \nline system as one dimensional, omit 0 and take i  as locations along the line.  A lack \nof inhibition between line segments gives: \n\n-axxi - gy(Yi) + Jogx(Xi) + L Jijgx(Xj) + 10  + Iline-input \n\n#i \n\n(5) \n\n(6) \n\nIf line  is  infinite,  by  symmetry,  each edge  segment  has  the same  average  activity \ngx(Xi)  rv  gx(xo)  for  all  i.  This system  can then  be  seen[12]  either as  a  giant edge \nwith self-excitatory connection (Jo + 2::#i Jij ),  or a single edge with extra external \ninput  t1I =  (2::#i Jij)gx(xo).  Either way,  activities  gx(xo)  are enhanced  for  each \nedge element  in the line  (figure 3 and 2). \n\nThis analysis  is  also  applicable to constant  curvature  curves[12].  It can  be shown \nthat, in the linear range of gx 0, the response ratio between a curve segment and an \nisolated segment is  (g~(yo) + 1- Jo)/(g~(yo) + 1 - Jo - 2::i#j  Jij ).  Since  2::ih Jij \ndecreases with increasing curvature, so does the response enhancement.  Translation \ninvariance along the curve  breaks  down  in  a  finite  length  curve  near its two ends, \nwhere activity enhancement decays by a decreased excitation from fewer neighboring \nsegments.  This suggests  that a  closed  or  longer  curve  has higher  saliency  than  an \nopen or shorter one  (figure  (3)).  This prediction is expected to hold also for  curves \nof non-constant curvature, and should  playa significant role  in  the psychophysical \nobservation[lO] showing a decreased detection threshold for  closed curves from  that \nof the open ones. \n\nFurther analysis[12] shows that the edge segments in a curve normally exhibit neural \noscillations around their mean activity levels with near-zero phase delays from each \nother.  The  model  predicts  that,  like  the  contour  enhancement,  the  oscillation  is \nstronger for  longer,  smoother,  and closed  curves  than open  and shorter ones,  and \ntapers  off  near curve  endings  where oscillation  synchrony also  deteriorates  (figure \n(3)). \n\n2.3 Central feedback control for contour enhancement, snpression, filling \nin, and segmentation \n\n\fA Neural Model o/VlSual Contour Integration \n\n73 \n\nModel visual inputs \n\n/ \n\nI \n-I  , \n\n\" \n\n\\ \n1 \nI \n\\ \n,-I \n\nModel outputs \n\n-- I \n\n, \n\n/ \n\n1 \n\n-\n\n( \n\\ \n\\  ( \n\\ \n\n\\ \n) \n\n,  , \n\" \n\\ \n) \n\n-I \\ \n\n,I \n\\1-/ \n\n.......... # \n\n/#- .... \\ \n\n-I  , \n\n\\  . \n\n-I \n\n# \n\nI, \n\" \ni l \nI\" \nI.. \ni:, \n1\"f--L---;-1-..-_.--;;--.J  I,ol--'  .L.-:-=----_---,,--;r-----J  I'ol--'  -'---,,..-_----;-----;;r-----J  I,ol--'  .L.-,---o-----;;--;---' \n\nNeural signals in time \nI,' \nI,' \nI' \nI, \nit \ni. \nI:: \n\\: \nU~fi9IllP5lll1  j\" \n!, \ni:, \ni:, \n\nI, \n\" \ni' \nI\" \nI .. \ni:, \n\n.' \n\nFigure 3:  Model  performance for  input curves  and noises,  Each  column  is  dedicated  to \none  input  condition.  The  top  row  is  the  visual  input;  the  middle  row  is  the  maximum \nneural  responses,  and  the  bottom  row  the  segment  outputs  as  a  function  of  time.  The \nneural  signals  in  the  bottom  row  are  shown  superposed.  The  solid  curves  plot  outputs \nfor  the segments away  from  the curve endings,  the dash-dotted curves for  segments near \nthe curve endings,  and the dashed  curve in  each  plot,  usually  the lowest  lying  one,  is  an \nexample from  a noise  segment.  Note the decrease in neural signal synchrony between the \ncurve segments,  in  the oscillation  amplitudes,  and  average  neural  activities  for  the curve \nsegments,  as  the curve  becomes  open,  shorter,  and  more  curled  or  when  the segment  is \nnear the curve endings.  Because the model employs discrete a grid input space,  the figure \n'8' curve in the right column is  almost not a smooth curve,  hence the contour is  only  very \nweakly  enhanced.  The enhancement for  this  curve  could  be increased  by  a  denser  input \ngrid or  a multiscale input space. \n\nThe model assumes that higher visual centers send inputs Ie  to the inhibitory cells, \nto influence neural activities by equation (4).  Take Ie  =  Ie,baekground+Ie,eontrol  ~ 0, \nwhere  Ie,baekground  is  the same for  all  edge segments and  is  useful  for  modulating \nthe overall  visual alertness  level.  By setting  Ie,eontrol  differently for  different  edge \nsegments, higher centers can selectively suppress or enhance activities for  some vi(cid:173)\nsual objects (contours)  (compare figure 4D,G), and even effectively achieve contour \nsegmentation (figure (4H)) by silencing segments from a given curve.  A similar feed(cid:173)\nback control mechanism was used in an olfactory model for  odor sensitivity control \nand odor segmentation[ll].  Nevertheless, the feedback cannot completely substitute \nfor the visual input Iii}.  By equation (4),  Ie  is effective only when g~(Yo)g~(xo) \"#  0. \nWith insufficient  background input 10 ,  some  visual  input or excitation from  other \nsegments is  needed  to have  g~(xo) > 0,  even  when  all  the inhibition from  Ie  is  re(cid:173)\nmoved by central control.  However, a segment with weak visual input or excitation \nfrom  aligned neighboring segments can increase its activity or become  active from \nsubthreshold.  Therefore, this model central feedback can enhance a weak contour or \nfill in an incomplete one (compare figure 4F ,I), but can not enhance or \"hallucinate\" \nany contour not existing at least  partially in the visual input I  (figure 4E), \n\n3.  Summary and Discussion \n\nWe  have  presented  a  model  which  accounts  for  phenomena  of contour  enhance-\n\n\f74 \n\nA:  Input \n\nB: Input \n\n;:\\ - ~ \n' \\   , \n\n/ \n\n, \n\n\\ \n\nI \n\nc: Input \n\nD: Outpl,lt for  A \nno central control \nI:'  ':x \n, \n-..-+-\n'\\-',) \n-\n, \n.. \n\n.... \n\nI~-:-\u00b7,\\ \n- \\ \n\n~ .,'  . \n'. '.1 -\n\nF: Output f~r C \nno  central control \n\nE: Qutout for  B,  enh~cement H: Same.as G,  except with \nfOr  clrcre  and non-exlstmg line \n\nstronger line suppreslOn \n\nZ.Li \n\nG.:  Output for  A,  line suppresion \ncircle enhancement \n\nI~-:-\u00b7,\\ \n- \\ \n-a-------a--\n\" \n\\. '.1 \n:--\n\n.-\n\n, \n\nI \n\n\u2022 \n\nI~'-\u00b7,\\ - \\ \n'. '.1 -\n\n\u2022 \n\nI; Outp~t.for-C,  en):lajlcement \nlOr  both hne and circle \n\n' \n\nI~'-\u00b7~ \n... , \n- \\ \n~\u00b7\\\u00b7~T \n\n,I ... \\ \n\n:\\ \n\n\\ \n\nI  \" \n\" \n\n... , \n- \\ \n''I. \n-... \n\nI \n\nI \n\n, \n\n.,. \n\n.. \n\n.. \n\n... \n... -... \n\n. .... \n\n,-'.-; \n\n7~\\ -'--.- -7--\nFigure  4:  Central  feedback  control.  The  visual  inputs  are  in  A,  B  and  C.  The  rest \nare  the  model  responses  under  different  feedback  conditions.  D:  model  response  for \ninput  A  without  central  control.  G:  model  response  for  input  A  with  line  suppres(cid:173)\nsion  by  Ic,control  =  O.25Ic ,background  on  the  line  segments,  and  circle  enhancement  by \nIc,control  = -O.29Ic ,background  on the circle elements.  H:  Same as  G  except the line sup(cid:173)\npression signal  Ic,control  is  doubled.  E:  Model response to input B  with enhancement for \nthe line  and the circle  by central feedback  Ic,control  =  -O.29Ic ,background'  F: Response to \ninput C  with no central control.  I: Response to input C  with line and circle enhancement \nby  Ic ,control  = -O.29Ic ,background'  Note how  the input gaps  are  partiallly filled  in  F  and \nalmost  completely filled  in  I.  Note that the apparent gaps in the circle  are caused by the \nunderlying discrete grid in the model  input space,  and  hence  no  gap  actually exists,  and \nno filling  in is needed, for  this circle.  Also, with the wrap around boundary condition, the \nline  is  actually  a  closed  or  infinitely  long  line,  and  is  thus  naturally  more salient  in  this \nmodel without feedback. \n\nment using  neurally plausible elements in VI.  It is  shown analytically and empiri(cid:173)\ncally that both the contour enhancement and the neural oscillation amplitudes are \nstronger for longer, closed, and smaller curvature curves, agreeing with experimental \nobservations [8,  4,  10,  6,  2].  The model  predicts that horizontal connections target \npreferentially excitatory or inhibitory post-synaptic cells when the linked edges are \naligned or less aligned (Fig.  2).  In addition, we introduce a possible feedback mech(cid:173)\nanism by which higher visual centers could selectively enhance or suppress contour \nactivities, and achieve contour segmentation.  This feedback mechanism has the de(cid:173)\nsirable property that while  the higher  centers can enhance or complete an existing \nweak and/or fragmented input contour, they cannot enhance a non-exist ant contour \nin the input, thus preventing  \"hallucination\".  This property could be exploited by \nhigher visual centers for  hypothesis testing and object reconstruction by cooperat(cid:173)\ning with lower visual centers.  Analogous computational mechanisms have been used \nin an olfactory model to achieve odor segmentation and sensitivity modulation[ll]. \nIt will  be interesting to explore the universality of such computational mechanisms \nacross sensory modalities. \n\nThe  organization  of the  model  is  based  on  various  experimental  finding[17,  5,  1, \n8,  14]:  recurrent excitatory-inhibitory interactions;  excitatory and  inhibitory link(cid:173)\ning  of  edge  elements  with  similar  orientation  preferences;  and  neural  connection \n\n\fA Neural Model of Visual Contour Integration \n\n75 \n\npatterns.  At  the  cost  of analytical tractability without essential changes in  model \nperformance, one can relax the model's idealization of a  1:1  ratio in  the excitatory \nand  inhibitory  cell  numbers,  the  lack  of connections  between  the  inhibitory  cells, \nand the excitatory cells as the exclusive recipients of visual input.  While abundant \nfeedback connections are observed from  higher visual centers to the primary visual \ncortex,  there is  as  yet  no clear  indication of cell  types  of their targets [3,  16].  It is \ndesirable to find  out whether the feedback is indeed directed to the inhibitory cells \nas  predicted. \n\nThis  model  can  be  extended  to  stereo,  temporal,  and  chromatic  dimensions,  by \nlinking  edge  segments  aligned  in  orientation,  depth,  motion  direction  and  color. \nV1  cells  have  receptive  field  tuning  in  all  these  dimensions,  and  cortical  connec(cid:173)\ntions  are indeed  observed to link  cells  of similar receptive field  properties[5].  This \nmodel  does  not model  many other apparently non-contour related  visual  phenom(cid:173)\nena such as  receptive field  adaptations[5].  It is  also  beyond  this  model  to explain \nhow  the higher visual centers decide which segments belong to one contour in order \nto  achieve  feedback  control,  although it  has  been  hypothesized  that  phase  locked \nneural oscillations and neural correlations can play such a  role[13]. \n\nReferences \n[1]  Douglas  R .J.  and  Martin K.  A.  \"Neocortex\"  in  Synaptic  Organization  of the  Brain \n\n3rd Edition,  Ed.  G. M.  Shepherd, Oxford University Press  1990. \n\n[2]  Eckhorn R,  Bauer, R.  Jordan W.  Brosch,  M. Kruse W.,  Munk M.  and Reitboeck,  H. \n\nJ.  1988.  Bioi.  Cybern.  60:121-130. \n\n[3]  van  Essen D. Peters  and E  G  Jones,  Plenum Press,  New York,  1985.  p.  259-329 \n[4]  Field DJ; Hayes A;  Hess  RF 1993.  Vision  Res.  1993  Jan; 33(2):  173-93 \n[5]  Gilbert  CD  Neuron.  1992  Julj  9(1):  1-13 \n[6]  Gray C.M.  and Singer W.  1989  Proc.  Natl.  Acad.  Sci.  USA  86:  1698-1702. \n[7]  Grossberg  S;  Mingolla  E  Percept  Psychophys.  1985  Aug;  38(2):  141-71 \n[8]  Kapadia MKj  Ito M;  Gilbert  CD;  Westheimer  G,  Neuron.  1995  Oct;  15(4):  843-56 \n[9]  Knierim J.  J.  and van  Essen D.  C.  1992,  J.  Neurophysiol.  67,  961-980. \n[10]  Kovacs  I;  Julesz  B  Proc  Natl  Acad Sci  USA . 1993  Aug  15;  90(16):  7495-7 \n[11]  Li  Zhaoping,  Biological  Cybernetics,  62/4  (1990),  P. 349-361 \n[12]  Li  Zhaoping,  \"A  neural  model  contour  integration  in  primary  visual  cortex\" \n\nmanuscript submitted for  publication. \n\n[13]  von der Malsburg C.  1981  \"The correlation theory of brain function .\"  Internal report, \n\nMax-Planck-Institute for  Biophysical Chemistry,  Gottingen,  West  Germany. \n\n[14]  Polat U;  Sagi D,  1994  Vision  Res.  1994  Jan;  34(1):  73-8 \n[15]  Shashua A.  and Ullman S.  1988  Proceedings  of the  International  Conference  on  Com(cid:173)\n\nputer  Vision.  Tempa,  Florida,  482-488. \n\n[16]  Valverde  F .  in  Cerebral  Cortex  Eds.  A  Peters  and  E  G  Jones,  Plenum  Press,  New \n\nYork,  1985.  p.  207-258. \n\n[17]  White E. L.  Cortical  circuits  46-82,  Birkhauser,  Boston,  1989 \n[18]  Zucker S. W., David C., Dobbins A,  and Iverson L.  in Second international conference \n\non  computer vision pp.  568-577,  IEEE computer society press,  1988. \n\n\f", "award": [], "sourceid": 1244, "authors": [{"given_name": "Zhaoping", "family_name": "Li", "institution": null}]}