{"title": "Recurrent Cortical Competition: Strengthen or Weaken?", "book": "Advances in Neural Information Processing Systems", "page_first": 89, "page_last": 95, "abstract": null, "full_text": "Recurrent cortical competition: Strengthen or \n\nweaken? \n\nPeter Adorjan*, Lars Schwabe, \n\nChristian Piepenbrock* , and Klaus Obennayer \n\nDept. of Compo Sci., FR2-I, Technical University Berlin \n\nFranklinstrasse 28/29 10587 Berlin, Germany \n\nadorjan@epigenomics.com, {schwabe, oby} @cs.tu-berlin.de, \n\npiepenbrock@epigenomics.com \n\nhttp://www.ni.cs.tu-berlin.de \n\nAbstract \n\nWe investigate the short term .dynamics of the recurrent competition and \nneural activity in the primary visual cortex in terms of information pro(cid:173)\ncessing and in the context of orientation selectivity. We propose that af(cid:173)\nter stimulus onset, the strength of the recurrent excitation decreases due \nto fast synaptic depression. As a consequence, the network shifts from \nan initially highly nonlinear to a more linear operating regime. Sharp \norientation tuning is established in the first highly competitive phase. In \nthe second and less competitive phase, precise signaling of multiple ori(cid:173)\nentations and long range modulation, e.g., by intra- and inter-areal con(cid:173)\nnections becomes possible (surround effects). Thus the network first ex(cid:173)\ntracts the salient features from the stimulus, and then starts to process \nthe details. We show that this signal processing strategy is optimal if \nthe neurons have limited bandwidth and their objective is to transmit the \nmaximum amount of information in any time interval beginning with the \nstimulus onset. \n\n1 \n\nIntroduction \n\nIn the last four decades there has been a vivid and highly polarized discussion about the \nrole of recurrent competition in the primary visual cortex (VI) (see [12] for review). The \nmain question is whether the recurrent excitation sharpens a weakly orientation tuned feed(cid:173)\nforward input, or the feed-forward input is already sharply tuned, hence the massive re(cid:173)\ncurrent circuitry has a different function. Strong cortical recurrency implements a highly \nnonlinear mapping of the feed-forward input, and obtains robust and sharply tuned corti(cid:173)\ncal response even if only a weak or no feed-forward orientation bias is present [6, 11, 2]. \nHowever, such a competitive network in most cases fails to process mUltiple orientations \nwithin the classical receptive field and may signal spurious orientations [7]. This moti(cid:173)\nvates the concept that the primary visual cortex maps an already sharply orientation tuned \nfeed-forward input in a less competitive (more linear) fashion [9, 13]. \n\nAlthough these models for orientation selectivity in VI vary on a wide scale, they have \none common feature: each of them assumes that the synaptic strength is constant on the \nshort time scale on which the network operates. Given the phenomenon of fast synaptic \n\n*Current address: Epigenomics GmbH, Kastanienallee 24,0-10435 Berlin, Germany \n\n\f90 \n\nP. Adorjan, L. Schwabe, C. Piepenbrock and K. Obermayer \n\ndynamics this, however, does not need to be the case. Short term synaptic dynamics, e.g., \nof the recurrent excitatory synapses would allow a cortical network to operate in both(cid:173)\ncompetitive and linear-regimes. We will show below (Section 2) that such a dynamic \ncortical amplifier network can establish sharp contrast invariant orientation tuning from \na broadly tuned feed-forward input, while it is still able to respond correctly to mUltiple \norientations. \n\nWe then show (Section 3) that decreasing the recurrent competition with time naturally fol(cid:173)\nlows from functional considerations, i.e. from the requirement that the mutual information \nbetween stimuli and representations is maximal for any time interval beginning with stimu(cid:173)\nlus onset. We consider a free-viewing scenario, where the cortical layer represents a series \nof static images that are flashed onto the retina for a fixation period (~T = 200 - 300 ms) \nbetween saccades. We also assume that the spike count in increasing time windows after \nstimulus onset carries the information. The key observations are that the signal-to-noise \nratio of the cortical representation increases with time (because more spikes are available) \nand that the optimal strength of the recurrent connections (w.r.t. information transfer) de(cid:173)\ncreases with the decreasing output noise. Consequently the model predicts that the infor(cid:173)\nmation content per spike (or the SNR for ajixed sliding time window) decreases with time \nfor a flashed static stimulus in accordance with recent experimental studies. The neural \nsystem thus adapts to its own internal changes by modifying its coding strategy, a phe(cid:173)\nnomenon which one may refer to as \"dynamic coding\". \n\n2 Cortical amplifier with fast synaptic plasticity \n\nTo investigate our first hypothesis, we set up a model for an orientation-hypercolumn in \nthe primary visual cortex with similar structure and parameters as in [7]. The important \nnovel feature of our model is that fast synaptic depression is present at the recurrent ex(cid:173)\ncitatory connections. Neurons in the cortical layer receive orientation-tuned feed-forward \ninput from the LGN and they are connected via a Mexican-hat shaped recurrent kernel in \norientation space. In addition, the recurrent and feed-forward excitatory synapses exhibit \nfast depression due to the activity dependent depletion of the synaptic transmitter [1, 14]. \nWe compare the response of the cortical amplifier models with and without fast synap(cid:173)\ntic plasticity at the recurrent excitatory connections to single and mUltiple bars within the \nclassical receptive field. \n\nThe membrane potential V (0, t) of a cortical cell tuned to an orientation 0 decreases due \nto the leakage and the recurrent inhibition, and increases due to the recurrent excitation \n\na \nT at V(O, t) + V(O, t) \n\n(1) \n\nwhere T = 15 ms is the membrane time constant and ILGN (0, t) is the input received from \nthe LGN. The recurrent excitatory and inhibitory cortical inputs are given by \n\nr:\"(O, t) \n\n(2) \n\nwhere ~ (Of, 0) is a 1T periodic circular difference between the preferred orientations, \nJCX(O, Of , t) are the excitato~ and inhibitory connection strengths (with a E {exc, inh}, \nJ~x~x = 0.2 m V /Hz and J:::ax = 0.8m V /Hz), and f is the presynaptic firing rate. The ex(cid:173)\ncitatory synaptic efficacy r xc is time dependent due to the fast synaptic depression, while \nthe efficacy of inhibitory synapses Jinh is assumed to be constant. The recurrent excitation \nis sharply tuned (j exc = 7.5 0 , while the inhibition has broad tuning (jinh = 90 0 \u2022 The map(cid:173)\nping from the membrane potential to firing rate is approximated by a linear function with \na threshold at 0 (f(O) = ,6max(O, V(O)),,6 = 15Hz/mV). Gaussian-noise with variances \n\n\fRecurrent Cortical Competition: Strengthen or Weaken? \n\nFeedforward Input \n\n,--,., ---\n\n15 \n\nN \nX \n' - ' \nQ) \n= \nrJJ \n0 \no.. \nrJJ \nQ) \n~ \n\n1 \n\n,....., \n> \nE \n.... \n' - ' \n::l \ng-\n\n-\n\nStatic \n\nn \nj! \n\n, \n, \n, \n\n\\ \n\n15 \n\nN \nX \n= \n\n' - ' \nQ) \nrJJ \n0 \n0.. \nrJJ \nQ) \n~ \n\nDepressing \n\n91 \n\n90 \n\n~90 -45 \n\n0 \n\n45 \n\n90 \n\nOrientation [deg] \n\n~90 -45 \n\n0 \n\n45 \n\n90 \n\nOrientation [deg] \n\n~90 -45 \n\n0 \n\n45 \n\nOrientation [deg] \n\n(a) \n\n(b) \n\n(c) \n\nFigure 1: The feed-forward input (a), and the response ofthe cortical amplifier model with \nstatic recurrent synaptic strength (b), and a network with fast synaptic depression (c) if the \nstimulus is single bar with different stimulus contrasts (40%dotted; 60%dashed; 80%solid \nline). The cortical response is averaged over the first 100 illS after stimulus onset. \n\nof 6 Hz and 1.6 Hz is added to the input intensities and to the output of cortical neurons. \nThe orientation tuning curves of the feed-forward input ILGN are Gaussians (O'\"LGN = 18\u00b0) \nresting on a strong additive orientation independent component which would correspond to \na geniculo-cortical connectivity pattern with an approximate aspect ratio of 1 :2. Both, the \norientation dependent and independent components increase with contrast. Considering a \nfree-viewing scenario where the environment is scanned by saccading around and fixating \nfor short periods of 200 - 300 illS we model stationary stimuli present for 300 illS. The \nstimuli are one or more bars with different orientations. \n\nFeed-forward and recurrent excitatory synapses exhibit fast depression. Fast synaptic de(cid:173)\nI'ression is modeled by the dynamics of the expected synaptic transmitter or \"resource\" \nR( t) for each synapse. The amount of the available transmitter decreases proportionally to \nthe release probability p and to the presynaptic firing rate /, and it recovers exponentially \n(T~~N = 120 illS, Tr~~x = 850 illS, pLGN = 0.35 and pCtx = 0.55), \n\n1 - R(t) \n\nTree \n\n-\n\n- /(t)p(t)R(t) = -\n\nR(t) \n(f() ()) + -. \n\n1 \nTree \n\nt, P t \n\nTeff \n\n(3) \n\nThe change of the membrane potential on the postsynaptic cell at time t is proportional \nto the released transmitter pR(t). The excitatory connectivity strength between neurons \ntuned to orientations 0 and 0' is expressed as j\u20acxe(o , 0', t) = J:::~xpR991(t). Similarly this \napplies to the feed-forward synapses. Fast synaptic plasticity at the feed-forward synapses \nhas been investigated in more detail in previous studies [3, 4]. \n\nIn the following, we compare the predictions of the cortical amplifier model with and with(cid:173)\nout fast synaptic depression at the recurrent excitatory connections. In both cases fast \nsynaptic depression is present at the feed-forward connections limiting the duration of the \neffective feed-forward input to 200 - 400 illS. Figure 1 shows the orientation tuning curves \nat different stimulus contrasts. The feed-forward input is noisy and broadly tuned (Fig. \nla). Both models exhibit contrast invariant tuning (Fig. 1 b, c). If fast synaptic depression \nis present at the recurrent excitation, the cortical network sharpens the broadly tuned feed(cid:173)\nforward input in the initial response phase. Once sharply tuned input is established, the \ntuning width does not change, only the response amplitude decreases in time. \n\nThe predictions of the two models differ substantially if multiple orientations are present \n(Fig. 2). At first, we test the cortical response to two bars separated by 60\u00b0 with differ(cid:173)\nent intensities (Figs. 2a, b). If the recurrent synaptic weights are static and strong enough \n(Fig. 2a), then only one orientation is signaled. The cortical network selects the orientation \n\n\f92 \n\n(a) \n\n(b) \n\n(d) \n\n90 \n\n0 \n\nf~1 \n\n~90 -45 \n\n45 \nOrientation [deg] \n\ng \n.: \nS 0 \n~ \n-90 \n90 \n\n90 I~I~--~_~~\"~,------~ \nI \n, ' \n.b~---~#\"\" \n;;20 A \nI \nOrienretionldeg) r:S:;;1 (: \n\n. ., \ng \n~ 0 \n\u00b7c o \n-90 \n90 \ng \n.~ 0 \n~ \n-90 \n\n110 \n90 ~ 0 \n\n/', \n\n.. ,----, ... '.-'--<\" \n.. \n\n1~ . . .. \u00b7\u00b7\u00b7INI \n:> \nE \n'i \n\n',,--,'\" \n'. \n\n, .... --\n\n,-,,, \n\n45 \n\n---\n\n''''' \n\n\" \n\na) \n\nUl \n\nc: \n\n(c) - ~90 -45 \n\n0 \n\n,\n\n~~~_ o \n'\" ~90- -45 \n\n. 0 '. 45 \nOrientation [deg] \n\n'90 -900 \n\nP Adorjim. L. Schwabe. C. Piepenbrock and K. Obermayer \n\nFeedforward Input \n\nAverage Cortical Response \n\nActivity Profile \n\n:;a \u2022\u2022\u2022\u2022 \\~~~\\$~* ~'> ~ \\m0 '\" s \nm\u00b7\" \n~ . \n\n~-... \n\n: \n\n150 \n\nTime [ms] \n\n300 \n\nFigure 2: The response of the cortical amplifier model with static (a,c) and fast depressing \nrecurrent synapses (b, d). In both models the feed-forward synapses are fast depressing. In \nthe left column the feed-forward input is shown, that is same for both models. Two types \nof stimuli were applied. The first stimulus consists of a stronger (a = -30\u00b0) and a weaker \nbar (a = +30\u00b0) (a, b); the second stimulus consists of three equal intensity bars with \norientations that are separated by 60\u00b0 (c, d). In the middle column the cortical response is \nshown averaged for different time windows ([0 .. 30] dotted; [0 .. 80] dashed; [200 .. 300] solid \nline). In the right column the cortical activity profile is plotted as a function of time. Gray \nvalues indicate the activity with bright denoting high activities. \n\nwith the highest amplitude in a winner-take-all fashion. In contrast, if synaptic depression \nis present at the recurrent excitatory synapses, both bars are signaled in parallel (at low \nrelease probability, Fig. 2b) or after each other (high release probability, data not shown). \nFirst, those cells fire which are tuned to the orientation of the bar with the stronger inten(cid:173)\nsity, and a sharply tuned response emerges at a single orientation-the network operates in \na winner-take-all regime. The synapses of these highly active cells then become strongly \ndepressed and cortical competition decreases. As the network is shifted to a more linear \noperation regime, the second orientation is signaled too. Note that this phenomenon(cid:173)\ntogether with the observed contrast invariant tuning-cannot be reproduced by simply de(cid:173)\ncreasing the static synaptic weights in the cortical amplifier model. The recurrent synap(cid:173)\ntic efficacy changes inhomogeneously in the network depending on the activity. Only the \nsynapses of the highly active cells depress strongly, and therefore a sharply tuned response \ncan be evoked by a bar with weak intensity. Fast synaptic depression thus behaves as a lo(cid:173)\ncal self-regulation that modulates competition with a certain delay. This delay, and there(cid:173)\nfore the delay of the rise of the response to the second bar depends on the effective time \nconstant reff(f(t),p) = rrec/(l + pf(t)rrec) of the synaptic depression at the recurrent \nconnections. If the depression becomes faster due to an increase in the release probabil(cid:173)\nity p, then the delay decreases. The delay also scales with the difference between the bar \nintensities. The closer to each other they are, the shorter the delay will be. \n\nIn Figs. 2c, d the cortical response to three bars with equal intensities is presented. Cells \ntuned to the presented three orientations respond in parallel if fast synaptic depression at \nthe recurrent excitation is present (Figs. 2d). The cortical network with strong static recur(cid:173)\nrent synapses again fails to signal faithfully its feed-forward input. Additive noise on the \n\n\fRecurrent Cortical Competition: Strengthen or Weaken? \n\n93 \n\nfeed-forward input introduces a slight symmetry breaking and the network with static re(cid:173)\ncurrent weights responds strongly at the orientation of only one of the presented bars (Fig. \n2c). \n\nIn summary, our simulations revealed that a recurrent network with fast synaptic depres(cid:173)\nsion is capable of obtaining robust sharpening of its feed-forward input and it also re(cid:173)\nsponds correctly to multiple orientations. Note that other local activity dependent adapta(cid:173)\ntion mechanisms, such as slow potassium current, would have similar effects as the synap(cid:173)\ntic depression on the highly orientation specific excitatory connections. An experimentally \ntestable prediction of our model is that the response to a flashed bar with lower contrast \ncan be delayed by masking it with a second bar with higher contrast (Fig. 2b, right). We \nalso suggest that long range integration from outside of the classical receptive field could \nemerge with a similar delay. In the initial phase of the cortical response, strong local fea(cid:173)\ntures are amplified. In the longer, second phase, recurrent competition decreases and then \nweak modulatory recurrent or feed-forward input has a stronger relative effect. In the fol(cid:173)\nlowing, we investigate whether this strategy is favorable from the point of view of cortical \nencoding. \n\n3 Dynamic coding \n\nIn the previous section we have proposed that during cortical processing a highly nonlin(cid:173)\near phase is followed by a more linear mode if we consider a short stimulus presentation \nor a fixation period. The simulations demonstrated that unless the recurrent competition \nis modulated in time, the network fails to account for more than one feature in its input. \nFrom a strictly functional point of view the question arises, why not to use weak recurrent \ncompetition during the whole processing period. We investigate this problem in an abstract \nsignal-encoder framework \n\ni7 = g( i) + 1] , \n\n(4) \nwhere i is the input to the \"cortical network\", g(i) is a nonlinear mapping and-for the \nsake of simplicitY-1] is additive Gaussian noise. Naturally, in a real recurrent network \noutput noise becomes input noise because of the feedback. Here we use the simplifying \nassumption that only output noise is present on the transformed input signal (input noise \nwould lead to different predictions that should be further investigated). Output noise can \nbe interpreted as a noisy channel that projects out from, e.g., the primary visual cortex. \nThe nonlinear transformation g(i) here is considered as a functional description of a cor(cid:173)\ntical amplifier network without analyzing how actually it is \"implemented\". Considering \norientation selectivity, the signal i can be interpreted as a vector of intensities (or contrasts) \nof edges with different orientations. Edges which are not present have zero intensity. The \ncoding capacity of a realistic neural network is limited. Among several other noise sources, \nthis limitation could arise from imprecision in spike timing and a constraint on the maximal \nor average firi ng rate. \nThe input-output mapping g( i) of a cortical amplifier network is approximated with the \nsoft-max function \n\n(5) \n\nThe f3 parameter can be interpreted as the level of recurrent competition. As f3 -+ 0 the \nnetwork operates in a more linear mode, while f3 -+ 00 puts it into a highly nonlinear \nwinner-take-all mode. In all cases the average activity in the network is constrained which \nhas been suggested to minimize metabolic costs [5]. Let us consider a factorizing input \ndistribution, \n\n(_x a ) \np( i) = Z IIi exp ---t-\n\n1 \n\nfor x ~ 0 , \n\n(6) \n\n\f94 \n\nP. Adorjfm, L. Schwabe, C. Piepenbrock and K. Obermayer \n\n8rr===~~----~----~ \n\n--- 0.5 \n0--0 1.0 \n\n6 \n\n'J:;-----Q(cid:173)\n\n.c!1 - - ,,-s-' \n\n00 \n\n0.05 \n\n0.1 \nNoise (stdev) \n\n0.15 \n\nFigure 3: The optimal competition \nparameter j3 as a function of the \nstandard deviation of the Gaussian \noutput noise 'f}. The optimal j3 is cal(cid:173)\nculated for highly super-Gaussian, \nGaussian, and sub-Gaussian stimu(cid:173)\nlus densities. The sparsity parame(cid:173)\nter a is indicated in the legend. \n\nwhere the exponent a detennines the sparsity of the probability density function, Z is a \nnonnalizing constant, and ~ detennines the variance. If a = 2, the input density is the \npositive half of a multivariate Gaussian distribution. With a > 2 the signal distribution \nbecomes sub-Gaussian, and with a < 2 it becomes super-Gaussian. \nFor optimal processing in time one needs to gain the maximal infonnation about the signal \nfor any increasing time window. Let us assume that the stimulus is static and it is pre(cid:173)\nsented for a limited time. As time goes ahead after stimulus onset, the time window for the \nencoding and the read-out mechanism increases. During a longer period more samples of \nthe noisy network output are available, and thus the output noise level decreases with time. \nWe suggest that the optimal competition parameter j3opt_at which the mutual infonna(cid:173)\ntion between input i and output if (Eq. 4) is maximized-