{"title": "Learning Curves for Stochastic Gradient Descent in Linear Feedforward Networks", "book": "Advances in Neural Information Processing Systems", "page_first": 1197, "page_last": 1204, "abstract": "", "full_text": "Learning curves for stochastic gradient descent\n\nin linear feedforward networks\n\nJustin Werfel\nDept. of EECS\n\nMIT\n\nCambridge, MA 02139\njkwerfel@mit.edu\n\nXiaohui Xie\n\nH. Sebastian Seung\n\nHHMI\n\nMIT\n\nDept. of Molecular Biology\n\nPrinceton University\nPrinceton, NJ 08544\n\nxhx@princeton.edu\n\nAbstract\n\nDept. of Brain & Cog. Sci.\n\nCambridge, MA 02139\n\nseung@mit.edu\n\nGradient-following learning methods can encounter problems of imple-\nmentation in many applications, and stochastic variants are frequently\nused to overcome these dif\ufb01culties. We derive quantitative learning\ncurves for three online training methods used with a linear perceptron:\ndirect gradient descent, node perturbation, and weight perturbation. The\nmaximum learning rate for the stochastic methods scales inversely with\nthe \ufb01rst power of the dimensionality of the noise injected into the sys-\ntem; with suf\ufb01ciently small learning rate, all three methods give iden-\ntical learning curves. These results suggest guidelines for when these\nstochastic methods will be limited in their utility, and considerations for\narchitectures in which they will be effective.\n\n1\n\nIntroduction\n\nLearning in arti\ufb01cial systems can be formulated as optimization of an objective function\nwhich quanti\ufb01es the system\u2019s performance. A typical approach to this optimization is to\nfollow the gradient of the objective function with respect to the tunable parameters of the\nsystem. Frequently this is accomplished directly, by calculating the gradient explicitly and\nupdating the parameters by a small step in the direction of locally greatest improvement.\n\nIn many circumstances, however, attempts at direct gradient-following can encounter prob-\nlems. In VLSI and other hardware implementations, computation of the gradient may be\nexcessively unwieldy, if not impossible due to unavoidable imperfections in manufacturing\n[1]-[5]. In some cases, as with many where the reinforcement learning framework is used,\nthere may be no explicit form for the objective function and hence no way of calculating its\ngradient [6]. And in biological systems, any argument that direct gradient calculation might\nbe what the system is actually doing typically encounters severe obstacles. For instance,\nbackpropagation, the standard method for training arti\ufb01cial neural networks, requires two-\nway, multipurpose synapses, units with global knowledge about the system that are able to\nrecognize different kinds of signals and treat them in very different ways, and (in the case\nof trajectory learning) the ability to run backwards in time, all of which strain the bounds\nof biological plausibility [1, 7]. For reasons such as these, there has been broad interest in\nstochastic methods which approximate the gradient on average.\n\n\fCompared to a method that follows the true gradient directly, we would intuitively expect\na stochastic gradient-following approach to learn more slowly. The stochastic algorithms\nin this study use a reinforcement-learning framework with a single reward signal, which is\nassigned based on the contributions of all the tunable parameters of the system; that sin-\ngle reward is all that is available to evaluate how every one of the parameters should be\nupdated, in contrast to a true-gradient method where the optimal updates are all speci\ufb01ed.\nMoreover, if the network is made larger and the number of parameters thereby increased,\nthis credit assignment problem becomes still more dif\ufb01cult; thus we expect the performance\nof stochastic gradient methods to scale up with network size more poorly than determin-\nistic methods. However, under some circumstances stochastic methods can be equally as\neffective as direct ones in training even large networks, generating near-identical learning\ncurves (see, e.g., Fig. 2 below). Under what circumstances, then, will stochastic gradient\ndescent have performance comparable to that of the deterministic variety? And how good\ncan that performance be?\n\nIn this paper, we investigate these issues quantitatively by calculating the learning curves\nfor a linear perceptron using a direct gradient method and two stochastic methods, node\nperturbation and weight perturbation. We \ufb01nd that the maximum learning speed for each\nalgorithm scales inversely with the \ufb01rst power of the dimensionality of the noise injected\ninto the system; this result is in contradiction to previous work, which reported maximum\nlearning speed scaling inversely with the square root of the dimensionality of the injected\nnoise [4]. Additionally, when learning rates are chosen to be very low, and such that the\nweight updates prescribed by each method are equal on average, we \ufb01nd that all three\nmethods give identical learning curves.\n\n2 Perceptron comparison\n\nDirect and stochastic gradient approaches are general classes of training methods. We\nstudy the operation of exemplars of both on a feedforward linear perceptron, which has the\nadvantage over the nonlinear case that the learning curves can be calculated exactly [8]. We\nhave N input units and M output units, connected by a weight matrix w of M N elements;\noutputs in response to an input x are given by y = wx. For the ensemble of possible inputs,\nwe want to train the network to produce desired corresponding outputs y = d; in order to\nensure that this task is realizable by the network, we assume the existence of a teacher\nnetwork w(cid:3) such that d = w(cid:3)x. We use the squared error function\n\n1\n2jy (cid:0) dj2 =\n\nE =\n\n(1)\nwhere we have de\ufb01ned the matrix W (cid:17) w (cid:0) w(cid:3). We train the network with an online\napproach, choosing at each time step an input vector x with components drawn from a\nGaussian distribution with mean 0 and unit variance, and using it to construct a weight\nupdate according to one of the three prescriptions below.\n\n1\n2j(w (cid:0) w(cid:3))xj2 =\n\n1\n2jW xj2\n\nThe online gradient-following approach explicitly uses the gradient of the error function\nfor a given input to determine the weight update:\n\n(cid:1)WOL = (cid:0)(cid:17)rE\n\nwhere (cid:17) > 0 is the learning rate. This is the approach taken, e.g., by backpropagation.\nIn the stochastic algorithms, the gradient is not calculated directly; instead, some noise is\nintroduced into the system, affecting its error for a given input, and the difference between\nthe error with and without noise is used to estimate the gradient. The simplest case is when\nnoise is added directly to the weight matrix:\n1\n2j(W +  )xj2\n\nWP =\n\nE0\n\n\fSuch an approach is sometimes termed \u2018weight perturbation\u2019 [2, 4]. We choose each el-\nement of the noise matrix   from a Gaussian distribution with mean 0 and variance (cid:27)2.\nIntuitively, if the addition of the noise lowers the error, that perturbation to the weight ma-\ntrix is retained, which will mean lower error for that input in future. Conversely, if the noise\nleads to an increase in error, the opposite change is made to the weights; the effect of small\nnoise on error can be approximated as linear, and the opposite change in weights will lead\nto the opposite change in error, again decreasing error for that input in future. These two\ncases can be combined into the single weight update\n\n(cid:1)WWP = (cid:0)\n\n(cid:17)\n(cid:27)2 (E0\n\nWP (cid:0) E) \n\nA more subtle way to introduce stochasticity involves adding the noise to the output of each\noutput unit rather than to every weight:\n\nE0\n\nNP =\n\n1\n2jW x + (cid:24)j2\n\nSuch an approach is sometimes called \u2018node perturbation\u2019 [1, 3]. Here if the noise leads\nto a decrease in error, the weights are adjusted in such a way as to move the outputs in the\ndirection of that noise. The degree of freedom for each output unit corresponds to the ad-\njustment of its threshold, making the unit more or less responsive to a given pattern of input\nactivity. The elements of (cid:24) are again chosen independently from a Gaussian distribution\nwith variance (cid:27)2; here (cid:24) has M elements, whereas   in the previous case had M N. The\nREINFORCE framework [9] gives for the weight update\n\n(cid:1)WNP = (cid:0)\n\n(cid:17)\n(cid:27)2 (E0\n\nNP (cid:0) E)(cid:24)xT\n\nThese stochastic frameworks produce weight updates identical to that of direct gradient\ndescent on the error function when averaged over all values of the noise [4, 9], which is the\nsense in which they constitute stochastic gradient descent. This result is easy to verify in\nthe particular forms taken by (cid:1)WNP and (cid:1)WWP here, shown below.\n\n2.1 Online gradient method\n\nTaking the gradient of the error function of Eq. 1 gives\n(cid:1)WOL = (cid:0)(cid:17)W xxT\n\n(2)\nas the individual weight update for particular values of W and x. This rule lets us calculate\na recursion relation specifying how kWk2 changes from one time step to the next:\n\nXij\n\nh(W (t)\n\nij )2it = (1 (cid:0) 2(cid:17) + (N + 2)(cid:17)2)Xij (cid:16)W (t(cid:0)1)\n\nij\n\n(cid:17)2\n\n(3)\n\nwhere the parenthesized superscript is a time index, and the subscripted angle brackets\ndenote an average over the ensemble of all inputs at that time. Applying this recursion rela-\ntion gives an expression for the average error as a function of time, where the unsubscripted\nbrackets indicate a mean taken over all inputs at every time step:\nOLi = (1 (cid:0) 2(cid:17) + (N + 2)(cid:17)2)tE(0)\n\nhE(t)\n\nIn a single online learning run, E(t) would depend on the particular values of x that were\nrandomly chosen; averaging over the ensemble of possible inputs x removes this variation.\nWe therefore use this averaged error hE (t)i as the learning curve measuring the perfor-\nmance of the system.\n\nWe have the condition for convergence of the average error\n\n(cid:17) <\n\n2\n\nN + 2\n\n\fThe limit on (cid:17) has this dependence on N because of the randomness inherent in an online\ntraining regimen; the exact gradient for error due to a given single input x will not in general\nmatch that for error averaged over the entire ensemble of inputs. We can write an expression\nfor the ij-component of the weight update, explicitly in terms of \u2018gradient signal\u2019 (term\nmultiplying Wij) plus \u2018gradient noise\u2019 [1] (contamination from other components of W\ndue to projection onto x):\n\n(cid:1)Wij = (cid:0)(cid:17)0\n\n@Wijx2\n\nj +Xk6=j\n\nWikxkxj1\nA\n\nWe can similarly rewrite Eq. 3 as\n\nhW (1)2\n\nij\n\nXij\n\ni = Xij\n\nW (0)2\n\nij\n\n(1 (cid:0) 2(cid:17) + 3(cid:17)2) + (cid:17)2(N (cid:0) 1)Xij\n\nW (0)\n\nij\n\nwhere the \ufb01rst term is due entirely to the gradient signal and the second to the gradient\nnoise; choosing (cid:17) . 1=N allows the signal to be revealed via averaging over & N samples\n(see also the Discussion). This gradient noise is common to all three algorithms considered\nhere.\n\n2.2 Node perturbation\n\nHere averages are taken at each step not only over the inputs x but also over the noise (cid:24).\nThe weight update, recursion relation, learning curve, and convergence condition are\n\n(cid:1)WNP = (cid:0)\nhW (t)2\nit = Xij\n\nij\n\nXij\n\n(cid:17)\n(cid:27)2 ((cid:24)T W x +\nW (t(cid:0)1)2\n\nij\n\n1\n2\n\n(cid:24)T (cid:24))(cid:24)xT\n\n(1 (cid:0) 2(cid:17) + (cid:17)2(M + 2)(N + 2))\n\n+\n\n1\n4\n\n(cid:17)2(cid:27)2M N (M + 2)(M + 4))\n\nNPi = (cid:18)E(0) (cid:0)\nhE(t)\n\n(cid:17)(cid:27)2(M + 2)(M + 4)M N=8\n\n2 (cid:0) (N + 2)(M + 2)(cid:17) (cid:19) (1 (cid:0) 2(cid:17) + (M + 2)(N + 2)(cid:17)2)t\n\n(cid:17)(cid:27)2(M + 2)(M + 4)M N=8\n\n+\n\n2 (cid:0) (M + 2)(N + 2)(cid:17)\n2\n\n(cid:17) <\n\n(M + 2)(N + 2)\n\nIn this case the recursion relation has not only a multiplicative term as before but also an\nadditive one. The latter is a result of the noise (cid:24); when W is far from the minimum of the\nobjective function, (cid:24) will typically be small in comparison to W x and the additive term will\nbe negligible, but close to the minimum the noise will prevent the system from attaining\narbitrarily low error. This effect appears also in the learning curve. The limit on (cid:17) is stricter\nby a factor of M, the dimensionality of the noise, as discussed below.\n\n2.3 Weight perturbation\n\nThe same approach as before gives in this case\n\n(cid:1)WWP = (cid:0)\nhW (t)2\nit = Xij\n\nij\n\nXij\n\n(cid:17)\n(cid:27)2 (xT  T W x +\nW (t(cid:0)1)2\n\nij\n\n1\n2\n\nxT  T  x) \n\n(1 (cid:0) 2(cid:17) + (cid:17)2(M N + 2)(N + 2))\n\n\f+\n\n1\n4\n\n(cid:17)2(cid:27)2(M 3N 3 + 2M 2N 3 + 2M 3N 2 + 16M 2N 2 + 24M N )\n\n(cid:17)(cid:27)2M N (M N (M + 2)(N + 2) + 12(M N + 2))\n\n)\n\nhE(t)\n\nWPi = (E(0) (cid:0)\n\n8(2 (cid:0) (N + 2)(M N + 2)(cid:17))\n\n(cid:1)(1 (cid:0) 2(cid:17) + (N + 2)(M N + 2)(cid:17)2)t\n+\n\n(cid:17)(cid:27)2M N (M N (M + 2)(N + 2) + 12(M N + 2))\n\n8(2 (cid:0) (N + 2)(M N + 2)(cid:17))\n\n2\n\n(cid:17) <\n\n(M N + 2)(N + 2)\n\nAs with node perturbation, the recursion relation involves both multiplicative and additive\nterms, and the learning curve shows nonzero residual error even at in\ufb01nite time. The limit\non (cid:17) is a further factor of N smaller, corresponding to the greater dimensionality of  \ncompared to (cid:24).\n\n3 Comparison of learning curves\n\nAll three of the above learning curves hE (t)i take the form (cid:22)E(a((cid:17)))t + b((cid:17); (cid:27)), where b\nis the residual error which the network will approach as t ! 1 if learning converges,\n(cid:22)E (cid:17) E(0) (cid:0) b is the transient error, and a is a multiplicative factor by which (cid:22)E changes\nat each time step. The magnitude of a, which depends on the parameter (cid:17) but not on (cid:27),\ndetermines whether the average error will converge and the rate at which it will do so.\nFor the online gradient method, b = 0; a network trained this way, if it converges, will\napproach zero error as t ! 1. The stochastic algorithms have positive residual noise b,\nwhich depends on both (cid:17) and (cid:27); in the limit (cid:27) ! 0, this residual error vanishes. Of course,\n(cid:27) cannot be set directly to 0 or the stochastic algorithms will cease to function.\n\n3.1 Maximal learning rates\n\nThe analysis of the previous section suggests at least two reasonable ways to compare these\ndifferent algorithms with respect to performance. One is to choose the optimal learning rate\nfor each, that value of (cid:17) for which the average error converges most quickly. The learning\ncurves, to highest order in (cid:17), M, and N, then become\n\nOLi = (cid:22)E(cid:18)1 (cid:0)\nhE(t)\nNPi = (cid:22)E(cid:18)1 (cid:0)\nhE(t)\nWPi = (cid:22)E(cid:18)1 (cid:0)\nhE(t)\n\n1\n\n1\n\nN(cid:19)t\nM N(cid:19)t\nM N 2(cid:19)t\n\n1\n\n+\n\n1\n8\n\n(cid:27)2M 2\n\n+\n\n1\n8\n\n(cid:27)2M 2N\n\nDirect gradient descent, then, can train a network faster than can node perturbation, which\nin turn is faster than weight perturbation.\nThe noise takes different forms in the two stochastic variants. For node perturbation, (cid:24)i is\nadded directly to the ith output unit; for weight perturbation, the quantity added to the same\n\noutput unit is Pij  ijxj. By the central limit theorem, the latter approaches a Gaussian\nwith mean 0 and variance N (cid:27)2 for large N. For most direct comparison of the two stochas-\ntic variants, therefore, (cid:27) for (cid:24) should be chosen a factor pN larger than for  . With this\nchoice, the residual error for the two stochastic variants becomes identical, and the learning\ncurves differ only in their rates of convergence.\n\n\f3.2 Equal average updates\n\nA second way to compare the algorithms is to choose learning rates such that all three have\nthe same average weight update. As noted above, choosing the same value of (cid:17) in all three\ncases will ensure this condition. That common value of (cid:17) must be small enough that all\nthree algorithms converge; if we take (cid:17) (cid:28) 1\n\nM N 2 , the learning curves become\n\nhE(t)\nhE(t)\nhE(t)\n\nOLi = (cid:22)E(1 (cid:0) 2(cid:17))t\nNPi = (cid:22)E (1 (cid:0) 2(cid:17))t +\nWPi = (cid:22)E (1 (cid:0) 2(cid:17))t +\n\n1\n16\n1\n16\n\n(cid:17)(cid:27)2M 3N\n\n(cid:17)(cid:27)2M 3N 3\n\nWe began by saying that, because of the credit assignment problem of choosing updates\nto many parameters based on a single reward signal, intuition is that a stochastic gradient-\nfollowing approach should learn more slowly than a direct one. However, for equal small\n(cid:17), the average error for all three algorithms converges at the same rate. Weight perturbation\napproaches a larger value of residual error than does node perturbation; however, in the\n(cid:27) ! 0 limit, the residual error vanishes for both.\n4 Discussion\n\nIn a linear feedforward network of N input and M output units, in terms of the maximum\npossible rate of convergence of average error, online gradient descent on a squared error\nfunction is faster by a factor of M than node perturbation, which in turn is faster by a factor\nof N than weight perturbation. The difference in the rate of convergence is the dimen-\nsionality of the noise. Weight perturbation operates by explicit exploration of the entire\nM N-dimensional weight space; only one component of a particular update will be in the\ndirection of the true gradient for a given input, while the other components can be viewed as\nnoise masking that signal. That is, an update can be written as (cid:1)W = h(cid:1)Wi (the \u2018learning\nsignal\u2019, the actual gradient) + ((cid:1)W (cid:0) h(cid:1)Wi) (the \u2018learning noise\u2019), where the average is\ntaken over all values of  . This learning noise will typically have magnitude pM N larger\nthan the learning signal, and so M N samples are required in order to average it away. Di-\nrect gradient descent gives weight updates that are purely signal in this sense; while still\noccurring in an M N-dimensional space, they are by de\ufb01nition exactly in the direction of\nthe gradient for a given input. Thus no exploration of the weight space nor averaging over\nmultiple samples is necessary, and the maximum learning speed is correspondingly greater.\nNode perturbation is a stochastic algorithm like weight perturbation, but it explores the M-\ndimensional output space rather than the larger weight space; the learning noise is of lower\ndimension, and correspondingly fewer samples need to be averaged to reveal a learning\nsignal of a given size.\n\nIt has previously been argued that the maximum learning rate should scale, not with the\ndimensionality of the update as shown here, but with the square root of that dimensionality\n[4]. That claim is based on the fact that the squared magnitude of the update goes as the\nnumber of dimensions, and for a given error landscape and position in parameter space,\nthere will be a maximum update size, greater than which instability will result. However,\na more quantitative approach is to examine the conditions under which error will decrease,\nas we have done above. Rather than stopping with the statement that the size of the weight\nupdate scales as the square root of the number of dimensions, we have shown that this fact\nimplies that the restriction on convergence scales with the \ufb01rst power of the dimensionality.\nNumerical simulations of error curves, averaged over many individual trials with online\nupdating, support these conclusions with respect to both the quantitative shapes of the\nlearning curves and the scaling behavior of the conditions on convergence (Fig. 1).\n\n\fOnline gradient method\n\nNode perturbation\n\nWeight perturbation\n\n)\ns\nt\ni\nn\nu\n \ny\nr\na\nr\nt\ni\nb\nr\na\n(\n \nr\no\nr\nr\n\nE\n\n100\n\n102\n# of examples\n\n100\n\n102\n\n# of examples\n\n104\n\n100\n\n# of examples\n\n105\n\nFigure 1: Sample learning curves for the three algorithms applied to a linear feedforward\nnetwork as described in the text, showing the agreement between theory (black) and experi-\nment (gray). In each case, a network of linear units with N = 20; M = 25; (cid:27) = 10(cid:0)3, and\noptimal (cid:17) was trained on successive input examples for the number of iterations shown.\n100 such runs were averaged together in each case; the three gray lines show the mean\n(solid) and standard deviation (dashed) of squared error among those runs.\n\nThis scaling result means that, for these stochastic methods, there is no net advantage in\nspeed of training when all degrees of freedom are varied at the same time, compared to\nwhen they are varied sequentially, in terms of scaling with M and N. For instance, in the\ncase of weight perturbation, varying only one weight at a time would allow the learning\nrate to be increased by a factor on the order of M N; but each of the M N weights would\nneed to be trained in this way, so that the total training time required would scale in the\nsame way as if all were varied at once. (The speed of learning for parallel vs. sequential\nvariation, however, can differ by a constant ratio, though we do not pursue this issue here.)\n\nThe analysis here describes the behavior in a worst case of sorts, where the objective func-\ntion and distribution of inputs are isotropic. In the anisotropic case, where the problem is\neffectively lower-dimensional, the scaling behavior of all three methods can be correspond-\ningly more favorable than that derived here, and the relative performance of the stochastic\nmethods can be better.\n\nThe results described in this paper extend at least qualitatively to more complicated net-\nworks and architectures. For instance, Fig. 2 shows learning curves that result from apply-\ning the three algorithms to a two-layer feedforward network of nonlinear units. All three\nalgorithms give identical learning curves if the learning rate is set small enough; as (cid:17) is\nincreased, the weight perturbation curve fails to converge to low error, while the other two\ncurves continue to match; increasing (cid:17) further leads to the node perturbation curve also\nfailing to converge.\n\nIn the above, we have shown that stochastic gradient descent techniques can be expected\nto scale with increasing network size more poorly than direct ones, in terms of maximum\nlearning rate. This may serve as a caution regarding the size of networks they may use-\nfully be applied to. However, with learning rates in the regime where error converges,\nequal learning curves in each of the three will follow from equal learning rates, although\nindividual weight updates will typically be considerably different. This is because for cor-\nrespondingly small adjustments to the weights, only the component parallel to the gradient\nwill have a signi\ufb01cant effect on error; orthogonal components will not affect the error to\n\ufb01rst order. Moreover, node perturbation can have performance comparable to that of di-\nrect gradient descent even in training very large networks, so long as the number of output\nunits is small [6]. Thus these stochastic methods may be of considerable utility for train-\ning networks in some situations, particularly in reinforcement learning frameworks and\nthose where the gradient of the objective function is dif\ufb01cult or impossible to calculate, for\nmathematical or practical reasons.\n\n\f)\ns\nt\ni\nn\nu\n \ny\nr\na\nr\nt\ni\nb\nr\na\n(\n \nr\no\nr\nr\n\nE\n\n100\n\n = 4 \u00b7  10\u22124\n\n = 4 \u00b7  10\u22123\n\n = 4 \u00b7  10\u22122\n\n# of examples\n\n105\n\n100\n\n# of examples\n\n105\n\n100\n\n102\n# of examples\n\nFigure 2: Sample learning curves for the three algorithms applied to a two-layer nonlinear\nfeedforward network (gradient descent, black dotted; node perturbation, dark gray dashed;\nweight perturbation, light gray solid). The input, hidden, and output layers each had 10\nunits, whose output was equal to the hyperbolic tangent of their weighted input. Inputs and\nnoises were drawn from the same distributions as in the linear case; (cid:27) = 10(cid:0)3; (cid:17) had the\nvalue shown for all three algorithms in each panel. In each case, the network was trained\non successive input examples for the number of iterations shown; curves show single rep-\nresentative runs. Error was evaluated based on the total squared difference between the\noutput of the network and that of a teacher network with randomly chosen weights; the test\nerror shown was the mean of that for 100 random inputs not used in training.\n\nAcknowledgments\n\nWe thank Ila Fiete and Gert Cauwenberghs for useful discussions and comments. This\nwork was supported in part by a Packard Foundation Fellowship (to H.S. Seung) and NIH\ngrants (GM07484 to MIT and MH60651 to H.S. Seung).\n\nReferences\n\n[1] Widrow, B. & Lehr, M. A. 30 years of adaptive neural networks: Perceptron, Madaline, and\nbackpropagation. Proc. IEEE 78(9):1415\u20131442, 1990.\n\n[2] Jabri, M. & Flower, B. Weight perturbation: an optimal architecture and learning technique for\nanalog VLSI feedforward and recurrent multilayered networks. IEEE Transactions on Neural Net-\nworks 3(1):154\u2013157, 1992.\n\n[3] Flower, B. & Jabri, M. Summed weight neuron perturbation: an O(n) improvement over weight\nperturbation. In Advances in Neural Information Processing Systems 5, San Mateo, CA: Morgan\nKaufman Publishers: 212\u2013219, 1993.\n\n[4] Cauwenberghs, G. A fast stochastic error-descent algorithm for supervised learning and optimiza-\ntion. In Advances in Neural Information Processing Systems 5, San Mateo, CA: Morgan Kaufman\nPublishers: 244\u2013251, 1993.\n\n[5] Cauwenberghs, G. An analog VLSI recurrent neural network learning a continuous-time trajec-\ntory. IEEE Transactions on Neural Networks 7(2):346\u2013361, 1996.\n\n[6] Fiete, I. Private communication.\n\n[7] Bartlett, P. & Baxter, J. Hebbian synaptic modi\ufb01cations in spiking neurons that learn. Technical\nreport, November 27 1999.\n\n[8] Baldi, P. & Hornik, K. Learning in linear neural networks: a survey. IEEE Transactions on Neural\nNetworks 6(4):837\u2013858, 1995.\n\n[9] Williams, R.J. Simple statistical gradient-following algorithms for connectionist reinforcement\nlearning. Machine Learning 8:229\u2013256, 1992.\n\nh\nh\nh\n\f", "award": [], "sourceid": 2511, "authors": [{"given_name": "Justin", "family_name": "Werfel", "institution": null}, {"given_name": "Xiaohui", "family_name": "Xie", "institution": null}, {"given_name": "H.", "family_name": "Seung", "institution": null}]}