{"title": "On the Computational Complexity of Networks of Spiking Neurons", "book": "Advances in Neural Information Processing Systems", "page_first": 183, "page_last": 190, "abstract": null, "full_text": "On the Computational Complexity of Networks of \n\nSpiking Neurons \n(Extended Abstract) \n\nWolfgang Maass \n\nInstitute for Theoretical Computer Science \n\nTechnische Universitaet Graz \n\nA-80lO Graz, Austria \n\ne-mail: maass@igi.tu-graz.ac.at \n\nAbstract \n\nWe investigate the computational power of a formal model for net(cid:173)\nworks of spiking neurons, both for the assumption of an unlimited \ntiming precision, and for the case of a limited timing precision. We \nalso prove upper and lower bounds for the number of examples that \nare needed to train such networks. \n\n1 \n\nIntroduction and Basic Definitions \n\nThere exists substantial evidence that timing phenomena such as temporal differ(cid:173)\nences between spikes and frequencies of oscillating subsystems are integral parts \nof various information processing mechanisms in biological neural systems (for a \nsurvey and references see e.g. Abeles, 1991; Churchland and Sejnowski, 1992; Aert(cid:173)\nsen, 1993). Furthermore simulations of a variety of specific mathematical models \nfor networks of spiking neurons have shown that temporal coding offers interesting \npossibilities for solving classical benchmark-problems such as associative memory, \nbinding, and pattern segmentation (for an overview see Gerstner et al., 1992). Some \naspects of these models have also been studied analytically, but almost nothing is \nknown about their computational complexity (see Judd and Aihara, 1993, for some \nfirst results in this direction). In this article we introduce a simple formal model \nSNN for networks of spiking neurons that allows us to model the most important \ntiming phenomena of neural nets (including synaptic modulation), and we prove up(cid:173)\nper and lower bounds for its computational power and learning complexity. Further \n\n\f184 \n\nWolfgang Maass \n\ndetails to the results reported in this article may be found in Maass, 1994a,1994b, \n1994c. \nDefinition of a Spiking Neuron Network (SNN): An SNN N consists of \n- a finite directed graph {V, E} (we refer to the elements of V as \"neurons\" \n\nand to the elements of E as \"synapses\") \n\n- a subset Yin S; V of input neurons \n- a subset Vout S; V of output neurons \n- for each neuron v E V - Yin a threshold-function 9 v : R+ -+ R U {oo} \n\n{where R+ := {x E R : x ~ O}) \n\n- for each synapse {u, v} E E a response-function \u00a3u,v \n\n: R+ -+ R and a \n\nweight- function Wu,v : R+ -+ R \n\n. \n\nWe assume that the firing of the input neurons v E Yin is determined from outside \nof N, i.e. the sets Fv S; R+ of firing times (\"spike trains\") for the neurons v E \nYin are given as the input of N. Furthermore we assume that a set T S; R+ of \npotential firing times 7iiiSfjeen fixed. \nFor a neuron v E V - Yin one defines its set Fv of firing times recursively. The \n, and for any s E Fv the next \nfirst element of Fv is \nlarger element of Fv is \n,where the \npotential function Pv : R+ -+ R is defined by \nL \n\ninf{t E T : t > sand Pv(t) ~ 0 v(t - s)} \n\ninf{t E T : Pv(t) ~ 0 v(O)} \n\nPv(t) := 0 + L \n\nu : {u, v} E EsE Fu : s < t \n\nwu,v(s) . \u00a3u,v(t - s) \n\nThe firing times (\"spike trains\") Fv of the output neurons v E Vout that result in \nthis way are interpreted as the output of N. \nRegarding the set T of potential firing times we consider in this article the case \nT = R+ (SNN with continuous time) and the case T = {i\u00b7 JJ : i E N} for some JJ \nwith 1/ JJ E N (SNN with discrete time). \nWe assume that for each SNN N there exists a bound TN E R with TN > 0 such \nthat 0 v(x) = 00 for all x E (0, TN) and all v E V - Yin (TN may be interpreted \nas the minimum of all \"refractory periods\" Tref of neurons in N). Furthermore we \nassume that all \"input spike trains\" Fv with v E Yin satisfy IFv n [0, t]l < 00 for \nall t E R+. On the basis of these assumptions one can also in the continuous case \neasily show that the firing times are well-defined for all v E V - Yin (and occur in \ndistances of at least TN)' \nInput- and Output-Conventions: For simulations between SNN's and Turing \nmachines we assume that the SNN either gets an input (or produces an output) \nfrom {O, 1}* in the form of a spike-train (i.e. one bit per unit of time), or encoded \ninto the phase-difference of just two spikes. Real-valued input or output for an SNN \nis always encoded into the phase-difference of two spikes. \n\nRemarks \na) In models for biological neural systems one assumes that if x time-units have \n\n\fOn the Computational Complexity of Networks of Spiking Neurons \n\n/85 \n\npassed since its last firing, the current threshold 0 11 (z) of a neuron v is \"infinite\" \nfor z < TreJ (where TreJ = refractory period of neuron v), and then approaches \nquite rapidly from above some constant value. A neuron v \"fires\" (i.e. it sends an \n\"action potential\" or \"spike\" along its axon) when its current membrane potential \nPII (t) at the axon hillock exceeds its current threshold 0 11 . PII (t) is the sum of \nvarious postsynaptic potentials W U ,II(S). t: U ,II(t - s). Each of these terms describes an \nexcitatory (EPSP) or inhibitory (IPSP) postsynaptic potential at the axon hillock of \nneuron v at time t, as a result of a spike that had been generated by a \"presynaptic\" \nneuron u at time s, and which has been transmitted through a synapse between both \nneurons. Recordings of an EPSP typically show a function that has a constant value \nc (c = resting membrane potential; e.g. c = -70m V) for some initial time-interval \n(reflecting the axonal and synaptic transmission time), then rises to a peak-value, \nand finally drops back to the same constant value c. An IPSP tends to have the \nnegative shape of an EPSP. For the sake of mathematical simplicity we assume in \nthe SNN-model that the constant initial and final value of all response-functions \nt:U ,1I is equal to 0 (in other words: t:U ,1I models the difference between a postsynaptic \npotential and the resting membrane potential c). Different presynaptic neurons u \ngenerate postsynaptic potentials of different sizes at the axon hillock of a neuron \nv, depending on the size, location and current state of the synapse (or synapses) \nbetween u an? v. This effect is modelled by the weight-factors W U ,II(S). \nThe precise shapes of threshold-, response-, and weight-functions vary among dif(cid:173)\nferent biological neural systems, and even within the same system. Fortunately one \ncan prove significant upper bounds for the computational complexity of SNN's N \nwithout any assumptions about the specific shapes of these functions of N. Instead, \nwe only assume that they are of a reasonably simple mathematical structure. \nb) In order to prove lower bounds for the computational complexity of an SNN N \none is forced to make more specific assumptions about these functions . All lower \nbound results that are reported in this article require only some rather weak basic \nassumptions about the response- and threshold-functions. They mainly require \nthat EPSP's have some (arbitrarily short) segment where they increase linearly, \nand some (arbitrarily short) segment where they decrease linearly (for details see \nMaass, 1994a, 1994b). \nc) Although the model SNN is apparently more \"realistic\" than all models for bio(cid:173)\nlogical neural nets whose computational complexity has previously been analyzed, \nit deliberately sacrifices a large number of more intricate biological details for the \nsake of mathematical tractability. Our model is closely related to those of (Buh(cid:173)\nmann and Schulten, 1986), and (Gerstner, 1991, 1992). Similarly as in (Buhmann \nand Schulten, 1986) we consider here only the deterministic case. \nd) The model SNN is also suitable for investigating algorithms that involve synaptic \nmodulation at various time-scales. Hence one can investigate within this framework \nnot only the complexity of algorithms for supervised and unsupervised learning, but \nalso the potential computational power of rapid weight-changes within the course of \na computation. In the theorems of this paper we allow that the value of a weight \nWU,II(S) at a firing time s E Fu is defined by an algebraic computation tree (see van \nLeeuwen, 1990) in terms of its value at previous firing times s' E Fu with s' < s, \nsome preceding firing times s < s of arbitrary other neurons, and arbitrary real(cid:173)\nvalued parameters. In this way WU,II(S) can be defined by different rational functions \n\n\f/86 \n\nWolfgang Maass \n\nof the abovementioned arguments, depending on the numerical relationship between \nthese arguments (which can be evaluated by comparing first the relative size of \narbitrary rational functions of these arguments). As a simple special case one can \nfor example increase wu \u2022tI (perhaps up to some specified saturation-value) as long \nas neurons u and v fire coherently, and decrease wu \u2022tI otherwise. \n\nFor the sake of simplicity in the statements of our results we assume in this extended \nabstract that the algebraic computation tree for each weight w U \u2022tI involves only \n0(1) tests and rational functions of degree 0(1) that depend only on 0(1) of the \nabovementioned arguments. Furthermore we assume in Theorems 3, 4 and 5 that \neither each weight is an arbitrary time-invariant real, or that each current weight is \nrounded off to bit-length poly(1ogpN') in binary representation, and does not depend \non the times of firings that occured longer than time 0(1) ago. Furthermore we \nassume in Theorems 3 and 5 that the parameters in the algebraic computation tree \nare rationals of bit-length O(1ogpN'). \n\ne) It is well-known that the Vapnik-Chervonenkis dimension {\"VC-dimension\"} of \na neural net N (and the pseudo-dimension for the case of a neural net N with real(cid:173)\nvalued output, with some suitable fixed norm for measuring the error) can be used \nto bound the number of examples that are needed to train N (see Haussler, 1992). \nObviously these notions have to be defined differently for a network with time(cid:173)\ndependent weights. We propose to define the VC-dimension (pseudo-dimension)of \nan SNN N with time-dependent weights as the VC-dimension (pseudo-dimension) \nof the class of all functions that can be computed by N with different assignments of \nvalues to the real-valued (or rational-valued) parameters of N that are involved in \nthe definitions of the piecewise rational response-, threshold-, and weight-functions \nof N. In a biological neural system N these parameters might for example reflect \nthe concentrations of certain chemical substances that are known to modulate the \nbehavior of N. \nf) The focus in the investigation of computations in biological neural systems differs \nin two essential aspects from that of classical computational complexity theory. \nFirst, one is not only interested in single computations of a neural net for unrelated \ninputs z, but also in its ability to process an interrelated sequence \u00ab(z( i), y( i)} )ieN \nof inputs and outputs, which may for example include an initial training sequence \nfor learning or associative memory. Secondly, exact timing of computations is all(cid:173)\nimportant in biological neural nets, and many tasks have to be solved within a \nspecific number of steps. Therefore an analysis in terms of the notion of a real-time \ncomputation and real-time simulation appears to be more adequate for models of \nbiological neural nets than the more traditional analysis via complexity classes. \nOne says that a sequence \u00ab(z(i),y(i)})ieN is processed in real-time by a machine \nM, if for every i E N the machine M outputs y( i) within a constant number c of \ncomputation steps after having received input z(i). One says that M' simulates M \nin real-time (with delay factor ~), if every sequence that is processed in real-time \nby M (with some constant c), can also be processed in real-time by M' (with a \nconstant ~ . c). For SNN's M we count each spike in M as a computation step. \nThese definitions imply that a real-time simulation of M by M' is a special case of \na linear-time simulation, and hence that any problem that can be solved by M with \na certain time complexity ten), can be solved by M' with time complexity O(t(n\u00bb \n\n\fOn the Computational Complexity of Networks of Spiking Neurons \n\n187 \n\n(see Maass, 1994a, 1994b, for details). \n\n2 Networks of Spiking Neurons with Continuous Time \n\nTheorem 1: If the response- and threshold-functions of the neurons satisfy some \nrather weak basic assumptions (see Maass, 1994a, 1994b), then one can build from \nsuch neurons for any given dEN an SNN NTM(d) of finite size with rational \ndelays that can simulate with a suitable assignment of rational values from [0, 1] to \nits weights any Turing machine with at most d tapes in real-time. \n\nFurthermore NTM(2) can compute any function F : {0,1}* -- {0,1}* with a \n\nsuitable assignment of real values from [0,\"1] to its weights. \n\nThe fixed SNN NTM(d) of Theorem 1 can simulate Turing machines whose tape \ncontent is much larger than the size of NTM (d), by encoding such tape content into \nthe phase-difference between two oscillators. The proof of Theorem 1 transforms ar(cid:173)\nbitrary computations of Turing machines into operations on such phase-differences. \n\nThe last part of Theorem 1 implies that the VC-dimension of some finite SNN's \nis infinite. In contrast to that the following result shows that one can give finite \nbounds for the VC-dimension of those SNN's that only use a bounded numbers of \nspikes in their computation. Furthermore the last part of the claim of Theorem 2 \nimplies that their VC-dimension may in fact grow linearly with the number S of \nspikes that occur in a computation. \n\nTheorem 2: The VC-dimension and pseudo-dimension of any SNN N with piece(cid:173)\nwise linear response- and threshold-functions, arbitrary real-valued parameters and \ntime-dependent weights (as specified in section 1) can be bounded (even for real(cid:173)\nvalued inputs and outputs) by D(IEI . WI . S(log IVI + log S\u00bb \nif N uses in each \ncomputation at most S spikes. \n\nFurthermore one can construct SNN's (with any response- and threshold-functions \n\nthat satisfy our basic assumptions, with fixed rational parameters and rational time(cid:173)\ninvariant weights) whose VC-dimension is for computations with up to S spikes as \nlarge as O(IEI . S). \nWe refer to Maass, 1994a, 1994c, for upper bounds on the computational power of \nSNN's with continuous time. \n\n3 Networks of Spiking Neurons with Discrete Time \n\nIn this section we consider the case where all firing times of neurons in N are \nmultiples of some J.l with 1/ J.l EN. We restrict our attention to the biologically \nplausible case where there exists some tN ~ 1 such that for all z > tN all response \n\nfunctions \u00a3U,II(Z) have the value \u00b0 and all threshold functions ell(z) have some \narbitrary constant value. If tN is chosen minimal with this property, we refer to \nPN := rtN/J.ll as the timing-precision ofN. Obviously for PN = 1 the SNN is \nequivalent to a \"non-spiking\" neural net that consists of linear threshold gates, \nwhereas a SNN with continuous time may be viewed as the opposite extremal case \nfor PN -- 00. \n\n\f188 \n\nWolfgang Maass \n\nThe following result provides a significant upper bound for the computational power \nof an SNN with discrete time, even in the presence of arbitrary real-valued parame(cid:173)\nters and weights. Its proof is technically rather involved. \n\nTheorem 3: Assume that N is an SNN with timing-precision PJII, piecewise polyno(cid:173)\nmial response- and piecewise rational threshold-functions with arbitrary real-valued \nparameters, and weight-functions as specified in section 1. \n\nThen one can simulate N for boolean valued inputs in real-time by a Turing ma(cid:173)\nchine with poly(lVl, logpJII,log l/TJII) states and poly(lVl, logpJII, tJII/TJII) tape-cells. \nOn the other hand any Turing machine with q states that uses at most s tape(cid:173)\ncells can be simulated in real-time by an SNN N with any response- and threshold(cid:173)\nfunctions that satisfy our basic assumptions, with rational parameters and time(cid:173)\ninvariant rational weights, with O(q) neurons, logpJII = O(s), and tJII/TJII = 0(1). \n\nThe next result shows that the VC-dimension of any SNN with discrete time is \nfinite, and grows proportionally to logpJII. The proof of its lower bound combines a \nnew explicit construction with that of Maass, 1993. \n\nTheorem 4: Assume that the SNN N has the same properties as in Theorem 3. \nThen the VC-dimension and the pseudo-dimension of N (for arbitrary real valued \ninputs) can be bounded by O(IEI\u00b7IVI\u00b7logpJII), independently of the number of spikes \nin its computations. \n\nFurthermore one can construct SNN's N of this type with any response- and \nthreshold-functions that satisfy our basic assumptions, with rational parameters and \ntime-invariant rational weights, so that N has (already for boolean inputs) a VC(cid:173)\ndimension of at least O(IEI(logpJII + log IE!\u00bb. \n\n4 Relationships to other Computational Models \nWe consider here the relationship between SNN's with discrete time and recurrent \nanalog neural nets. In the latter no \"spikes\" or other non-trivial timing-phenomena \noccur, but the output of a gate consists of the \"analog\" value of some squashing(cid:173)\nor activation function that is applied to the weighted sum of its inputs. See e.g. \n(Siegelmann and Sontag, 1992) or (Maass, 1993) for recent results about the compu(cid:173)\ntational power of such models. We consider in this section a perhaps more \"realistic\" \nversion of such modelsN, where the output of each gate is rounded off to an integer \nmultiple of some ~ (with a EN). We refer to a as the number of activation levels \nof N. \nIt is an interesting open problem whether such analog neural nets (with gate-outputs \ninterpreted as firing rates) or networks of spiking neurons provide a more adequate \ncomputational model for biological neural systems. Theorem 5 shows that in spite \nof their quite different structure the computational power of these two models is in \nfact closely related. \n\nOn the side the following theorem also exhibits a new subclass of deterministic \nfinite automata (DFA's) which turns out to be of particular interest in the context \nof neural nets. We say that a DFA M is a sparse DFA of size s if M can be realized \nby a Turing machine with s states and space-bound s (such that each step of M \ncorresponds to one step of the Turing machine). Note that a sparse DFA may have \nexponentially in s many states, but that only poly(s) bits are needed to describe its \n\n\fOn the Computational Complexity of Networks of Spiking Neurons \n\n189 \n\ntransition function. Sparse DFA's are relatively easy to construct, and hence are \nvery useful for demonstrating (via Theorem 5) that a specific task can be carried \nout on a \"spiking\" neural net with a realistic timing precision (respectively on an \nanalog neural net with a realistic number of activation levels). \n\nTheorem 5: The following classes of machines have closely related computational \npower in the sense that there is a polynomial p such that each computational model \nfrom any of these classes can be simulated in real-time (with delay-factor ~ p(s\u00bb) by \nsome computational model from any other class (with the size-parameter s replaced \nby p(s\u00bb): \n\n\u2022 sparse DFA's of size s \n\u2022 SNN's with 0(1) neurons and timing precision 23 \n\u2022 recurrent analog neural nets that consist of O( 1) gates with piecewise ra(cid:173)\n\ntional activation functions with 23 activation levels, and parameters and \nweights of bit-length $ s \n\n\u2022 neural nets that consist of s linear threshold gates (with recurrencies) with \n\narbitrary real weights. \n\nThe result of Theorem 5 is remarkably stable since it holds no matter whether one \nconsiders just SNN's N with 0(1) neurons that employ very simple fixed piecewise \nlinear response- and threshold-functions with parameters of bit-length 0(1) (with \ntN/TN = 0(1) and time-invariant weights of bit-length $ s), or if one considers \nSNN's N with s neurons with arbitrary piecewise polynomial response- and piece(cid:173)\nwise rational threshold-functions with arbitrary real-valued parameters, tN/TN ~ s, \nand time-dependent weights (as specified in section 1). \n\n5 Conclusion \n\nWe have introduced a simple formal model SNN for networks of spiking neurons, \nand have shown that significant bounds for its computational power and sample \ncomplexity can be derived from rather weak assumptions about the mathematical \nstructure of its response-, threshold-, and weight-functions. Furthermore we have \nestablished quantitative relationships between the computational power of a model \nfor networks of spiking neurons with a limited timing precision (i.e. SNN's with \ndiscrete time) and a quite realistic version of recurrent analog neural nets (with a \nbounded number of activation levels). The simulations which provide the proof of \nthis result create an interesting link between computations with spike-coding (in \nan SNN) and computations with frequency-coding (in analog neural nets). We also \nhave established such relationships for the case of SNN's with continuous time (see \nMaass 1994a, 1994b, 1994c), but space does not permit to report these results in \nthis article. \n\nThe Theorems 1 and 5 of this article establish the existence of mechanisms for sim(cid:173)\nulating arbitrary Turing machines (and hence any common computational model) \non an SNN. As a consequence one can now demonstrate that a concrete task (such \nas binding, pattern-matching, associative memory) can be carried out on an SNN \nby simply showing that some arbitrary common computational model can carry out \nthat task. Furthermore one can bound the required timing-precision of the SNN in \nterms of the space needed on a Turing machine. \n\n\f190 \n\nWolfgang Maass \n\nSince we have based our investigations on the rather refined notion of a real-time \nsimulation, our results provide information not only about the possibility to imple(cid:173)\nment computations, but also adaptive behavior on networks of spiking neurons. \n\nAcknowledgement \nI would like to thank Wulfram Gerstner for helpful discussions. \nReferences \nM. Abeles. (1991) Corticonics: Neural Circuits of the Cerebral Cortex. Cambridge \nUniversity Press. \nA. Aertsen. ed. (1993) Brain Theory: Spatio-Temporal Aspects of Brain Function. \nElsevier. \nJ. Buhmann, K. Schulten. (1986) Associative recognition and storage in a model \nnetwork of physiological neurons. Bioi. Cybern. 54: 319-335. \n\nP. S. Churchland, T. J. Sejnowski. (1992) The Computational Brain. MIT-Press. \n\nW. Gerstner. \n(1991) Associative memory in a network of \"biological\" neurons. \nAdvances in Neural Information Processing Systems, vol. 3, Morgan Kaufmann: \n84-90. \nW. Gerstner, R. Ritz, J. L. van Hemmen. (1992) A biologically motivated and \nanalytically soluble model of collective oscillations in the cortex. Bioi. Cybern. 68: \n363-374. \nD. Haussler. (1992) Decision theoretic generalizations of the PAC model for neural \nnets and other learning applications. Inf and Comput. 95: 129-161. \nK. T. Judd, K. Aihara. (1993) Pulse propagation networks: A neural network \nmodel that uses temporal coding by action potentials. Neural Networks 6: 203-215. \n\nJ. van Leeuwen, ed. (1990) Handbook of Theoretical Computer Science, vol. A: \nAlgorithms and Complexity. MIT-Press. \n\nW. Maass. (1993) Bounds for the computational power and learning complexity \nof analog neural nets. Proc. 25th Annual ACM Symposium on the Theory of \nComputing, 335-344. \n\nW. Maass. (1994a) On the computational complexity of networks of spiking neurons \n(extended abstract). TR 393 from May 1994 of the Institutes for Information \nProcessing Graz (for a more detailed version see the file maass.spiking.ps.Z in the \nneuroprose archive). \nW. Maass. \nspiking neurons. Neural Computation, to appear. \n\n(1994b) Lower bounds for the computational power of networks of \n\nW. Maass. (1994c) Analog computations on networks of spiking neurons (extended \nabstract). Submitted for publication. \n\nH. T. Siegelmann, E. D. Sontag. (1992) On the computational power of neural nets. \nProc. 5th ACM- Workshop on Computational Learning Theory, 440-449. \n\n\f", "award": [], "sourceid": 926, "authors": [{"given_name": "Wolfgang", "family_name": "Maass", "institution": null}]}