{"title": "Asynchronous Dynamics of Continuous Time Neural Networks", "book": "Advances in Neural Information Processing Systems", "page_first": 493, "page_last": 500, "abstract": null, "full_text": "Asynchronous Dynamics of Continuous \n\nTime Neural Networks \n\nComputer Science Department \n\nUniversity of California at Los Angeles \n\nXin Wang \n\nLos Angeles, CA 90024 \n\nQingnan Li \n\nDepartment of Mathematics \n\nUniversity of Southern California \n\nLos Angeles, CA 90089-1113 \n\nEdward K. Blum \n\nDepartment of Mathematics \n\nUniversity of Southern California \n\nLos Angeles, CA 90089-1113 \n\nABSTRACT \n\nMotivated by mathematical modeling, analog implementation and \ndistributed simulation of neural networks, we present a definition of \nasynchronous dynamics of general CT dynamical systems defined \nby ordinary differential equations, based on notions of local times \nand communication times. We provide some preliminary results \non globally asymptotical convergence of asynchronous dynamics \nfor contractive and monotone CT dynamical systems. When ap(cid:173)\nplying the results to neural networks, we obtain some conditions \nthat ensure additive-type neural networks to be asynchronizable. \n\n1 \n\nINTRODUCTION \n\nNeural networks are massively distributed computing systems. A major issue in par(cid:173)\nallel and distributed computation is synchronization versus asynchronization (Bert(cid:173)\nsekas and Tsitsiklis, 1989). To fix our idea, we consider a much studied additive-type \nmodel (Cohen and Grossberg, 1983; Hopfield, 1984; Hirsch, 1989) of a continuous(cid:173)\ntime (CT) neural network of n neurons, whose dynamics is governed by \n\nXi(t) = -ajXi(t) + L WijO'j (Jlj Xj (t)) + Ii, \n\nn \n\ni = 1,2, ... , n, \n\nj=1 \n\n(1) \n\n493 \n\n\f494 \n\nWang. Li. and Blum \n\nwith neuron states Xi (t) at time t, constant decay rates ai, external inputs h, gains \nJJj, neuron activation functions Uj and synaptic connection weights Wij. Simu(cid:173)\nlation and implementation of idealized models of neural networks such as (1) on \ncentralized computers not only limit the size of networks, but more importantly \npreclude exploiting the inherent massive parallelism in network computations. A \ntruly faithful analog implementation or simulation of neural networks defined by \n(1) over a distributed network requires that neurons follow a global clock t, com(cid:173)\nmunicate timed states Xj(t) to all others instantaneously and synchronize global \ndynamics precisely all the time (e.g., the same Xj(t) should be used in evolution of \nall Xi(t) at time t). Clearly, hardware and software realities make it very hard and \nsometimes impossible to fulfill these requirements; any mechanism used to enforce \nsuch synchronization may have an important effect on performance of the net(cid:173)\nwork. Moreover, absolutely insisting on synchronization contradicts the biological \nmanifestation of inherent asynchrony caused by delays in nerve signal propagation, \nvariability of neuron parameters such as refractory periods and adaptive neuron \ngains. On the other hand, introduction of asynchrony may change network dynam(cid:173)\nics, for example, from convergent to oscillatory. Therefore, validity of asynchronous \ndynamics of neural networks must be assessed in order to ensure desirable dynamics \nin a distributed environment. \n\nMotivated by the above issues, we study asynchronous dynamics of general CT dy(cid:173)\nnamical systems with neural networks in particular. Asynchronous dynamics has \nbeen thoroughly studied in the context of iterative maps or discrete-time (DT) dy(cid:173)\nnamical systems; see, e.g., (Bertsekas and Tsitsiklis, 1989) and references therein. \nAmong other results are that P-contractive maps on Rn (Baudet, 1978) and contin(cid:173)\nuous maps on partially ordered sets (Wang and Parker, 1992) are asynchronizable, \ni.e., any asynchronous iterations of these maps will converge to the fixed points \nunder synchronous (or parallel) iterations. The synchronization issue has also been \naddressed in the context of neural networks. In fact, the celebrated DT Hopfield \nmodel (Hopfield, 1982) adopts a special kind of asynchronous dynamics: only one \nrandomly chosen neuron is allowed to update its state at each iterative step. The \nissue is also studied in (Barhen and Gulati, 1989) for CT neural networks. The \napproach there is, however, to convert the additive model (1) into a DT version \nthrough the Euler discretization and then to apply the existing result for contrac(cid:173)\ntive mappings in (Baudet, 1978) to ensure the discretized system to be asynchro(cid:173)\nnizable. Overall, studies for asynchronous dynamics of CT dynamical systems are \nstill lacking; there are even no reasonable definitions for what it means, at least to \nour knowledge. \n\nIn this paper, we continue our studies on relationships between CT and DT dy(cid:173)\nnamical systems and neural networks (Wang and Blum, 1992; Wang, Blum and Li, \n1993) and concentrate on their asynchronous dynamics. We first extend a concept \nof asynchronous dynamics of DT systems to CT systems, by identifying the distinc(cid:173)\ntion between synchronous and asynchronous dynamics as (i) presence or absence of \na common global clock used to synchronize the dynamics of the different neurons \nand (ii) exclusion or inclusion of delay times in communication between neurons, \nand present some preliminary results for asynchronous dynamics of contractive and \nmonotone CT systems. \n\n\fAsynchronous Dynamics of Continuous Time Neural Networks \n\n495 \n\n2 MATHEMATICAL FORMULATION \n\nTo be general, we consider a CT dynamical system defined by an n-dimensional \nsystem of ordinary differential equations, \n\n(2) \nwhere Ii : Rn --+ R are continuously differentiable and x(t) E Rn for all t in R+ (the \nset of all nonnegative real numbers). In contrast to the asynchronous dynamics \ngiven below, dynamics of this system will be called synchronous. An asynchronous \n: R+ --+ R+ and rj : R+ --+ R+, \nscheme consists of two families of functions Ci \ni, j = 1, ... , n, satisfying the following constraints: for any t > 0, \n(i) Initiation: Ci(t) ~ 0 and rJ(t) ~ 0; \n(ii) Non-starvation: Ci'S are differentiable and l\\(t) > 0; \n(iii) Liveness: limt_oo Ci(t) = 00 and limt_oo rJ(t) = 00; \n\n(iv) Accessibility: rj(t) ~ Cj(t). \n\nGiven an asynchronous scheme ({cd, {rJ}), the associated asynchronous dynamics \nof the system (2) is the solution of the following parametrized system: \n\n(3) \n\nWe shall call this system an asynchronized system of the original one (2). \nThe functions Ci(t) should be viewed as respective \"local\" times (or clocks) of com(cid:173)\nponents i, as compared to the \"global\" time (or clock) t. As each component i \nevolves its state according to its local time Ci(t), no shared global time t is needed \nexplicitly; t only occurs implicitly. The functions rj(t) should be considered as time \ninstants at which corresponding values Xi of components j are used by component \ni; hence the differences (ci(t) - rj(t\u00bb ~ 0 can be interprated as delay times in \ncommunication between the components j and i. Constraint (i) reflects the fact \nthat we are interested in the system dynamics after some global time instance, say \n0; constraint (ii) states that the functions Ci are monotone increasing and hence the \nlocal times evolve only forward; constraint (iii) characterizes the live ness property \nof the components and communication channels between components; and, finally, \nconstraint (iv) precludes the possibility that component i accesses states x j ahead \nof the local times Cj(t) of components j which have not yet been generated. \n\nNotice that, under the assumption on monotonicity of Ci(t), the inverses C;l(t) exist \nand the asynchronized system (3) can be transformed into \n\n(4) \nby letting Yi(t) = Xi( Ci(t\u00bb and y} (t) = Xj (rJ(t\u00bb = Yj (c;l (rJ(t\u00bb for i, j = 1,2, ... , n. \nThe vector form of (4) can be given by \n\niJ = Cf F[Y] \n\n(5) \n\n\f496 \n\nWang, Li, and Blum \n\nwhere yet) = [Yl (t), \"\" Yn(t)]T, C' = diag(dcl (t)/dt, \"\" dcn(t)/dt) , F = [/1, \"\" fn]T, \ny = [Y;] and \n\n/1 cYi(t) , yHt), \"\" y~(t)) 1 \n' \n\nhcYr(t), y~(t), \"\" y~(t)) \n\n, \n\n_ \n\n[ \nF[Y] = \n\nfn (i/'l (t), y~(t), \"\" y~(t)) \n\nNotice that the complication in the way F applies to Y ~imply means ,that every \ncomponent i will use possibly different \"global\" states [Yi(t) , y2(t) , \"\" y~(t)] , This \npeculiarity makes the equation (5) fit into none ofthe categories of general functional \ndifferential equations (Hale, 1977), However, if rJ(t) for i = 1, \"., n are equal, \nall the components will use a same global state y = [yHt) , y~(t), .. \" y~(t)] and \nthe asynchronized system (5) assumes a form of retarded functional differential \nequations, \n\niJ = c' FcY), \n\n(6) \nWe shall call this case uniformly-delayed, which will be a main consideration in the \nnext section where we discuss asynchronizable systems, \nThe system (5) includes some special cases. In a no communication delay situation, \nrj(t) = Cj(t) for all i and the system (5) reduces to iJ = C' F(y), This includes the \nsimplest case where the local times Ci(t) are taken as constant-time scalings cit of \nthe global time t; specially, when all Ci(t) = t the system goes back to the original \none (2), If, on the other hand, all the local time~ are identi~al to the global time t \nand the communication times take the form of rJ(t) = t - OJ(t) one obtains a most \ngeneral delayed system \n\n(7) \n\nwhere the state Yj(t) of component j may have different delay times O)(t) for dif(cid:173)\nferent other components i. \n\nFinally, we should point out that the above definitions of asynchronous schemes and \ndynamics are analogues of their counterparts for DT dynamical systems (Bertsekas \nand Tsitsiklis, 1989; Blum, 1990), Usually, an asynchronous scheme for a DT \nsystem defined by a map f : X -+ X, where X = Xl X X2 X '\" X X n , consists of a \nfamily {Ti ~ N I i = 1, , .. , n} of subset~ of discrete times (N) at which components \ni update their states and a family {rJ : N -+ N I i = 1,2\"\", n} of communication \ntimes, Asynchronous dynamics (or chaotic iteration, relaxation) is then given by \n\nX.(t + 1) = { fi(xl(rt(t)), \"', xn(r~(t))) if t E ~ \notherwise. \n\nXi(t) \n\nI \n\nNotice that the sets Ti can be interpreted as local times of components i . In fact, \none can define local time functions Ci : N -+ N as Ci(O) = 0 and Ci(t + 1) = Ci(t) + 1 \nif t E 11 and Ci(t) otherwise. The asynchronous dynamics can then be defined by \n\nXi(t + 1) - Xi(t) = (Ci(t + 1) - ci(t))(fi(xl(rf(t)), ... ,Xn(r~(t))) - Xi(t)), \n\nwhich is analogous to the definition given in (4). \n\n\fAsynchronous Dynamics of Continuous Time Neural Networks \n\n497 \n\n3 ASYNCHRONIZABLE SYSTEMS \n\nIn general, we consider a CT dynamical system as asynchronizable ifits synchronous \ndynamics (limit sets and their asymptotic stability) persists for some set of asyn(cid:173)\nchronous schemes. In many cases, asynchronous dynamics of an arbitrary CT sys(cid:173)\ntem will be different from its synchronous dynamics, especially when delay times \nin communication are present. An example can be given for the network (1) with \nsymmetric matrix W. It is well-known that (synchronous) dynamics of such net(cid:173)\nworks is quasi-convergent, namely, all trajectories approach a set of fixed points \n(Hirsch, 1989). But when delay times are taken into consideration, the networks \nmay have sustained oscillation when the delays exceed some threshold (Marcus and \nWestervelt, 1989). A more careful analysis on oscillation induced by delays is given \nin (Wu, 1993) for the networks with symmetric circulant weight matrices. \n\nHere, we focus on asynchronizable systems. We consider CT dynamical systems on \nRn of the following general form \n\nAx(t) = -x(t) + F(x(t\u00bb \n\n(8) \nwhere x(t) ERn, A = diag(a1,a2, ... ,an) with aj > 0 and F = [Ji] E G1(Rn). It \nis easy to see that a point x E Rn is a fixed point of (8) if and only if x is a fixed \npoint of the map F. Without loss of generality, we assume that 0 is a fixed point \nof the map F. According to (5), the asynchronized version of (8) for an arbitrary \nasynchronous scheme ({ cd, { rj}) is \n\nAy = G'( -y + F[Y]), \n\n(9) \n\nwhere jj = (jjtct), jj~(t), ... , y~(t)]. \n\n3.1 Contractive Systems \n\nOur first effort attempts to obtain a result similar to the one for P-contractive \nmaps in (Baudet, 1978). We call the system (8) strongly P-contractive if there is a \nsymmetric and invertible matrix S such that IS- 1 F(Sx)1 < Ixl for all x E Rn and \nIS- 1 F(Sx)1 = Ixl only for x = 0; here Ixl denotes the vector with components Ixil \nand < is component-wise. \n\nTheorem 1 If the system (8) is strongly P-contractive, then it is asynchronizable \nfor any asynchronous schemes without self time delays (i. e., rf (t) = Ci(t) for all \ni=1,2, ... ,n). \n\nProof. It is not hard to see that synchronous dynamics of a strongly P-contractive \nsystem is globally convergent to the fixed point O. Now, consider the transformation \nz = A- 1 y and the system for z \n\nAi = G'( -z + S-1 F[SZ]) = G'( -z + G[Z]), \n\nwhere G[Z] = S-1 FS[Z]. This system has the same type of dynamics as (9). \nDefine a function E : R+ x Rn --+ R+ by E(t) = z T (t)Az(t)j2, whose derivative \nwith respect to t is \nE = z T G' (-z + G(Z\u00bb < IIG'II (-z T z + IzlT IG(Z)!) < IIG'II( -z T z + IzlT Izl) ::; O. \n\n\f498 \n\nWang, Li, and Blum \n\nHence E is an energy function and the asynchronous dynamics converges to the \nfixed point O. \n0 \nOur second result is for asynchronous dynamics of contractive systems with no \ncommunication delay. The system (8) is called contractive if there is a real constant \no ~ a < 1 such that \n\nIIF(x) - F(y)1I ~ allz - yll \n\nfor all x, y E Rn; here II . II denotes the usual Euclidean norm on Rn. \n\nTheorem 2 If the system (8) is contractive, then it is asynchronizable for asyn(cid:173)\nchronous schemes with no communication delay. \n\nProof. The synchronous dynamics of contractive systems is known to be globally \nconvergent to a unique fixed point (Kelly, 1990). For an asynchronous scheme with \nno communication delay, the system (8) is simplified to Ali = G'( -y + F(y\u00bb. We \nconsider again the function E = y T Ay/2, which is an energy function as shown \nbelow. \n\nE = Y T G' (-y + F(y\u00bb ~ IIG/II( -lIyll2 + lIyIlIlF(y)ID < O. \n\nTherefore, the asynchronous dynamics converges to the fixed point O. \nFor the additive-type neural networks (1), we have \n\no \n\nCorollary 1 Let the network (1) have neuron activation functions Ui of sigmoidal \ntype with 0 < uHz) ~ SUPzER ui(z) = 1. If it satisfies the condition \n\n(10) \n\nwhere M = diag(J-ll, ... , J-ln), \nschemes with no communication delay. \nProof. The condition (10) ensures the map F(x) = A-I Wu(M x) + A- 1 I to be \ncontractive. \n0 \n\nthen it is asynchronizable for any asynchronous \n\nNotice that the condition (10) is equivalent to many existing ones on globally asymp(cid:173)\ntotical stability based on various norms of matrix W, especially the contraction con(cid:173)\ndition given in (Kelly, 1990) and some very recent ones in (Matsuoka, 1992). The \ncondition (10) is also related very closely to the condition in (Barhen and Gulati, \n1989) for asynchronous dynamics of a discretized version of (1) and the condition \nin (Marcus and Westervelt, 1989) for the networks with delay. \n\nWe should emphasize that the results in Theorem 2 and Corollary 1 do not directly \nfollow from the result in (Kelly, 1990); this is because local times Ci(t) are allowed \nto be much more general functions than linear ones Ci t. \n\n3.2 Monotone Systems \n\nA binary relation ~ on Rn is called a partial order if it satisfies that, for all x, y, z E \nRn, (i) x ~ x; (ii) x ~ y and y ~ x imply x = y; and (iii) x -< y and y -< z \nimply x -< z. For a partial order ~ on Rn, define ~ on Rn by x ~ y iff x < y \nand Xi # Yi for all i = 1, .. \" n. A map F : Rn -I- Rn is monotone if x ~ y implies \n\n\fAsynchronous Dynamics of Continuous Time Neural Networks \n\n499 \n\nF(x) -< F(y). A CT dynamical system of the form (2) is monotone if Xl ~ X2 implies \nthe trajectories Xl(t), X2(t) with Xl(O) = Xl and X2(0) = X2 satisfy Xl(t) ::5 X2(t) \nfor all t ~ 0 (Hirsch, 1988). \n\nTheorem 3 If the map F in (8) is monotone, then the system (8) is asynchroniz(cid:173)\nable for uniformly-delayed asynchronous schemes, provided that all orbits x(t) have \ncompact orbit closure and there is a to > 0 with x(to) ~ x(O) or x(to) ~ x(O). \n\nProof. This is an application of a Henry's theorem (see Hirsch, 1988) that im(cid:173)\nplies that the asynchronized system (9) in the no communication delay situation \nis monotone and Hirsch's theorem (Hirsch, 1988) that guarantees the asymptotic \nconvergence of monotone systems to fixed points. \n0 \n\nCorollary 2 If the additive-type neural network (1) with sigmoidal activation func(cid:173)\ntions is cooperative (i.e., Wij > 0 for i # j (Hirsch, 1988 and 1989)), then it is \nasynchronizable for uniformly-delayed asynchronous schemes, provided that there is \na to > 0 with x(to) ~ x(O) or x(to) ~ x(O). \n\nProof. According to (Hirsch, 1988), cooperative systems are monotone. As the \nnetwork has only bounded dynamics, the result follows from the above theorem. 0 \n\n4 CONCLUSION \n\nBy incorporating the concepts of local times and communication times, we have \nprovided a mathematical formulation of asynchronous dynamics of continuous-time \ndynamical systems. Asynchronized systems in the most general form haven't been \nstudied in theories of dynamical systems and functional differential equations. For \ncontractive and monotone systems, we have shown that for some asynchronous \nschemes, the systems are asynchronizable, namely, their asynchronizations preserve \nconvergent dynamics of the original (synchronous) systems. When applying these \nresults to the additive-type neural networks, we have obtained some special condi(cid:173)\ntions for the networks to be asynchronizable. \n\nWe are currently investigating more general results for asynchronizable dynamical \nsystems, with a main interest in oscillatory dynamics. \n\nReferences \n\nG. M. Baudet (1978). Asynchronous iterative methods for multiprocessors. Journal \nof the Association for Computing Machinery, 25:226-244. \nJ. Barhen and S. Gulati (1989). \"Chaotic relaxation\" in concurrently asynchronous \nneurodynamics. In Proceedings of International Conference on Neural Networks, \nvolume I, pages 619-626, San Diego, California. \n\nBertsekas and Tsitsiklis (1989). Parallel and Distributed Computation: Numerical \nMethods. Englewood Cliffs, N J: Prentice Hall. \nE. K. Blum (1990). Mathematical aspects of outer-product asynchronous content(cid:173)\naddressable memories. Biological Cybernetics, 62:337-348, 1990. \n\n\f500 \n\nWang, Li, and Blum \n\nE. K. Blum and X. Wang (1992). Stability of fixed-points and periodic orbits, and \nbifurcations in analog neural networks. Neural Networks, 5:577-587. \nJ. Hale (1977). Theory of Functional Differential Equations. New York: Springer(cid:173)\nVerlag. \nM. W. Hirsch (1988). Stability and convergence in strongly monotone dynamical \nsystems. J. reine angew. Math., 383:1-53. \nM. W. Hirsch (1989). Convergent activation dynamics in continuous time networks. \nNeural Networks, 2:331-349. \nJ. Hopfield (1982). Neural networks and physical systems with emergent computa(cid:173)\ntional abilities. Proc. Nat. Acad. Sci. USA, 79:2554-2558. \nJ. Hopfield (1984). Neurons with graded response have collective computational \nproperties like those of two-state neurons. Proc. Nat. Acad. Sci. USA, 81:3088-\n3092. \nD. G. Kelly (1990). Stability in contractive nonlinear neural networks. IEEE Trans. \nBiomedi. Eng., 37:231-242. \nQ. Li (1993). Mathematical and Numerical Analysis of Biological Neural Networks. \nUnpublished Ph.D. Thesis, Mathematics Department, University of Southern Cali-\nfornia. \n\nC. M. Marcus and R. M. Westervelt (1989). Stability of analog neural networks \nwith delay. Physical Review A, 39(1):347-359. \n\nK. Matsuoka (1992) . Stability conditions for nonlinear continuous neural networks \nwith asymmetric connection weights. Neural Networks, 5:495-500 . \nJ. M. Ortega and W. C. Rheinboldt (1970). Iterative solution of nonlinear equations \nin several variables. New York: Academic Press. \nX. Wang, E. K. Blum, and Q. Li (1993). Consistency on Local Dynamics and \nBifurcation of Continuous-Time Dynamical Systems and Their Discretizations. To \nappear in the AMS proceedings of Symposia in Applied Mathematics, Mathematics \nof Computation 1943 - 1993, Vancouver, BC, August, 1993, edited by W. Gautschi. \n\nX. Wang and E. K. Blum (1992). Discrete-time versus continuous-time neural \nnetworks. Computer and System Sciences, 49:1-17. \n\nX. Wang and D. S. Parker (1992). Computing least fixed points by asynchronous \niterations and random iterations. Technical Report CSD-920025, Computer Science \nDepartment, UCLA. \nJ .-H. Wu (1993). Delay-Induced Discrete Waves of Large Amplitudes in Neural Net(cid:173)\nworks with Circulant Connection Matrices. Preprint, Department of Mathematics \nand Statistics, York University. \n\n\f", "award": [], "sourceid": 804, "authors": [{"given_name": "Xin", "family_name": "Wang", "institution": null}, {"given_name": "Qingnan", "family_name": "Li", "institution": null}, {"given_name": "Edward", "family_name": "Blum", "institution": null}]}