{"title": "Neural Network Weight Matrix Synthesis Using Optimal Control Techniques", "book": "Advances in Neural Information Processing Systems", "page_first": 348, "page_last": 354, "abstract": null, "full_text": "348 \n\nFarotimi, Demho and Kailath \n\nNeural  Network  Weight  Matrix  Synthesis  Using \n\nOptimal Control  Techniques \n\nO.  Farotimi \n\nA.  Dembo \n\nInformation Systems  Lab. \nElectrical  Engineering  Dept. \n\nStanford  University, \nStanford,  CA  94305 \n\nABSTRACT \n\nT.  Kailath \n\nGiven a set  of input-output  training samples,  we  describe  a  proce(cid:173)\ndure  for  determining  the  time sequence  of weights  for  a  dynamic \nneural  network  to  model  an  arbitrary  input-output  process.  We \nformulate  the  input-output  mapping  problem  as  an  optimal con(cid:173)\ntrol  problem,  defining  a  performance  index  to  be  minimized  as  a \nfunction  of time-varying  weights.  We  solve  the  resulting  nonlin(cid:173)\near two-point-boundary-value problem, and this yields the  training \nrule.  For the performance index chosen,  this rule  turns out  to be  a \ncontinuous time generalization of the outer product rule earlier sug(cid:173)\ngested heuristically by Hopfield for designing associative memories. \nLearning curves for  the new  technique are  presented. \n\nINTRODUCTION \n\n1 \nSuppose  that  we  desire  to  model  as  best  as  possible  some  unknown  map 4>  : u -\nV,  where U, V  ~ nn.  One  way  we  might go  about doing this  is  to  collect  as  many \ninput-output samples {(9in, 90ud : 4>(9in )  = 9 0ud as possible and \"find\" some func-\ntion f  : U - V such that a suitable distance metric d(f( z(t)), 4>(z(t)))I ZE{9 ... :4>c9 ... )=9 o .. d \nis  minimized. \n\nIn the foregoing, we assume a system of ordinary differential equations motivated by \ndynamic neural  network structures[l]  [2].  In  particular we  set  up  an  n-dimensional \n\n\fNeural Network Weight Matrix Synthesis \n\n349 \n\nneural network; call it N.  Our goal is  to synthesize a  possibly time varying weight \nmatrix for N  such that for  initial conditions zeta), the input-output transformation, \nor  flow  1  : zeta)  -- I(z(t,\u00bb  associated  with  N  approximates closely  the  desired \nmap 4>. \nFor the purposes of synthesizing the weight program for N, we consider another sys(cid:173)\ntem, say S,  a formal  nL-dimensional system of differential equations comprising L \nn-dimensional subsystems.  With the exception that all L n-dimensional subsystems \nare  constrained  to  have  the  same  weight  matrix,  they  are  otherwise  identical  and \ndecoupled.  We shall use  this system to determine the optimal weight program given \nL  input-output samples.  The resulting  time program of weights is  then  applied  to \nthe  original n-dimensional system N  during  normal operation.  We  emphasize  the \ndifference  between  this  scheme  and  a  simple  L-fold  replication  of N: \nthe  latter \nwill  yield  a  practically unwieldy nL x  nL weight matrix sequence,  and  in fact  will \ngenerally  not discover the  underlying map from U  to V,  discovering instead  differ(cid:173)\nent  maps  for  each  input-output  sample  pair.  By  constraining  the  weight  matrix \nsequence  to  be  an identical  n  x  n  matrix for  each subsystem  during this synthesis \nphase, our scheme in essence  forces  the weight sequence  to capture some underlying \nrelationship between  all the  input-output pairs.  This is  arguably  the best estimate \nof the  map given the  information we  have. \n\nUsing formal optimal control techniques[3],  we  set  up a  performance index  to max(cid:173)\nimize  the  correlation  between  the system  S  output  and  the  desired  output.  This \noptimization  technique  leads  in  general  to  a  nonlinear  two-point-boundary-value \nproblem,  and  is  not  usually  solvable  analytically.  For this  particular  performance \nindex  we  are  able  to  derive  an  analytical  solution  to  the  optimization  problem. \nThe optimal interconnection  matrix at each  time is  the  sum  (over  the  index  of all \nsamples) of the  outer  products  between  each  desired  output  n-vector  and  the  cor(cid:173)\nresponding  subsystem  output.  At the  end  of this  synthesis  procedure,  the  weight \nmatrix sequence  represents  an optimal time-varying program for  the  weights of the \nn-dimensional neural  network N  that will  approximate 4>  : U -- V. \nWe remark that in the ideal case, the weight matrix at the final time (i.e one element \nof the time sequence)  corresponds to the symmetric matrix suggested empirically by \nHopfield  for  associative memory applications[4].  It becomes clear that the  Hopfield \nmatrix is  suboptimal for  associative memory,  being just one  point on  the  optimal \nweight trajectory;  it is  optimal only in  the special case  where  the initial conditions \ncoincide exactly  with  the desired  output. \n\nIn Section  2 we  outline the  mathematical formulation and solution of the synthesis \ntechnique, and in Section 3 we  present the learning curves.  The learning curves also \nby default yield the system performance  over the  training samples, and we  compare \nthis performance to that of the outer product rule.  In Section 4 we give concluding \nremarks and give the  directions of our future  work. \n\nAlthough the results here are derived for a specific case of the neuron state equation, \nand  a  specific  choice  of performance  index,  in  further  work  we  have  extended  the \nresults  to very general  state equations and performance indices. \n\n\f350 \n\nFarotimi, Dembo and Kailath \n\n2  SYNTHESIS OF WEIGHT MATRIX TIME SEQUENCE \nSuppose  we  have  a  training  set  consisting  of  L  pairs  of  n-dimensional  vectors \n(o(r)i, e(r\\), r = 1,2, ... , L, i = 1,2, ... , n.  For example, in  an  autoassociative sys(cid:173)\ntem in  which  we  desire  to store  e(r)i,r = 1,2, ... ,L,i = 1,2, ... ,n,  we  can  choose \nthe  o(r)i, r = 1,2, ... , L, i = 1,2, ... , n  to be  sample points in  the  neighborhood of \n(}(r)i  in  n-dimensional space.  The idea here  is  that by  training the network to map \nsamples  in  the  neighborhood  of an  exemplar  to  the  exemplar,  it  will  have  devel(cid:173)\noped  a  map  that  can  smoothly interpolate  (or  generalize)  to other  points around \nthe  exemplar  that  may not  be  in  the  training set.  In  this  paper  we  deal  with  the \nissue of finding the weight matrix that transforms the neural network dynamics into \nsuch  a  map.  We  demonstrate  through  simulation results  that such  a  map  can  be \nachieved.  For autoassociation, and using error  vectors  drawn from  the  training set, \nwe  show  that the  method here  performs  better  (in  an error-correcting  sense)  than \nthe  outer  product  rule.  We  are  still  investigating the  performance  of the  network \nin generalizing to samples outside  the  training set. \nWe  construct  an  n-dimensional  neural network system N  to model the  underlying \ninput-output map according  to \n\nN:  z(t) = -z(t) + W(t)g(z(t), \n\n(1) \n\nWe  interpret z  as the neuron  activation, g(z(t)) is the  neuron output, and  W(t) is \nthe  neural  network  weight matrix. \nTo determine the appropriate W(t),  we  define  an nL-dimensional formal system of \ndifferential equations, S \n\nS:  z\u00b7(t) = -z.(t) + W.(t)g(z.),  g(z.(to\u00bb  = iJ \n\n(2) \n\nformed by concatenating the equations for N  L times.  W. (t)  is block-diagonal with \nidentical blocks W(t).  8  is  the  concatenated  vector of sample desired  outputs, iJ  is \nthe concatenated  vector of sample inputs. \nThe  performance index for  S  is \n\nminJ = min {-z.T(tI)8 + 41t' (-2Z. T(t)8 + /3Q  + /3-1 t WJ(t)Wi(t\u00bb)  dt} \n\ni=1 \n\n\" \n\nto \n\n(3) \nThe performance index is chosen to minimize the  negative of the  correlation between \nthe (concatenated) neuron activation and the (concatenated) desired output vectors, \nor  equivalently  maximize  the  correlation  between  the  activation  and  the  desired \noutput at the final  time tl, (the term  -Z.T(t1 )8).  Along the  way from  initial time \nto  to final  time t I, the term -z. T (t)8  under  the integral penalizes  decorrelation of \nthe  neuron  activation and the desired  output.  Wj(t), j  = 1,2, ... , n  are  the  rows of \nW(t), and /3  is a positive constant.  The term /3-1 Ei=l wJ(t)Wj(t) effects  a bound \n\n\fNeural Network Weight Matrix Synthesis \n\n351 \n\non the magnitude of the weights.  The term \n\nQ(g(Z(t\u00bb) = L L L L o/r)o/v)g(zu(v\u00bbg(zu(r\u00bb, \n\nn \n\nL \n\nn \n\nL \n\nj=lr=lu=lv=1 \n\nand  its  meaning  will  be  clear  when  we  examine  the  optimal  path  later.  g(.)  is \nassumed  Cl  differentiable. \nProceeding formally[3],  we  define  the  Hamiltonian: \n\nH  =  ~ ( _2zT(I)9 + Q + t WJ<I)BWj(I\u00bb)  + >7(1)( -z(l) + W.(I)g(z(l))) \n\n~ ( _2\",T(I)9 + Q + t WJ<I)BWj(I\u00bb)  - >7(1)\",(1) + t. t, A(r)jwJ<l)g(r)(z(l\u00bb \n\n(4) \n\nwhere \n\n.\\T (t)  =  [  ,\\(1)1 (t) \n\n,\\(1)2(t) \n\n...  ,\\(L)n (t)  ] \n\nis the vector of Lagrange multipliers, and we  have used  the fact  that W.(t) is block(cid:173)\ndiagonal  with  identical  blocks  W(t)  in  writing  the  summation of the  last  term in \nthe second  line  of equation  (4).  The  Euler-Lagrange  equations are  then  given by \n\n( OH)T \n\nOZ \n\n1 (OQ)T \n\n= 2  oz \n\n-9 \n\n- (9 + .\\(t\u00bb +  oz  W.  (t)'\\(t) \n\n(Og)T \n\nT \n\no \n\noH  = w'f B  + \"\"\" ,\\(r) .g(r)T (z(t\u00bb \now. \n\nJ  ~  J \n\nL \n\nr=1 \n\nJ \n\nFrom equation  (7)  we  have \n\nChoosing \n\nWij(t) = -f3 L ,\\(r) jg(z;<r)(t\u00bb \n\nL \n\nr=l \n\n.\\(t) = -9 \n\n(5) \n\n(6) \n\n(7) \n\n(8) \n\n(9) \n\nsatisfies  the  final  condition  (6),  and  with  some  algebra  we  find  that  this  choice  is \nalso consistent with equations (5) and (7).  The optimal weight program is therefore \n\nWij (t)  = f3 L o(r\\g(z;<r)(t\u00bb \n\nL \n\nr=l \n\n(10) \n\nThis describes  the  weight  paradigm to be  applied to the n-dimensional neural net(cid:173)\nwork .. system  /II  in  order  to  model  the  underlying  map  described  by  the  sample \n\n\f352 \n\nFarotimi, Dembo and KaiIath \n\npoints.  A similar  result  can  be  derived  for  the  discrete-time  network  z(k + 1)  = \nW(k)g(z(k\u00bb: \n\nwi;(k) = f3 L o(r)ig(x/r)(k\u00bb \n\nL \n\nr=l \n\n2.1  REMARKS \n\n\u2022  Meaning  ofQ. \n\nOn  the optimal path, using equation (10),  it is  straightforward to show that \n\nn \n\nf3Q = f3- 1 L wT(t)w;(t) \n\n;=1 \n\nThus Q  acts like another integral constraint  term on  the weights. \n\n\u2022  The  Optimal Return  Function. \n\nThe optimal return  function[3],  which  is  the  value  of the  performance  index \non the optimal path can  be  shown to be \n\nThus the optimal weight  matrix W(t) seeks  at every instant to minimize the \nnegative correlation (or maximize the correlation) on  the optimal path in  the \nformal system  S  (and hence  in  the neural  network N). \n\n\u2022  Comparison  with  outer product rule. \n\nIt is  worthwhile to compare equation (10)  with  the outer product  rule: \n\nL \n\nWi;  = f3 L o(r) jo(r); \n\nr=l \n\n(11) \n\nWe  see  that the  outer product  rule  is just one  point on the weight trajectory \ndefined  by equation (10) - the point at final time tf when  g(X/r)(tf\u00bb) =  o(r)j' \n\n3  LEARNING  CURVES \nIn  our  simulation  we  considered  14  8-dimensional  vectors  as  the  desired  outputs. \nThe weight synthesis or learning phase is as follows:  we initialized the 112-dimensional \nformal synthesis system S  with a corrupted version of the vector set,  and used equa(cid:173)\ntion  (10)  to find  the  optimal  8  x  8  weight  matrix sequence  for  an  8-dimensional \nneural network N  to correctly  classify  any of the corrupted  14  vectors.  The  weight \nsequence  is  recorded.  This  procedure  is  required  only  once for  any given  training \nset.  After  this learning is completed, the normal operation of the  neural network N \nconsists  in  running  it  using  the  weights obtained  from  the  synthesis  phase  above. \nThe resulting network describes  a continuous input-output map.  At points belong(cid:173)\ning  to  the  training set  this  map coincides  with  the  underlying  map  we  are  trying \nto model.  For points outside the training set,  it performs a  nonlinear interpolation \n\n\fNeural Network Weight Matrix Synthesis \n\n353 \n\n(generalization) the nature of which is  determined by the training set as  well as the \nneuron  state  equation.  Figure  1 shows  the  learning  procedure  through  time.  The \ncurves  labeled  \"Optimally  Trained Network\" shows  the  behavior of two correlation \nmeasures  as  the  training  proceeds.  One  correlation  measure  used  was  the  cosine \nof the  angle  between  the  desired  vector  (8)  and  the  neuron  activation  (z)  vector. \nThe other correlation measure  was the cosine of the  angle  between the desired  vec(cid:173)\ntor  (8)  and  the  neuron  output (g(z(t\u00bb)  vector.  Given our system  initialization in \nequation  (2),  the  correlation  g(z(t\u00bb)T8  more  accurately  represents  our  objective) \nalthough  the  performance  index  (3)  reflects  the  correlation  zT 8.  The  reason  for \nour  performance  index  choice  is  that  the  weight  trajectory  yielded  by  g(z(t\u00bb)T8 \nleads  the system to an all-zero,  trivial equilibri um for  a sigmoid gC)  (we  used  such \na  g(.)  with  saturation  values  at  +1  and  -1  in  our  simulations).  This  is  not  the \ncase  for  the  weight  trajectory yielded  by  z T8.  Since  g(z(t\u00bb \nis  monotonic with  z, \nzT 8  represented  an  admissible  alternative  choice  for  the  performance  index.  The \nresults  bear this out.  Another  possible  choice  is  (g(z(t\u00bb  + z)T8.  This gives simi(cid:173)\nlar  results  upon simulation.  The  correlation  measures  are  plotted  on  the  ordinate. \nThe  abscissa  is  the  number of computer  iterations.  A  discrete-time  network  with \nreal-valued parameters was used.  The total number of errors  in the  14  8-bit binary \n{I, -I} vectors  used  to  initialize  the  system  was  21.  This  results  in  an  average of \n1.5  errors  per  8-bit  vector.  We  note  that  the  learning  was  completed  in  two time \nsteps.  Therefore,  in  this  case  at  least)  we  see  that  the  storage  requirement  is  not \nintensive - only  two weight matrices need  to be stored  during  the synthesis  phase. \n\nWe  note that  the  learning phase by default  also  represents  the autoassociative sys(cid:173)\ntem error-correcting  performance  over  input  samples  drawn  from  the  training  set. \nTherefore  over  the  training set  we  can  compare  this  performance  with  that  of the \nouter  product  rule  (11).  By  considering  corrupted  input  vectors  from  the  train(cid:173)\ning set,  we  compare the  error-correcting  capabilities of the  two methods,  not  their \ncapacities  to  store  uncorrupted  vectors.  In  fact  we  see  that  the  two  weight  rules \nbecome identical when  we  initialize with  the  true  vectors  (this equivalence is  not  a \npeculiarity of the new  technique, but merely a consequence  of the  particular perfor(cid:173)\nmance index  chosen).  In  other words,  this comparison is  a  test  of the extent of the \nbasins of attraction around  the  desired  memories for  the  two techniques.  Looking \nat the curves labeled  \"Conventional  Outer Product\",  we  see  that the new  technique \nperforms better  than  the outer product  rule. \n\n4  CONCLUSIONS  AND FURTHER WORK \nWe  have  described  a  technique  for  training neural  networks based  on  formal  tools \nfrom optimal control theory.  For a specific example consisting of learning the input(cid:173)\noutput map in  a training set we derived the relevant weight equations and illustrated \nthe  learning  phase  of the  method.  This  example  gives  a  weight  rule  that  turns \nout  to  be  a  continuous-time  generalization  of the  outer-product  rule.  Using  cor(cid:173)\nrupted  vectors  from  the  training  set,  we  show  that  the  new  rule  performs  better \nin  error-correction  than  the  outer-product  rule.  Simulations on  the  generalization \ncapa5ilities of the  method are ongoing and are  not  included  in  the present  work. \n\n\f354 \n\nFarotimi, Dembo and Kailath \n\nDISCItEn CUI \n\n\u00b7-~T\"\"\"Hc __ \n... c.........aa- ........ \n\n0.1 \n\n0.6 \n\nlo  , \nO.9~ I \n\n0.1 \n\nI \nt \n\n0.9 \n\n0.1 \n\n0.1 \n\nI  Q.6 t 0-' \n\n0.4 \n\nD.l \n\n0.1  \u2022 \n\n4.1 \n\n0 \n\n10 \n\n0-' \n\n0.4 \n\n0.3 \n\n0.1 \n\n0.1 \n\n0 \n\n10 \n\n\u00bb \n\n30 \n\n40 \n\n50 \n\n60 \n\n10 \n\n10 \n\n90 \n\nlaD \n\n~ \n\nFigure 1:  Learning  Curves \n\nAlthough  we  considered  a  training  set  consisting  of input-output  vector  pairs  as \nthe  starting  point for  the  procedure,  a  closer  examination  shows  that  this  is  not \nrequired.  More  generally,  what is  required  is  a  performance index that  reflects  the \nobjective  of the  training.  Also  in  our  ongoing  work  we  have extended  the  results \nto  more  general  forms of the  state equation and  the  performance  index.  Using  an \nappropriate  performance  index  we  are  investigating  a  network  for  the  Travelling \nSalesman Problem and related  applications like Tracking and Data Association. \n\nReferences \n\n[1]  Michael  A.  Cohen  &  Stephen  Grossberg,  \"Absolute  Stability  of  Global  Pat(cid:173)\ntern Formation and Parallel Memory Storage by Competitive Neural Networks,\" \nIEEE Transactions on  Systems,  Man  and  Cybernetics SMC-13 (1983),815-826. \n\n[2]  J. J. Hopfield &  D.  W. Tank, \"Neural Computation of Decisions in Optimization \n\nProblems,\"  Biological Cybernetics 52 (1985),  141-152. \n\n[3]  Arthur  E.  Bryson  & Yu-Chi Ho,  Applied Optimal Control, Hemisphere,  1975. \n\n[4]  J. J.  Hopfield,  \"Neural Networks and Physical Systems with Emergent Collective \nComputational Abilities,\"  Proceedings  of the  National Academy of Sciences 79 \n(1982),  2554-2558. \n\n\f", "award": [], "sourceid": 191, "authors": [{"given_name": "O.", "family_name": "Farotimi", "institution": null}, {"given_name": "Amir", "family_name": "Dembo", "institution": null}, {"given_name": "Thomas", "family_name": "Kailath", "institution": null}]}