{"title": "Speech Production Using A Neural Network with a Cooperative Learning Mechanism", "book": "Advances in Neural Information Processing Systems", "page_first": 232, "page_last": 239, "abstract": null, "full_text": "232 \n\nSPEECH  PRODUCTION  USING  A  NEURAL \n\nNETWORK  WITH  A  COOPERATIVE \n\nLEARNING  MECHANISM \n\nInternational Institute for Advanced Study of Social Information Science, \n\nMitsuo Komura \n\nAkio Tanaka \n\n140 Miyamoto, Numazu-shi Shizuoka, 410-03 Japan \n\nFujitsu Limited \n\nABSTRACT \n\nWe  propose  a  new  neural  network  model  and  its  learning \nalgorithm. The proposed neural network consists of four layers \n- input, hidden, output and final output layers. The hidden and \noutput layers are multiple.  Using the proposed  SICL(Spread \nPattern Information and Cooperative  Learning)  algorithm,  it \nis  possible  to  learn  analog  data  accurately  and  to  obtain \nsmooth outputs. Using this neural network, we have developed \na  speech production system consisting of a  phonemic  symbol \nproduction  subsystem  and  a  speech  parameter  production \nsubsystem.  We  have  succeeded  in  producing  natural  speech \nwaves with high accuracy. \n\nINTRODUCTION \n\nOur purpose is  to produce natural speech waves. In general, speech synthesis \nby  rule  is  used  for  producing  speech  waves.  However,  there  are  some \ndifficulties in speech synthesis by rule. First, the rules are very complicated. \nSecond,  extracting  a  generalized  rule  is  difficult.  Therefore,  it  is  hard  to \nsynthesize a natural speech wave by using rules. We use a neural network for \nproducing speech waves. Using a neural network, it is possible to learn speech \nparameters without rules.  (Instead of describing rules explicitly,  selecting a \ntraining data set becomes an important subject.) In this paper, we  propose  a \nnew  neural  network  model  and  its  learning  algorithm.  Using  the  proposed \nneural network, it is possible to learn and produce analog data accurately. We \napply  the  network  to  a  speech  production  system  and  examine  the  system \nperformance. \n\nPROPOSED  NEURAL  NETWORK  AND  ITS  LEARNING \n\nALGORITHM \n\nWe use an analog neuron-like element in a neural network. The element has a \nlogistic activation function presented by equation (3). As a learning algorithm, \n\n\fSpeech Production Using A Neural Network \n\n233 \n\nthe BP(Back Propagation) method is widely  used.  By  using this method  it is \npossible to learn the weighting coefficients of the units whose target values are \nnot given directly. However, there are disadvantages. First, there are singular \npoints  at 0  and  1  (outputs  of the  neuron-like  element).  Second,  finding  the \noptimum  values of learning constants  is  not  easy.  We  have  proposed  a  new \nneural network model and its learning algorithm to  solve  this  problem.  The \nproposed SICL(Spread Pattern Information and Cooperative Learning) method \nhas the  following features. \n(a)The singular points of the BP method are removed. (Outputs are not simply \no or 1.)  This improves the convergence rate. \n(b)A spread pattern information(SI) learning algorithm is proposed.  In the SI \nlearning algorithm, the weighting coefficients from  the  hidden  layers to  the \noutput layers are fixed  to  random values.  Pattern information is spread over \nthe memory space of the weighting coefficients. As a  result, the network can \nlearn analog data accurately. \n(c)A cooperative learning(CL) algorithm is proposed. This algorithm makes  it \npossible to obtain smooth and stable output. The CL system is shown in Fig.1 \nwhere D(L)  is a delay line which delays L time units. \nIn the following  sections,  we  define  a  three-layer network,  introduce  the BP \nmethod, and propose the SICL method . \n\n. ' \n.. ' \n\n\", \n..... , ..\u2022.\u2022. \n\n.... \". '. -'. \n\nk=K \n\n~,;\" \n\nI ' ; ;  \n\n,:yi \n\n,/ \n\nInput  layar \n\nHlddan  layar. \n\nOutput layar. \n\nFinal  output  layar \n\nFigure 1.  Cooperative Learning System \n\n(Speech Parameter I Phonemic Symbol Production Subsystem) \n\n\f234 \n\nKomura and Tanaka \n\nTHREE\u00b7LA YER  NETWORK \nWe define a three-layer network that has an input layer, a hidden  layer, and \nan output layer. The propagation of a signal from the input layer to the hidden \nlayer is represented as follows. \n\nUj = 'EiwIHijXi, \n\nYj = f(Uj -OJ)  , \n\n(1) \n\nwhere  i= 1,2, ... ,1; j= 1,2, ... ,J and  Xi  is  an input, Yj  the output of the  hidden \nlayer, and OJ a  threshold value. The propagation of a  signal from  the  hidden \nlayer to the output layer is represented as follows. \n\n(2) \nwhere Zk is the output of the output layer and k= 1,2, ... ,K.  The activation \nfunction f(u) is defined by \n\nf(u)  =  (1+exp(-u+O\u00bb-l. \n\n(3) \n\nSettingy = f(u), the derivation off(u) is then given by f' (u) = y(1-y). \nBACK  PROPAGATION (BP)  METHOD \n\nThe three-layer BP algorithm is shown as follows. The back propagation error, \n80 k(n) , for an output unit is given by \n\n(4) \n\nwhere n is an iteration number. \nThe back propagation error, 8Hin), for a hidden unit is given by \n\n8Hin) = ('Ek80k(n) WHOjk) f' (u)<n\u00bb  . \n\n(5) \nThe change to the weight from the i-th to thej-th unit, flwIH;,in) is given by \n(6) \n\nflwIHi/n) =a8H/n)xi(n) + jJflwIH;,in-l) . \n\nThe change to the weight from thej-th to the k-th unit, flwH0jk(n) is given by \n\nflwHOjk(n) = a80k(n) Yin) + jJflwHOjk(n-l) , \n\n(7) \n\nwhere a and jJ are learning constants, and have positive values. \nSPREAD  PATTERN  INFORMATION  AND \nCOOPERATIVE  LEARNING  (SICL)  METHOD \nThe  proposed  learning algorithm  - SICL  method  is  shown  as  follows.  The \npropagation of a signal from the input layer to the hidden layer is given by \n(8) \n\nu/l,n) = 'EiwlHij(l,n-l) xi(n) , \n\ny/l,n) =f(u/l,n) -oin) , \n\nwhere l is a stage number (l = -LI2, ... ,LI2). The propagation of a signal from the \nhidden layer to the output layer is given by \n\nvk(l,n) = 'EjwH0jk(n Yin) , \n\nzk(l,n) = f(vk(l,n) -Ok(n) . \n\n(9) \n\nThe back propagation error, 80k(n), for an output unit is given by \n\n\fSpeech Production Using A Neural Network \n\n235 \n\n(10) \n\nwhere y is a constant for removing singular points. \nThe back propagation error, OHj(l,n), for a hidden unit is given by \n\n8HjCI,n) = (EkoOk(l,n) wH0jk(l)  (Yil,n)(1-yjCl,n\u00bb + y) . \n\n(11) \nThe change to the weight from the i-th to thej-th unit, ll.wIHi/l,n) is given by \n(12) \n\nAwIHij(l,n) =aOHjCl,n)xln) + jJAwIHij(l,n-l) . \n\nThe weight from thej-th to the k-th unit,wHOjk(l) is given by \n\nwH\u00b0jk(l) = Cjk(l), \n\n(13) \n\nwhere Cjk(l) is a fixed value and a random number with normal distribution. A \nfinal output is a weighting sum of outputs zk(l,n) ,and is given by \n\nZFk( n) = E,Wl zk(l,n). \n\n(14) \n\nA  SPEECH  PRODUCTION  SYSTEM  USING  THE \n\nPROPOSED  NEURAL  NETWORK \n\nThe block diagram of a speech production system is shown in Fig.2. The system \nconsists of a phonemic symbol production subsystem and a  speech parameter \nproduction subsystem using the proposed neural network. \n\nPhonemic  Symbols \n\nSpeech  ParameterslAnalog  detal \n\nPhonemic  SYlnbol =  Speech  Parameter \n\n!  \u00ae \n\n1 \"\"\"\"\u00ae~-=0--1 \n\nInput  String \n\nI'Iaduclim  SiRy.1n  _ \n\nProduction  Subsystern \n\nSynthesized  Speech \n\nInput  Speech  S~lg...Lnll~I-------li Speech  Psremeter \nExtlClIon  S.mystem \n\n@  Automatic \n\nFigure 2.  The Speech Production System Using the Proposed Neural Network \n\nIn  the  automatic  speech  parameter  extraction  subsystem@,  speech \nparameters are extracted automatically from  an input speech signal.  Speech \nparameters are  composed  of source  parameters( voiced/unvoiced  ratio,  pitch \nand source power) and a  vocal tract area function(PARCOR coefficients). The \nextracted speech parameters are used as training data of the speech parameter \nproduction subsystem. \n\nIn  the  speech  parameter  production  subsystem\u00ae,  input  is  the  string  of \nphonemic symbols. (In the training stage, phonemic symbols are decided by the \nteacher using input speech data. After training, phonemic symbols are given \nby  phonemic symbol  production  subsystem.)  Targets  are  speech  parameters \nextracted in @ . \nThe phonemic symbol production subsystem CD  consists of a preprocessor and a \nlearning  system  using  the  proposed  neural  network.  In  the  preprocessor,  a \n\n\f236 \n\nKomura and Tanaka \n\nstring ofinput characters is converted to a string of phonemic symbols with the \nmean  length  of utterance.  The  input  is  the  string  of phonemic  symbols \nconverted by the preprocessor.  In the  training stage,  the  targets  are  actual \nphonemic symbols, namely, the inputs of subsystem @  which are decided  by \nthe teacher. \n\nThe output speech parameters are converted into the synthesized speech wave \nusing the PARCOR synthesizer circuit~. \nEXPERIMENTS \nWe performed two separate experiments. \nExperiment I \nAutomatic Speech Parameter Extraction Subsystem@ \n\nSampling frequency: 8 KHz \nFrame period: 10 ms \n\nFrame length \n\n: 20 ms \n\nSpeech Parameter Production Subsystem @ \nInput layer: 1,044(36 X 29) units \nOutput layers: 13 units X 9 stages \nFor source parameters, 3(VIUV, pitch, source power) units are assigned. For a \nvocal  tract area function,  10(PARCOR coefficients)  units  are  assigned.  This \nsystem has an input layer, 9 hidden layers, 9 output layers, and a final output \nlayer.  For each output layer, different target data are  assigned.  In the final \noutput layer, these outputs are summed with weighting coefficients. \nPhonemic Symbol Production SubsystemCD \n\nHidden layers: 80 units X 9 stages \nFinal output layer: 13 units \n\nInput layer: 1,044(36 X 29) units  Hidden layers: 80 units X 9 stages \nOutput layers: 36 units X 9 stages  Final output layer: 36 units \n\nInput Speech Signal No.1 \nThe input speech signal No.1  is  r ASAHA YAKU BANGARONI  DENPOGA \nTODOITA  J, that is, Japanese sentence which means that \"A telegram was \nsent to the bungalow early in the morning.\". This signal is a 408 frame (4.08s) \nsequence. \nExperimental Results \nIn  Fig.3,  the  targets  and  the  outputs  of the  speech  parameter  production \nsubsystem are shown.  The dotted  line  is  the sequence  of target values.  The \nsolid line is the sequence of output values after the training stage. In this case, \nthe learning constant a is equal to 0.7 and jJ  to 0.2.  After training, the actual \noutputs produced by SICL system agree well with the targets. \nFig.4  (a)  shows  the  learning behaviors  of the  speech  parameter  production \nsubsystem. In this case, the input speech signal is No.1. The learning constant \na is equal to  0.7  and P to  0.2.  The  learning curve  based on  the  SICL and SI \n\n\fSpeech Production Using A Neural Network \n\n237 \n\nmethods converged. However, the learning curve based on the BP method did \nnot converge. \nFig.4 (b) also shows the learning behaviors. The learning constant a is equal to \n0.07  and jJ  to  0.02.  In this case,  all  of the  learning curves converged.  These \nresults show that if we use the SICL or SI method instead of the BP method, we \ncan obtain better results. \n\n1.0 \n\n0.5 \n\n0.0 \n\n~. If '. . \\ \n\nI \n\nI \nI \n\no \n\nSource Power \n\n1.0 \na =  0.7,/1 =  0.2 \n\n2.0 \n\n3.0 \n\n4.0 \n\ns \n\nInput speech signal : No.1 \n\nFigure 3. Targets and Outputs of Speech Parameter Production Subsystem \n\nMSE(log) \n\n1 \n\n10-1 \n\n10-2 \n\n10-3 \n\n10-4 \n\n10-5 \n\n1 \n\n10 \n\n102 \n\n103 \n\nMSE(log) \n\n1 \n\n10-1 \n\n10-2 \n\n10-3 \n\n10-4 \n\n10.5 \n\n(a)  a =  0.7 ,jJ =  0.2 \n\n(NNI)  1 \n\n10 \n(b)  a = 0.07 ,p = 0.02 \nNNI : Normalized number of iteration \n\n102 \n\n103 \n\nMSE: Mean square error \n\nG-\u00a3J : SICL method (9stage)  G-B: SI method (l = 0)  *  *: BP method \n\nFigure 4.  Learning Behavior of Speech Parameter Production Subsystem \n\nTable 1 shows mean square errors of the system using SICL method for a, p. It \nshould  be  noted  that  the  domain  of convergence  is  very  wide.  From  these \n\n\f238 \n\nKomura and Tanaka \n\nexperiments, it is seen that the SICL method almost always allows stable and \nsmooth output to be obtained. \n\nWe  also  examined  the  system  performance  for  another  input  speech  signal \nwhich is a  1,60S frame (16.0S s) sequence. The learning curves converged and \nwere similar to those for the input speech signal No.1 \n\n0.07 \n\n0.20 \n\n0.7 \n\n0.01 \n0.02 \n0.1 \n0.2 \n0.35 \n0.7 \n\n7.9S X 10-5 \n7.97X10-5 \n7.8SX 10-5 \n7.85 X 10-5 \n7.82x 10-5 \n7.80X 10-5 \n\n7.S2X 10-5 \n7.S2X10-5 \n7.S1 X 10-5 \n7.S0X 10-5 \n7.S1 X 10-5 \n8_26X 10-5 \n\n1.03 X 10-4 \n1.09X 10-4 \n1.26 X 10-4 \n1.72 X 10-4 \n2.S6X 10-4 \n1.72x 10-3 \n\nInput speech signal: No.1. \n\nTABLE 1. Mean Square Errors for a, p \n\nExperiment II \nSpeech Parameter Production Subsystem @ \nInput layer: 665 (35X 19)units \nOutput layers: 14 (12 + 2)units X 9 stages \nFor the first target data(VIUV, power ratio, 10 PARCOR coefficients), 12 units \nare . assigned.  For  the  second  target data(pitch,  source  power),  2  units  are \nassigned. \n\nHidden layers: 400 units X 9 stages \n\n. \n\nInput Speech Signal No.2 \nThis signal is a 1:),700 frame (137s) sequence shown as follows. \nIARAARN,IARAIRN, .... ,1 ARAKARN,  I ARAKIRN, ..... ,1ARASARN, ..... . \nI ARARAN,I!RARA!/, ..... ,1KARARAKN ,1KIRARAKI/, ... ,1SARARASN, .... . \nIn experiment II , a learning test for producing an arbitrary combination of the \ninput phonemes (ai) was done. The learning behavior of subsystem\u00ae is shown \nin Fig.5.  In this case,  the fIrst 12  target data items are trained. The other 2 \ntarget data it~ms should be trained using another network. The upper line is a \nlearning curve for  the  SI  method,  and  the  lower  line  for  the  SICL  method. \nSome of the learning curves for  the SI  method didn't converge.  However,  the \nlearning curve for the SICL method did. We can thus say that the SICL method \nis very powerful for actual use. (In this case, 400 hidden units is not sufficient \nfor the size of input data. If more hidden units are used, the MSE will be small.) \n\nCONCLUSION \n\n\fSpeech Production Using A Neural Network \n\n239 \n\nMSE (log) ~---------..., \n\n10-1 \n\n10-2 \n\n10-3 \n\n10-4 \n\nG-B : SICL method (9stage) \n\nG-E) : SI method (l= 0) \n\na = 0.2, 11 = 0.035 \n\nInput speech signal: No.2 \n\n1 \n\n10 \n\n102  NNI \n\nFigure 5.  Learning Behavior of Speech Parameter Production Subsystem \n\nFrom the experiments shown in previous section, it should be noted that using \nthe  SICL  (Spread  Pattern  Information  and  Cooperative  Learning)  method \nmakes it possible to learn speech parameters or phonemic symbols stably and \nto produce more natural speech waves than those synthesized by rules.  If the \ninput is  the combination of a  word  and  postpositional  particle,  it is  easy  to \nproduce sound for  unknown input data using the proposed speech production \nsystem.  However,  the number of the  hidden units and  training data will  be \ngreat. Therefore we have to make the system learn phonemes among phoneme \nsequence.  Using this training data makes it possible to produce an arbitrary \nsequence of phonemes. In experiment II , phonemic information (V IUV I  power \nratio, PARCOR coefficients)  is trained. Then the range of input window  was \nset to be 190 ms.  For prosodic information (pitch, source power), we  must use \nanother  network.  Because,  if we  want  to  make  the  system  learn  prosodic \ninformation,  we  must set the range of input data wider  than  that of words. \nU sing these strategies, it is possible to produce  arbitrary natural speech. \n\nAcknowledgments \nThe  authors  thank  Dr.  Tosio  Kitagawa,  the  president of liAS-SIS  for  his \nencouragement. \n\nReferences \nD.  E.  Rumelhart,  J.  L.  McClelland  and  PDP  Research  Group  Parallel \nDistributed Processing. VOL.l The MIT Press (1987). \nT.J.Sejnowski and C.R.Rosenberg  Parallel Networks that Learn to Pronounce \nEnglish Text.  Complex Systems, 1, pp.145-168 (1987). \nM.Komura and A.Tanaka  Speech Synthesis Using a  Neural Network with a \nCooperative  Learning Mechanism. \nIEICE  Tech.  Rep.  MBE88-8  (1988)  (in \nJapanese). \n\n\f", "award": [], "sourceid": 141, "authors": [{"given_name": "Mitsuo", "family_name": "Komura", "institution": null}, {"given_name": "Akio", "family_name": "Tanaka", "institution": null}]}