{"title": "Pairwise Neural Network Classifiers with Probabilistic Outputs", "book": "Advances in Neural Information Processing Systems", "page_first": 1109, "page_last": 1116, "abstract": null, "full_text": "Pairwise  Neural  Network  Classifiers  with \n\nProbabilistic  Outputs \n\nDavid  Price \nA2iA and ESPCI \n\n3 Rue de l'Arrivee, BP 59 \n\n75749 Paris Cedex 15, France \n\na2ia@dialup.francenet.fr \n\nStefan  Knerr \n\nESPCI and CNRS  (UPR AOOO5) \n\n10, Rue Vauquelin, 75005 Paris, France \n\nknerr@neurones.espci.fr \n\nLeon  Personnaz,  Gerard  Dreyfus \n\nESPeI, Laboratoire d'Electronique \n\n10, Rue Vauquelin, 75005 Paris, France \n\ndreyfus@neurones.espci.fr \n\nAbstract \n\nMulti-class  classification  problems  can  be  efficiently  solved  by \npartitioning the  original problem into  sub-problems  involving  only  two \nclasses:  for each pair of classes,  a (potentially small)  neural  network is \ntrained  using  only  the  data  of  these  two  classes.  We  show  how  to \ncombine the outputs of the  two-class  neural  networks in order to  obtain \nposterior probabilities for  the class decisions.  The resulting probabilistic \npairwise classifier is part of a handwriting recognition  system which  is \ncurrently applied to check reading. We present results on real world data \nbases and show that, from a practical point of view, these results compare \nfavorably to other neural network approaches. \n\n1 \n\nIntroduction \n\nGenerally,  a  pattern  classifier  consists  of  two  main  parts:  a  feature  extractor  and  a \nclassification  algorithm.  Both parts  have  the  same ultimate  goal,  namely  to  transform  a \ngiven input pattern into a representation that is easily interpretable as  a class decision.  In \nthe case of feedforward neural networks, the interpretation is particularly easy if each class \nis  represented by  one output unit.  For many pattern recognition problems, it suffices  that \nthe classifier compute the class of the input pattern, in which case it is common practice to \nassociate  the pattern  to the  class corresponding  to  the maximum output  of the classifier. \nOther problems require graded (soft) decisions,  such as probabilities, at the output of the \n\n\f1110 \n\nDavid Price,  Stefan Knerr,  Leon Personnaz,  Gerard Dreyfus \n\nclassifier for  further  use  in  higher context levels:  in  speech  or character recognition  for \ninstance, the probabilistic outputs of the phoneme (character) recognizer are often used by a \nHidden-Markov-Model  algorithm  or  by  some  other  dynamic  programming algorithm  to \ncompute the most probable word hypothesis. \nIn  the  context  of classification,  it  has  been  shown  that  the  minimization  of the  Mean \nSquare  Error  (MSE)  yields  estimates  of a  posteriori  class  probabilities  [Bourlard  & \nWellekens,  1990;  Duda  &  Hart,  1973].  The  minimization  can  be  performed  by  a \nfeedforward multilayer perceptrons (MLP's) using the backpropagation algorithm, which is \none  of the  reasons  why  MLP's  are  widely  used  for  pattern  recognition  tasks.  However, \nMLPs have  well-known limitations when  coping  with real-world problems,  namely  long \ntraining times and unknown architecture. \nIn  the  present paper,  we  show that the estimation of posterior probabilities for  a K-class \nproblem can be performed efficiently using estimates of posterior probabilities for K(K -1 )/2 \ntwo-class sub-problems. Since the number of sub-problems increases as  K2,  this procedure \nwas  originally  intended for  applications  involving  a relatively  small  number  of classes, \nsuch as  the  10 classes for the recognition of handwritten digits  [Knerr et aI.,  1992]. In this \npaper we  show that this approach is also viable for applications with K\u00bb  10. \nThe  probabilistic  pairwise  classifier  presented  in  this  paper  is  part  of a  handwriting \nrecognition system, discussed elsewhere [Simon,  1992], which is currently applied to check \nreading. The purpose of our character recognizer is to classify pre-segmented characters from \ncursive handwriting. The probabilistic outputs of the recognizer are used to estimate word \nprobabilities.  We  present results  on real  world data involving 27  classes,  compare  these \nresults  to  other  neural  network  approaches,  and  show  that  our  probabilistic  pairwise \nclassifier  is  a  powerful  tool  for  computing  posterior  class  probabilities  in  pattern \nrecognition problems. \n\n2  Probabilistic  Outputs  from  Two-class  Classifiers \n\nMulti-class  classification  problems  can  be  efficiently  solved  by  \"divide  and  conquer\" \nstrategies which partition the original problem into a set of K(K-l)/2 two-class  problems. \nFor each  pair  of classes  (OJ  and (OJ,  a  (potentially  small)  neural  network  with  a  single \noutput unit  is  trained  on  the  data  of the  two  classes  [Knerr et aI.,  1990,  and  references \ntherein].  In  this  section,  we  show  how  to  obtain probabilistic  outputs  from  each  of the \ntwo-class classifiers in the pairwise neural network classifier (Figure  1). \n\nK(K-I)12 \n\ntwo-class networks \n\ninputs \n\nFigure 1:  Pairwise neural network classifier. \n\n\fPairwise Neural Network  Classifiers with  Probabilistic Outputs \n\n1111 \n\nIt has  been  shown  that  the  \\llinimization  of the  MSE  cost  function  (or likewise  a  cost \nfunction  based  on  an  entropy  measure,  [Bridle,  1990])  leads  to  estimates  of posterior \nprobabilities. Of course, the quality of the estimates depends on the number and distribution \nof examples in the training set and on the minimization method used. \nIn  the  theoretical case of two classes  <01  and <02,  each Gaussian distributed,  with means \nm 1 and m2,  a  priori  probabilities  Pq  and  Pr2,  and  equal  covariance  matrices  ~, the \nposterior probability of class <01  given the pattern x is: \n\nPr(class=<o\\  I X=x) = _ _ __ ______  --'1'----___ ______  _ \n1 + Pr2  exp( _ !-(2xT~-\\(m\\-m2) + m!~-\\m2 - m T~-lm\\)) \n\nPrJ \n\n2 \n\n(1) \n\nThus  a  single  neuron  with  a  sigmoidal  transfer  function  can  compute  the  posterior \nprobabilities for the two classes. \nHowever,  in  the  case  of real  world  data  bases,  classes  are  not  necessarily  Gaussian \ndistributed, and therefore the transformation of the K(K-l )/2 outputs of our pairwise neural \nnetwork classifier to posterior probabilities proceeds in two steps. \nIn  the  first  step,  a  class-conditional  probability  density  estimation  is  performed  on  the \nlinear output of each two-class neural network:  for both classes <OJ  and <OJ  of a  given two(cid:173)\nclass neural network, we fit the probability density over Vjj  (the weighted sum of the inputs \nof the output neuron) to a function.  We denote by <Ojj  the union of classes <OJ  and <OJ.  The \nresulting  class-conditional  densities  p(vij I <OJ)  and  p(Vjj  I <OJ)  can  be  transformed  to \nprobabilities  Pr(<Oj  I <OJ'  /\\ (Vij=Vjj\u00bb  and Pr(<Oj  I <Ojj  /\\ (Vjj=Vjj\u00bb  via the Bayes rule (note \nthat Pr(<Ojj  /\\ (Vij=Vjj) 1 <OJ)  =  Pr\u00abVij=Vjj) I <OJ)): \n\np( VjJ\"  I <OJ)  Pr( <OJ) \nPr(<Oj  I <Ojj/\\(Vij=Vij\u00bb  = - - - - - \" - - - - - -\nL  p(Vjj  I <Ok)  Pr(<Ok) \nke{j,j} \n\n(2) \n\nIt is  a  central  assumption  of our  approach  that  the  linear  classifier  output  Vij  is  as \ninformative  as  the  input vector x.  Hence,  we approximate  Prij  = Pr(<Oj  I <Ojj  /\\  (X=x\u00bb  by \nPr(<Oi  I <Ojj  /\\  (V=Vjj\u00bb.  Note  that Pji =  I-Pjj. \nIn  the  second  step,  the  probabilities  Prij  are  combined  to  obtain  posterior probabilities \nPr(<Oj  I (X=x\u00bb for all classes <Oi  given a pattern x.  Thus, the network can be considered as \ngenerating an intermediate data representation in the recognition chain,  subject to  further \nprocessing [Denker &  LeCun,  1991].  In  other words, the neural network becomes part of \nthe preprocessing and contributes to dimensionality reduction. \n\n3  Combining  the  Probabilities  Prij  of  the  Two-class  Classifiers \n\nto  a  posteriori  Probabilities \n\nThe set of two-class neural  network classifiers discussed in the previous section results  in \n\nprobabilities  Prjj  for  all  pairs  (i, j) with  i * j.  Here,  the  task  is  to  express  the  posterior \nprobabilities Pr(<Oj  I (X=x\u00bb  as functions of the Prjj-\n\n\f1112 \n\nDavid Price,  Stefan Knerr,  Leon Personnaz,  Gerard Dreyfus \n\nWe assume that each pattern belongs to only one class: \n\nK \n\nPr( U  Olj  I (X=x\u00bb  = 1 \n\nj=1 \n\nFrom the definition of Olij.  it follows for any given i: \n\nK \n\nK \n\nPr(U  Olj  I (X=x\u00bb  = Pr(  U  Olij  I (X=x\u00bb  = 1 \n\nJ=I \n\nj=l,j*i \n\nU sing the closed form expression for the probability of the union of N events Ei: \n\n(3) \n\n(4) \n\nPr(U  Ei) = L  Pr(Ej) + ... + (_I)k.1  L  Pr(EhA ... AEh) + ... + (-I)N.lpr(EIA ... AEN) \n\nN \n\nN \n\ni= 1 \n\ni= 1 \n\nN \n\ni}< ... <ik \n\nit follows from (4): \n\nK L  Pr(Olij  I (X=x\u00bb  - (K-2) Pr(Oli I (X=x\u00bb  = 1 \n\nj=l,j*i \n\nWith \n\nPrij  = Pr(Oli  I OlijA(X=X\u00bb = \n\nPr(OliAOli'A(X=X\u00bb \n\nJ \n\nPr(OlijA(X=X\u00bb \n\n= --'----~ \n\nPr(Oli  I (X=x\u00bb \nPr(Olij  I (X=x\u00bb \n\n(5) \n\n(6) \n\none obtains the  final expression for the  K posterior probabilities given the K(K-l)12  two(cid:173)\nclass probabilities Prji : \n\nPr(Oli I (X=x\u00bb  = __  --\"-1 __  _ \n\nf  _1 __ (K-2) \n\nj=I,#i Prij \n\n(7) \n\nIn [Refregier et aI.,  1991], a method was derived which allows to compute the K posterior \nprobabilities from only (K-l) two-class probabilities using the following relation between \nposterior probabilities and two-class probabilities: \n\nPrij  = Pr(Oli  I (X=x\u00bb \nPrji  Pr(Olj  I (X=x\u00bb \n\n(8) \n\nHowever,  this  approach  has  several  practical  drawbacks.  For instance,  in  practice,  the \nquality of the estimation of the posterior probabilities depends critically  on  the  choice of \nthe  set  of (K-l)  two-class  probabilities,  and  finding  the  optimal  subset  of (K-l)  Prij  is \ncostly, since it has to be performed for each pattern at recognition time. \n\n\fPairwise Neural Network  Classifiers with  Probabilistic Outputs \n\n1113 \n\n4  Application  to  Cursive  Handwriting  Recognition \n\nWe  applied  the  concepts  described  in  the previous  sections  to  the  classification  of pre(cid:173)\nsegmented characters from cursive words originating from real-world French postal checks. \nFor cursive  word  recognition  it  is  important to  obtain  probabilities at the  output of the \ncharacter classifier since it is necessary to establish an ordered list of hypotheses along with \na confidence value for further processing at the word recognition level: the probabilities can \nbe passed to an Edit Distance algorithm [Wagner et at, 1974] or to a Hidden-Markov-Model \nalgorithm [Kundu et aI.,  1989]  in  order to compute recognition scores for  words.  For the \nrecognition of the amounts on French postal checks we used an Edit Distance algorithm and \nmade extensive use of the fact that we are dealing with a limited vocabulary (28 words). \nThe 27 character classes are particularly chosen for this task and include pairs of letters such \nas  \"fr\",  \"gttl,  and  \"tr\"  because  these  combinations  of letters  are  often  difficult  to  pre(cid:173)\nsegment. Other characters, such as tlk\"  and \"y\" are not included because they do not appear \nin the given 28 word vocabulary. \n\n0~)(\"  C.A~ t:;  ~\"U::l._ r;.., \n~ \\\\:..~~~ (~tSl\\h~~ \\J'~ ~~~ \n~~~~~~ \n\n&.nl' tL'i.upr  rm'A \n\nFigure 2:  Some examples of literal amounts from live French postal checks. \n\nA  data  base  of about  3,300  literal  amounts  from  postal  checks  (approximately  16,000 \nwords) was annotated and, based on this annotation, segmented into words and letters using \nheuristic methods [Simon et aI.,  1994]. Figure 2 shows some examples of literal amounts. \nThe writing styles vary strongly throughout the data base and many checks are difficult to \nread even for humans.  Note that the images of the pre-segmented letters may still contain \nsome of the ligatures or other extraneous parts and do not in general resemble hand-printed \nletters.  The  total  of about  55,000  characters  was  divided  into  three  sets:  training  set \n(20,000),  validation set (20,000),  and test  set (15,000).  All  three sets were  used without \nany further data base cleaning. Therefore, many letters are not only of very bad quality, but \nthey  are  truly  ambiguous:  it  is  not  possible  to  recognize  them  uniquely  without  word \ncontext. \n\nFigure 3: Reference lines indicating upper and lower limit of lower case letters. \n\nBefore  segmentation,  two  reference lines  were detected for  each check (Figure 3).  They \nindicate  an  estimated upper  and  lower limit  of the  lower  case  letters  and  are  used  for \n\n\f1114 \n\nDavid Price,  Stefan Knerr,  Leon Personnaz,  Gerard Dreyfus \n\nnonnalization of the pre-segmented characters (Figure 4) to  10 by 24 pixel matrices with 16 \ngray values (Figure 5). This is the representation used as input to the classifiers. \n\nFigure 4:  Segmentation of words into isolated letters (ligatures are removed later). \n\nFigure 5:  Size nonnalized letters:  10 by 24 pixel matrices with 16 gray values. \n\nThe  simplest two-class  classifier  is  a  single  neuron;  thus,  351  neurons  of the  resulting \npairwise  classifier  were  trained  on  the  training  data  using  the  generalized  delta  rule \n(sigmoidal  transfer  function).  In  order to  avoid  overfitting,  training  was  stopped  at the \nminimum  of  MSE  on  the  validation  set.  The  probability  densities  P(Vij  I IDi)  were \nestimated on the validation set:  for both classes IDi  and IDj  of a given neuron,  we fitted the \nprobability densities  over the linear output Vij  to  a  Gaussian.  The two-class  probabilities \nPrij  and Prji  were then obtained via Bayes rule.  The 351  probabilities Prij  were combined \nusing equation (7) in order to obtain a posteriori probabilities Pr(IDi  I (X=x\u00bb, i E  {1, .. ,27}. \nHowever, the a priori probabilities for letters as  given by the training set are different from \nthe prior probabilities in a given word context [Bourlard &  Morgan,  1994]. Therefore, we \ncomputed the posterior probabilities either by using,  in Bayes rule,  the prior probabilities \nof the letters in the training set, or by assuming that the prior probabilities are equal. In the \nfirst case, many infonnative letters, for instance those having ascenders or descenders, have \nlittle chance to be recognized at all due to small a priori probabilities. \nTable 1 gives the recognition perfonnances on the test set for classes assumed to have equal \na priori probabilities as  well as  for the  true  a priori  probabilities of the  test set.  For each \npattern, an ordered list (in descending order) of posterior class probabilities was generated; \nthe recognition perfonnance is  given (i) in tenns of percentage of true classes found in first \nposition,  and  (ii)  in  tenns  of average  position  of the  true  class  in  the  ordered  list.  As \nmentioned  above,  the  results  of the  first  column  are  the  most relevant  ones,  since  the \nclassifier outputs are subsequently used for word recognition. Note that the recognition rate \n(first position) of isolated letters without context for a human reader can be estimated to be \naround 70% to 80%. \nWe compared the  results  of the  pairwise classifier to  a  number of other neural  network \nclassification  algorithms.  First,  we  trained  MLPs  with  one  and  two  hidden  layers  and \nvarious numbers of hidden units using stochastic backpropagation. Here again, training was \nstopped based on the minimum MSE on the  validation set. Second, we trained MLPs with \na  single  hidden  layer using  the  Softmax  training  algorithm  [Bridle,  1990].  As  a  third \napproach, we  trained 27  MLPs with  10 hidden units each, each MLP separating one class \nfrom  all others.  Table  1 gives the recognition perfonnances on the test set.  The  Softmax \n\n\fPairwise Neural Network  Classifiers with  Probabilistic Outputs \n\n1115 \n\ntraining  algorithm  clearly  gives  the  best  results  in  terms  of recognition  performance. \nHowever, the pairwise classifier has three very attractive features for classifier design: \n(i) training is faster than for MLP's by more than one order of magnitude; therefore, many \ndifferent designs  (changing pattern representations  for  instance) can be  tested at a  small \ncomputational cost; \n(ii) in the same spirit, adding a new class or modifying the training set of an existing one \ncan be done without retraining all two-class classifiers; \n(iii)  at  least  as  importantly,  the  procedure  gives  more  insight  into  the  classification \nproblem than MLP's do. \n\nClassifier \n\nPairwise \nClassifier \n\nMLP \n\n(100 hid. units) \n\nSoftmax \n\n(100 hid. units) \n\n27  MLPs \n\nA veragePosition \nequal prior probs  equal  prior probs \n\nFirst  Position  A veragePosition \ntrue  prior  probs \n\nFirst  Position \ntrue  prior  probs \n\n2.9 \n\n3.6 \n\n2.6 \n\n3.2 \n\n48.9 % \n\n48.9  % \n\n54.9  % \n\n41.6 % \n\n2.6 \n\n2.7 \n\n2.2 \n\n2.4 \n\n52.2 % \n\n60.0 % \n\n61.9  % \n\n55.8 % \n\nTable  I: Recognition performances on the test set in terms of average position and \n\nrecognition rate (first position) for the various neural networks used. \n\nOur pairwise  classifier  is  part  of a  handwriting  recognition  system  which  is  currently \napplied to check reading. The complete system also incorporates other character recognition \nalgorithms as  well  as  a  word  recognizer which  operates without pre-segmentation.  The \nresult of the complete check recognition chain on a set of test checks is the following:  (i) at \nthe  word  level,  83.3%  of  true  words  are  found  in  first  position;  (ii)  64.1 %  of  well \nrecognized  literal amounts  are  found  in  first  position  [Simon et al.,  1994].  Recognizing \nalso the numeral amount, we obtained 80% well recognized checks for 1 % error. \n\n5  Conclusion \n\nWe have shown how to obtain posterior class probabilities from a set of pairwise classifiers \nby  (i) performing class density estimations on the  network outputs and using  Bayes rule, \nand (ii)  combining the resulting  two-class  probabilities.  The application of our pairwise \nclassifier to the recognition of real world French postal checks shows that the procedure is a \nvaluable tool for designing a recognizer, experimenting with various data representations at \na small computational cost and, generally, getting insight into the classification problem. \n\nAcknowledgments \n\nThe authors wish to  thank J.C.  Simon, N.  Gorsky,  O.  Baret, and J.C.  Deledicq for  many \ninformative and stimulating discussions. \n\n\f1116 \n\nDavid Price,  Stefan Knerr,  Leon Personnaz,  Gerard Dreyfus \n\nReferences \n\nH.A.  Bourlard, N.  Morgan  (1994).  Connectionist Speech  Recognition.  Kluwer Academic \n\nPublishers. \n\nH.A.  Bourlard,  C.  Wellekens  (1990).  Links  between  Markov  Models  and  Multilayer \nPerceptrons. IEEE Transactions on Pattern Analysis and Machine  Intelligence, Vol. \n12, No.  12,  1167-1178. \n\nJ.S.  Bridle  (1990).  Probabilistic  Interpretation  of Feedforward  Classification  Network \nOutputs,  with Relationships to  Statistical Pattern  Recognition.  In Neurocomputing: \nAlgorithms,  Architectures  and Applications,  Fogelman-Soulie,  and Herault  (eds.). \nNATO ASI Series, Springer. \n\nJ.S.  Denker,  Y.LeCun  (1991).  Transforming  Neural-Net  Output  Levels  to  Probability \nDistributions.  In Advances in  Neural Information  Processing  Systems 3,  Lippmann, \nMoody, Touretzky (eds.). Morgan Kaufman. \n\nR.O. Duda, P.E. Hart (1973). Pattern Classification and Scene Analysis. Wiley. \nS.  Knerr, L. Personnaz, G.  Dreyfus (1990).  Single-Layer Learning Revisited:  A Stepwise \nProcedure  for  Building  and  Training  a  Neural  Network.  In  Neurocomputing: \nAlgorithms,  Architectures  and Applications,  Fogelman-Soulie  and  Herault  (eds.). \nNATO AS! Series, Springer. \n\nS.  Knerr,  L.  Personnaz,  G.  Dreyfus  (1992).  Handwritten  Digit  Recognition  by  Neural \nNetworks with Single-Layer Training. IEEE Transactions on Neural Networks,  Vol. \n3,  No.6, 962-968. \n\nA.  Kundu,  Y.  He,  P.  Bahl (1989).  Recognition  of Handwritten Words:  First and  Second \n\nOrder Hidden Markov Model Based Approach. Pattern Recognition, Vol. 22, No.3. \n\nlC. Simon (1992).  Off-Line Cursive Word  Recognition.  Proceedings  of the  IEEE,  Vol. \n\n80,  No.7,  1150-1161. \n\nPh.  Refregier, F.  Vallet (1991).  Probabilistic Approach for Multiclass Classification with \n\nNeural Networks.  Int.  Conference on Artificial Networks, Vol. 2,  1003-1007. \n\nJ.C.  Simon,  O.  Baret,  N.  Gorski  (1994).  Reconnaisance  d'ecriture  manuscrite.  Compte \n\nRendu Academie des Sciences, Paris, t.  318, Serie II, 745-752. \n\nR.A.  Wagner,  M.J.  Fisher  (1974).  The  String  to  String  Correction  Problem.  J.A.C.M. \n. \n\nVol.  21,  No.5,  168-173. \n\n\f", "award": [], "sourceid": 883, "authors": [{"given_name": "David", "family_name": "Price", "institution": null}, {"given_name": "Stefan", "family_name": "Knerr", "institution": null}, {"given_name": "L\u00e9on", "family_name": "Personnaz", "institution": null}, {"given_name": "G\u00e9rard", "family_name": "Dreyfus", "institution": null}]}