{"title": "Representing Face Images for Emotion Classification", "book": "Advances in Neural Information Processing Systems", "page_first": 894, "page_last": 900, "abstract": null, "full_text": "Representing Face Images for  Emotion \n\nClassification \n\nCurtis Padgett \n\nDepartment of Computer Science \nUniversity of California, San Diego \n\nLa Jolla, CA  92034 \n\nGarrison Cottrell \n\nDepartment of Computer Science \nUniversity of California, San  Diego \n\nLa Jolla,  CA  92034 \n\nAbstract \n\nWe  compare  the  generalization  performance of three  distinct  rep(cid:173)\nresentation schemes for facial emotions using a  single classification \nstrategy  (neural  network).  The face  images presented  to the  clas(cid:173)\nsifiers  are  represented  as:  full  face  projections  of the  dataset  onto \ntheir  eigenvectors  (eigenfaces);  a  similar projection  constrained  to \neye  and  mouth  areas  (eigenfeatures);  and  finally  a  projection  of \nthe eye and mouth areas onto the eigenvectors obtained from 32x32 \nrandom image patches from the dataset.  The latter system achieves \n86%  generalization on novel face  images  (individuals the networks \nwere  not trained  on)  drawn from  a  database in  which  human sub(cid:173)\njects  consistently  identify a single emotion for  the face . \n\n1 \n\nIntroduction \n\nSome  of the  most  successful  research  in  machine  perception  of  complex  natural \nimage  objects  (like  faces),  has  relied  heavily  on  reduction  strategies  that  encode \nan  object  as  a  set  of values  that  span  the  principal  component  sub-space  of the \nobject's  images [Cottrell and  Metcalfe,  1991,  Pentland et al.,  1994].  This approach \nhas gained wide acceptance for its success in classification, for the efficiency in which \nthe  eigenvectors  can  be  calculated,  and  because  the  technique  permits  an  imple(cid:173)\nmentation that is  biologically plausible.  The procedure followed  in generating these \nface representations requires normalizing a large set of face views (\" mug-shots\") and \nfrom these,  identifying a  statistically relevant sub-space.  Typically the sub-space is \nlocated  by finding either the eigenvectors  of the faces  [Pentland et  al.,  1994]  or the \nweights  of the  connections  in  a  neural network  [Cottrell and  Metcalfe,  1991]. \n\nIn this work,  we  classify face  images based on their emotional content and examine \nhow  various representational  strategies  impact the  generalization results  of a  clas(cid:173)\nsifier.  Previous  work using whole face  representations  for  emotion classification  by \n\n\fRepresenting Face Images for Emotion Classification \n\n895 \n\nCottrell  and  Metcalfe  [Cottrell and  Metcalfe,  1991]  was  less  encouraging  than  re(cid:173)\nsults obtained for  face  recognition.  We  seek to determine if the problem in Cottrell \nand  Metcalfe's work stems from bad data (i.e.,  the inability of the undergraduates \nto demonstrate emotion), or an inadequate representation  (i.e.  eigenfaces). \n\nThree  distinct  representations  of faces  are  considered  in  this  work- a  whole  face \nrepresentation  similar to that used  in previous  work on  recognition,  sex,  and emo(cid:173)\ntion [Cottrell and Metcalfe,  1991]; a more localized representation based on the eyes \n(eigeneyes  and eigenmouths)  and  mouth  [Pentland et aI.,  1994];  and  a  representa(cid:173)\ntion  of the  eyes  and  mouth  that  makes  use  of  basis  vectors  obtained  by  princi(cid:173)\npal  components  of random image blocks.  By examining the  generalization  rate  of \nthe  classifiers  for  these  different  face  representations,  we  attempt  to  ascertain  the \nsensitivity  of the  representation  and  its  potential for  broader  use  in  other  vision \nclassification  problems. \n\n2  Face Data \n\nThe dataset used  in Cottrell and Metcalfe's work on emotions consisted of the faces \nof undergraduates  who  were  asked  to  pose  for  particular  expressions.  However, \nfeigned  emotions  by  untrained  individuals exhibit  significant  differences  from  the \nprototypical face  expression  [Ekman and  Friesen,  1977].  These  differences  often re(cid:173)\nsult  in  disagreement  between  the  observed  emotion  and  the  expression  the  actor \nis  attempting to feign.  A feigned  smile for  instance,  differs  around the eyes  when \ncompared with  a  \" natural\"  smile.  The  quality of the  displayed  emotion  is  one  of \nthe  reasons  cited  by  Cottrell  and  Metcalfe for  the poor recognition  rates  achieved \nby their classifier. \n\nTo reduce  this possibility, we  made use  of a validated facial emotion database (Pic(cid:173)\ntures of Facial Affect)  assembled by Ekman and Friesen  [Ekman and  Friesen,  1976]. \nEach  of the  face  images  in  this  set  exhibits  a  substantial  agreement  between  the \nlabeled  emotion and the observed  response  of human subjects.  The actors  used  in \nthis database were trained to reliably produce emotions using Facial Action Coding \nSystem  [Ekman and  Friesen,  1977]  and  their  images were  presented  to undergrad(cid:173)\nuates  for  testing.  The  agreement  between  the  emotion the  actor was  required  to \nexpress  and  the  students'  observations  was  at  least  70%  on  all  the  images  incor(cid:173)\nporated  in  the  database.  We  digitized  a  total  of 97  images from  12  individuals  (6 \nmale, 6 female).  Each  portrays one  of 7 emotions- happy, sad, fear,  anger, surprise, \ndisgust or neutral.  With the exception of the neutral faces,  each image in the set  is \nlabeled with a  response vector of the remaining six emotions indicating the fraction \nof total respondents  classifying the image with a  particular emotion. \n\nEach  of the  images  was  linearly  stretched  over  the  8  bit  greyscale  range  to  re(cid:173)\nduce lighting variations. Although care was taken in collecting the original images, \nnatural  variations  in  head  size  and  the  mouth's  expression  resulted  in  significant \nvariation in  the  distance  between  the eyes  (2.7  pixels)  and in the vertical  distance \nfrom the eyes to the mouth (5.0 pixels).  To achieve scale invariance, each image was \nscaled so that prominent facial features  were located in the same image region.  Eye \nand mouth templates were constructed from a  number of images, and the most cor(cid:173)\nrelated template was used to localize the respective feature.  Similar techniques have \nbeen  employed in  previous work on faces  [Brunelli and  Poggio, 1993] .  Examples of \nthe normalized images and typical facial expressions  can be found  in  Figure 1. \n\n\f896 \n\nC.  Padgen and G.  W.  Conrell \n\nA \n\nB \n\nC \n\nFigure  1:  The image regions  from  which  the  representations  are  derived.  Image A \nis  a  typical  normalized and  cropped  image used  to generate  the full  face  eigenvec(cid:173)\ntors.  Image  B  depicts  the feature  regions  from  which  the feature  eigenvectors  are \ncalculated.  Image C  indicates  each  of the  block  areas  projected  onto  the  random \nblock eigenvectors. \n\n3  Representation \n\nFrom the  normalized database, we  develop three  distinct representations that form \nindependent pattern sets for a single classification scheme.  The selected  representa(cid:173)\ntions differ  in their scope (features or whole face)  and in the nature ofthe sub-space \n(eigen- faces/features  or eigenvectors of random image patches).  The more familiar \nrepresentational  schemes  (eigenfaces,  eigenfeatures)  are  based  on  PCA  of aligned \nfeatures or faces.  They have been shown to provide a  reasonably compact represen(cid:173)\ntation for  recognition  purposes  but little is  known  about their suitability for  other \nclassification tasks. \n\nRandom image patches  are  used  to  identify an  alternative sub-space  from  which a \nset  of localized  face  feature  patterns are  generated.  This  space  is  different  in  that \nthe sub-space  is  more general,  the variance  captured  by the leading eigenvectors  is \nderived  from  patches  drawn  randomly over  the  set  of face  images.  As  we  seek  to \ndevelop  generalizations  across  the  rather  small  portion  of image space  containing \nfaces  or features , perturbations in this space will hopefully reflect  more about class \ncharacteristics  than individual distinctions. \n\nFor  each  of the  pattern  sets,  we  normalized  the  resultant  set  of values  obtained \nfrom their projections on the eigenvectors by their standard deviation to produce Z \nscores.  The Z score obtained from each image constitutes a single input to the neural \nnetwork  classifier.  The  highest  valued eigenvectors  typically contain  more average \nfeatures  so  that  presumably they  would  be  more  suitable for  object  classification. \nAll  the  representations  will  make use  of the top k  principal components. \n\nThe  full-faced  pattern  has  proved  to  be  quite  useful  in  identification  and  the \nsame techniques  using face  features  have  also  been valuable [Pentland et  al.,  1994, \nCottrell  and  Metcalfe,  1991].  However  representations  useful  for  identification  of \nindividuals  may  not  be  suitable  for  emotion  recognition.  In  determining  the  ap(cid:173)\npropriate  emotion, structural  differences  in  faces  need  to  be  suppressed.  One way \nto  accomplish this  is  to eliminate portions  of the  face  image where  variation  pro(cid:173)\nvides  little  information  with  respect  to  emotion.  Local  changes  in  facial  muscles \naround  the  eyes  and  mouth  are  generally  associated  with  our  perception  of emo(cid:173)\ntions  [Ekman and Friesen,  1977].  The  full  face  images  presumably  contain  much \ninformation that  is  simply irrelevant  to  the  task  at  hand  which  could  impact  the \nability of the  classifier  to uncover  the signal. \n\n\fRepresenting Face Images for Emotion Classification \n\n897 \n\nThe feature  based  representations  are  derived  from local windows  around the eyes \nand  mouth of the  normalized whole  face  images (see  Fig.  IB).  The eigenvectors  of \nthe feature sub-space  are determined independently for  each feature  (left/right eye \nand mouth).  A face pattern is  generated by projecting the particular facial features \non their respective  eigenvectors. \n\nThe  random block  pattern  set  is  formed  from  image blocks  extracted  around  the \nfeature locations (see  Fig.  lC). The areas around each eye  are divided into two ver(cid:173)\ntically overlapping blocks of size 32x32 and the mouth is sectioned into three.  How(cid:173)\never,  instead  of performing PCA  on each  individual block  or all  of them together, \na  more general  PCA of random 32x32 blocks taken over the entire image was  used \nto generate the eigenvectors.  We  used random blocks to reduce the uniqueness  of a \nprojection for  a single individual and provide a  more reasonable model of the early \nvisual system.  The final  input pattern  consists  of the normalized  projection of the \nseven  extracted  blocks for  the image on  the top n  principal components. \n\n4  Classifier design  and  training \n\nThe principal goal of classification for this study is to examine how the different rep(cid:173)\nresentational spaces facilitate  a  classifiers  ability to generalize to novel  individuals. \nComparing expected  recognition  rate error  using  the  same classification technique \nwith  different  representations  should  provide  an  indication  of how  well  the signal \nof interest  is  preserved  by  the  respective  representation.  A  neural  network  with \na  hidden  layer  employing  a  non-linear  activation function  (sigmoid)  is  trained  to \nlearn  the  input-output  mapping between  the representation  of the face  image and \nthe associated  response  vector given by  human subjects. \n\nA  simple, fully  connected,  feed-forward  neural  network  containing  a  single  hidden \nlayer with  10  nodes,  when  trained  using  back  propagation,  is  capable  of correctly \nclassifying the input of training sets  from each  of the three  representations  (tested \nfor  pattern  sizes  up  to  140  dimensions).  The  architecture  of the  network  is  fixed \nfor  a  particular  input  size  (based  on  the  number  of projections  on  the  respective \nsub-space)  and the generalization of the network is found  on a set  of images from a \nnovel  individual.  An  overview  of the network design  is  shown in  Fig.  2. \n\nTo minimize the impact of choosing a  poor hold out set from the training set, each \nof the 11  individuals in the training set was in turn used as a hold out.  The results of \nthe 11  networks  were  then  combined to evaluate the classification  error on the test \nset.  A number of different techniques are possible:  winner take all, weighted average \noutput,  voting,  etc.  The  method  that  we  found  to  consistently  give  the  highest \ngeneralization rate involved combining Z scores  from the  11  networks.  The average \noutput for  each  possible emotion across  all the networks was  calculated along with \nits deviation over the entire training set.  These values were  used  to normalize each \noutput of the  11  networks  and the highest  weighted sum for  a  particular input was \nthe associated  emotion. \n\nDue  to  the  limited  amount  of  data  available  for  testing  and  training,  a  cross(cid:173)\nvalidation technique  using  each  set  of an  individual's  images  for  testing  was  em(cid:173)\nployed to increase the confidence ofthe generalization measurement.  Thus, for each \nindividual, 11  networks  were  combined to evaluate the generalization  on  the single \ntest individual, and this procedure is repeated for all 12  individuals to give an aver(cid:173)\nage generalization error.  This results in a total of 132 networks to evaluate the entire \ndatabase.  A single trial consisted of the generalization rate obtained over the whole \ndatabase  for  a  particular  size  of input  pattern .  By  varying  the  initial  weights  of \nthe network, we  can determine the expected generalization performance of this type \n\n\f898 \n\nC.  Padgett and G.  W.  Cottrell \n\nHuman ReSPOllge. \n\nEnsemble \n\nNetworks \n\nFigure 2:  The processing path used to evaluate the pattern set of each representation \nscheme.  The  original  image  data  (after  normalization)  is  used  to  generate  the \neigenvectors  and construct  the pattern sets.  Human responses  are used  in training \nthe  classifiers  and determining generalization percentages  on the test  data. \n\nclassifier on each representation.  The number of projections on the relevant space is \nalso varied to determine a  generalization curve for  each  representation.  Construct(cid:173)\ning,  training, and evaluating the  132  networks takes  approximately 2 minutes on a \nSparcStation 10 for  input pattern size of 15 and 4 minutes for  an input pattern size \nof 80. \n\n5  Results \n\nFig.  3  provides  the expected  generalization  achIeved  by the  neural  network  archi(cid:173)\ntecture  initially  seeded  with  small  random  weights  for  an  increasing  number  of \nprojections  in  the  respective  representational  spaces.  Each  data  point  represents \nthe  average  of 20  trials,  1  (T  error  bars show  the  amount  of error  with  respect  to \nthe mean.  The  curve  (generalization  rate vs.  input pattern  size)  was evaluated  at \n6  points for  the whole  face  and  at  8  points for  each  feature  based  approach.  The \neigenfeature  representation  made use  of up to 40 eigenvectors  for  the three regions \nwhile the random block  representation  made use  of up to  17 eigenvectors  for  each \nof its seven  regions. \n\nFor  the  most  part,  all  the  representations  show  improvement  as  the  number  of \nprojections  increase.  Variations  as  input  size  increases  are  most  likely  due  to  a \ncombination of two factors:  lower signal to noise  ratios (SNR) for  higher order pro(cid:173)\njections; and the increasing number of para~eters with a fixed  number of patterns, \nmaking generalization  difficult.  The  highest  average  recognition  rate  achieved  by \nthe  neural  network  ensembles  is  86%,  found  using  the  random  block  representa(cid:173)\ntion  with  15  projections  per  block.  The  results  indicate  that  the  generalization \nrate for emotion classification varies significantly depending on the representational \nstrategy.  Both  local  feature-based  approaches  (eigenfeatures  and  random  block) \ndid  significantly  better  over  their  shared  range  than  the eigenface  representation. \nOver  most of the range,  the random block  representation  is  clearly superior  to the \neigenfeature representation even though both are derived from the same image area. \n\n\fRepresenting Face Images for Emotion Classification \n\n899 \n\no . 9 r - - - - , - - - . . , . . - - - - , . - - - - , . - - - - - - , , . - - - - - ,  \n\nrandom block \n\n0.85 \n\n~  0.8 \nc: \n.Q \n\n~ \n\u00b7ill 0.75 \n~ \n~ .. ~  0.7 \n~ \n\neigenface \n\nO\u00b7~':-O --r--4:':-O---6\"':0---BO-.l-:----l~OO-:------'12-0--~140 \n\nNumber of inputs (projections on eignevectors) \n\nFigure  3:  Generalization  curves for  feature-based  representation  and full-face  rep(cid:173)\nresentation. \n\n6  Discussion \n\nFig.  3  clearly  demonstrates  that  reasonable  recognition  rates  can  be  obtained  for \nnovel individuals using representational techniques that were found useful for  iden(cid:173)\ntity.  The  86%  generalization  rate  achieved  by  the  neural  network  ensemble  us(cid:173)\ning  random  block  patterns  with  105  projections  compares  favorably  with  the  re(cid:173)\nsults obtained from techniques  that use  an expression  sequence  (neutral to expres(cid:173)\nsion)  [Mase,  1991,  Yacoob  and  Davis,  1996,  Bartlett et  al.,  1996].  Such  schemes \nmake  use  of a  neutral  mask  which  enhances  the  sequence's  expression  by  simple \nsubtraction,  a technique that is  not  possible on novel,  static face  images.  That our \ntechnique  works  as  well  or  better  indicates  the  possibility  that  the  human visual \nsystem need not rely on  difference image strategies over sequences of images in clas(cid:173)\nsifying emotions.  As  many psychological studies  are  performed on  static images of \nindividuals,  models that  can  accommodate this  aspect  of emotion recognition  can \nmake predictions that directly guide research  [Padgett et  al.,  1996] . \nAs  for  the suitability of the various representations  for  fine  grained  discrimination \nover different individual objects (as required by emotion classification), Fig. 3 clearly \ndemonstrates the benefits  accrued  by  concentrating on facial features  important to \nemotion .  The  generalization  of the  trained  networks  making use  of the  two  local \nfeature-based  representations  averages  6-15% higher  than  do  the networks  trained \nusing  projections  on  the eigenfaces.  The increased  performance  can  be  attributed \nto a  better signal to noise  ratio for  the feature  regions.  As  much of the face  is rigid \n(e.g.  the  chin  and forehead),  these  regions  provide little in the way  of information \nuseful  in  classifying  emotions.  However,  there  are  substantial  differences  in  these \nareas  between  individuals,  which  will  be  expressed  by  the  principal  component \nanalysis  of the images and  thus reflected  in the projected  values.  These  variations \nare  essentially  noise  with  respect  to  emotion  recognition  making it  more  difficult \nfor  the classifier  to extract useful generalizations during learning. \n\nThe final  point is  the superiority of the random block representation over the range \n\n\f900 \n\nC.  Padgett and G.  W.  Cottrell \n\nexamined.  One  possible  explanation  for  its  significant  performance  edge  is  that \nmajor  feature  variations  (e.g.  open  mouth,  open  eyes,  etc.)  are  more  effectively \npreserved  by this  representation  than the  eigenfeature  approach,  which  covers  the \nsame image area.  Due to individual differences  in  mouth/eye structure,  one would \nexpect  that many of the eigenvectors of the feature  space would be devoted  to this \nvariance.  Facial  expressions  could  be substantially orthogonal to this variance,  so \nthat information pertinent  to emotion discrimination is  effectively  hidden.  This  of \ncourse  would  imply that the eigenfeature  representation  should be better than the \nrandom  block  representation  for  face  recognition  purposes.  However,  this  is  not \nthe  case.  Nearest  neighbor  classification of individuals using the same pattern sets \nshows that the random block representation does better for this task as well  (results \nnot shown).  We  are  currently developing  a  noise  model that looks promising as  an \nexplanation for  this  phenomenon. \n\n7  Conclusion \n\nWe  have  demonstrated  that  average  generalization  rates  of 86%  can  be  obtained \nfor  emotion recognition on novel  individuals using techniques  similar to work  done \nin  face  recognition.  Previous  work  on  emotion  recognition  has  relied  on  image \nsequences  and  obtained  recognition  rates  of nearly  the  same  generalization.  The \nmodel  we  developed  here  is  potentially of more  interest  to  researchers  in  emotion \nthat make use of static images of novel individuals in conducting their tests.  Future \nwork will  compare aspects  of the network  model with human performance. \n\nReferences \n\n[Bartlett et  al.,  1996]  Bartlett,  M.,  Viola,  P.,  Sejnowski,  T .,  Larsen,  J.,  Hager,  J., \nand  Ekman,  P.  (1996).  Classifying facial  action.  In  Touretzky,  D.,  Mozer,  M. , \nand  Hasselmo,  M.,  editors,  Advances  in  Neural  Information  Processing  Systems \n8,  Cambridge, MA . MIT Press. \n\n[Brunelli  and  Poggio, 1993]  Brunelli,  R.  and  Poggio,  T.  (1993) .  Face  recognition: \n\nFeature versus  templates.  IEEE  Trans.  Patt.  Anal.  Machine  Intell.,  15(10). \n\n[Cottrell and  Metcalfe,  1991]  Cottrell,  G.  W.  and  Metcalfe,  J.  (1991).  Empath: \nFace,  gender  and emotion recognition  using  holons.  In  Lippman, R.,  Moody,  J ., \nand Touretzky,  D.,  editors,  Advances  in  Neural  Information  Processing  Systems \n3,  pages  564-571, San Mateo.  Morgan Kaufmann. \n\n[Ekman and  Friesen,  1976]  Ekman,  P.  and  Friesen,  W.  (1976).  Pictures  of facial \n\naffect. \n\n[Ekman and  Friesen,  1977]  Ekman, P. and Friesen, W. (1977).  Facial Action Coding \n\nSystem.  Consulting Psychologists,  Palo Alto,  CA. \n\n[Mase,  1991]  Mase,  K.  (1991).  Recognition of facial  expression  from  optical  flow. \n\nIEICE  Transactions,  74(10):3474-3483 . \n\n[Padgett et al.,  1996]  Padgett, C., Cottrell, G., and Adolphs, R.  (1996).  Categorical \nperception  in facial emotion classification.  In  Cottrell,  G., editor,  Proceedings  of \nthe  18th  Annual Cognitive  Science  Conference,  San  Diego  CA . \n\n[Pentland et al.,  1994]  Pentland,  A.  P.,  Moghaddam,  B.,  and  Starner,  T.  (1994). \nView-based  and  modular eigenspaces  for  face  recognition.  In  IEEE  Conference \non  Computer  Vision  fj  Pattern  Recognition. \n\n[Yacoob  and  Davis,  1996]  Yacoob, Y.  and Davis, L.  (1996).  Recognizing human fa(cid:173)\n\ncial expressions from long image sequences  using optical flow. IEEE Transactions \non  Pattern  Analysis  and  Machine  Intelligence,  18:636-642. \n\n\f", "award": [], "sourceid": 1180, "authors": [{"given_name": "Curtis", "family_name": "Padgett", "institution": null}, {"given_name": "Garrison", "family_name": "Cottrell", "institution": null}]}