{"title": "EMPATH: Face, Emotion, and Gender Recognition Using Holons", "book": "Advances in Neural Information Processing Systems", "page_first": 564, "page_last": 571, "abstract": null, "full_text": "EMPATH:  Face, Emotion, and Gender Recognition Using Holons \n\n565 \n\nMunro & Zipser (1987) showed that a back  propagation network could be used \ncompression.  The network is trained to simply reproduce its input, and so can \nas a non-linear version  of Kohonen's  (1977) auto-associator.  However  it must \nthrough a narrow channel of hidden units, so  it must extract regularities from  the \nduring learning.  Empirical analysis of the trained network showed that the \nspan  the principal  subspace  of the  image  vectors,  with  some noise on  the \ncomponent due to network nonlinearity (Cottrell & Munro,  1988). \n\nthe  network  uses  error-correction  learning, no  teacher  other  than  the  input  is \nso  the  learning  can  be  regarded  as  unsupervised.  We  suggested  that  this \nbe used for  automatic feature  extraction  in a pattern recognition  system. \n\napproach taken here. \nis shown in Figure  1.  The image compression network extracts the features, \nhidden  unit representation  so  developed  is  given  as  input to one and  two \n')lll', ...... J  . .   networks  which  yield  identity,  gender, and emotion  as  output.  In \nwe  showed that the features  developed  by  the  model  are  holistic rather \nfea'ture:s,  that they can combine to form faces thatthe model has never been \nthat they form  a redintegrative memory, able to complete noisy or partial \net aI.,  1977).  We have dubbed them holons. \n\ncomprised the input to  the network were selected from  full  face pictures \nand  10 males.  All subjects were introductory psychology studentc; at \nof  California,  San  Diego  who  received  partial  course  credit  for \nthe procedure outlined  by  Galton  (1878),  images  were aligned \neyes and  mouth.  These images were captured  by a fraOl~ grabber, \npixels by averaging.  To prevent the use of first order statistics for \nwere  normalized  to  have equal brightness and variance.  The \nscaled  to  the  range  [0 \u2022. 8].  Part of the  training  set and  its \n\n~I'; ~lIItclenl~oder are shown in Figure 2. \n\nfaceness \n\ngender \n\nO~\u00b7\u00b7\u00b7 \u2022.  \u00b7~ \n\ninputs (hiddeils from auto-associator) \n\nFace informa.ticnextractor \n\nb \n\n'nrt'~iti, .....  model.  (a)  An  image compression network  is  trained \n4096 inputs into 40 hidden  units.  (b) The hiddenunits \n\n'.:\"\"llti1(\\lr1l\" are used as inputs to various recognition networks. \n\nEMPATH: \n\nFace, Emotion, and Gender \nRecognition Using Ho10ns \n\n. \n\nGarrison w. CottreU \nComputer Science and Engm~enng Dept \nInstitute for Neural Computati~n \nUniversity of California San Diego \nLa Jolla, CA 92093 \n\n. \n\nAbstract \n\nThe  dimens~onali~y of a  set Off  160 1:~ :a:~s ~~\u00b7.10 . \nfemale  subjects  IS  reduced \n........ . \nnetwork  The extracted features do not correspond to \nin previ~us face  recognition systems (KaR\u00b7 na~e, 19~;)y' ......\u2022\u2022.\u2022.. \nf~tures we  call  holons.  The  hol.ons  are fV~~ t~!  ..... . \n..  ......\\ \nd'  tances  between  facial  elements. \n\nat. er, \n\nrom \n\npropagation networks th.at are teamed  toc ~:~~~\nidentity.  feigned  emouonal  state. and  g.  f .. \nextracted  holons  provide  a.  suf~cient  basIS  or \ndiscriminations, 99% of the Idenuty \nemotion  discriminations ~mong the  traml: S~~d .. \n'ud  ements  of  the  emouons  are  compar  .' \n~et!orkS tend to confuse more distant emouons \n\n.' \n\n. t \n\n. \n\ny \n. ... \n\nd \n\nribe  further  research  on  the  use of \n\n1  Introduction and motivation \n:og:on first  described  in  (Cottrell  ~ Flemmg. \nThere, we demonstrated that an U~SUpe~lSed autoerIC~[llnl~\\ \nfeatures  from  faces  sufficient  for  Idenuty \nshow that a network so trained can also recognIZe \n\n. \n\n. \n\n564 \n\n\f566 \n\nCottrell and Metcalfe \n\nEMP ATH:  Face, Emotion, and Gender Recognition Using Holons \n\n567 \n\nEach  column  corresponds  to one of 8  different  feigned emotional  eXI)r~;sioins) \n(1980)  has shown that subjects' judgements of adjectives describing human \nbe  represented  in  a  two-dimensional  \"emotion  space\"  (Figure  3). \ndimension can  be characterized as  pleasure/displeasure;  the  vertical  <1lnllensiril \narousal/low  arousal.  Russell  and  his  colleagues  have  shown  using \nscaling  techniques  (RusseU,  1980,  1983;  Russell  &  Bullock,  1986)  that \nhuman emotions fallon a circle within this space.  We chose adjectives from \nbe the  emotions  that we  asked  our subjects  to feign.  The adjectives  for \nthose in  the numbered circles in Figure 3.  If the subject did  not respond \nthe adjectives, others from  the  circled region  were given as en<;ouragement \nproper facial expression.  We labeled  these classes with one adjective from \nastonished,  happy,  pleased, relaxed, sleepy, bored, miserable and angry. \nwere presented in randomized order to offset possible carry over effects. \n\nWe found  that subjects  were enthusiastically expressive  with certain  of \nsuch  as  astonished  and  delighted.  However,  despite claims  of negative \ncued  with  adjectives  such  as  miserable,  bored and  sleepy,  the  subjects \nexpress these negative emotions very clearly. \n\n3  Procedure \nThe whole  image is input at once to our network, so the input layer is \n40  hidden  units,  and  a  64x64  output  layer,  with  a  sigmoidal  activation \nrange  [-1,11.  Due  to  the  extreme  difference  in  fan-in  at  the  hidden  and \n(4096  vs.  40),  differential  learning  rates  were  used  at  the  two  layers. \nlearning rate led to most of the hidden units becoming pinned at full off or \n\nWe used a learning rate of .25 at the output layer during the first epoch  in \nlearn  the bias, or \"palette\",  then a rate of .1  was  used for the remaining \nwhere  an  epoch  corresponds  to  the  presentation  of all  160  images.  The \nused  a constant learning rate of .0001.  The initial weight span  was  .1  (+/_ \nno momentum or weight decay.  The average squared error per unit at the \n17.  This  corresponds  to  about  12  gray  levels  per  pixel.  Sample \nof trained images are shown in Figure 2. \n\nvectors  produced  by  the  hidden  units  of the compression  network  are \ninput to a single layer network that has a Iocalist unit for every name and a \nA two-layer network with 20 hidden  units is used for identifying which \nadjectives that were given  to  the subjects pertains  to each  image.  The \nto produce .5 for the wrong answer, and  1 for the correct answer.  The \nis trained for 1000 epochs, which reduces the total sum squared error to \nhow  performance  changed  with  further  training,  we  trained  this \nepochs.  9  other  networks  were  trained using  the features  from \nnetwork for  1000 epochs from different initial random weights for \n\n2000 \n. \nhuman subjects on the same task. \n\n.. \n\ncorrectness  was  that the output unit with  the maximum  activation  must \nthe  correct  answer.  The  network  learned  to  discriminate  99%  of the \nidentity.  One  image  of  one  woman  was  taken  for  another.  Sex \nwas  perfect.  It  was  found  that  performance  was . better  on  these  tasks \n\nlayer.  The emotion classification network performed better with a \n\nAIAIiMEO' \n\nFigure 2:  Three subjects and their reproductions by the cOInm:ess:ionl J' \n\nM ultidim('Mional  ~(':lIi\"~ !nrulion  (0C'\" 18 affect  ~Yords. \n\ntwo-dimensional  emotion space extracted via multi-dimensional scaling \nsimilarity ratings. Data from Russell (1980). \n\n\f568 \n\nCottrell and Metcalfe \n\nTable 1:  Percentage hits on each emotion (Generalization in prurenlthe:seSl \n\ndelighted \npleased \nrelaxed \nsleepy \nbored \nmiserable \n\nhidden  layer.  However,  the observation  during  data acquisition  that \nare poorly portrayed was confirmed by the network's early performance \n\nInitially,  the  network  was  much  better  at detecting  positive  states  than \nLater  training  improved  some categories  at the  expense of others,  .. \" ... \"'~~ .... \nnetwork did not have enough capacity to perform all the discriminations. \ntests were performed on a small set of 5 subjects (40 faces  in all), with the \nin parentheses in Table  1.  Generalization gets w~rse with more  ~in~ng, \nnetwork  is  becoming overtrained.  The network IS  best at generalization \nemotions.  This  also suggests  that the negative emotions are not easily \nolir data set.  The generalization  results,  while not spectacular,  should be  . \nthe light of the fact that thetrainirig set o~ly contain~ 20 SU?ject.s, and it \nthat the compression netwodc: was not tramed on the Images In thIS test set \n\n5  Internal representation \n\nWe  investigated  the representation  formed  by  the compression  network. \nreceptive fields of the hidden units in this network to be white noise.  In \nthe actual features used, we recorded the hidden unit activations as the \"plUlnrlt \nall  160 images.  We  formed  the  covariance  matrix  of the  hidden  unit \nextracted the principal components.  Note that this operation \ncomponents  from  the  distributed  representation  used.  T~e r~sulting \ndecompressed for viewing purposes.  The results are shown 10 Figure 4. \n\nFigure 4:  Sixteen botons derived by PCA from hidden unit re~,nol~se: \n\nEMPATH:  Face, Emotion, and Gender Recognition Using Holons \n\n569 \n\nsimilar to  the  \"eigenfaces\"  found  by  Turk  &  Pentland  (submitted)  in  their \ncomponents  analysis  of  faces.  Such  a  representation,  if  localized  by  a \nsuch as Sanger's  (1990),  or as  we have  found  in  previous  work  develops  at \nrates (Cottrell &  Fleming, 1990), could provide a computational basis for \ncell recordings  found in the STS of monkey cortex, without resorting \n\n\"grandmother\" cells for each face. \n\nwith human subjects \n\n.. \n\ncompare our network performance to that of human subjects on the same task, \nhuman  subjects  on  the  same  discriminations  the networks  were  required  to \n\u00b7\u00b7\u00b7 10 subjects were presented with one quarter of the training set (40 images of 5 \ntimes in randomized order in  each  block (320 presentations  total).  On  each \n.... \n\u00b7  of an  image, the subject was asked to  make a  Yes/No discrimination  as  to \n\nimage is a good example of one of the adjective class names. \n\n(small sample size, large heterogeneity of variance) prevented a reliable \ntest of the  model  vs. the subjects.  However,  it is informative to compare the \n.  ..  matrices  for  the  two pools  of subjects  (the  other 9  network  simulations  are \nhere).  All \"yes\" responses to each kind of face with each adjective are summed \n.. \nfor  the  humans  and  the  networks.  The  networks'  responses  were \n\"yes/no\"  responses  by thresholding the  outputs of the  networks for each \n8 yes/no responses.  The threshold was chosen to produce approximately \n\u2022.. \n. ..  overall number of \"yes\" responses for the 10 networks as the  10 humans.  The \nshown  in Table 2.  The rows correspond  to  the portrayals of the emotions, \n. .. \nare the adjectives presented with them.  So, for example, across  10 subjects \n45  instances of calling a \"pleased\" face a good example of \"delighted\" (out of \n\nthe tables  that there is a  lot of regularity  in the the  human subjects data \ntotally  captured  by  the  model.  The  first  three  emotions/adjectives  form  a \n\u00b7 do the last four.  Since these adjectives are listed in the order of a  tour around \n. \n\u00b7 \nof Russell's circomptex  model,  the confusability of nearby emotions \nthe clustering of descriptive adjectives is  matched by a perceptual clustering \n.. \n\u00b7 expressions produced by a subject in response to those adjectives.  However, \n\u00b7  diagonal band of confusabitity, as would be predicted by the circomplex, the \nne~ath,e dimension  appears  to  separate  into  two  clusters.  For example,  anger \nrure  separated more than  would be expected from Russell's circomplex \n.  is  no  \"wrap-around\"  in  this  matrix).  In  between  these  two  clusters,  the \nis  seen  by  the  subjects  as  compatible  with  nearly  every  facial \ndegree,  but the  \"relaxed\"  faces  rure  not compatible with the first three \n\n\u00b7 \n\nlIIUI,luna! categories. \n\nwhile  displaying  some of the  clustering  shown  in the human  data,  have \nalong  the  diagonals,  due  to  having  been  trained  on  this  data,  and  more \n()nlUSJ1ons  in regions where the human subjects (upper right and lower left) have \nentries, such as  \"angry\"  labels  on  \"delighted\"  and  \"pleased\"  faces.  This \nto  forcing  the  netwodc:s  to  make  as  many  responses  as  the  humans.  We \nminor threshold change leads  to  many more responses,  suggesting that we \n\nuractll1lO' responses from  the network. \n\n\f570 \n\nCottrell and Metcalfe \n\nEMPATH:  Face, Emotion, and Gender RecognitioriUshlg Hol()l:ts \n\n571 \n\nTable 2:  Confusion matrices for human and network subjects \n\nseveral category confusions that humans do not. \n\nastonished \ndelighted \npleased \nrelaxed \nsleepy \nbored \n\n1 \n24 \n6 \n6  29 \n2  45  48  18 \n0 \n1  0  7  42  22  32 \no  0  0  28  31  33 \no  0  1  33  24  38 \no  0  1  24  17 \n\n5  14  13 \n27  7  8 \n50  1  3 \n2  46  43 \n2  39  49 \n15  26  21 \n9  3  18 \n\n7  Holons \nThis work demonstrates that, at least for our data set, dimensionality reduction \npreprocessing step that can maintain enough information for the recognition \nterm  the representational units used by  the compression network \"holons\". \nthan  just another  name  for  a  distributed representation.  By  this  we  simply \nrepresentational element is a holon  if its receptive field  sub tends  the  whole \nrepresented.  Ideally  we  want  to  require  that  the  infonnation  in  a  set  of \nmaximally distributed:  i.e., the average unit entropy is maximized.  The latter \neliminates grandmother cells,  insures that the  representation be noise  resls~mt a \ndistributes the processing Joadeventy.  A weak point of our definition is the \ndefining precisely the notion of a \"whole object\". \nThis definition applies to many distributed representational  schemes, but does  .. \nto articulated ones such as the Wickelfeatures used by Rumelhart and \n. \nin their past tense model as these only represent portions of the verb.  On the \nwe  would not have holons for a \"room\", simply because we can  not get a \nnot  extend  beyond  our  sensory  surface  at once.  Given  this  meaning  for \nunits of area 17 are not holons, but the units in Superior Temporal Sulcus \nmain motivation for this definition is to give an alternative notion to the ...... ntir .... \none for face cells in STS (Desimone et al., 1984). \n\n8  Conclusions \nWe  have  shown  that a network  model  that extracts  features  from  its  \"n\"irr,rim \nunsupervised  manner  can  achieve  near  perfect  recognition  rales \ndiscrimination  and  sex  discrimination,  even  though  the  features  were  not \nthat purpose.  Where categories become \"fuzzier\", as  in  emotional states, \nabilities  are  also  limited.  In  particular,  generali7..ation  to  new  faces  is \npreliminary study of human perception of these faces, we found  support for \nwhen subjects are asked to produce expressions based on \"near\"  aUII\"\"\",'L.n~~.,,,\u00b7 .. , \nspace,  they  produce  \"near\"  expressions  in  perceptual  space.  These \npositive/negative  clusters  much  more  than  the  circomplex  model \nHowever,  this  could  be  a  fault of the  subjects'  abilities  to  portray  the \nrather  than  a  fault  of  the  Circomplex  model.  Finally,  we  cornoatred \nperfonnance to that of humans.  We found  that the networks (when  l\"n\"C!tr<1lin,~rl \nas  many responses as humans), while generally  following  the pattern of the \n\nG.  &  Fleming,  M.  (1990).  Face  recognition  using  unsupervised  feature \nIn Proceedings of the International Neural Network Conference, Paris. \n\n:,  ~ .. :. : \n\n'0, Munro,  P.  &  Zipser,  D.  (1987)~ Learning  internal  representations  of gray \n\u00b7!.;o..ft~~,~,  An example of extensional programming.  In Proc. Ninth Annual Cognitive \n\nConference, Seattle, Wa. \n\n\u00b7\u00b7 G.W.  and Munro, P.  (1988)  Principal components analysis of images via back \n\nIn Proc. Soc. of Photo-Optical Instr. Eng., Cambridge, MA. \n\nR.,  Albright.  T.,  Gross,  C.,  and  Bruce,  C.  (1984).  Stimulus-selective \n\nof inferior temporal neurons in the Macaque.  J. Neuroscience. 4, 2051-2062. \n\n. \u00b7\u00b7 M.  &  Cottrell,  G.  (1990).  A  neural  network  model  of  face  recognition.  In \n\nof the Int. Joint Con/. on Neural Networks, San Diego, CA. \n\n. R. S.  (1878).  Composite Portraits.  Nature, 23. 97-100. \n\n'rakeo (1973). Picture processing  system by computer complex and recognition \nfaces.  Unpublished Ph.D. Thesis, Dept. of Info. Science, Kyoto University. \nt. Lehtio, P., Oja, E., Kortekangas, A., & Makisara, K.  (1977).  Demoristralion \nprocessing properties of the optimal associative mappings.  In Proc Inti.  Conj. \n\n\u00b7\"'\"r, .. \",ti,,. .. and Society, Wash., D.C. \n.... 1. A.  (1980).  A circomplex  model  of affect.  Journal  of Personality and Social \n\n39,1161-1178. \n\n:. \n\n. \n\n..  J.  A.  (1983).  Pancultural  aspects  of  the  human  conceptual  organi7..ation  of \n. '''.''''Pf.nI of Personality and Social Psychology, 45, 1281-1288. \n\nA.  &  Bullock,  M.  (1986).  On  the  dimensions  preschoolers  use  to  interpret \n\n~xDressilonsofemotion. Developmental Psychology. 22,97-102. \n\n. D. E., Hinton, G. E., and Williams, R. J.  (1986).  Learning repreSentations by \n\nropa~atin~ errors.  Nature, 323, 533-536. \n\n. \n\n. \n\nD~ &  McClelland,  J.  On  learning  the  past  tenses  of English  verbs.  In  J.L. \n.... &  D.E.  Rumelhart  (Eds.),  Parallel  Distributed  Processing.  Vol  2., \n\u00b7 MA: MIT Press. \n\n(1989) ()ptimal unsupervised learning iri a single-layer linear feMf()rward \n\nnet'worlc  Neural Networks, 2, pp. 459-473. \n\nPentland, A. (1990) Eigenfaces for recognition.  (Submitted for publiCation). \n\n\f", "award": [], "sourceid": 361, "authors": [{"given_name": "Garrison", "family_name": "Cottrell", "institution": null}, {"given_name": "Janet", "family_name": "Metcalfe", "institution": null}]}