{"title": "Categorization by Learning and Combining Object Parts", "book": "Advances in Neural Information Processing Systems", "page_first": 1239, "page_last": 1245, "abstract": null, "full_text": "Categorization by Learning\nand Combining Object Parts\n\nBernd Heisele\u0002\u0001 Thomas Serre Massimiliano Pontil\u0003 Thomas Vetter\n\n Center for Biological and Computational Learning, M.I.T., Cambridge, MA, USA\n\nTomaso Poggio\n\n\u0001 Honda R&D Americas, Inc., Boston, MA, USA\n\n\u0003 Department of Information Engineering, University of Siena, Siena, Italy\n\nComputer Graphics Research Group, University of Freiburg, Freiburg, Germany\n\n\u0005 heisele,serre,tp\n\n@ai.mit.edu pontil@ing.unisi.it\n\nvetter@informatik.uni-freiburg.de\n\nAbstract\n\nWe describe an algorithm for automatically learning discriminative com-\nponents of objects with SVM classi\ufb01ers. It is based on growing image\nparts by minimizing theoretical bounds on the error probability of an\nSVM. Component-based face classi\ufb01ers are then combined in a second\nstage to yield a hierarchical SVM classi\ufb01er. Experimental results in face\nclassi\ufb01cation show considerable robustness against rotations in depth and\nsuggest performance at signi\ufb01cantly better level than other face detection\nsystems. Novel aspects of our approach are: a) an algorithm to learn\ncomponent-based classi\ufb01cation experts and their combination, b) the use\nof 3-D morphable models for training, and c) a maximum operation on\nthe output of each component classi\ufb01er which may be relevant for bio-\nlogical models of visual recognition.\n\n1 Introduction\n\nWe study the problem of automatically synthesizing hierarchical classi\ufb01ers by learning dis-\ncriminative object parts in images. Our motivation is that most object classes (e.g. faces,\ncars) seem to be naturally described by a few characteristic parts or components and their\ngeometrical relation. Greater invariance to viewpoint changes and robustness against par-\ntial occlusions are the two main potential advantages of component-based approaches com-\npared to a global approach.\n\nThe \ufb01rst challenge in developing component-based systems is how to choose automatically\na set of discriminative object components. Instead of manually selecting the components,\nit is desirable to learn the components from a set of examples based on their discriminative\npower and their robustness against pose and illumination changes. The second challenge is\nto combine the component-based experts to perform the \ufb01nal classi\ufb01cation.\n\n\u0004\n\u0004\n\u0006\n\f2 Background\n\nGlobal approaches in which the whole pattern of an object is used as input to a single\nclassi\ufb01er were successfully applied to tasks where the pose of the object was \ufb01xed. In [6]\nHaar wavelet features are used to detect frontal and back views of pedestrians with an SVM\nclassi\ufb01er. Learning-based systems for detecting frontal faces based on a gray value features\nare described in [14, 13, 10, 2].\n\nComponent-based techniques promise to provide more invariance since the individual com-\nponents vary less under pose changes than the whole object. Variations induced by pose\nchanges occur mainly in the locations of the components. A component-based method for\ndetecting faces based on the empirical probabilities of overlapping rectangular image parts\nis proposed in [11]. Another probabilistic approach which detects small parts of faces is\nproposed in [4]. It uses local feature extractors to detect the eyes, the corner of the mouth,\nand the tip of the nose. The geometrical con\ufb01guration of these features is matched with\na model con\ufb01guration by conditional search. A related method using statistical models is\npublished in [9]. Local features are extracted by applying multi-scale and multi-orientation\n\ufb01lters to the input image. The responses of the \ufb01lters on the training set are modeled as\nGaussian distributions. In [5] pedestrian detection is performed by a set of SVM classi\ufb01ers\neach of which was trained to detect a speci\ufb01c part of the human body.\n\nIn this paper we present a technique for learning relevant object components. The technique\nstarts with a set of small seed regions which are gradually grown by minimizing a bound\non the expected error probability of an SVM. Once the components have been determined,\nwe train a system consisting of a two-level hierarchy of SVM classi\ufb01ers. First, component\nclassi\ufb01ers independently detect facial components. Second, a combination classi\ufb01er learns\nthe geometrical relation between the components and performs the \ufb01nal detection of the\nobject.\n\n3 Learning Components with Support Vector Machines\n\n3.1 Linear Support Vector Machines\n\n\u000b\u0011\u0010\u0012\u000b\u0014\u0013\n\u000b! \n\n\u0003\u0018\u0017\u001a\u0019\u001c\u001b\u001e\u001d\n\nis the class label of the data point\n\n(1)\n\n(2)\n\n(3)\n\nis the distance of the support vectors to the hyperplane, it is given by:\n\nLinear SVMs [15] perform pattern recognition for two-class problems by determining the\nseparating hyperplane with maximum distance to the closest points in the training set.\nThese points are called support vectors. The decision function of the SVM has the form:\n\n\u000b\u0016\u0015\n\u0002\u0001\u0004\u0003\u0006\u0005\b\u0007\n\f\u0006\r\u000f\u000e\nis the number of data points and\u0010\n\u000b are the solution of a quadratic programming problem. The margin\n\u000b . The coef\ufb01cients\u000e\n\u0005#\"%$\nwhere\u001f\n\u000b\u0012*\nbility of the SVM,+-,\u0002.0/1/ , satis\ufb01es the following bound [15]:\n+87:9<;\n;>=\nwhere9\n\nThe margin is an indicator of the separability of the data. In fact, the expected error proba-\n\nis the diameter of the smallest sphere containing all data points in the training set.\n\n+2,3.4/5/%6\n\n\t\n\n\u000b\n\u0003\n\u001d\n$\n\u0006\n&\n'\n'\n\u0007\n$\n(\n)\n\t\n\u000b\n\u000e\n$\n\u001f\n'\n\u001d\n\f3.2 Learning Components\n\nOur method automatically determines rectangular components from a set of object images.\nThe algorithm starts with a small rectangular component located around a pre-selected\npoint in the object image (e.g. for faces this could be the center of the left eye). The com-\nponent is extracted from each object image to build a training set of positive examples. We\nalso generate a training set of background patterns that have the same rectangular shape as\nthe component. After training an SVM on the component data we estimate the performance\nof the SVM based on the upper bound on the error probability. According to Eq. (3) we\ncalculate:\n\nAs shown in [15] this quantity can be computed by solving a quadratic programming prob-\nlem. After determining  we enlarge the component by expanding the rectangle by one\npixel into one of the four directions (up, down, left, right). Again, we generate training\ndata, train an SVM and determine  . We do this for expansions into all four directions\nand \ufb01nally keep the expansion which decreases \nthe most. This process is continued until\n. In order to learn a set of\nthe expansions into all four directions lead to an increase of \ncomponents this process can be applied to different seed regions.\n\n(4)\n\n9<;\n\n4 Learning Facial Components\n\nExtracting face patterns is usually a tedious and time-consuming work that has to be done\nmanually. Taking the component-based approach we would have to manually extract each\nsingle component from all images in the training set. This procedure would only be feasible\nfor a small number of components. For this reason we used textured 3-D head models [16]\nto generate the training data. By rendering the 3-D head models we could automatically\ngenerate large numbers of faces in arbitrary poses and with arbitrary illumination. In ad-\ndition to the 3-D information we also knew the 3-D correspondences for a set of reference\npoints shown in Fig. 1a). These correspondences allowed us to automatically extract facial\ncomponents located around the reference points. Originally we had seven textured head\nmodels acquired by a 3-D scanner. Additional head models were generated by 3-D morph-\n\ning between all pairs of the original head models. The heads were rotated between\"\u0002\u0001\u0004\u0003\u0006\u0005\npointing towards the center of the face. The position of the light varied between\"\u0002\u0001\u0004\u0003\b\u0005 and\n\nin elevation. Overall, we generated 2,457 face\n58. Some examples of synthetic face images used for training are shown\n\nand \u0001\u0007\u0003\n\u0001\u0004\u0003\t\u0005\nimages of size 58\nin Fig. 1b).\n\nin depth. The faces were illuminated by ambient light and a single directional light\n\nin azimuth and between \u0001\u0007\u0003\t\u0005 and\n\n58 non-face patterns randomly\nThe negative training set initially consisted of 10,209 58\nextracted from 502 non-face images. We then applied bootstrapping to enlarge the training\ndata by non-face patterns that look similar to faces. To do so we trained a single linear\nSVM classi\ufb01er and applied it to the previously used set of 502 non-face images. The false\npositives (FPs) were added to the non-face training data to build the \ufb01nal training set of\nsize 13,654.\n\n\u0003\u0007\u0005\n\nWe started with fourteen manually selected seed regions of size 5\nponents were located around the eyes (17\n17 pixels), the nose (15\n(31\npixels), the corners of the mouth (18\nbridge of the nose (18\n\n15 pixels), the cheeks (21\n\n20 pixels), the lip (13\n\n11 pixels), the eyebrows (19\n\n16 pixels).\n\n5. The resulting com-\n20 pixels), the mouth\n\n16 pixels), the nostrils (\n\n15 pixels), and the\n\n\f\u0004\f\r\u000b\n\n\n\u0007\n'\n;\n*\n\u0005\n\n\u000b\n\u000b\n\u000b\n\u000b\n\u000b\n\u000b\n\u000b\n\u000b\n$\n\f\n\u000b\n\u000b\n\u000b\n\fa)\n\nb)\n\nFigure 1: a) Reference points on the head models which were used for 3-D morphing and\nautomatic extraction of facial components. b) Examples of synthetic faces.\n\n5 Combining Components\nAn overview of our two-level component-based classi\ufb01er is shown in Fig. 2. On the \ufb01rst\nlevel the component classi\ufb01ers independently detect components of the face. Each classi-\n\ufb01er was trained on a set of facial components and on a set of non-face patterns generated\nfrom the training set described in Section 4. On the second level the combination classi-\n\ufb01er performs the detection of the face based on the outputs of the component classi\ufb01ers.\nThe maximum real-valued outputs of each component classi\ufb01er within rectangular search\nregions around the expected positions of the components are used as inputs to the combina-\ntion classi\ufb01er. The size of the search regions was estimated from the mean and the standard\ndeviation of the locations of the components in the training images. The maximum op-\neration is performed both during training and at run-time. Interestingly it turns out to be\nsimilar to the key pooling mechanism postulated in a recent model of object recognition in\nthe visual cortex [8]. We also provide the combination classi\ufb01er with the precise positions\n58 window. Overall\nof the detected components relative to the upper left corner of the 58\nwe have three values per component classi\ufb01er that are propagated to the combination clas-\nimage coordinates of\n\nsi\ufb01er: the maximum output of the component classi\ufb01er and the& -\u0010\n\nthe maximum.\n\n*Outputs of component\nexperts: bright intensities\nindicate high confidence.\n\n*\n\nLeft Eye\nLeft Eye\nexpert:\nexpert:\n\nLinear SVM\nLinear SVM\n\n.\n\n.\n\n.\n\nNose expert:\nNose expert:\nLinear SVM\nLinear SVM\n\n.\n\n.\n\n.\n\nMouth\nMouth\nexpert:\nexpert:\n\nLinear SVM\nLinear SVM\n\n1. Shift 58x58 window\nover input image\n\n2. Shift component\nexperts over\n58x58 window\n\n(\nYXO\n1\n\n,\n\n,\n\n1\n\n1\n\n)\n\n(\nYXO\n1\n\n,\n\n,\n\n1\n\n1\n\n,...,\n\nYXO\n\n,\n\n,\n\n14\n\n14\n\n14\n\n)\n\nCombination\nCombination\n\nclassifier:\nclassifier:\nLinear SVM\nLinear SVM\n\n)\n\n(\nYXO\nk\n\n,\n\n,\n\nk\n\nk\n\n*\n\n*\n\n(\nYXO\n\n,\n\n,\n\n14\n\n14\n\n14\n\n3. For each component k,\ndetermine its maximum\noutput within a search\nregion and its location:\n(\nYXO\nk\n\n)\n\n,\n\n,\n\nk\n\nk\n\n)\n\n4. Final decision:\nface / background\n\nFigure 2: System overview of the component-based classi\ufb01er.\n\n\u000b\n\f6 Experiments\n\ntest set consisted of 1,834 faces rotated between about\"\u0002\u0001\u0007\u0003\n\nIn our experiments we compared the component-based system to global classi\ufb01ers. The\ncomponent system consisted of fourteen linear SVM classi\ufb01ers for detecting the compo-\nnents and a single linear SVM as combination classi\ufb01er. The global classi\ufb01ers were a single\nlinear SVM and a single second-degree polynomial SVM both trained on the gray values\nof the whole face pattern. The training data for these three classi\ufb01ers consisted of 2,457\nsynthetic gray face images and 13,654 non-face gray images of size 58\n58. The positive\nin depth. The faces\nwere manually extracted from the CMU PIE database [12]. The negative test set consisted\nof 24,464 dif\ufb01cult non-face patterns that were collected by a fast face detector [3] from\nweb images. The FP rate was calculated relative to the number of non-face test images.\nBecause of the resolution required by the component-based system, a direct comparison\nwith other published systems on the standard MIT-CMU test set [10] was impossible. For\nan indirect comparison, we used a second-degree polynomial SVM [2] which was trained\non a large set of 19\n19 real face images. This classi\ufb01er performed amongst the best face\ndetection systems on the MIT-CMU test set. The ROC curves in Fig. 3 show that the\ncomponent-based classi\ufb01er is signi\ufb01cantly better than the three global classi\ufb01ers. Some\ndetection results generated by the component system are shown in Fig. 4.\n\n\u0005 and \u0001\u0007\u0003\u0007\u0005\n\nFigure 3: Comparison between global classi\ufb01ers and the component-based classi\ufb01er.\n\nFigure 4: Faces detected by the component-based classi\ufb01er.\n\n\u000b\n\u000b\n\fA natural question that arises is about the role of geometrical information. To answer this\nquestion\u2013which has relevant implications for models of cortex\u2013we tested another system in\nwhich the combination classi\ufb01er receives as inputs only the output of each component clas-\nsi\ufb01er but not the position of its maximum. As shown in Fig. 5 this system still outperforms\nthe whole face systems but it is worse than the system with position information.\n\nFigure 5: Comparison between a component-based classi\ufb01er trained with position infor-\nmation and a component-based classi\ufb01er without position information.\n\n7 Open Questions\n\nAn extension under way of the component-based approach to face identi\ufb01cation is already\nshowing good performances [1]. Another natural generalization of the work described here\ninvolves the application of our system to various classes of objects such as cars, animals,\nand people. Still another extension regards the question of view-invariant object detec-\ntion. As suggested by [7] in a biological context and demonstrated recently by [11] in\nmachine vision, full pose invariance in recognition tasks can be achieved by combining\nview-dependent classi\ufb01ers. It is interesting to ask whether the approach described here\ncould also be used to learn which views are most discriminative and how to combine them\noptimally. Finally, the role of geometry and in particular how to compute and represent\nposition information in biologically plausible networks, is an important open question at\nthe interface between machine and biological vision.\n\nReferences\n\n[1] B. Heisele, P. Ho, and T. Poggio. Face recognition with support vector machines:\nglobal versus component-based approach. In Proc. 8th International Conference on\nComputer Vision, Vancouver, 2001.\n\n[2] B. Heisele, T. Poggio, and M. Pontil. Face detection in still gray images. A.I.\nmemo 1687, Center for Biological and Computational Learning, MIT, Cambridge,\nMA, 2000.\n\n[3] B. Heisele, T. Serre, S. Mukherjee, and T. Poggio. Feature reduction and hierarchy\nof classi\ufb01ers for fast object detection in video images. In Proc. IEEE Conference on\nComputer Vision and Pattern Recognition, Hawaii, 2001.\n\n\f[4] T. K. Leung, M. C. Burl, and P. Perona. Finding faces in cluttered scenes using\nIn Proc. International Conference on Computer\n\nrandom labeled graph matching.\nVision, pages 637\u2013644, Cambridge, MA, 1995.\n\n[5] A. Mohan, C. Papageorgiou, and T. Poggio. Example-based object detection in im-\nages by components. In IEEE Transactions on Pattern Analysis and Machine Intelli-\ngence, volume 23, pages 349\u2013361, April 2001.\n\n[6] C. Papageorgiou and T. Poggio. A trainable system for object detection. In Interna-\n\ntional Journal of Computer Vision, volume 38, 1, pages 15\u201333, 2000.\n\n[7] T. Poggio and S. Edelman. A network that learns to recognize 3-D objects. Nature,\n\n343:163\u2013266, 1990.\n\n[8] M. Riesenhuber and T. Poggio. Hierarchical models of object recognition in cortex.\n\nNature Neuroscience, 2(11):1019\u20131025, 1999.\n\n[9] T. D. Rikert, M. J. Jones, and P. Viola. A cluster-based statistical model for object\ndetection. In Proc. IEEE Conference on Computer Vision and Pattern Recognition,\nvolume 2, pages 1046\u20131053, Fort Collins, 1999.\n\n[10] H. A. Rowley, S. Baluja, and T. Kanade. Rotation invariant neural network-based face\ndetection. Computer Science Technical Report CMU-CS-97-201, CMU, Pittsburgh,\n1997.\n\n[11] H. Schneiderman and T. Kanade. A statistical method for 3d object detection ap-\nplied to faces and cars. In Proc. IEEE Conference on Computer Vision and Pattern\nRecognition, pages 746\u2013751, 2000.\n\n[12] T. Sim, S. Baker, and M. Bsat. The CMU pose, illumination, and expression (PIE)\n\ndatabase of human faces. Computer Science Technical Report 01-02, CMU, 2001.\n\n[13] K.-K. Sung. Learning and Example Selection for Object and Pattern Recognition.\nPhD thesis, MIT, Arti\ufb01cial Intelligence Laboratory and Center for Biological and\nComputational Learning, Cambridge, MA, 1996.\n\n[14] R. Vaillant, C. Monrocq, and Y. Le Cun. An original approach for the localisation of\nobjects in images. In International Conference on Arti\ufb01cial Neural Networks, pages\n26\u201330, 1993.\n\n[15] V. Vapnik. Statistical learning theory. John Wiley and Sons, New York, 1998.\n[16] T. Vetter. Synthesis of novel views from a single face.\n\nInternational Journal of\n\nComputer Vision, 28(2):103\u2013116, 1998.\n\n\f", "award": [], "sourceid": 2109, "authors": [{"given_name": "Bernd", "family_name": "Heisele", "institution": null}, {"given_name": "Thomas", "family_name": "Serre", "institution": null}, {"given_name": "Massimiliano", "family_name": "Pontil", "institution": null}, {"given_name": "Thomas", "family_name": "Vetter", "institution": null}, {"given_name": "Tomaso", "family_name": "Poggio", "institution": null}]}