{"title": "Using Manifold Stucture for Partially Labeled Classification", "book": "Advances in Neural Information Processing Systems", "page_first": 953, "page_last": 960, "abstract": null, "full_text": "Using Manifold Structure for Partially \n\nLabelled Classification \n\nMikhail B elkin \n\nUniversity of Chicago \n\nDepartment of Mathematics \nmisha@math .uchicago .edu \n\nPartha Niyogi \n\nUniversity of Chicago \n\nDepts of Computer Science and Statistics \n\nniyogi@cs.uchicago .edu \n\nAbstract \n\nWe consider the general problem of utilizing both labeled and un(cid:173)\nlabeled data to improve classification accuracy. Under t he assump(cid:173)\ntion that the data lie on a submanifold in a high dimensional space, \nwe develop an algorithmic framework to classify a partially labeled \ndata set in a principled manner . The central idea of our approach is \nthat classification functions are naturally defined only on t he sub(cid:173)\nmanifold in question rather than the total ambient space. Using the \nLaplace Beltrami operator one produces a basis for a Hilbert space \nof square integrable functions on the submanifold. To recover such \na basis , only unlab eled examples are required. Once a basis is ob(cid:173)\ntained , training can be performed using the labeled data set. Our \nalgorithm models the manifold using the adjacency graph for the \ndata and approximates the Laplace Beltrami operator by the graph \nLaplacian. Practical applications to image and text classification \nare considered. \n\n1 \n\nIntroduction \n\nIn many practical applications of data classification and data mining , one finds a \nwealth of easily available unlabeled examples , while collecting labeled examples can \nbe costly and time-consuming . Standard examples include object recognition in im(cid:173)\nages, speech recognition, classifying news articles by topic. In recent times , genetics \nhas also provided enormous amounts of readily accessible data. However, classi(cid:173)\nfication of this data involves experimentation and can be very resource intensive. \nConsequently it is of interest to develop algorithms that are able to utilize both \nlabeled and unlabeled data for classification and other purposes. Although the area \nof partially labeled classification is fairly new, a considerable amount of work has \nbeen done in that field since the early 90 's , see [2, 4, 7]. In this paper we address \nthe problem of classifying a partially labeled set by developing the ideas proposed \nin [1] for data representation. In particular , we exploit the intrinsic structure of \nthe data to improve classification with unlab eled examples under the assumption \n\n\fthat the data resides on a low-dimensional manifold within a high-dimensional rep(cid:173)\nresentation space. In some cases it seems to be a reasonable assumption that the \ndata lies on or close to a manifold. For example a handwritten digit 0 can be \nfairly accurately represented as an ellipse , which is completely determined by the \ncoordinates of its foci and the sum of the distances from the foci to any point. \nThus the space of ellipses is a five-dimensional manifold. An actual handwritten 0 \nwould require more parameters, but perhaps not more than 15 or 20. On the other \nhand the dimensionality of the ambient representation space is the number of pixels \nwhich is typically far higher. For other types of data the question of the manifold \nstructure seems significantly more involved. While there has been recent work on \nusing manifold structure for data representation ([6, 8]), the only other application \nto classification problems that we are aware of, was in [7] , where the authors use a \nrandom walk on the data adjacency graph for partially labeled classification. \n\n2 Why Manifold Structure IS Useful for Partially \n\nSupervised Learning \n\nTo provide a motivation for using a manifold structure, consider a simple synthetic \nexample shown in Figure l. The two classes consist of two parts of the curve shown \nin the first panel (row 1). We are given a few labeled points and 500 unlabeled \npoints shown in panels 2 and 3 respectively. The goal is to establish the identity of \nthe point labeled with a question mark. By observing the picture in panel 2 (row 1) \nwe see that we cannot confidently classify\"?\" by using the labeled examples alone. \nOn the other hand, the problem seems much more feasible given the unlabeled \ndata shown in panel 3. Since there is an underlying manifold, it seems clear at \nthe outset that the (geodesic) distances along the curve are more meaningful than \nEuclidean distances in the plane. Therefore rather than building classifiers defined \non the plane (lR 2) it seems preferable to have classifiers defined on the curve itself. \nEven though the data has an underlying manifold, the problem is still not quite \ntrivial since the two different parts of the curve come confusingly close to each \nother. There are many possible potential representations of the manifold and the \none provided by the curve itself is unsatisfactory. Ideally, we would like to have a \nrepresentation of the data which captures the fact that it is a closed curve. More \nspecifically, we would like an embedding of the curve where the coordinates vary \nas slowly as possible when one traverses the curve. Such an ideal representation \nis shown in the panel 4 (first panel of the second row). Note that both represent \nthe same underlying manifold structure but with different coordinate functions. It \nturns out (panel 6) that by taking a two-dimensional representation of the data \nwith Laplacian Eigenmaps [1] , we get very close to the desired embedding. Panel 5 \nshows the locations of labeled points in the new representation space. We see that \n\"?\" now falls squarely in the middle of \"+\" signs and can easily be identified as a \n\"+\". \nThis artificial example illustrates that recovering the manifold and developing clas(cid:173)\nsifiers on the manifold itself might give us an advantage in classification problems. \nTo recover the manifold, all we need is unlab eled data. The labeled data is then \nused to develop a classifier defined on this manifold. However we need a model for \nthe manifold to utilize this structure. The model used here is that of a weighted \ngraph whose vertices are data points. Two data points are connected with an edge if \n\n\f3 \n\n2 \n\nr--\" \n\n0 ~\\-) \n\n- 1 \n\n-2 \n\n-3 \n\n-2 \n\n2 \n\n'''',-) \n\n2 \n\n? 00 \na \n+8 \n\n- 1 \n\n- 1 \n\n-2 \n\n-3 \n\n-2 \n\n-2 \n\n+ \n\n& to \n0.5 0 \n\n-0.1 \n\n-3 \n\n-2 \n\n-1 \n\n-1 \n\n0 \n\n0 \n\n-0.5 \n\n?+-++- a C) \n\n<a \n. \n+ + C\u00a7P \n\n0 \n\n0.1 \n\n-0.1 \n\n0 .1 \n\nFigure 1: Top row: Panel l. Two classes on a plane curve. Panel 2. Labeled \nexamples. \"?\" is a point to be classified. Panel 3. 500 random unlab eled examples. \nBottom row: Panel 4. \nIdeal representation of the curve. Panel 5. Positions of \nlabeled points and \"?\" after applying eigenfunctions of the Laplacian. Panel 6. \nPositions of all examples. \n\nand only if the points are sufficiently close. To each edge we can associate a distance \nbetween the corresponding points. The \"geodesic distance\" between two vertices is \nthe length of the shortest path between them on the adjacency graph. Once we set \nup an approximation to the manifold, we need a method to exploit the structure \nof the model to build a classifier. One possible simple approach would be to use \nthe \"geodesic nearest neighbors\" . However , while simple and well-motivated , this \nmethod is potentially unstable. A related more sophisticated method based on a \nrandom walk on the adjacency graph is proposed in [7]. We also note the approach \ntaken in [2] which uses mincuts of certain graphs for partially labeled classifications. \n\nOur approach is based on the Laplace-Beltrami operator defined on Riemannian \nmanifolds (see [5]). The eigenfunctions of the Laplace Beltrami operator provide a \nnatural basis for functions on the manifold and the desired classification function \ncan be expressed in such a basis. The Laplace Beltrami operator can be estimated \nusing unlabeled examples alone and the classification function is then approximated \nusing the labeled data. In the next two sections we describ e our algorithm and the \ntheoretical underpinnings in some detail. \n\n3 Description of the Algorithm \n\nGiven k points X l, . . . , X k E IR I , we assume that the first s < k points have labels \nCi, where Ci E {- I, I} and the rest are unlab eled. The goal is to label th e unlab eled \npoints. We also introduce a straightforward extension of the algorithm for the case \nof more than two classes. \n\nStep 1 [Constru cting the Adja cenc y Graph with n nearest neighbors]. Nodes i and \n\n\fj corresponding to the points Xi and Xj are connected by an edge if i is among n \nnearest neighbors of j or j is among n nearest neighbors of i. The distance can be \nthe standard Euclidean distance in II{ I or some other appropriately defined distance. \nWe take Wij = 1 if points Xi and Xj are connected and Wij = 0 otherwise. For a \ndiscussion about the appropriate choice of weights, and connections to the heat \nkernel see [1]. \n\nStep 2. [Eigenfunctions] Compute p eigenvectors e1 , ... , e p corresponding to the p \nsmallest eigenvalues for the eigenvector problem Le = Ae where L = D - W is the \ngraph Laplacian for the adjacency graph. Here W is the adjacency matrix defined \nabove and D is a diagonal matrix of the same size as W satisfying Dii = 2::j Wij. \nLaplacian is a symmetric , positive semidefinite matrix which can be thought of as \nan operator on functions defined on vertices of the graph . \n\nStep 3. [Building the classifier] To approximate the class we minimize t he error func(cid:173)\ntion Err(a) = 2:::=1 (Ci - 2::~=1 ajej(i)) 2 where p is the number of eigenfunctions \nwe wish to employ, the sum is taken over all labeled points and the minimization is \nconsidered over the space of coefficients a = (a1' ... , apf. The solution is given by \n\na = E 1ab Elab \n\n( T \n\n)-1 T \n\nE 1ab C \n\nwhere c = (C1 , ' .. ,Cs f and El ab is an s x p matrix whose i, j entry is ej (i). For \nthe case of several classes , we build a one-against-all classifier for each individual \nclass. \nStep 4. [Classifying unlabeled points] If Xi, i > s is an unlabeled point we put \n\n{\n\nCi = \n\nI , \n-1 , \n\nThis, of course, is just applying a linear classifier constructed in Step 3. If there are \nseveral classes , one-against-all classifiers compete using 2::~ =1 aj ej (i) as a confidence \nmeasure. \n\n4 Theoretical Interpretation \n\nLet M C \nII{ k be an n-dimensional compact Riemannian manifold isometrically \nembedded in II{ k for some k. Intuitively M can be thought of as an n-dimensional \n\"surface\" in II{ k. Riemannian structure on M induces a volume form that allows \nus to integrate functions defined on M. The square integrable functions form a \nHilbert space .c2(M). The Laplace-Beltrami operator 6.M (or just 6.) acts on \ntwice differentiable functions on M. There are three important points that are \nrelevant to our discussion here. \nThe Laplacian provides a basis on .c 2(M): \nIt can be shown (e.g. , [5]) that 6. is a self-adjoint positive semidefinite operator and \nthat its eigenfunctions form a basis for the Hilbert space .c 2(M) . The spectrum of 6. \nis discret e (provided M is compact) , with the smallest eigenvalue 0 corresponding \nto the constant eigenfunction. Therefore any f E .c 2(M) can be written as f(x) = \n2::~o ai ei(x) , where ei are eigenfunctions, 6. ei = Ai ei. \n\nThe simplest nontrivial example is a circle Sl. 6. S1 f( \u00a2) \n\n- d'li,</\u00bb . Therefore \n\n\fthe eigenfunctions are given by - d:121\u00bb = e( if;), where I( if;) is a 7r-periodic func(cid:173)\ntion. It is easy to see that all eigenfunctions of 6. are of the form e( if;) = sin( nif; ) \nor e( if;) = cos( nif;) with eigenvalues {l2, 22, ... }. Therefore, we see that any 7r(cid:173)\nperiodic \u00a32 function 1 has a convergent Fourier series expansion given by I( if;) = \n2::~= o an sin( nif; ) + bn cos( nif;). In general, for any manifold M , the eigenfunctions \nof the Laplace-Beltrami operator provide a natural basis for \u00a32(M). However 6. \nprovides more than just a basis , it also yields a measure of smoothness for functions \non the manifold. \n\nThe Laplacian as a snlOothness functional: \nA simple measure of the degree of smoothness for a function 1 on a unit circle 51 is \nthe \"smoothness functional\" S(J) = J I/( if;)' 12dif;. If S(J) is close to zero, we think \nof 1 as being \"smooth\" . Naturally, constant functions are the most \"smooth\" . \nIntegration by parts yields S(J) \nIn \ngeneral , if I: M ----+ ~, then \n\nJ 16.ldif; = (6./,1)\u00a3.2(51)' \n\nJ f'( if; )dl \n\n5' \n\n5' \n\n5' \n\nS(J) d~f J IV/1 2dp = J 16.ldp = (6./ , I)\u00a3.2(M ) \n\nM \n\nM \n\nX t \n\nIf the manifold is ~ n then VI = \nwhere Viis the gradient vector field of f. \n\" , n_ 1 !!La a -aa .' In general, for an n-manifold , the expression in a local coordinate \nL ~ _ X t \nchart involves the coefficients of the m etric tensor. Therefore the smoothness of a \nunit norm eigenfunction ei of 6. is controlled by the corresponding eigenvalue Ai \nsince 5(ei) = (6. ei, ei)\u00a3.2(M) = Ai. For an arbitrary 1 = 2::i [ti ei, we can write S(J) \nas \n\nA Reproducing Kernel Hilbert Space can be constructed from S. A1 = 0 is the \nsmallest eigenvalue for which the corresponding eigenfunction is the constant func(cid:173)\ntion e1 = 1'(1). It can also be shown that if M is compact and connected there \nare no other eigenfunctions with eigenvalue O. Therefore approximating a function \nI( x) :::::: 2::; ai ei (x) in terms of the first p eigenfunctions of 6. is a way of controlling \nthe smoothness of the approximation. The optimal approximation is obtained by \n\nargmin J (/ (X) - t aiei( X)) 2 dp. \nThis approximation is given by a projection in \u00a32 onto the span of the first p \neigenfunctions ai = J ei(x )/(x)dp = (ei ' I) \u00a3.2(M) In practice we only know the \nvalues of 1 at a finite number of points X l, ... , X n and therefore have to solve a \ndiscrete version of this problem a = _ a~gmi~ .t (/(Xi ) - t O,jej(Xi )) 2 The so-\n\nminimizing the \u00a32 norm of the error : a = \n\na=(a\" ... ,ap ) M \n\nM \n\n, \n\na=(a\" ... ,a p ), =l \n\n) =1 \n\nlution to this standard least squares problem is given by aT = (ET E)- l EyT, where \nEij = ei (Xj) and y = (J(xd , \u00b7 .. , I(xn )). \nConection with the Graph Laplacian: \nAs we are approximating a manifold with a graph, we need a suitable m easure \nof smoothness for functions defined on the graph. It turns out that many of the \nconcepts in the previous section have parallels in graph theory (e.g., see [3]). Let \nG = (V, E) be a weighted graph on n vertices. We assume that the vertices are \n\n\fnumbered and use the notation i ~ j for adjacent vertices i and j. The graph \nLaplacian of G is defined as L = D - W , where W is the weight matrix and D is a \ndiagonal matrix, Dii = I:j Wj i. L can be thought of as an operator on functions \ndefined on vertices of the graph. It is not hard to see that L is a self-adj oint positive \nsemidefinite operator. By the (finite dimensional) spectral theorem any function on \nG can be decomposed as a sum of eigenfunctions of L. If we think of G as a model \nfor the manifold M it is reasonable to assume that a function on G is smooth if it \ndoes not change too much between nearby points. If f = (11 , ... , In) is a function \non G, then we can formalize that intuition by defining the smoothness functional \nSG(f) = I: Wij(Ji - h)2. It is not hard to show that SG(f) = f LfT = (f , Lf)G = \nn I: Ai (f , ei) G which is the discrete analogue of the integration by parts from the \ni =l \nprevious section . The inner product here is the usual Euclidean inner product on \nthe vector space with coordinates indexed by the vertices of G , ei are normalized \neigenvectors of L, Lei = Aiei, Ileill = 1. All eigenvalues are non-negative and the \neigenfunctions corresponding to the smaller eigenvalues can be thought as \"more \nsmooth\". The smallest eigenvalue A1 = 0 corresponds to the constant eigenvector \ne1\u00b7 \n\n5 Experimental Results \n\n5.1 Handwritten Digit Recognition \n\nWe apply our techniques to the problem of optical character recognition. We use \nthe popular MNIST dataset which contains 28x28 grayscale images of handwritten \ndigits. 1 We use the 60000 image training set for our experiments. For all experi(cid:173)\nments we use 8 nearest neighbours to compute the adjacency matrix. The adjacency \nmatrices are very sparse which makes solving eigenvector problems for matrices as \nbig as 60000 by 60000 possible. For a particular trial, we fix the number of labeled \nexamples we wish to use. A random subset of the 60000 images is used with labels \nto form the labeled set L. The rest of the images are used without labels to form the \nunlab eled data U. The classification results (for U) are averaged over 20 different \nrandom draws for L. Shown in fig. 2 is a summary plot of classification accuracy on \nthe unlab eled set comparing the nearest neighbors baseline with our algorithm that \nretains the numb er of eigenvectors by following taking it to be 20% of the numb er \nof labeled points. The improvements over the base line are significant, sometimes \nexceeding 70% depending on the number of labeled and unlabeled examples . With \nonly 100 labeled examples (and 59900 unlabeled examples), the Laplacian classifier \ndoes nearly as well as the nearest neighbor classifier with 5000 labeled examples. \nSimilarly, with 500/59500 labeled/unlabeled examples, it does slightly better than \nthe nearest neighbor base line using 20000 labeled examples By comparing the re(cid:173)\nsults for the total 60000 point data set, and 10000 and 1000 subsets we see that \nadding unlab eled data consistently improves classification accuracy. When almost \nall of the data is labeled , the performance of our classifier is close to that of k-NN. It \nis not particularly surprising as our method uses the nearest neighbor information. \n\n1 We use the first 100 prin cipal components of the set of all images to represent each \n\nimage as a 100 dimensional vector. \n\n\f60 ~-----'------'------'------'--r~~====~~====~ \n\n--e- Laplacian 60,000 points total \nLaplacian 10,000 points total \n-\n-A- Laplacian 1 ,QOO points total \n+ best k-NN, k=1 ,3,5 \n\n40 \n\n20 \n\n2 L-____ -L ______ L -____ -L ______ L -____ -L ______ L -____ - \" \n20 \n\n20000 \n\n500 \n\n1000 \n\n5000 \n\n50 \n\n100 \n\n50000 \n\nNumber of Labeled Points \n\nFigure 2: MNIST data set, Percentage error rates for different numb ers of labeled \nand unlabeled points compared to best k-NN base line, \n\n5.2 Text Classification \n\nThe second application is text classification using the popular 20 Newsgroups data \nset, This data set contains approximately 1000 postings from each of 20 different \nnewsgroups, Given an article, the problem is to determine to which newsgroup it \nwas posted, We tokenize the articles using the software package Rainbow written \nby Andrew McCallum, We use a \"stop-list\" of 500 most common words to be \nexcluded and also exclude headers , which among other things contain the correct \nidentification of the newsgroup, Each document is then represented by the counts \nof the most frequent 6000 words normalized to sum to L Documents with 0 total \ncount are removed , thus leaving us with 19935 vectors in a 6000-dimensional space, \nWe follow the same procedure as wit h the MNIST digit data above, A random \nsubset of a fixed size is taken with labels to form L, The rest of the dataset is \nconsidered to be U, We average the results over 20 random splits 2 , As with the \ndigits, we take the number of nearest neighbors for the algorithm to be 8, In fig, 3 \nwe summarize the results by taking 19935 , 2000 and 600 total points respectively \nand calculating the error rate for different numbers oflabeled points, The numb er of \neigenvectors used is always 20% of the number of labeled points, We see that having \nmore unlabeled points improves the classification error in most cases although when \nthere are very few labeled points , the differences are smalL \n\nReferences \n\n[1] M. Belkin , P. Niyogi , Laplacian Eigenmaps for Dim ensionality Redu ction and \nData Representation, Technical Report, TR-2002-01 , Department of Computer \nScience, The University of Chicago, 2002 . \n\n2In the case of 2000 eigenvectors we take just 10 random splits since the computations \n\nare rather time-consuming . \n\n\f80 ~------'-------'-------'------r~======~7=====~ \n\n-e- Laplacian 19,935 points total \n~ Laplacian 2,000 pOints total \n---A- Laplacian 600 points total \n+ best k-NN, k=1,3,5 \n\n60 \n\n40 \n\n30 \n\n22 L-------~-------L------~--------~-------L------~ \n\n1000 \n\n5000 \n\n10000 \n\n18000 \n\n50 \n\n100 \n\n500 \n\nNumber of Labeled Points \n\nFigure 3: 20 Newsgroups data set. Error rates for different numbers of labeled and \nunlabeled points compared to best k-NN baseline . \n\n[2] A. Blum , S. Chawla, Learning from Labeled and Un labeled Data using Graph \n\nMincuts, ICML, 2001 , \n\n[3] Fan R. K. Chung, Spectra l Graph Theory, Regional Conference Series in Math(cid:173)\n\nematics , number 92, 1997 \n\n[4] K. Nigam , A.K. McCallum , S. Thrun , T. Mitchell , Text Classifi cation from \n\nLabeled in Unlabeled Data , Machine Learning 39(2/3),2000, \n\n[5] S. Rosenberg, The Laplacian on a Riemmannian Manifold, Cambridge Uni(cid:173)\n\nversity Press, 1997, \n\n[6] Sam T. Roweis, Lawrence K. Saul , N onlin ear Dimensionality Reduction by \n\nLocally Linear Embedding, Science, vol 290 , 22 December 2000 , \n\n[7] Martin Szummer , Tommi Jaakkola, Partially labeled classification with Markov \n\nrandom walks, Neural Information Processing Systems (NIPS) 2001 , vol 14. , \n\n[8] Joshua B. Tenenbaum, Vin de Silva, John C. Langford , A Global Geomet(cid:173)\nric Framework for N onlin ear Dimensionality Reduction, Science, Vol 290, 22 \nDecember 2000, \n\n\f", "award": [], "sourceid": 2306, "authors": [{"given_name": "Mikhail", "family_name": "Belkin", "institution": null}, {"given_name": "Partha", "family_name": "Niyogi", "institution": null}]}