{"title": "Zero-shot Learning with Semantic Output Codes", "book": "Advances in Neural Information Processing Systems", "page_first": 1410, "page_last": 1418, "abstract": "We consider the problem of zero-shot learning, where the goal is to learn a classifier $f: X \\rightarrow Y$ that must predict novel values of $Y$ that were omitted from the training set. To achieve this, we define the notion of a semantic output code classifier (SOC) which utilizes a knowledge base of semantic properties of $Y$ to extrapolate to novel classes. We provide a formalism for this type of classifier and study its theoretical properties in a PAC framework, showing conditions under which the classifier can accurately predict novel classes.  As a case study, we build a SOC classifier for a neural decoding task and show that it can often predict words that people are thinking about from functional magnetic resonance images (fMRI) of their neural activity, even without training examples for those words.", "full_text": "Zero-Shot Learning with Semantic Output Codes\n\nMark Palatucci\nRobotics Institute\n\nCarnegie Mellon University\n\nPittsburgh, PA 15213\n\nmpalatuc@cs.cmu.edu\n\nDean Pomerleau\n\nIntel Labs\n\nPittsburgh, PA 15213\n\ndean.a.pomerleau@intel.com\n\nGeoffrey Hinton\n\nComputer Science Department\n\nUniversity of Toronto\n\nToronto, Ontario M5S 3G4, Canada\n\nhinton@cs.toronto.edu\n\nTom M. Mitchell\n\nMachine Learning Department\nCarnegie Mellon University\n\nPittsburgh, PA 15213\n\ntom.mitchell@cs.cmu.edu\n\nAbstract\n\nWe consider the problem of zero-shot learning, where the goal is to learn a clas-\nsi\ufb01er f : X \u2192 Y that must predict novel values of Y that were omitted from\nthe training set. To achieve this, we de\ufb01ne the notion of a semantic output code\nclassi\ufb01er (SOC) which utilizes a knowledge base of semantic properties of Y to\nextrapolate to novel classes. We provide a formalism for this type of classi\ufb01er\nand study its theoretical properties in a PAC framework, showing conditions un-\nder which the classi\ufb01er can accurately predict novel classes. As a case study, we\nbuild a SOC classi\ufb01er for a neural decoding task and show that it can often predict\nwords that people are thinking about from functional magnetic resonance images\n(fMRI) of their neural activity, even without training examples for those words.\n\n1 Introduction\n\nMachine learning algorithms have been successfully applied to learning classi\ufb01ers in many domains\nsuch as computer vision, fraud detection, and brain image analysis. Typically, classi\ufb01ers are trained\nto approximate a target function f : X \u2192 Y , given a set of labeled training data that includes all\npossible values for Y , and sometimes additional unlabeled training data.\nLittle research has been performed on zero-shot learning, where the possible values for the class\nvariable Y include values that have been omitted from the training examples. This is an important\nproblem setting, especially in domains where Y can take on many values, and the cost of obtaining\nlabeled examples for all values is high. One obvious example is computer vision, where there are\ntens of thousands of objects which we might want a computer to recognize.\nAnother example is in neural activity decoding, where the goal is to determine the word or object\na person is thinking about by observing an image of that person\u2019s neural activity. It is intractable\nto collect neural training images for every possible word in English, so to build a practical neural\ndecoder we must have a way to extrapolate to recognizing words beyond those in the training set.\nThis problem is similar to the challenges of automatic speech recognition, where it is desirable to\nrecognize words without explicitly including them during classi\ufb01er training. To achieve vocabulary\nindependence, speech recognition systems typically employ a phoneme-based recognition strategy\n(Waibel, 1989). Phonemes are the component parts which can be combined to construct the words\nof a language. Speech recognition systems succeed by leveraging a relatively small set of phoneme\n\n1\n\n\frecognizers in conjunction with a large knowledge base representing words as combinations of\nphonemes.\nTo apply a similar approach to neural activity decoding, we must discover how to infer the compo-\nnent parts of a word\u2019s meaning from neural activity. While there is no clear consensus as to how the\nbrain encodes semantic information (Plaut, 2002), there are several proposed representations that\nmight serve as a knowledge base of neural activity, thus enabling a neural decoder to recognize a\nlarge set of possible words, even when those words are omitted from a training set.\nThe general question this paper asks is:\n\nGiven a semantic encoding of a large set of concept classes, can we build a classi\ufb01er to\nrecognize classes that were omitted from the training set?\nWe provide a formal framework for addressing this question and a concrete example for the task\nof neural activity decoding. We show it is possible to build a classi\ufb01er that can recognize words a\nperson is thinking about, even without training examples for those particular words.\n\n1.1 Related Work\n\nThe problem of zero-shot learning has received little attention in the machine learning community.\nSome work by Larochelle et al. (2008) on zero-data learning has shown the ability to predict novel\nclasses of digits that were omitted from a training set. In computer vision, techniques for sharing\nfeatures across object classes have been investigated (Torralba & Murphy, 2007; Bart & Ullman,\n2005) but relatively little work has focused on recognizing entirely novel classes, with the exception\nof Lampert et al. (2009) predicting visual properties of new objects and Farhadi et al. (2009) using\nvisual property predictions for object recognition.\nIn the neural imaging community, Kay et al. (2008) has shown the ability to decode (from visual\ncortex activity) which novel visual scenes a person is viewing from a large set of possible images,\nbut without recognizing the image content per se.\nThe work most similar to our own is Mitchell (2008). They use semantic features derived from\ncorpus statistics to generate a neural activity pattern for any noun in English.\nIn our work, by\ncontrast, we focus on word decoding, where given a novel neural image, we wish to predict the\nword from a large set of possible words. We also consider semantic features that are derived from\nhuman labeling in addition to corpus statistics. Further, we introduce a formalism for a zero-shot\nlearner and provide theoretical guarantees on its ability to recognize novel classes omitted from a\ntraining set.\n\n2 Classi\ufb01cation with Semantic Knowledge\n\nIn this section we formalize the notion of a zero-shot learner that uses semantic knowledge to extrap-\nolate to novel classes. While a zero-shot learner could take many forms, we present one such model\nthat utilizes an intermediate set of features derived from a semantic knowledge base. Intuitively,\nour goal is to treat each class not as simply an atomic label, but instead represent it using a vector\nof semantic features characterizing a large number of possible classes. Our models will learn the\nrelationship between input data and the semantic features. They will use this learned relationship in\na two step prediction procedure to recover the class label for novel input data. Given new input data,\nthe models will predict a set of semantic features corresponding to that input, and then \ufb01nd the class\nin the knowledge base that best matches that set of predicted features. Signi\ufb01cantly, this procedure\nwill even work for input data from a novel class if that class is included in the semantic knowledge\nbase (i.e. even if no input space representation is available for the class, but a feature encoding of it\nexists in the semantic knowledge base).\n\n2.1 Formalism\n\nDe\ufb01nition 1. Semantic Feature Space\nA semantic feature space of p dimensions is a metric space in which each of the p dimensions\nencodes the value of a semantic property. These properties may be categorical in nature or may\ncontain real-valued data.\n\n2\n\n\fAs an example, consider a semantic space for describing high-level properties of animals. In this\nexample, we\u2019ll consider a small space with only p = 5 dimensions. Each dimension encodes a\nis it furry? does it have a tail? can it breathe underwater? is it carnivorous? is it\nbinary feature:\nslow moving? In this semantic feature space, the prototypical concept of dog might be represented\nas the point {1, 1, 0, 1, 0}.\nDe\ufb01nition 2. Semantic Knowledge Base\nA semantic knowledge base K of M examples is a collection of pairs {f, y}1:M such that f \u2208 F p\nis a point in a p dimensional semantic space F p and y \u2208 Y is a class label from a set Y . We assume\na one-to-one encoding between class labels and points in the semantic feature space.\n\nA knowledge base of animals would contain the semantic encoding and label for many animals.\n\nDe\ufb01nition 3. Semantic Output Code Classi\ufb01er\nA semantic output code classi\ufb01er H : X d \u2192 Y maps points from some d dimensional raw-input\nspace X d to a label from a set Y such that H is the composition of two other functions, S and L,\nsuch that:\n\nH = L(S(\u00b7))\nS\n: X d \u2192 F p\nL : F p \u2192 Y\n\nThis model of a zero-shot classi\ufb01er \ufb01rst maps from a d dimensional raw-input space X d into a\nsemantic space of p dimensions F p, and then maps this semantic encoding to a class label. For\nexample, we may imagine some raw-input features from a digital image of a dog \ufb01rst mapped into\nthe semantic encoding of a dog described earlier, which is then mapped to the class label dog. As\na result, our class labels can be thought of as a semantic output code, similar in spirit to the error-\ncorrecting output codes of Dietterich and Bakiri (1995).\nAs part of its training input, this classi\ufb01er is given a set of N examples D that consists of pairs\n{x, y}1:N such that x \u2208 X d and y \u2208 Y . The classi\ufb01er is also given a knowledge base K of M\nexamples that is a collection of pairs {f, y}1:M such that f \u2208 F p and y \u2208 Y . Typically, M >> N,\nmeaning that data in semantic space is available for many more class labels than in the raw-input\nspace. Thus,\nA semantic output code classi\ufb01er can be useful when the knowledge base K covers more of the\npossible values for Y than are covered by the input data D.\nTo learn the mapping S, the classi\ufb01er \ufb01rst builds a new set of N examples {x, f}1:N by replacing\neach y with the respective semantic encoding f according to its knowledge base K.\nThe intuition behind using this two-stage process is that the classi\ufb01er may be able to learn the\nrelationship between the raw-input space and the individual dimensions of the semantic feature\nspace from a relatively small number of training examples in the input space. When a new example\nis presented, the classi\ufb01er will make a prediction about its semantic encoding using the learned S\nmap. Even when a new example belongs to a class that did not appear in the training set D, if the\nprediction produced by the S map is close to the true encoding of that class, then the L map will\nhave a reasonable chance of recovering the correct label. As a concrete example, if the model can\npredict the object has fur and a tail, it would have a good chance of recovering the class label dog,\neven without having seen images of dogs during training. In short:\nBy using a rich semantic encoding of the classes, the classi\ufb01er may be able to extrapolate and\nrecognize novel classes.\n\n3 Theoretical Analysis\n\nIn this section we consider theoretical properties of a semantic output code classi\ufb01er that determine\nits ability to recognize instances of novel classes. In other words, we will address the question:\nUnder what conditions will the semantic output code classi\ufb01er recognize examples from classes\nomitted from its training set?\n\n3\n\n\fIn answering this question, our goal is to obtain a PAC-style bound: we want to know how much\nerror can be tolerated in the prediction of the semantic properties while still recovering the novel\nclass with high probability. We will then use this error bound to obtain a bound on the number of\nexamples necessary to achieve that level of error in the \ufb01rst stage of the classi\ufb01er. The idea is that\nif the \ufb01rst stage S(\u00b7) of the classi\ufb01er can predict the semantic properties well, then the second stage\nL(\u00b7) will have a good chance of recovering the correct label for instances from novel classes.\nAs a \ufb01rst step towards a general theory of zero-shot learning, we will consider one instantiation\nof a semantic output code classi\ufb01er. We will assume that semantic features are binary labels, the\n\ufb01rst stage S(\u00b7) is a collection of PAC-learnable linear classi\ufb01ers (one classi\ufb01er per feature), and the\nsecond stage L(\u00b7) is a 1-nearest neighbor classi\ufb01er using the Hamming distance metric. By making\nthese assumptions, we can leverage existing PAC theory for linear classi\ufb01ers as well as theory for\napproximate nearest neighbor search. Much of our nearest-neighbor analysis parallels the work of\nCiaccia and Patella (2000).\nWe \ufb01rst want to bound the amount of error we can tolerate given a prediction of semantic features.\nTo \ufb01nd this bound, we de\ufb01ne F to be the distribution in semantic feature space of points from the\nknowledge base K. Clearly points (classes) in semantic space may not be equidistant from each\nother. A point might be far from others, which would allow more room for error in the prediction of\nsemantic features for this point, while maintaining the ability to recover its unique identity (label).\nConversely, a point close to others in semantic space will have lower tolerance of error. In short, the\ntolerance for error is relative to a particular point in relation to other points in semantic space.\nWe next de\ufb01ne a prediction q to be the output of the S(\u00b7) map applied to some raw-input example\nx \u2208 X d. Let d(q, q(cid:48)) be the distance between the prediction q and some other point q(cid:48) representing\na class in the semantic space. We de\ufb01ne the relative distribution Rq for a point q as the probability\nthat the distance from q to q(cid:48) is less than some distance z:\n\nThis empirical distribution depends on F and is just the fraction of sampled points from F that are\nless than some distance z away from q. Using this distribution, we can also de\ufb01ne a distribution on\nthe distance to the nearest neighbor of q, de\ufb01ned as \u03b7q:\n\nRq(z) = P (d(q, q(cid:48)) \u2264 z)\n\nGq(z) = P (\u03b7q \u2264 z)\n\nwhich is given in Ciaccia (2000) as:\n\nGq(z) = 1 \u2212 (1 \u2212 Rq(z))n\n\nwhere n is the number of actual points drawn from the distribution F. Now suppose that we de\ufb01ne\n\u03c4q to be the distance a prediction q for raw-input example x is from the true semantic encoding of\nthe class to which x belongs. Intuitively, the class we infer for input x is going to be the point closest\nto prediction q, so we want a small probability \u03b3 that the distance \u03c4q to the true class is larger than\nthe distance between q and its nearest neighbor, since that would mean there is a spurious neighbor\ncloser to q in semantic space than the point representing q\u2019s true class:\n\nP (\u03c4q \u2265 \u03b7q) \u2264 \u03b3\n\nRearranging we can put this in terms of the distribution Gq and then can solve for \u03c4q:\n\nP (\u03b7q \u2264 \u03c4q) \u2264 \u03b3\nGq(\u03c4q) \u2264 \u03b3\n\n(cid:104)\n\n(cid:105)\n\nIf Gq(\u00b7) were invertible, we could immediately recover the value \u03c4q for a desired \u03b3. For some\ndistributions, Gq(\u00b7) may not be a 1-to-1 function, so there may not be an inverse. But Gq(\u00b7) will\nnever decrease since it is a cumulative distribution function. We will therefore de\ufb01ne a function G\u22121\nsuch that: G\u22121\nSo using nearest neighbor for L(\u00b7), if \u03c4q \u2264 G\u22121\nq (\u03b3), then we will recover the correct class with\nat least 1 \u2212 \u03b3 probability. To ensure that we achieve this error bound, we need to make sure the\ntotal error of S(\u00b7) is less than G\u22121\n. We assume in this analysis\nthat we have p binary semantic features and a Hamming distance metric, so \u03c4 max\nde\ufb01nes the total\n\nq (\u03b3) which we de\ufb01ne as \u03c4 max\n\nq (\u03b3) = argmax\u03c4q\n\nGq(\u03c4q) \u2264 \u03b3\n\n.\n\nq\n\nq\n\nq\n\n4\n\n\fnumber of mistakes we can make predicting the binary features. Note with our assumptions, each\nsemantic feature is PAC-learnable using a linear classi\ufb01er from a d dimensional raw input space. To\nsimplify the analysis, we will treat each of the p semantic features as independently learned. By the\nPAC assumption, the true error (i.e. probability of the classi\ufb01er making a mistake) of each of the p\nlearned hypotheses is \u0001, then the expected number of mistakes over the p semantic features will be\nmistakes is given by\n\u03c4 max\nq\nthe binomial distribution: BinoCDF(\u03c4 max\nWe can obtain the desired error rate for each hypothesis by utilizing the standard PAC bound for\nVC-dimension1 (Mitchell, 1997). To obtain a hypothesis with (1 \u2212 \u03b4) probability that has true error\nat most \u0001 = \u03c4 max\n\n/p = G\u22121(\u03b3)/p, then the classi\ufb01er requires a number of examples Mq,\u03b4:\n\n/p. Further, the probability of making at most \u03c4 max\n\nif we set \u0001 = \u03c4 max\n\n; p, \u03c4 max\n\n/p)\n\nq\n\nq\n\nq\n\nq\n\nq\n\n(cid:105)\n\n(cid:104)\n\nMq,\u03b4 \u2265\n\np\n\n\u03c4 max\nq\n\n4log(2/\u03b4) + 8(d + 1)log(13p/\u03c4 max\n\nq\n\n)\n\n(1)\n\nIf each of the p classi\ufb01ers (feature predictors) is learned with this many examples, then with prob-\nability (1 \u2212 \u03b4)p, all feature predictors will achieve the desired error rate. But note that this is only\nthe probability of achieving p hypotheses with the desired true error rate. The binomial CDF yields\nmistakes total, and the (1 \u2212 \u03b3) term above speci\ufb01es the\nthe probability of making at most \u03c4 max\nprobability of recovering the true class if a maximum of this many mistakes were made. Therefore,\nthere are three probabilistic events required for the semantic output code classi\ufb01er to predict a novel\nclass and the total (joint) probability of these events is:\n\nq\n\nP (there are p feature predictors with true error \u2264 \u03c4 max\nP (at most \u03c4 max\nP (recovering true class | at most \u03c4 max\n\nmistakes made)\n\nq\n\nq\n\nq\n\n/p) \u00b7\n\nmistakes made | there are p feature predictors with true error \u2264 \u03c4 max\n\n/p ) \u00b7\n\nq\n\nand since \u03c4 max\n\nq\n\n= G\u22121\n\nq (\u03b3), the total probability is given by:\n(1 \u2212 \u03b4)p \u00b7 BinoCDF(G\u22121\n\nq (\u03b3); p, G\u22121\n\nq (\u03b3)/p) \u00b7 (1 \u2212 \u03b3)\n\n(2)\nIn summary, given desired error parameters (1\u2212\u03b3) and (1\u2212\u03b4) for the two classi\ufb01er stages, Equation\n2 provides the total probability of correctly predicting a novel class. Given the value for \u03b3 we can\ncompute the \u0001 necessary for each feature predictor. We are guaranteed to obtain the total probability\nif the feature predictors were trained with Mq,\u03b4 raw-input examples as speci\ufb01ed in Equation 1.\nTo our knowledge, Equations 1 and 2 specify the \ufb01rst formal guarantee that provides conditions\nunder which a classi\ufb01er can predict novel classes.\n\n4 Case Study: Neural Decoding of Novel Thoughts\n\nIn this section we empirically evaluate a semantic output code classi\ufb01er on a neural decoding task.\nThe objective is to decode novel words a person is thinking about from fMRI images of the person\u2019s\nneural activity, without including example fMRI images of those words during training.\n\n4.1 Datasets\n\nWe utilized the same fMRI dataset from Mitchell (2008). This dataset contains the neural activity\nobserved from nine human participants while viewing 60 different concrete words (5 examples from\n12 different categories). Some examples include animals: bear, dog, cat, cow, horse\nand vehicles: truck, car, train, airplane, bicycle. Each participant was shown a\nword and a small line drawing of the concrete object the word represents. The participants were\nasked to think about the properties of these objects for several seconds while images of their brain\nactivity were recorded.\nEach image measures the neural activity at roughly 20,000 locations (i.e. voxels) in the brain. Six\nfMRI scans were taken for each word. We used the same time-averaging described in Mitchell\n(2008) to create a single average brain activity pattern for each of the 60 words, for each participant.\n\n1The VC dimension of linear classi\ufb01ers in d dimensions is d + 1\n\n5\n\n\fIn the language of the semantic output code classi\ufb01er, this dataset represents the collection D of\nraw-input space examples.\nWe also collected two semantic knowledge bases for these 60 words. In the \ufb01rst semantic knowl-\nedge base, corpus5000, each word is represented as a co-occurrence vector with the 5000 most\nfrequent words from the Google Trillion-Word-Corpus2.\nThe second semantic knowledge base, human218, was created using the Mechanical Turk human\ncomputation service from Amazon.com. There were 218 semantic features collected for the 60\nwords, and the questions were selected to re\ufb02ect psychological conjectures about neural activity\nencoding. For example, the questions related to size, shape, surface properties, and typical usage.\nExample questions include is it manmade? and can you hold it?. Users of the Mechanical Turk\nservice answered these questions for each word on a scale of 1 to 5 (de\ufb01nitely not to de\ufb01nitely yes).\n\n4.2 Model\nIn our experiments, we use multiple output linear regression to learn the S(\u00b7) map of the semantic\noutput code classi\ufb01er. Let X \u2208 (cid:60)N\u2217d be a training set of fMRI examples where each row is the\nimage for a particular word and d is the number of dimensions of the fMRI image. During training,\nwe use the voxel-stability-criterion that does not use the class labels described in Mitchell (2008) to\nreduce d from about 20,000 voxels to 500. Let Y \u2208 (cid:60)N\u2217p be a matrix of semantic features for those\nwords (obtained from the knowledge base K) where p is the number of semantic features for that\n\nword (e.g. 218 for the human218 knowledge base). We learn a matrix of weights (cid:98)W \u2208 (cid:60)d\u2217p. In\n\nthis model, each output is treated independently, so we can solve all of them quickly in one matrix\noperation (even with thousands of semantic features):\n\n(cid:98)W = (XT X + \u03bbI)\u22121XT Y\n\n(3)\nwhere I is the identity matrix and \u03bb is a regularization parameter chosen automatically using the\ncross-validation scoring function (Hastie et al., 2001)3. Given a novel fMRI image x, we can obtain\n\na prediction(cid:98)f of the semantic features for this image by multiplying the image by the weights:\n(cid:98)f = x \u00b7 (cid:98)W\nclassi\ufb01er. In other words, L((cid:98)f) will take the prediction of features and return the closest point in a\n\nFor the second stage of the semantic output code classi\ufb01er, L(\u00b7), we simply use a 1-nearest neighbor\n\ngiven knowledge base according the Euclidean distance (L2) metric.\n\n4.3 Experiments\n\nUsing the model and datasets described above, we now pose and answer three important questions.\n1. Can we build a classi\ufb01er to discriminate between two classes, where neither class appeared in\nthe training set?\nTo answer this question, we performed a leave-two-out-cross-validation. Speci\ufb01cally, we trained the\nmodel in Equation 3 to learn the mapping between 58 fMRI images and the semantic features for\ntheir respective words. For the \ufb01rst held out image, we applied the learned weight matrix to obtain\na prediction of the semantic features, and then we used a 1-nearest neighbor classi\ufb01er to compare\nthe vector of predictions to the true semantic encodings of the two held-out words. The label was\nchosen by selecting the word with the encoding closest to the prediction for the fMRI image. We\nthen performed the same test using the second held-out image. Thus, for each iteration of the cross-\n\nvalidation, two separate comparisons were made. This process was repeated for all(cid:0)60\n\n(cid:1) = 1, 770\n\npossible leave-two-out combinations leading to 3,540 total comparisons.\nTable 1 shows the results for two different semantic feature encodings. We see that the human218\nsemantic features signi\ufb01cantly outperformed the corpus5000 features, with mean accuracies over\nthe nine participants of 80.9% and 69.7% respectively. But for both feature sets, we see that it is\npossible to discriminate between two novel classes for each of the nine participants.\n\n2\n\n2Vectors are normalized to unit length and do not include 100 stop words like a, the, is\n3We compute the cross-validation score for each task and choose the parameter that minimizes the average\n\nloss across all output tasks.\n\n6\n\n\fTable 1: Percent accuracies for leave-two-out-cross-validation for 9 fMRI participants (labeled P1-\nP9). The values represent classi\ufb01er percentage accuracy over 3,540 trials when discriminating be-\ntween two fMRI images, both of which were omitted from the training set.\n\ncorpus5000\nhuman218\n\nP1\n79.6\n90.3\n\nP2\n67.0\n82.9\n\nP3\n69.5\n86.6\n\nP4\n56.2\n71.9\n\nP5\n77.7\n89.5\n\nP6\n65.5\n75.3\n\nP7\n71.2\n78.0\n\nP8\n72.9\n77.7\n\nP9\n67.9\n76.2\n\nMean\n69.7\n80.9\n\nFigure 1: Ten semantic features from the human218 knowledge base for the words bear and dog.\nThe true encoding is shown along with the predicted encoding when fMRI images for bear and dog\nwere left out of the training set.\n\n2. How is the classi\ufb01er able to discriminate between closely related novel classes?\nFigure 1 shows ten semantic questions (features) from the human218 dataset. The graph shows the\ntrue values along with the predicted feature values for both bear and dog when trained on the other\n58 words. We see the model is able to learn to predict many of the key features that bears and dogs\nhave in common such as is it an animal? as well as those that differentiate between the two, such as\ndo you see it daily? and can you hold it? For both of these novel words, the features predicted from\nthe neural data were closest to the true word.\n3. Can we decode the word from a large set of possible words?\nGiven the success of the semantic output code classi\ufb01er at discriminating between the brain images\nfor two novel words, we now consider the much harder problem of discriminating a novel word from\na large set of candidate words. To test this ability, we performed a leave-one-out-cross-validation,\nwhere we trained using Equation 3 on images and semantic features for 59 words. We then pre-\ndicted the features for the held-out image of the 60th word, and then performed a 1-nearest neighbor\nclassi\ufb01cation in a large set of candidate words.\nWe tested two different word sets. The \ufb01rst was mri60 which is the collection of all 60 concrete\nnouns for which we collected fMRI data, including the 59 training words and the single held out\nword. The second set was noun940, a collection of 940 English nouns with high familiarity,\nconcreteness and imagineability, compiled from Wilson (1988) and Snodgrass (1980). For this set\nof words, we added the true held-out word to the set of 940 on each cross-validation iteration. We\nperformed this experiment using both the corpus5000 and human218 feature sets. The rank\naccuracy results (over 60 cross-validation iterations) of the four experiments are shown in Figure 2.\nThe human218 features again signi\ufb01cantly outperform corpus5000 on both mean and median\nrank accuracy measures, and both feature sets perform well above chance. On 12 of 540 total\npresentations of the mri60 words (60 presentations for each of nine participants), the human218\nfeatures predicted the single held-out word above all 59 other words in its training set. While just\na bit above chance level (9/540), the fact that the model ever chooses the held-out word over all\nthe training words is noteworthy since the model is undoubtedly biased towards predicting feature\nvalues similar to the words on which it was trained. On the noun940 words, the model predicted\nthe correct word from the set of 941 alternatives a total of 26 times for the human218 features and\n22 times for the corpus5000 features. For some subjects, the model correctly picked the right\n\n7\n\nBear & Dog Prediction Match-0.10-0.08-0.06-0.04-0.020.000.020.040.060.080.10Is it ananimal?Is it man-made?Do you seeit daily?Is ithelpful?Can youhold it?Would youfind it in ahouse?Do youlove it?Does itstand ontwo legs?Is it wild?Does itprovideprotection?AnswerBear PredictedBear TargetDog TargetDog Predicted\fFigure 2: The mean and median rank accuracies across nine participants for two different semantic\nfeature sets. Both the original 60 fMRI words and a set of 940 nouns were considered.\n\nTable 2: The top \ufb01ve predicted words for a novel fMRI image taken for the word in bold (all fMRI\nimages taken from participant P1). The number in the parentheses contains the rank of the correct\nword selected from 941 concrete nouns in English.\n\nBear\n(1)\nbear\nfox\nwolf\nyak\ngorilla\n\nFoot\n(1)\nfoot\nfeet\nankle\nknee\nface\n\nScrewdriver Train\n(1)\nscrewdriver\npin\nnail\nwrench\ndagger\n\n(1)\ntrain\njet\njail\nfactory\nbus\n\nTruck\n(2)\njeep\ntruck\nminivan\nbus\nsedan\n\nCelery\n(5)\nbeet\nartichoke\ngrape\ncabbage\ncelery\n\nHouse\n(6)\nsupermarket\nhotel\ntheater\nschool\nfactory\n\nPants\n(21)\nclothing\nvest\nt-shirt\nclothes\npanties\n\nword from the set of 941 more than 10% of the time. The chance accuracy of predicting a word\ncorrectly is only 0.1%, meaning we would expect less than one correct prediction across all 540\npresentations.\nAs Figure 2 shows, the median rank accuracies are often signi\ufb01cantly higher than the mean rank\naccuracies. Using the human218 features on the noun940 words, the median rank accuracy is\nabove 90% for each participant while the mean is typically about 10% lower. This is due to the fact\nthat several words are consistently predicted poorly. The prediction of words in the categories ani-\nmals, body parts, foods, tools, and vehicles typically perform well, while the words in the categories\nfurniture, man-made items, and insects often perform poorly.\nEven when the correct word is not the closest match, the words that best match the predicted features\nare often very similar to the held-out word. Table 2 shows the top \ufb01ve predicted words for eight\ndifferent held-out fMRI images for participant P1 (i.e. the 5 closest words in the set of 941 to the\npredicted vector of semantic features).\n5 Conclusion\nWe presented a formalism for a zero-shot learning algorithm known as the semantic output code\nclassi\ufb01er. This classi\ufb01er can predict novel classes that were omitted from a training set by leveraging\na semantic knowledge base that encodes features common to both the novel classes and the training\nset. We also proved the \ufb01rst formal guarantee that shows conditions under which this classi\ufb01er will\npredict novel classes.\nWe demonstrated this semantic output code classi\ufb01er on the task of neural decoding using semantic\nknowledge bases derived from both human labeling and corpus statistics. We showed this classi\ufb01er\ncan predict the word a person is thinking about from a recorded fMRI image of that person\u2019s neural\nactivity with accuracy much higher than chance, even when training examples for that particular\nword were omitted from the training set and the classi\ufb01er was forced to pick the word from among\nnearly 1,000 alternatives.\nWe have shown that training images of brain activity are not required for every word we would like\na classi\ufb01er to recognize. These results signi\ufb01cantly advance the state-of-the-art in neural decoding\nand are a promising step towards a large vocabulary brain-computer interface.\n\n8\n\nRank Accuracyhuman218corpus5000human218corpus500040%50%60%70%80%90%100%AccuracyMean RankMedian Rank50%Chancemri60 Word Setnoun940 Word Set\fReferences\nBart, E., & Ullman, S. (2005). Cross-generalization: learning novel classes from a single example\nby feature replacement. Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE\nComputer Society Conference on, 1, 672\u2013679 vol. 1.\n\nCiaccia, P., & Patella, M. (2000). PAC nearest neighbor queries: Approximate and controlled search\n\nin high-dimensional and metric spaces. Data Engineering, International Conference on, 244.\n\nDietterich, T. G., & Bakiri, G. (1995). Solving multiclass learning problems via error-correcting\n\noutput codes. Journal of Arti\ufb01cial Intelligence Research.\n\nFarhadi, A., Endres, I., Hoiem, D., & Forsyth, D. (2009). Describing objects by their attributes. Pro-\nceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition\n(CVPR).\n\nHastie, T., Tibshirani, R., & Friedman, J. H. (2001). The elements of statistical learning. Springer.\nKay, K. N., Naselaris, T., Prenger, R. J., & Gallant, J. L. (2008). Identifying natural images from\n\nhuman brain activity. Nature, 452, 352\u2013355.\n\nLampert, C. H., Nickisch, H., & Harmeling, S. (2009). Learning to detect unseen object classes\nby between-class attribute transfer. Proceedings of the IEEE Computer Society Conference on\nComputer Vision and Pattern Recognition (CVPR).\n\nLarochelle, H., Erhan, D., & Bengio, Y. (2008). Zero-data learning of new tasks. AAAI Conference\n\non Arti\ufb01cial Intelligence.\n\nMitchell, T., et al. (2008). Predicting human brain activity associated with the meanings of nouns.\n\nScience, 320, 1191\u20131195.\n\nMitchell, T. M. (1997). Machine learning. New York: McGraw-Hill.\nMitchell, T. M., Hutchinson, R., Niculescu, R. S., Pereira, F., Wang, X., Just, M., & Newman, S.\n(2004). Learning to decode cognitive states from brain images. Machine Learning, 57, 145\u2013175.\nPlaut, D. C. (2002). Graded modality-speci\ufb01c specialization in semantics: A computational account\n\nof optic aphasia. Cognitive Neuropsychology, 19, 603\u2013639.\n\nSnodgrass, J., & Vanderwart, M. (1980). A standardized set of 260 pictures: Norms for name agree-\nment, image agreement, familiarity and visual complexity. Journal of Experimental Psychology:\nHuman Learning and Memory, 174\u2013215.\n\nTorralba, A., & Murphy, K. P. (2007). Sharing visual features for multiclass and multiview object\n\ndetection. IEEE Trans. Pattern Anal. Mach. Intell., 29, 854\u2013869.\n\nvan der Maaten, L., & Hinton, G. (2008). Visualizing data using t-SNE. Journal of Machine\n\nLearning Research, 9(Nov), 2579\u20132605.\n\nWaibel, A. (1989). Modular construction of time-delay neural networks for speech recognition.\n\nNeural Computation, 1, 39\u201346.\n\nWilson, M. (1988). The MRC psycholinguistic database: Machine readable dictionary, version 2.\n\nBehavioral Research Methods, 6\u201311.\n\n9\n\n\f", "award": [], "sourceid": 395, "authors": [{"given_name": "Mark", "family_name": "Palatucci", "institution": null}, {"given_name": "Dean", "family_name": "Pomerleau", "institution": null}, {"given_name": "Geoffrey", "family_name": "Hinton", "institution": null}, {"given_name": "Tom", "family_name": "Mitchell", "institution": null}]}