{"title": "Delta-encoder: an effective sample synthesis method for few-shot object recognition", "book": "Advances in Neural Information Processing Systems", "page_first": 2845, "page_last": 2855, "abstract": "Learning to classify new categories based on just one or a few examples is a long-standing challenge in modern computer vision. In this work, we propose a simple yet effective method for few-shot (and one-shot) object recognition. Our approach is based on a modified auto-encoder, denoted delta-encoder, that learns to synthesize new samples for an unseen category just by seeing few examples from it. The synthesized samples are then used to train a classifier. The proposed approach learns to both extract transferable intra-class deformations, or \"deltas\", between same-class pairs of training examples, and to apply those deltas to the few provided examples of a novel class (unseen during training) in order to efficiently synthesize samples from that new class. The proposed method improves the state-of-the-art of one-shot object-recognition and performs comparably in the few-shot case.", "full_text": "\u2206-encoder: an effective sample synthesis method for\n\nfew-shot object recognition\n\nEli Schwartz*1,2, Leonid Karlinsky*1,\n\nJoseph Shtok1, Sivan Harary1, Mattias Marder1, Abhishek Kumar1,\n\nRogerio Feris1, Raja Giryes2 and Alex M. Bronstein3\n\n2School of Electrical Engineering, Tel-Aviv University, Tel-Aviv, Israel\n\n3Department of Computer Science, Technion, Haifa, Israel\n\n1IBM Research AI\n\nAbstract\n\nLearning to classify new categories based on just one or a few examples is a\nlong-standing challenge in modern computer vision. In this work, we propose a\nsimple yet effective method for few-shot (and one-shot) object recognition. Our\napproach is based on a modi\ufb01ed auto-encoder, denoted \u2206-encoder, that learns to\nsynthesize new samples for an unseen category just by seeing few examples from it.\nThe synthesized samples are then used to train a classi\ufb01er. The proposed approach\nlearns to both extract transferable intra-class deformations, or \"deltas\", between\nsame-class pairs of training examples, and to apply those deltas to the few provided\nexamples of a novel class (unseen during training) in order to ef\ufb01ciently synthesize\nsamples from that new class. The proposed method improves the state-of-the-art of\none-shot object-recognition and performs comparably in the few-shot case.\n\n* The authors have contributed equally to this work\nCorresponding author: Leonid Karlinsky (leonidka@il.ibm.com)\n\nFigure 1: Visualization of two-way one-shot classi\ufb01cation trained on synthesized examples. Cor-\nrectly classi\ufb01ed images are framed in magenta (Golden retriever) and yellow (African wild dog). The\nonly two images seen at training time and used for sample synthesis are framed in blue. Note the\nnon-trivial relative arrangement of examples belonging to different classes handled successfully by\nour approach. The \ufb01gure is plotted using t-SNE applied to VGG features. Best viewed in color.\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\f1\n\nIntroduction\n\nFollowing the great success of deep learning, the \ufb01eld of visual classi\ufb01cation has made a signi\ufb01-\ncant leap forward, reaching \u2013 and in some cases, surpassing \u2013 human levels performance (usually\nwhen expertise is required) [24, 37]. Starting from AlexNet [23], followed by VGG [38], Google\nInception [42], ResNet [18], DenseNet [20] and NASNet [54] the \ufb01eld made tremendous advances in\nclassi\ufb01cation performance on large-scale datasets, such as ImageNet [5], with thousands of examples\nper category. However, it is known that we humans are particularly good at learning new categories\non the go, from seeing just a few or even a single example [24]. This is especially evident in early\nchildhood, when a parent points and names an object and a child can immediately start \ufb01nding more\nof its kind in the surroundings.\nWhile the exact workings of the human brain are very far from being fully understood, one can\nconjecture that humans are likely to learn from analogies. That is, we identify in new objects elements\nof some latent semantic structure, present in other, already familiar categories, and use this structure\nto construct our internal classi\ufb01er for the new category. Similarly, in the domain of computer vision,\nwe assume that we can use the plentiful set of examples (instances) of the known classes (represented\nin some latent semantic space), in order to learn to sample from the distributions of the new classes,\nthe ones for which we are given just one or a few examples.\nTeaching a neural network to sample from distributions of new visual categories, based on just a few\nobserved examples, is the essence of our proposed approach. First, the proposed approach learns\nto extract and later to sample (synthesize) transferable non-linear deformations between pairs of\nexamples of seen (training) classes. We refer to these deformations as \"deltas\" in the feature space.\nSecond, it learns to apply those deltas to the few provided examples of novel categories, unseen\nduring training, in order to ef\ufb01ciently synthesize new samples from these categories. Thus, in the\nfew-shot scenario, we are able to synthesize enough samples of each new category to train a classi\ufb01er\nin the standard supervised fashion.\nOur proposed solution is a simple, yet effective method (in the light of the obtained empirical results)\nfor learning to sample from the class distribution after being provided with one or a few examples of\nthat class. It exhibits improved performance compared to the state-of-the-art methods for few-shot\nclassi\ufb01cation on a variety of standard few-shot classi\ufb01cation benchmarks.\n\n2 Related work\n\nFew-shot learning by metric learning: a number of approaches [47, 39, 36] use a large corpus of\ninstances of known categories to learn an embedding into a metric space where some simple (usually\nL2) metric is then used to classify instances of new categories via proximity to the few labeled\ntraining examples embedded in the same space. In [13], a metric learning method based on graph\nneural networks, that goes beyond the L2 metric, have been proposed. The metric-learning-based\napproaches are either posed as a general discriminative distance metric learning (DML) scheme [36],\nor optimized to operate in the few shot scenario [39, 47, 13]. These approaches show great promise,\nand in some cases are able to learn embedding spaces with quite meaningful semantics embedded in\nthe metric [36]. Yet, their performance is in many cases inferior to the meta-learning and generative\n(synthesis) approaches that will be discussed next.\nFew-shot meta-learning (learning-to-learn): these approaches are trained on few-shot tasks instead\nof speci\ufb01c object instances, resulting in models that once trained can \"learn\" on new such tasks\nwith relatively few examples. In Matching Networks [43], a non-parametric k-NN classi\ufb01er is\nmeta-learned such that for each few-shot task the learned model generates an adaptive embedding\nspace for which the task can be better solved. In [39] the embedding space is optimized to best\nsupport task-adaptive category population centers (assuming uni-modal category distributions). In\napproaches such as MAML [10], Meta-SGD [26], DEML+Meta-SGD [52], Meta-Learn LSTM [34]\nand Meta-Networks [31], the meta-learned classi\ufb01ers are optimized to be easily \ufb01ne-tuned on new\nfew-shot tasks using small training data.\nGenerative and augmentation-based few-shot approaches: In this line of methods, either gen-\nerative models are trained to synthesize new data based on few examples, or additional examples\nare obtained by some other form of transfer learning from external data. These approaches can be\ncategorized as follows: (1) semi-supervised approaches using additional unlabeled data [6, 11]; (2)\n\n2\n\n\f\ufb01ne tuning from pre-trained models [25, 45, 46]; (3) applying domain transfer by borrowing examples\nfrom relevant categories [27] or using semantic vocabularies [2, 12]; (4) rendering synthetic examples\n[32, 7, 40]; (5) augmenting the training examples [23]; (6) example synthesis using Generative Ad-\nversarial Networks (GANs) [53, 21, 14, 35, 33, 29, 8, 20]; and (6) learning to use additional semantic\ninformation (e.g. attribute vector) per-instance for example synthesis [4, 51]. It is noteworthy that all\nthe augmentation and synthesis approaches can be used in combination with the metric learning or\nmeta-learning schemes, as we can always synthesize more data before using those approaches and\nthus (hopefully) improve their performance.\nSeveral insightful papers have recently emerged dealing with sample synthesis. In [17] it is conjec-\ntured that the relative linear offset in feature space between a pair of same-class examples conveys\ninformation on a valid deformation, and can be applied to instances of other classes. In their approach,\nsimilar (in terms of this offset) pairs of examples from different categories are mined during training\nand then used to train a generator optimized for applying the same offset to other examples. In our\ntechnique, we do not restrict our \u201cdeltas\u201d to be linear offsets, and in principle can have the encoder\nand the generator to learn more complex deformations than offsets in the feature space.\nIn [44], a generator sub-network is added to a classi\ufb01cation network in order to synthesize additional\nexamples on the \ufb02y in a way that helps training the classi\ufb01er on small data. This generator receives\nthe provided training examples accompanied by noise vectors (source of randomness). At the learning\nstage, the generator is optimized to perform random augmentation, jointly with the meta-learner\nparameters, via the classi\ufb01cation loss. In contrast, in our strategy the generator is explicitly trained,\nvia the reconstruction loss, to transfer deformations between examples and categories. A similar idea\nof learning to randomly augment class examples in a way that will improve classi\ufb01cation performance\nis explored in [1] using GANs. In [35], a few-shot class density estimation is performed with an\nautoregressive model, augmented with an attention mechanism, where examples are synthesized by a\nsequential process. Finally, the idea of learning to apply deformations on class examples has also\nbeen successfully explored in other domains, such as text synthesis [16].\n\n3 The \u2206-encoder\n\nWe propose a method for few-shot classi\ufb01cation by learning to synthesize samples of novel categories\n(unseen during training) when only a single or a few real examples are available. The generated\nsamples are then used to train a classi\ufb01er. Our proposed approach, dubbed as the \u2206-encoder, learns\nto sample from the category distribution, while being seeded by only one or few examples from\nthat distribution. Doing so, it belongs to the family of example synthesis methods. Yet, it does\nnot assume the existence of additional unlabeled data, e.g., transferable pre-trained models (on an\nexternal dataset) or any directly related examples from other categories or domains, and it does not\nrely on additional semantic information per-instance.\nThe proposed solution is to train a network comprised of an encoder and a decoder. The encoder learns\nto extract transferable deformations between pairs of examples of the same class, while the decoder\nlearns how to apply these deformations to other examples in order to learn to sample from new\ncategories. For the ease of notation, assume we are given a single example Y belonging to a certain\ncategory C, and our goal is to learn to sample additional examples X belonging to the same category.\nIn other words, we would like to learn to sample from the class posterior: P(X|C, Y ). Notice that the\nconditioning on Y implies that we may not learn to sample from the whole class posterior, but rather\nfrom its certain subset of \"modes\" that can be obtained from Y using the deformations we learned to\nextract. Our method is inspired by the one used for zero-shot classi\ufb01cation in [3], where the decoder\nis provided side information about the class, in the form of human-annotated attributes.\nOur generative model is a variant of an Auto-Encoder (AE). Standard AE learns to reconstruct a\nsignal X by minimizing (cid:107)X \u2212 \u02c6X(cid:107)1, where \u02c6X = D(E(X)) is the signal reconstructed by the AE,\nand E and D are the encoder and decoder sub-networks, respectively. A common assumption for\nan AE is that the intermediate bottleneck representation E(X), can be of much lower dimension\nthan X. This is driven by assuming the ability to extract the \"semantic essence\" of X \u2013 a minimal\nset of identifying features of X necessary for the reconstruction. The simple key idea of this work\nis to change the meaning of E(X) from representing the \"essence\" of X, to representing the delta,\nor \"additional information\" needed to reconstruct X from Y (an observed example from the same\ncategory). To this end, we propose the training architecture depicted in Figure 2a. The encoder gets\n\n3\n\n\fFigure 2: Proposed \u2206-encoder architecture. (a) Training phase: X s and Y s are a random pair of\nsamples from the same seen class; the \u2206-encoder learns to reconstruct X s. (b) Sample synthesis\nphase: X s and Y s are a random pair of samples from a random seen class, and Y u is a single example\nfrom a novel unseen class; the \u2206-encoder generates a new sample \u02c6X u from the new class.\n\nas an input both the signal X and the \"anchor\" example Y and learns to compute the representation\nof the additional information Z = E(X, Y ) needed by the decoder D in order to reconstruct the X\nfrom both Y and Z. Keeping the dimension of Z small, we ensure that the decoder D cannot use\njust Z in order to reconstruct X. This way, we regularize the encoder to strongly rely on the anchor\nexample Y for the reconstruction, thus, enabling synthesis as described next.\nFollowing training, at the sample synthesis phase, we use the trained network to sample from\nP(X|C, Y ). We use the non-parametric distribution of Z by sampling random pairs {X s, Y s} from\nthe classes seen during training (such that X s and Y s belong to the same category) and generating\nfrom them Z = E(X s, Y s) using the trained encoder. Thus, we end up with a set of samples {Zi}.\nIn each of the one-shot experiments, for a novel unseen class U we are provided with an example\nY u, from which we synthesize a set of samples for the class U using our trained generator model:\n{D(Zi, Y u)}. The process is illustrated in Figure 2b. Finally, we use the synthesized samples to\ntrain a linear classi\ufb01er (one dense layer followed by softmax). As a straightforward extension, for\nk-shot learning we repeat the process k times, independently synthesizing samples based on each of\nthe k examples provided.\n\n3.1\n\nImplementation details\n\ninclude adaptive weights:(cid:80)\n\nIn all the experiments, images are represented by pre-computed feature vectors. In all our experiments\nwe are using the VGG16 [38] or ResNet18 [18] models for feature extraction. For both models the\nhead, i.e., the layers after the last convolution, is replaced by two fully-connected layers with 2048\nunits with ReLU activations. The features used are the 2048-dimensional outputs of the last fully-\nconnected layer. Following the ideas of [28], we augment the L1 reconstruction loss ((cid:107)X \u2212 \u02c6X(cid:107)1) to\ni wi|Xi \u2212 \u02c6Xi|, where wi = |Xi \u2212 \u02c6Xi|2/(cid:107)X \u2212 \u02c6X(cid:107)2, encouraging larger\ngradients for feature dimensions with higher residual error. The encoder and decoder sub-networks\nare implemented as multi-layer perceptrons with a single hidden layer of 8192 units, where each layer\nis followed by a leaky ReLU activation (max(x, 0.2 \u00b7 x)). The encoder output Z is 16-dimensional.\nAll models are trained with Adam optimizer with the learning rate set to 10\u22125. Dropout with 50%\nrate is applied to all layers. In all experiments 1024 samples are synthesized for each unseen class.\nThe \u2206-encoder training takes about 10 epochs to reach convergence; each epoch takes about 20\nseconds running on an Nvidia Tesla K40m GPU (48K training samples, batch size 128). The data\ngeneration phase takes around 0.1 seconds per 1024 samples. The code is available here.\n\n4\n\n\ud835\udc4b\ud835\udc60\u0de0\ud835\udc4b\ud835\udc60EncoderDecoder\ud835\udc4d\ud835\udc4c\ud835\udc60\ud835\udc4b\ud835\udc60EncoderDecoder\ud835\udc4d\ud835\udc4c\ud835\udc60(a) Training phase:(b) Sample synthesis phase:\ud835\udc4c\ud835\udc62\u0de0\ud835\udc4b\ud835\udc62\f4 Results\n\nWe have evaluated the few-shot classi\ufb01cation performance of the proposed method on multiple\ndatasets, which are the benchmarks of choice for the majority of few-shot learning literature, namely:\nminiImageNet, CIFAR-100, Caltech-256, CUB, APY, SUN and AWA2. These datasets are common\nbenchmarks for the zero- and few-shot object recognition, and span a large variety of properties\nincluding high- and low-resolution images, tens to hundreds of \ufb01ne- and coarse-grained categories,\netc. We followed the standard splits used for few-shot learning for the \ufb01rst four datasets; for the other\ndatasets that are not commonly used for few-shot, we used the split suggested in [49] for zero-shot\nlearning. Table 3 summarizes the properties of the tested datasets.\nIn all of our experiments, the data samples X and Y are feature vectors computed by a pre-trained\nneural-network. The experimental protocol in terms of splitting of the dataset into disjoint sets of\ntraining and testing classes is the same as in all the other works evaluated on the same datasets. We\nuse the VGG16 [38] backbone network for computing the features in all of our experiments except\nthose on Caltech-256 and CUB. For these small-scale datasets we used ResNet18 [18], same as\n[4], to avoid over-\ufb01tting. We show that even in this simple setup of using pre-computed feature\nvectors, competitive results can be obtained by the proposed method compared to the few-shot\nstate-of-the-art. Combining the proposed approach with an end-to-end training of the backbone\nnetwork is an interesting future research direction beyond the scope of this work.\nAs in compared approaches, we evaluate our approach by constructing \"few-shot test episode\" tasks.\nIn each test episode for the N-way k-shot classi\ufb01cation task, we draw N random unseen categories,\nand draw k random samples from each category. Then, in order to evaluate performance on the\nepisode, we use our trained network to synthesize a total of 1024 samples per category based on\nthose k examples. This is followed by training a simple linear N-class classi\ufb01er over those 1024 \u00b7 N\nsamples, and \ufb01nally, the calculation of the few-shot classi\ufb01cation accuracy on a set of M real (query)\nsamples from the tested N categories. In our experiments, instead of using a \ufb01xed (large) value for\nM, we simply test the classi\ufb01cation accuracy on all of the remaining samples of the N categories that\nwere not used for one- or few-shot training. Average performance on 10 such experiments is reported.\n\n4.1 Standard benchmarks\n\nFor miniImageNet, CIFAR100, CUB and Caltech-256 datasets, we evaluate our approach using a\nbackbone network (for computing the feature vectors) trained from scratch on a subset of categories\nof each dataset. For few-shot testing, we use the remaining unseen categories. The proposed synthesis\nnetwork is trained on the same set of categories as the backbone network. The experimental protocol\nused here is the same as in all compared methods.\nThe performance achieved by our approach is summarized in Table 1; it competes favorably to the\nstate-of-the-art of few-shot classi\ufb01cation on these datasets. The performance of competing methods\nis taken from [4]. We remark in the table whenever a method uses some form of additional external\ndata, be it training on an external large-scale dataset, using word embedding applied to the category\nname, or using human-annotated class attributes.\n\n4.2 Additional experiments using a shared pre-trained feature extracting model\n\nFor fair comparison, in the experiments described above in Section 4.1 we only trained our feature\nextractor backbone on the subset of training categories of the target dataset (same as in other works).\nHowever, it is nonetheless interesting to see how our proposed method performs in a realistic setting\nof having a single pre-trained feature extractor backbone trained on a large set of external data. To\nthis end, we have conducted experiments on four public datasets (APY, AWA2, CUB and SUN),\nwhere we used features obtained from a VGG16 backbone pre-trained on ImageNet. The unseen test\ncategories were veri\ufb01ed to be disjoint from the ImageNet categories in [49] that dealt with dataset bias\nin zero-shot experiments. The results of our experiments as well as comparisons to some baselines\nare summarized in Table 2. The experiments in this section illustrate that the proposed method\ncan strongly bene\ufb01t from better features trained on more data. For CUB with \"stronger\" ImageNet\nfeatures (last column in Table 2) we achieved more than 10% improvement over training only using a\nsubset of CUB categories (last column in Table 1).\n\n5\n\n\fTable 1: 1-shot/5-shot 5-way accuracy results\n\nminiImageNet CIFAR100\nMethod\nCaltech-256\nNearest neighbor (baseline)\n56.1 / 68.3\n44.1 / 55.1\n51.3 / 67.5\nMACO [19]\n-\n41.1 / 58.3\n-\nMeta-Learner LSTM [34]\n43.4 / 60.6\n-\n-\nMatching Nets [43]\n50.5 / 60.3\n46.6 / 60.0\n48.1 / 57.5\nMAML [10]\n49.3 / 58.3\n48.7 / 63.1\n45.6 / 54.6\nPrototypical Networks [39]\n-\n49.4 / 68.2\n-\nSRPN [30]\n-\n55.2 / 69.6\n-\nRELATION NET [41]\n57.0 / 71.1\n-\n-\nDEML+Meta-SGD\u2665 [52]\n61.6 / 77.9 (cid:5)\n58.5 / 71.3 (cid:5)\n62.2 / 79.5 (cid:5)\n63.4 / 78.4 \u2020\n58.1 / 76.9 \u2020\n63.8 / 80.5 \u2020\nDual TriNet\u2665 [4]\n\u2206-encoder\u2665\n66.7 / 79.8\n59.9 / 69.7\n73.2 / 83.6\n(cid:5) Also trained on an a large external dataset\n\u2020 Using label embedding trained on large corpus\n\u2665 Using ResNet features\n(cid:63) Using human annotated class attributes\n\nCUB\n52.4 / 66.0\n60.8 / 75.0\n40.4 / 49.7\n49.3 / 59.3\n38.4 / 59.1\n-\n-\n-\n66.9 / 77.1 (cid:5)\n69.6 / 84.1 (cid:63)\n69.8 / 82.6\n\nTable 2: 1-shot/5-shot 5-way accuracy with ImageNet model features (trained on disjoint categories)\n\nMethod\nNearest neighbor (baseline)\nPrototypical Networks\n\u2206-encoder\n\nAWA2\n\n65.9 / 84.2\n80.8 / 95.3\n90.5 / 96.4\n\nAPY\n\nSUN\n\nCUB\n\n57.9 / 76.4\n69.8 / 90.1\n82.5 / 93.4\n\n72.7 / 86.7\n74.7 / 94.8\n82.0 / 93.0\n\n58.7 / 80.2\n71.9 / 92.4\n82.2 / 92.6\n\n4.3 Ablation study: evaluating different design choices\n\nIn this section we review and evaluate different design choices used in the architecture and the\napproach proposed in this paper. For this ablation study we use the performance estimates for the\nAWA, APY, SUN and CUB datasets to compare the different choices.\nIn [3] the authors suggested the usage of Denoising-Autoencoder (DAE) for zero-shot learning (Fig.\n3a). The noise is implemented as 20% dropout on the input. In training time, the DAE learns to\nreconstruct X s from it\u2019s noisy version, where the decoder uses the class attributes to perform the\nreconstruction. At test time, the decoder is used to synthesize examples from a novel class using its\nattributes vector and a random noise vector Z. Average accuracy for the zero-shot task is 64.4% (\ufb01rst\nrow in Table 4). As a \ufb01rst step towards one-shot learning, we have tested the same architecture but\nusing another sample from the same class instead of the attributes vector (Figure 3b). The intuition\nbehind this is that the decoder will learn to reconstruct the class instances by editing another instances\nfrom the same class instead of relying on the class attributes. This already yields a signi\ufb01cant\nimprovement to the average accuracy bringing it to 81.1%, hinting that even a single class instance\nconveys more information than the human chosen attributes for the class in these datasets.\nNext, we replaced the random sampling of Z with a non-parametric density estimate of it obtained\nfrom the training set. Instead of sampling entries of Z \u223c N (0, 1), we randomly sample an instance X s\nbelonging to a randomly chosen training class and run it through the encoder to produce Z = E(X s).\nThis variant assumes that the distribution of Z is similar between the seen and unseen classes. We\nobserved a slight improvement of 0.5% due to this change. We also tested a variant where no noise\nis injected to the input, i.e. replacing Denoising-Autoencoder with Autoencoder. Since we did not\nobserve a change in performance we chose the Autoencoder for being the simpler of the two. Finally,\nto get to our \ufb01nal architecture as described in Section 3 we add Y as input to the encoder. This\nimproved the performance by 2.4%.\nLinear offset delta To evaluate the effect of the learned non-linear \u2206-encoder we also experimented\nwith replacing it with a linear \"delta\" in the embedding space. In this experiment we set Z =\nE(X s, Y s) = X s \u2212 Y s and \u02c6X = D(Z, Y u) = Y u + Z. That means we sample linear shifts from\nsame-class pairs in the training set and use them to augment the single example of a new class Y u that\nwe have. For this experiment we got \u223c 10 points lower accuracy compared to \u2206-encoder, showing\nthe importance of the learned non-linear \"delta\" encoding.\n\n6\n\n\fTable 3: Summary of the datasets used in our experiments\n\nDataset\nminiImageNet [43]\nCIFAR-100 [22]\nCaltech-256 Object Category [15]\nCaltech-UCSD Birds 200 (CUB) [48]\nAttribute Pascal & Yahoo (aPY) [9]\nScene UNderstanding (SUN) [50]\nAnimals with Attributes 2 (AWA2) [49]\n\nFine\n\ngrained\n\n\u0017\n\u0017\n\u0017\n\u0013\n\u0017\n\u0013\n\u0017\n\nImage\nsize\n\nMedium\nSmall\nLarge\nLarge\nLarge\nLarge\nLarge\n\nTotal #\nimages\n\nSeen\nclasses\n\nUnseen\nclasses\n\n60K\n60K\n30K\n12K\n14K\n14K\n37K\n\n80\n80\n156\n150\n20\n645\n40\n\n20\n20\n50\n50\n12\n72\n10\n\nTable 4: Evaluating different design choices. All the numbers are one-shot accuracy in %.\n\nMethod\nZero-shot (Y is attribute vector) [3] (Fig. 3a)\nTransferring linear offsets in the embedding space\nReplacing attribute with sample from class (Fig. 3b)\nZ is sampled from training set, not random (Fig. 3c)\nAutoencoder instead of denoising-autoencoder (Fig. 3d)\nAdding Y as input to encoder too (Fig. 2)\n\nAWA2 APY SUN CUB Avg.\n63.4\n66.4\n75.2\n81.3\n81.4\n85.4\n86.6\n81.9\n81.9\n88.2\n90.5\n84.3\n\n42.8\n73.9\n81.1\n77.0\n79.1\n82.2\n\n62.0\n72.1\n78.1\n84.2\n80.9\n82.5\n\n82.5\n73.5\n81.1\n80.1\n79.5\n82\n\n4.4 Are we synthesizing non-trivial samples?\n\nWe have observed a signi\ufb01cant few-shot performance boost when using the proposed method for\nsample synthesis when compared to the baseline of using just the few provided examples. But are\nwe learning to generate any signi\ufb01cant new information on the class manifold in the feature space?\nOr are we simply augmenting the provided examples slightly? We chose two ways to approach this\nquestion. First, evaluating the performance as more samples are synthesized. Second, visualizing the\nsynthesized samples in the feature space.\nFigure 4 presents the accuracy as a function of number of generated samples in both 1-shot and 5-shot\nscenarios when evaluated on the miniImageNet dataset. As can be seen the performance improves\nwith the number of samples synthesized, converging after 512 \u2212 1024 samples. For this reason we\nuse 1024 synthesized samples in all of our experiments. It is interesting to note that at convergence,\nnot only using the 5-shot synthesized samples is signi\ufb01cantly better then the baseline of using just the\n\ufb01ve provided real examples, but also using the samples synthesized from just one real example is\nbetter then using \ufb01ve real examples (without synthesis). This may suggest that the proposed synthesis\napproach does learn something non-trivial surpassing the addition of four real examples.\nTo visualize the synthesized samples we plot\nthem for the case of 12 classes unseen during\ntraining (Figure 5a). The samples were synthe-\nsized from a single real example for each class\n(1-shot mode) and plotted in 2d using t-SNE.\nAs can be seen from the \ufb01gure, the synthesized\nsamples reveal a non trivial density structure\naround the seed examples. Moreover, the seed\nexamples are not the centers of the synthesized\npopulations (we veri\ufb01ed that same is true before\napplying t-SNE) as would be expected for naive\naugmentation by random perturbation. Hence,\nthe classi\ufb01er learned from the synthesized exam-\nples signi\ufb01cantly differs from the nearest neigh-\nbors baseline classi\ufb01er that is using the seed\nexamples alone (improving its performance by\n20 \u2212 30 points in tables 1 & 2). In addition,\nFigure 5b shows visualizations for some of the\n\nFigure 4: miniImageNet 5-way Accuracy vs.\nnumber of generated samples. As indicated by\nthe accuracy trend we keep generating new mean-\ningful samples till we reach convergence at \u223c 1K\nsamples.\n\n7\n\n23242526272829210Number of generated samples45505560657075Accuracy1-shot accuracy1-shot nearest-neighbor baseline5-shot accuracy5-shot nearest-neighbor baseline\fFigure 3: Different design choices tried. Classi\ufb01cation accuracy for each architecture is presented\nin Table 4. The \ufb01nal chosen architecture is depicted in Fig. 2.\n\nsynthesized feature vectors obtained from a given seed example. The images displayed are the nearest\nneighbors of the synthesized ones in the feature space.\n\nFigure 5: a. Generated samples for 12-way one-shot. The red crosses mark the original 12\nsingle-samples. The generated points are colored according to their class. b. Synthesized samples\nvisualization. The single image seen at training is framed in blue. All other images represent the\nsynthesized samples visualized using their nearest \"real image\" neighbors in the feature space. The\ntwo-dimensional embedding was produced by t-SNE. Best viewed in color.\n5 Summary and Future work\n\nIn this work, we proposed a novel auto-encoder like architecture, the \u2206-encoder. This model learns\nto generate novel samples from a distribution of a class unseen during training using as little as\none example from that class. The \u2206-encoder was shown to achieve state-of-the-art results in the\ntask of few-shot classi\ufb01cation. We believe that this new tool can be utilized in a variety of more\ngeneral settings challenged by the scarceness of labeled examples, e.g., in semi-supervised and active\nlearning. In the latter case, new candidate examples for labeling can be selected by \ufb01rst generating\nnew samples using the \u2206-encoder, and then picking the data points that are farthest from the generated\nsamples. Additional, more technical, research directions include iterative sampling from the generated\ndistribution by feeding the generated samples as reference examples, and conditioning the sampling\nof the \"deltas\" on the anchor example for better controlling the set of transformations suitable for\ntransfer. We leave these interesting directions for future research.\n\n8\n\n\ud835\udc4e.Zero-shot (Y is attributes vector)\ud835\udc4b\ud835\udc60+\ud835\udc41\u0de0\ud835\udc4bEncoderDecoder\ud835\udc4d\ud835\udc4b\ud835\udc60EncoderDecoder\ud835\udc4dTraining phase:Sample synthesis phase:\ud835\udc4c\ud835\udc62\u0de0\ud835\udc4bc.Z is sampled from training set (not random)\ud835\udc4c=\ud835\udc4e\ud835\udc61\ud835\udc61\ud835\udc5f\ud835\udc56\ud835\udc4f\ud835\udc62\ud835\udc61\ud835\udc52\ud835\udc60\ud835\udc4b\ud835\udc60+\ud835\udc41\u0de0\ud835\udc4bEncoderDecoder\ud835\udc4dDecoder\ud835\udc4d=\ud835\udc41(0,1)Training phase:Sample synthesis phase:\u0de0\ud835\udc4b\ud835\udc4c=\ud835\udc4e\ud835\udc61\ud835\udc61\ud835\udc5f\ud835\udc56\ud835\udc4f\ud835\udc62\ud835\udc61\ud835\udc52\ud835\udc60b.Replacing attribute with sample from class\ud835\udc4b\ud835\udc60+\ud835\udc41\u0de0\ud835\udc4bEncoderDecoder\ud835\udc4dDecoder\ud835\udc4d=\ud835\udc41(0,1)Training phase:Sample synthesis phase:\u0de0\ud835\udc4b\ud835\udc4c\ud835\udc60\ud835\udc4c\ud835\udc62\ud835\udc4c\ud835\udc60\ud835\udc4b\ud835\udc60\u0de0\ud835\udc4bEncoderDecoder\ud835\udc4d\ud835\udc4b\ud835\udc60EncoderDecoder\ud835\udc4dTraining phase:Sample synthesis phase:\ud835\udc4c\ud835\udc62\u0de0\ud835\udc4bd.Autoencoder instead of denoising-autoencoder\ud835\udc4c\ud835\udc60\ud835\udc4e.\ud835\udc4f.\fAcknowledgment: Part of this research was partially supported by the ERC-StG SPADE grant.\nRogerio Feris is partly supported by IARPA via DOI/IBC contract number D17PC00341. Alex\nBronstein is supported by ERC StG RAPID. The U.S. Government is authorized to reproduce and\ndistribute reprints for Governmental purposes notwithstanding any copyright annotation thereon.\nDisclaimer: The views and conclusions contained herein are those of the authors and should not\nbe interpreted as necessarily representing the of\ufb01cial policies or endorsements, either expressed or\nimplied, of IARPA, DOI/IBC, or the U.S. Government)\n\nReferences\n[1] A. Antoniou, A. Storkey, and H. Edwards. Data Augmentation Generative Adversarial Networks.\n\narXiv:1711.04340, 2017.\n\n[2] J. L. Ba, K. Swersky, S. Fidler, and R. Salakhutdinov. Predicting deep zero-shot convolutional\nneural networks using textual descriptions. Proceedings of the IEEE International Conference\non Computer Vision, 2015 Inter:4247\u20134255, 2015.\n\n[3] M. Bucher, S. Herbin, and F. Jurie. Generating Visual Representations for Zero-Shot Classi\ufb01ca-\n\ntion. arXiv:1708.06975, 8 2017.\n\n[4] Z. Chen, Y. Fu, Y. Zhang, Y.-G. Jiang, X. Xue, and L. Sigal. Semantic Feature Augmentation in\n\nFew-shot Learning. arXiv:1804.05298v2, 2018.\n\n[5] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and F.-F. Li. ImageNet: A large-scale hierarchical\nimage database. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages\n248\u2013255, 2009.\n\n[6] X. Dong, L. Zheng, F. Ma, Y. Yang, and D. Meng. Few-shot Object Detection. Arxiv:1706.08249,\n\npages 1\u201311, 2017.\n\n[7] A. Dosovitskiy, J. T. Springenberg, M. Tatarchenko, and T. Brox. Learning to Generate Chairs,\nTables and Cars with Convolutional Networks. IEEE Transactions on Pattern Analysis and\nMachine Intelligence, 39(4):692\u2013705, 2017.\n\n[8] I. Durugkar, I. Gemp, and S. Mahadevan. Generative Multi-Adversarial Networks. International\n\nConference on Learning Representations (ICLR), pages 1\u201314, 2017.\n\n[9] A. Farhadi, I. Endres, D. Hoiem, and D. Forsyth. Describing Objects by their Attributes. IEEE\n\nConference on Computer Vision and Pattern Recognition (CVPR), pages 1778\u20131785, 2009.\n\n[10] C. Finn, P. Abbeel, and S. Levine. Model-Agnostic Meta-Learning for Fast Adaptation of Deep\n\nNetworks. arXiv:1703.03400, 2017.\n\n[11] Y. Fu, T. M. Hospedales, T. Xiang, and S. Gong. Transductive Multi-View Zero-Shot Learning.\n\nIEEE Transactions on Pattern Analysis and Machine Intelligence, 37(11):2332\u20132345, 2015.\n\n[12] Y. Fu and L. Sigal. Semi-supervised Vocabulary-informed Learning. IEEE Conference on\n\nComputer Vision and Pattern Recognition (CVPR), pages 5337\u20135346, 2016.\n\n[13] V. Garcia and J. Bruna. Few-Shot Learning with Graph Neural Networks. arXiv:1711.04043,\n\npages 1\u201313, 2017.\n\n[14] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and\nY. Bengio. Generative Adversarial Nets. Advances in Neural Information Processing Systems\n27, pages 2672\u20132680, 2014.\n\n[15] G. Grif\ufb01n, a. Holub, and P. Perona. Caltech-256 object category dataset. Caltech mimeo,\n\n11(1):20, 2007.\n\n[16] K. Guu, T. B. Hashimoto, Y. Oren, and P. Liang. Generating Sentences by Editing Prototypes.\n\nArxiv:1709.08878, 2017.\n\n[17] B. Hariharan and R. Girshick. Low-shot Visual Recognition by Shrinking and Hallucinating\n\nFeatures. IEEE International Conference on Computer Vision (ICCV), 2017.\n\n9\n\n\f[18] K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition.\n\narXiv:1512.03385, 2015.\n\n[19] N. Hilliard, L. Phillips, S. Howland, A. Yankov, C. D. Corley, and N. O. Hodas. Few-Shot\n\nLearning with Metric-Agnostic Conditional Embeddings. 2 2018.\n\n[20] G. Huang, Z. Liu, L. v. d. Maaten, and K. Q. Weinberger. Densely Connected Convolutional\nNetworks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages\n2261\u20132269, 2017.\n\n[21] E. S. Jun-Yan Zhu, Philipp Krahenbuhl and A. Efros. Generative Visual Manipulation on the\nNatural Image Manifold. European Conference on Computer Vision (ECCV)., pages 597\u2013613,\n2016.\n\n[22] A. Krizhevsky. Learning Multiple Layers of Features from Tiny Images. Technical report.\n\nScience Department, University of Toronto, Tech., pages 1\u201360, 2009.\n\n[23] A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet Classi\ufb01cation with Deep Convolutional\n\nNeural Networks. Advances In Neural Information Processing Systems, pages 1\u20139, 2012.\n\n[24] B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum. Human-level concept learning through\n\nprobabilistic program induction. Science, 350(6266):1332\u20131338, 2015.\n\n[25] Z. Li and D. Hoiem. Learning without Forgetting. IEEE Transactions on Pattern Analysis and\n\nMachine Intelligence, pages 1\u201313, 2016.\n\n[26] Z. Li, F. Zhou, F. Chen, and H. Li. Meta-SGD: Learning to Learn Quickly for Few-Shot\n\nLearning. arXiv:1707.09835, 2017.\n\n[27] J. J. Lim, R. Salakhutdinov, and A. Torralba. Transfer Learning by Borrowing Examples for\nMulticlass Object Detection. Advances in Neural Information Processing Systems 26 (NIPS),\npages 1\u20139, 2012.\n\n[28] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Doll\u00e1r. Focal Loss for Dense Object Detection.\n\narXiv:1708.02002, 2017.\n\n[29] X. Mao, Q. Li, H. Xie, R. Y. K. Lau, Z. Wang, and S. P. Smolley. Least Squares Generative\nAdversarial Networks. IEEE International Conference on Computer Vision (ICCV), pages 1\u201316,\n2016.\n\n[30] A. Mehrotra and A. Dukkipati. Generative Adversarial Residual Pairwise Networks for One\n\nShot Learning. arXiv:1703.08033, 2017.\n\n[31] T. Munkhdalai and H. Yu. Meta Networks. arXiv:1703.00837, 2017.\n\n[32] D. Park and D. Ramanan. Articulated pose estimation with tiny synthetic videos.\n\nIEEE\nConference on Computer Vision and Pattern Recognition (CVPR), 2015-Octob:58\u201366, 2015.\n\n[33] A. Radford, L. Metz, and S. Chintala. Unsupervised Representation Learning with Deep\n\nConvolutional Generative Adversarial Networks. arXiv:1511.06434, pages 1\u201316, 2015.\n\n[34] S. Ravi and H. Larochelle. Optimization As a Model for Few-Shot Learning. International\n\nConference on Learning Representations (ICLR), pages 1\u201311, 2017.\n\n[35] S. Reed, Y. Chen, T. Paine, A. van den Oord, S. M. A. Eslami, D. Rezende, O. Vinyals, and\nN. de Freitas. Few-shot autoregressive density estimation: towards learning to learn distributions.\narXiv:1710.10304, (2016):1\u201311, 2018.\n\n[36] O. Rippel, M. Paluri, P. Dollar, and L. Bourdev. Metric Learning with Adaptive Density\n\nDiscrimination. arXiv:1511.05939, (Dml):1\u201315, 2015.\n\n[37] F. Schroff, D. Kalenichenko, and J. Philbin. FaceNet: A uni\ufb01ed embedding for face recognition\nand clustering. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages\n815\u2013823, 2015.\n\n10\n\n\f[38] K. Simonyan and A. Zisserman. Very Deep Convolutional Networks for Large-Scale Image\n\nRecognition. CoRR arXiv:1409.1556, abs/1409.1:1\u201314, 2014.\n\n[39] J. Snell, K. Swersky, and R. S. Zemel. Prototypical Networks for Few-shot Learning. Advances\n\nIn Neural Information Processing Systems (NIPS), 2017.\n\n[40] H. Su, C. R. Qi, Y. Li, and L. J. Guibas. Render for CNN Viewpoint Estimation in Images\nUsing CNNs Trained with Rendered 3D Model Views.pdf. IEEE International Conference on\nComputer Vision (ICCV), pages 2686\u20132694, 2015.\n\n[41] F. Sung, Y. Yang, L. Zhang, T. Xiang, P. H. S. Torr, and T. M. Hospedales. Learning to Compare:\n\nRelation Network for Few-Shot Learning. arXiv:1711.06025, 11 2017.\n\n[42] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and\nA. Rabinovich. Going deeper with convolutions. In IEEE Computer Society Conference on\nComputer Vision and Pattern Recognition (CVPR), volume 07-12-June, pages 1\u20139. IEEE, 6\n2015.\n\n[43] O. Vinyals, C. Blundell, T. Lillicrap, K. Kavukcuoglu, and D. Wierstra. Matching Networks for\n\nOne Shot Learning. Advances In Neural Information Processing Systems (NIPS), 2016.\n\n[44] Y.-X. Wang, R. Girshick, M. Hebert, and B. Hariharan. Low-Shot Learning from Imaginary\n\nData. arXiv:1801.05401, 2018.\n\n[45] Y.-X. Wang and M. Hebert. Learning from Small Sample Sets by Combining Unsupervised\nMeta-Training with CNNs. Advances In Neural Information Processing Systems (NIPS),\n(Nips):244\u2013252, 2016.\n\n[46] Y.-X. Wang and M. Hebert. Learning to Learn: Model Regression Networks for Easy Small\nSample Learning. European Conference on Computer Vision (ECCV), pages 616\u2013634, 2016.\n\n[47] K. Q. Weinberger and L. K. Saul. Distance Metric Learning for Large Margin Nearest Neighbor\n\nClassi\ufb01cation. The Journal of Machine Learning Research, 10:207\u2013244, 2009.\n\n[48] P. Welinder, S. Branson, T. Mita, and C. Wah. Caltech-UCSD birds 200. pages 1\u201315, 2010.\n\n[49] Y. Xian, C. H. Lampert, B. Schiele, and Z. Akata. Zero-Shot Learning - A Comprehensive\n\nEvaluation of the Good, the Bad and the Ugly. arXiv:1707.00600, pages 1\u201314, 2018.\n\n[50] J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. SUN database: Large-scale scene\nrecognition from abbey to zoo. In IEEE Conference on Computer Vision and Pattern Recognition\n(CVPR), pages 3485\u20133492. IEEE, 6 2010.\n\n[51] A. Yu and K. Grauman. Semantic Jitter: Dense Supervision for Visual Comparisons via\nSynthetic Images. Proceedings of the IEEE International Conference on Computer Vision,\n2017-Octob:5571\u20135580, 2017.\n\n[52] F. Zhou, B. Wu, and Z. Li. Deep Meta-Learning: Learning to Learn in the Concept Space.\n\narXiv:1802.03596, 2 2018.\n\n[53] J. Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired Image-to-Image Translation Using\nCycle-Consistent Adversarial Networks. Proceedings of the IEEE International Conference on\nComputer Vision, 2017-Octob:2242\u20132251, 2017.\n\n[54] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le. Learning Transferable Architectures for\n\nScalable Image Recognition. arXiv:1707.07012, 2017.\n\n11\n\n\f", "award": [], "sourceid": 1493, "authors": [{"given_name": "Eli", "family_name": "Schwartz", "institution": "IBM-Research"}, {"given_name": "Leonid", "family_name": "Karlinsky", "institution": "IBM-Research"}, {"given_name": "Joseph", "family_name": "Shtok", "institution": "IBM-Reseach"}, {"given_name": "Sivan", "family_name": "Harary", "institution": "IBM-Research"}, {"given_name": "Mattias", "family_name": "Marder", "institution": "IBM-Research"}, {"given_name": "Abhishek", "family_name": "Kumar", "institution": "Google"}, {"given_name": "Rogerio", "family_name": "Feris", "institution": "IBM Research AI"}, {"given_name": "Raja", "family_name": "Giryes", "institution": "Tel Aviv University"}, {"given_name": "Alex", "family_name": "Bronstein", "institution": "Technion"}]}