{"title": "Out-of-Distribution Detection using Multiple Semantic Label Representations", "book": "Advances in Neural Information Processing Systems", "page_first": 7375, "page_last": 7385, "abstract": "Deep Neural Networks are powerful models that attained remarkable results on a variety of tasks. These models are shown to be extremely efficient when training and test data are drawn from the same distribution. However, it is not clear how a network will act when it is fed with an out-of-distribution example. In this work, we consider the problem of out-of-distribution detection in neural networks. We propose to use multiple semantic dense representations instead of sparse representation as the target label. Specifically, we propose to use several word representations obtained from different corpora or architectures as target labels. We evaluated the proposed model on computer vision, and speech commands detection tasks and compared it to previous methods. Results suggest that our method compares favorably with previous work. Besides, we present the efficiency of our approach for detecting wrongly classified and adversarial examples.", "full_text": "Out-of-Distribution Detection using Multiple\n\nSemantic Label Representations\n\nGabi Shalev\n\nBar-Ilan University, Israel\nshalev.gabi@gmail.com\n\nYossi Adi\n\nBar-Ilan University, Israel\nyossiadidrum@gmail.com\n\nJoseph Keshet\n\nBar-Ilan University, Israel\njkeshet@cs.biu.ac.il\n\nAbstract\n\nDeep Neural Networks are powerful models that attained remarkable results on a\nvariety of tasks. These models are shown to be extremely ef\ufb01cient when training\nand test data are drawn from the same distribution. However, it is not clear how a\nnetwork will act when it is fed with an out-of-distribution example. In this work,\nwe consider the problem of out-of-distribution detection in neural networks. We\npropose to use multiple semantic dense representations instead of sparse representa-\ntion as the target label. Speci\ufb01cally, we propose to use several word representations\nobtained from different corpora or architectures as target labels. We evaluated\nthe proposed model on computer vision, and speech commands detection tasks\nand compared it to previous methods. Results suggest that our method compares\nfavorably with previous work. Besides, we present the ef\ufb01ciency of our approach\nfor detecting wrongly classi\ufb01ed and adversarial examples.\n\n1\n\nIntroduction\n\nDeep Neural Networks (DNNs) have gained lots of success after enabling several breakthroughs in\nnotably challenging problems such as image classi\ufb01cation [12], speech recognition [1] and machine\ntranslation [4]. These models are known to generalize well on inputs that are drawn from the same\ndistribution as of the examples that were used to train the model [43]. In real-world scenarios, the\ninput instances to the model can be drawn from different distributions, and in these cases, DNNs\ntend to perform poorly. Nevertheless, it was observed that DNNs often produce high con\ufb01dence\npredictions for unrecognizable inputs [33] or even for a random noise [13]. Moreover, recent works\nin the \ufb01eld of adversarial examples generation show that due to small input perturbations, DNNs tend\nto produce high probabilities while being greatly incorrect [11, 6, 17]. When considering AI Saftey,\nit is essential to train DNNs that are aware of the uncertainty in the predictions [2]. Since DNNs are\nubiquitous, present in nearly all segments of technology industry from self-driving cars to automated\ndialog agents, it becomes critical to design classi\ufb01ers that can express uncertainty when predicting\nout-of-distribution inputs.\nRecently, several studies proposed different approaches to handle this uncertainty [13, 25, 23, 19].\nIn [13] the authors proposed a baseline method to detect out-of-distribution examples based on the\nmodels\u2019 output probabilities. The work in [25] extended the baseline method by using temperature\nscaling of the softmax function and adding small controlled perturbations to inputs [14]. In [23] it\nwas suggested to add another term to the loss so as to minimize the Kullback-Leibler (KL) divergence\nbetween the models\u2019 output for out-of-distribution samples and the uniform distribution.\nEnsemble of classi\ufb01ers with optional adversarial training was proposed in [19] for detecting out-of-\ndistribution examples. Despite their high detection rate, ensemble methods require the optimization of\nseveral models and therefore are resource intensive. Additionally, each of the classi\ufb01ers participated\nin the ensemble is trained independently and the representation is not shared among them.\n\n32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montr\u00e9al, Canada.\n\n\fIn this work, we replace the traditional supervision during training by using several word embeddings\nas the model\u2019s supervision, where each of the embeddings was trained on a different corpus or with a\ndifferent architecture. More speci\ufb01cally, our classi\ufb01er is composed of several regression functions,\neach of which is trained to predict a word embedding of the target label. At inference time, we gain\nrobustness in the prediction by making decision based on the output of the regression functions.\nAdditionally, we use the L2-norm of the outputs as a score for detecting out-of-distribution instances.\nWe were inspired by several studies. In [26] the authors presented a novel technique for robust\ntransfer learning, where they proposed to optimize multiple orthogonal predictors while using a shared\nrepresentation. Although being orthogonal to each other, according to their results, the predictors\nwere likely to produce identical softmax probabilities. Similarly, we train multiple predictors that\nshare a common representation, but instead of using the same supervision and forcing orthogonality\nbetween them, we use different supervisions based on word representations. The idea of using word\nembeddings as a supervision was proposed in [8] for the task of zero-shot learning. As opposed to\nours, their model was composed of a single predictor. Last, [39] found a link between the L2-norm of\nthe input representation and the ability to discriminate in a target domain. We continue this thread\nhere, where we explore the use of the L2-norm for detecting out-of-distribution samples.\nThe contributions of this paper are as follows:\n\nredundancy in an ensemble model with a shared representation.\n\n\u2022 We propose using several different word embeddings as a supervision to gain diversity and\n\u2022 We propose utilizing the semantic structure between word embeddings to produce semantic\n\u2022 We propose using the L2-norm of the output vectors for detecting out-of-distribution inputs.\n\u2022 We examined the use of the above approach for detecting adversarial examples and wrongly\n\nquality predictions.\n\nclassi\ufb01ed examples.\n\nThe outline of this paper is as follows. In Section 2, we formulate the notations in the paper. In\nSection 3 we describe our approach in detail. Section 4 summarizes our empirical results. In\nSections 5 and Section 6 we explore the use of our method for detecting adversarial examples and\nwrongly classi\ufb01ed examples. In Section 7 we list the related works, and we conclude the paper in\nSection 8.\n\n2 Notations and De\ufb01nitions\nWe denote by X \u2286 Rp the set of instances, which are represented as p-dimensional feature vectors,\nand we denote by Y = {1, . . . , N} the set of class labels. Each label can be referred as a word, and\nthe set Y can be considered as a dictionary. We assume that each training example (x, y) \u2208 X \u00d7 Y\nis drawn from a \ufb01xed but unknown distribution \u03c1. Our goal is to train a classi\ufb01er that performs well\non unseen examples that are drawn from the distribution \u03c1, and can also identify out-of-distribution\nexamples, which are drawn from a different distribution, \u00b5.\nOur model is based on word embedding representations. A word embedding is a mapping of a word\nor a label in the dictionary Y to a real vector space Z \u2286 RD, so that words that are semantically\nclosed have their corresponding vectors close in Z. Formally, the word embedding is a function\ne : Y \u2192 Z from the set of labels Y to an abstract vector space Z. We assume that distances in\nthe embedding space Z are measured using the cosine distance which is de\ufb01ned for two vectors\nu, v \u2208 Z as follows:\n\ndcos(u, v) =\n\n.\n\n(1)\n\n(cid:16)\n\n1 \u2212 u \u00b7 v\n(cid:107)u(cid:107)(cid:107)v(cid:107)\n\n1\n2\n\n(cid:17)\n\nTwo labels are considered semantically similar if and only if their corresponding embeddings are\nclose in Z, namely, when dcos(e(y1), e(y2)) is close to 0. When the cosine distance is close to 1, the\ncorresponding labels are semantically far apart.\n\n3 Model\n\nOur goal is to build a robust classi\ufb01er that can identify out-of-distribution inputs. In communication\ntheory, robustness is gained by adding redundancy in different levels of the transmission encoding [20].\n\n2\n\n\fFigure 1: Our proposed K-embeddings model composed of K-predictors where each contains several\nfully-connected layers. The shared layers consisting a deep neural network.\n\nBorrowing ideas from this theory, our classi\ufb01er is designed to be trained on different supervisions\nfor each class. Rather than using direct supervision, our classi\ufb01er is composed of a set of regression\nfunctions, where each function is trained to predict a different word embedding (such as GloVe,\nFastText, etc.). The prediction of the model is based on the outcome of the regression functions.\nRefer to the model depicted schematically in Figure 1. Formally, the model is composed of K\n\u03b8k : X \u2192 Z for 1 \u2264 k \u2264 K. The input for each function is an instance\nregression functions f k\nx \u2208 X , and its output is a word embedding vector in Z. Note that the word embedding spaces are not\nthe same for the different regression functions, and speci\ufb01cally the notion of distance is unique for\neach function. Overall, given an instance x, the output of the model is K different word embedding\nvectors.\nThe set of parameters of a regression function k, \u03b8k = {\u03b8shared, \u03b8k\nexcl.}, is composed of a set of\nparameters, \u03b8shared, that is shared among all the K functions, and a set of parameters, \u03b8k\nexcl., which is\n\u03b8k is trained to predict a word embedding\nexclusive to the k-th function. Each regression function f k\nvector ek(y) corresponds to the word which represents the target label y \u2208 Y. In the next subsections,\nwe give a detailed account of how the model is trained and then present the inference procedure\nfollowing the procedure to detect out-of-distribution instances.\n\n3.1 Training\n\nIn classic supervised learning, the training set is composed of M instance-label pairs.\nIn our\nsetting, each example from the training set, Strain, is composed of an instance x \u2208 X and a set\nof K different word embeddings {e1(y), . . . , eK(y)} of a target label y \u2208 Y. Namely, Strain =\n{(xi, e1(yi), ..., eK(yi))}M\ni=1.\nOur goal in training is to minimize a loss function which measures the discrepancy between the\npredicted label and the desired label. Since our intermediate representation is based on word\nembeddings we cannot use the cross-entropy loss function. Moreover, we would like to keep the\nnotion of similarity between the embedding vectors from the same space. Our surrogate loss function\nis the sum of K cosine distances between the predicted embedding and the corresponding embedding\nvector of the target label, both from the same embedding space. Namely,\n\nK(cid:88)\n\n\u00af(cid:96)(x, y; \u03b8) =\n\ndcos(ek(y), f k\n\n\u03b8k (x)).\n\n(2)\n\nThe cosine distance (or the cosine similarity) is a function often used in ranking tasks, where the goal\nis to give a high score to similar embedding vectors and a low score otherwise [9, 30].\n\nk=1\n\n3.2\n\nInference\n\nAt inference time, the regression functions predict K vectors, each corresponds to a vector in a\ndifferent word embedding space. A straightforward solution is to predict the label using hard decision\nover the K output vectors by \ufb01rst predict the label of each output and then use a majority vote over\nthe predicted labels.\n\n3\n\nx<latexit sha1_base64=\"5wjrnOk9QrryL9pe/o3mQy34hyA=\">AAADi3icbZLdbtMwFMfPGj5GYbDBJTcWFdKuqgQkQGjSJiEkLsdKt25NVTmOs1nzR2Q7pZWVN+AWnoYLXoMH4R6nKWqaciRLx+d3Pv62TpJzZmwY/t7pBHfu3ru/+6D78NHe4yf7B0/PjSo0oUOiuNKjBBvKmaRDyyyno1xTLBJOL5LbDxW/mFFtmJJf7CKnE4GvJcsYwdaHzuL5dL8X9sOloW0nWjm945+Hf34BwOn0oGPiVJFCUGkJx8aMozC3E4e1ZYTTshsXhuaY3OJrOvauxIKaiVtKLdFLH0lRprQ/0qJltFnhsDBmIRKfKbC9MW1WBf/HxoXN3k0ck3lhqST1oKzgyCpUvRulTFNi+cI7mGjmtSJygzUm1v9ON5b0K1FCYJm6eF66uBqQZG5elpuMrhlts8yzRPG00q+4yyquaTOjWFcX23S2pjNP0Ubz0QoSzN2oPfmyAS/b8KoBr7anDhp40K71q8TNP1mJO2vzlCiP02mdoYWr7u0krkzVI8Ha/x/nbTywGjO5oWLdr2Zl6dc0ai/ltnP+qh+F/ehz2Dt5A7XtwnN4AYcQwVs4gU9wCkMgkME3+A4/gr3gdfA+OKpTOzurmmewYcHHvyeiORY=</latexit><latexit sha1_base64=\"uVP+j1JwXQcs2UtoXpFx1Oi8FWA=\">AAADi3icbZLdahNBFMenWT9qtNrqpTeLQag3YVdBRQQLInhZG9OmzYYwM3u2HTofy8xsTBj2DQSv9Gm88i18EO+dzUay2Xhg4Mz5nY//DIfknBkbRb93OsGNm7du797p3r23d//B/sHDU6MKTWFIFVd6RLABziQMLbMcRrkGLAiHM3L9vuJnM9CGKfnZLnKYCHwpWcYotj50ksyn+72oHy0t3HbildN79/Pwz69vybPj6UHHJKmihQBpKcfGjOMotxOHtWWUQ9lNCgM5ptf4EsbelViAmbil1DJ86iNpmCntj7ThMtqscFgYsxDEZwpsr0ybVcH/sXFhs9cTx2ReWJC0HpQVPLQqrN4dpkwDtXzhHUw181pDeoU1ptb/TjeR8IUqIbBMXTIvXVINIJmbl+UmgzWDNss8I4qnlX7FXVZxDc2MYl1dbNPZms48DTeaj1aQYu5G7cnnDXjehhcNeLE9ddDAg3atXyVu/ski7qTNU6o8Tqd1hhauureTuDJVD4K1/z/O23hgNWZyQ8W6X83K0q9p3F7Kbef0eT+O+vGnqHf0EtW2ix6jJ+gQxegVOkIf0TEaIooy9BV9Rz+CveBF8CZ4W6d2dlY1j9CGBR/+AmGJOiw=</latexit><latexit sha1_base64=\"uVP+j1JwXQcs2UtoXpFx1Oi8FWA=\">AAADi3icbZLdahNBFMenWT9qtNrqpTeLQag3YVdBRQQLInhZG9OmzYYwM3u2HTofy8xsTBj2DQSv9Gm88i18EO+dzUay2Xhg4Mz5nY//DIfknBkbRb93OsGNm7du797p3r23d//B/sHDU6MKTWFIFVd6RLABziQMLbMcRrkGLAiHM3L9vuJnM9CGKfnZLnKYCHwpWcYotj50ksyn+72oHy0t3HbildN79/Pwz69vybPj6UHHJKmihQBpKcfGjOMotxOHtWWUQ9lNCgM5ptf4EsbelViAmbil1DJ86iNpmCntj7ThMtqscFgYsxDEZwpsr0ybVcH/sXFhs9cTx2ReWJC0HpQVPLQqrN4dpkwDtXzhHUw181pDeoU1ptb/TjeR8IUqIbBMXTIvXVINIJmbl+UmgzWDNss8I4qnlX7FXVZxDc2MYl1dbNPZms48DTeaj1aQYu5G7cnnDXjehhcNeLE9ddDAg3atXyVu/ski7qTNU6o8Tqd1hhauureTuDJVD4K1/z/O23hgNWZyQ8W6X83K0q9p3F7Kbef0eT+O+vGnqHf0EtW2ix6jJ+gQxegVOkIf0TEaIooy9BV9Rz+CveBF8CZ4W6d2dlY1j9CGBR/+AmGJOiw=</latexit><latexit sha1_base64=\"irjMf6AI3iH2bTbZS+MwUqDa7yU=\">AAADi3icbZLdbtMwFMe9BtgoDDa45MaiQuKqSkAa07SLSQhpl2OlW7emqhznZLPmj8h2Sisrb7BbeDfeBqcNappyJEvH5/c/H7ZOknNmbBj+2ekET54+29173n3xcv/V64PDN1dGFZrCkCqu9CghBjiTMLTMchjlGohIOFwnD18rfj0DbZiSP+wih4kgd5JljBLrQ5fxfHrQC/vh0vC2E9VOD9V2MT3smDhVtBAgLeXEmHEU5nbiiLaMcii7cWEgJ/SB3MHYu5IIMBO3HLXEH3wkxZnS/kiLl9FmhiPCmIVIvFIQe2/arAr+j40Lmx1PHJN5YUHSVaOs4NgqXL0bp0wDtXzhHUI187Niek80odb/TjeW8JMqIYhMXTwvXVw1SDI3L8tNBmsGbZZ5liieVvMr7rKKa2gqinV2sU1nazrzFG8UH9WQEu5G7c43DXjThrcNeLvdddDAg3auXyVu/o2VuMs2T6nyOJ2uFFq46t4WcWWqGgnR/v84b+OB1YTJjSnW9VasLP2aRu2l3HauPvWjsB99D3tnR/XC7qF36D36iCL0BZ2hc3SBhoiiDD2iX+h3sB98Dk6C05W0s1PnvEUbFnz7C6u2Nj4=</latexit>\u2713shared<latexit sha1_base64=\"YAQrKWj1GCueT3QFCnq/dDhMRvw=\">AAADnnicbZJLb9NAEMe3MY+S8kjpkcuKColTZHMAjpWQgAuiNKRNG0fWej1uVt2HtbsOjVb+IJy4wkfi23QdB8VxGGml/85vXhpNWnBmbBj+3esF9+4/eLj/qH/w+MnTZ4PD5+dGlZrCmCqu9CQlBjiTMLbMcpgUGohIOVykNx9qfrEAbZiS3+2ygJkg15LljBLrXcngKLZzsCRxsRbYzImGrEoGx+EwXBneFdFaHJ8cfPqJvJ0mhz0TZ4qWAqSlnBgzjcLCzhzRllEOVT8uDRSE3pBrmHopiQAzc6vpK/zKezKcK+2ftHjlbWc4IoxZitRHCmLnpstq5//YtLT5+5ljsigtSNo0ykuOrcL1KnDGNFDLl14QqpmfFVO/AUKtX1g/lvCDKiGIzFx8W7m4bpDm7raqthlsGHRZ7lmqeFbPr7jLa66hHVFusstdutjQhad4q/hkDSnhbtLtfNmCl1141YJXu11HLTzq5vrr4ubfWKk76/KMKo+zpInQwtX/bhBXpq6REu33x3kXj6wmTG5NsanXsKo+06h7lLvi/M0wCofRN3+vb1Fj++gFeoleowi9QyfoMzpFY0TREv1Cv9GfAAcfgy/B1ya0t7fOOUJbFkzuAP7yPws=</latexit><latexit sha1_base64=\"Qz1Eq4zFzajuxUDxqvvOwrSu0Jw=\">AAADnnicbZLLbtNAFIanMZeScknpspsRFRKryGYBLCshARtEIaR1G0fWeHzcjDoXa2YcGo38IKzYwoPwELwN4zgoicORRvrnfOemo5OVnBkbhn/2esGdu/fu7z/oHzx89PjJ4PDpuVGVpjCmiisdZ8QAZxLGllkOcamBiIzDRXbztuEXc9CGKfnVLkqYCnItWcEosd6VDo4SOwNLUpdogc2MaMjrdHASDsOl4V0RrcTJ6cH778e/4/wsPeyZJFe0EiAt5cSYSRSWduqItoxyqPtJZaAk9IZcw8RLSQSYqVtOX+Pn3pPjQmn/pMVL72aGI8KYhch8pCB2Zrqscf6PTSpbvJk6JsvKgqRto6Li2CrcrALnTAO1fOEFoZr5WTH1GyDU+oX1EwnfqBKCyNwlt7VLmgZZ4W7repvBmkGXFZ5liufN/Iq7ouEaNiOqdXa1S+drOvcUbxWPV5AS7uJu58sNeNmFVxvwarfraAOPurn+urj5N1bmvnR5TpXHedpGaOGafzeIK9PUyIj2++O8i0dWEya3pljXa1ndnGnUPcpdcf5yGIXD6LO/11eotX10jJ6hFyhCr9Ep+oDO0BhRtEA/0E/0K8DBu+Bj8KkN7e2tco7QlgXxX8b0QKc=</latexit><latexit sha1_base64=\"Qz1Eq4zFzajuxUDxqvvOwrSu0Jw=\">AAADnnicbZLLbtNAFIanMZeScknpspsRFRKryGYBLCshARtEIaR1G0fWeHzcjDoXa2YcGo38IKzYwoPwELwN4zgoicORRvrnfOemo5OVnBkbhn/2esGdu/fu7z/oHzx89PjJ4PDpuVGVpjCmiisdZ8QAZxLGllkOcamBiIzDRXbztuEXc9CGKfnVLkqYCnItWcEosd6VDo4SOwNLUpdogc2MaMjrdHASDsOl4V0RrcTJ6cH778e/4/wsPeyZJFe0EiAt5cSYSRSWduqItoxyqPtJZaAk9IZcw8RLSQSYqVtOX+Pn3pPjQmn/pMVL72aGI8KYhch8pCB2Zrqscf6PTSpbvJk6JsvKgqRto6Li2CrcrALnTAO1fOEFoZr5WTH1GyDU+oX1EwnfqBKCyNwlt7VLmgZZ4W7repvBmkGXFZ5liufN/Iq7ouEaNiOqdXa1S+drOvcUbxWPV5AS7uJu58sNeNmFVxvwarfraAOPurn+urj5N1bmvnR5TpXHedpGaOGafzeIK9PUyIj2++O8i0dWEya3pljXa1ndnGnUPcpdcf5yGIXD6LO/11eotX10jJ6hFyhCr9Ep+oDO0BhRtEA/0E/0K8DBu+Bj8KkN7e2tco7QlgXxX8b0QKc=</latexit><latexit sha1_base64=\"wc3erbP5Dh01pgdrwpLGKifKKxc=\">AAADnnicbZJLb9NAEMe3MY8SXik9clkRIXGKbA6UYyWkiguiENKmjSNrvR43q+7D2l2njVb+LFzhI/FtWCdGcRxGWum/85uXRpMWnBkbhn8OesGDh48eHz7pP332/MXLwdGrC6NKTWFCFVd6mhIDnEmYWGY5TAsNRKQcLtPbTzW/XII2TMkfdlXAXJAbyXJGifWuZHAc2wVYkrhYC2wWRENWJYNhOArXhvdF1Ighauw8OeqZOFO0FCAt5cSYWRQWdu6ItoxyqPpxaaAg9JbcwMxLSQSYuVtPX+G33pPhXGn/pMVrbzvDEWHMSqQ+UhC7MF1WO//HZqXNP84dk0VpQdJNo7zk2CpcrwJnTAO1fOUFoZr5WTH1GyDU+oX1Ywl3VAlBZObi+8rFdYM0d/dVtctgy6DLcs9SxbN6fsVdXnMN7Yhym13u0+WWLj3FO8WnDaSEu2m381ULXnXhdQte73cdt/C4m+uvi5t/Y6Xue5dnVHmcJZsILVz97wZxZeoaKdF+f5x38dhqwuTOFNt6G1bVZxp1j3JfXLwfReEo+hYOTz80B3uIXqM36B2K0Ak6RZ/ROZogilboJ/qFfgc4OAu+BF83ob2DJucY7Vgw/QuCfz3g</latexit>\u27131excl<latexit sha1_base64=\"4J5pV5+06gHdt98S1WuhiZzWIQI=\">AAADnnicbZLLbtNAFIanMZcSbimV2LAZUSGximwWhWWlShUbRCGkdRsHazw+bkadizUzDolGlngTtvBIPAMvwTgJSuJwpJH+Od+56ehkJWfGhuHvvU5w5+69+/sPug8fPX7ytHfw7MKoSlMYUsWVjjNigDMJQ8ssh7jUQETG4TK7PW345RS0YUp+sfMSxoLcSFYwSqx3pb3DxE7Akq9R6hItMMwor9PeUdgPF4Z3RbQSRyfPoz/fEULn6UHHJLmilQBpKSfGjKKwtGNHtGWUQ91NKgMlobfkBkZeSiLAjN1i+hq/8p4cF0r7Jy1eeDczHBHGzEXmIwWxE9NmjfN/bFTZ4t3YMVlWFiRdNioqjq3CzSpwzjRQy+deEKqZnxXTCdGEWr+wbiLhG1VCEJm7ZFa7pGmQFW5W19sM1gzarPAsUzxv5lfcFQ3XsBlRrbOrXTpd06mneKt4vIKUcBe3O19twKs2vN6A17tdBxt40M7118XNv7Ey97nNc6o8ztNlhBau+beDuDJNjYxovz/O23hgNWFya4p1vSWrmzON2ke5Ky7e9KOwH33y93qMlraPXqCX6DWK0Ft0gt6jczREFM3RD/QT/QpwcBZ8CD4uQzt7q5xDtGVB/BeQSj+W</latexit><latexit sha1_base64=\"zqn2ayt59GS5dw5y5pZlanBQl5w=\">AAADnnicbZLdahNBFMenWT9q/EoteCPIYBG8CrteqJcFQbwRa2PatNm4zM6ebYbOxzIzGxOGvfQ5vNU38FV8Bl/C2WwkycYDA/85v/PF4aQFZ8aG4e+9TnDj5q3b+3e6d+/df/Cwd/DozKhSUxhSxZUepcQAZxKGllkOo0IDESmH8/T6bc3PZ6ANU/KzXRQwEeRKspxRYr0r6R3GdgqWfIkSF2uBYU55lfSOwn64NLwropU4On4c/fn29NfpSXLQMXGmaClAWsqJMeMoLOzEEW0Z5VB149JAQeg1uYKxl5IIMBO3nL7Cz70nw7nS/kmLl97NDEeEMQuR+khB7NS0We38HxuXNn8zcUwWpQVJm0Z5ybFVuF4FzpgGavnCC0I187NiOiWaUOsX1o0lfKVKCCIzF88rF9cN0tzNq2qbwZpBm+WepYpn9fyKu7zmGjYjynV2uUtnazrzFG8VH60gJdyN2p0vNuBFG15uwMvdroMNPGjn+uvi5t9YqTtt84wqj7OkidDC1f92EFemrpES7ffHeRsPrCZMbk2xrtewqj7TqH2Uu+LsZT8K+9Enf6+vUGP76Al6hl6gCL1Gx+g9OkFDRNECfUc/0M8AB++CD8HHJrSzt8o5RFsWjP4CDfRAww==</latexit><latexit sha1_base64=\"zqn2ayt59GS5dw5y5pZlanBQl5w=\">AAADnnicbZLdahNBFMenWT9q/EoteCPIYBG8CrteqJcFQbwRa2PatNm4zM6ebYbOxzIzGxOGvfQ5vNU38FV8Bl/C2WwkycYDA/85v/PF4aQFZ8aG4e+9TnDj5q3b+3e6d+/df/Cwd/DozKhSUxhSxZUepcQAZxKGllkOo0IDESmH8/T6bc3PZ6ANU/KzXRQwEeRKspxRYr0r6R3GdgqWfIkSF2uBYU55lfSOwn64NLwropU4On4c/fn29NfpSXLQMXGmaClAWsqJMeMoLOzEEW0Z5VB149JAQeg1uYKxl5IIMBO3nL7Cz70nw7nS/kmLl97NDEeEMQuR+khB7NS0We38HxuXNn8zcUwWpQVJm0Z5ybFVuF4FzpgGavnCC0I187NiOiWaUOsX1o0lfKVKCCIzF88rF9cN0tzNq2qbwZpBm+WepYpn9fyKu7zmGjYjynV2uUtnazrzFG8VH60gJdyN2p0vNuBFG15uwMvdroMNPGjn+uvi5t9YqTtt84wqj7OkidDC1f92EFemrpES7ffHeRsPrCZMbk2xrtewqj7TqH2Uu+LsZT8K+9Enf6+vUGP76Al6hl6gCL1Gx+g9OkFDRNECfUc/0M8AB++CD8HHJrSzt8o5RFsWjP4CDfRAww==</latexit><latexit sha1_base64=\"ad5rXwNAa/LW+gDXIs2HtKBhuAI=\">AAADnnicbZJLj9MwEMe9DY+lvLrskYtFhcSpSjgAx5WQEBfEQulud5sSOc5ka60fke2UVlY+C1f4SHwbnDaoacpIlv6e37w0mrTgzNgw/HPUC+7cvXf/+EH/4aPHT54OTp5dGFVqChOquNLTlBjgTMLEMsthWmggIuVwmd6+r/nlErRhSn6z6wLmgtxIljNKrHclg9PYLsCS71HiYi0wrCivksEwHIUbw4ciasQQNXaenPRMnClaCpCWcmLMLAoLO3dEW0Y5VP24NFAQektuYOalJALM3G2mr/BL78lwrrR/0uKNt53hiDBmLVIfKYhdmC6rnf9js9Lm7+aOyaK0IOm2UV5ybBWuV4EzpoFavvaCUM38rJguiCbU+oX1Ywk/qBKCyMzFq8rFdYM0d6uq2mewY9BluWep4lk9v+Iur7mGdkS5yy4P6XJHl57iveLTBlLC3bTb+aoFr7rwugWvD7uOW3jczfXXxc2/sVL3tcszqjzOkm2EFq7+d4O4MnWNlGi/P867eGw1YXJvil29LavqM426R3koLl6PonAUfQmHZ2+agz1Gz9EL9ApF6C06Qx/ROZogitboJ/qFfgc4+BB8Cj5vQ3tHTc4p2rNg+hfDDT2k</latexit>\u27132excl<latexit sha1_base64=\"L7MZMrF13/wJcd1gsYREjZr7DAc=\">AAADnnicbZLLbtNAFIanMZcSbimV2LAZUSGxiuwugGUlpIoNohDSuo2DNR4fN6POxZoZp4lGlngTtvBIPAMvwTgJSuJwpJH+Od+56ehkJWfGhuHvvU5w5+69+/sPug8fPX7ytHfw7NyoSlMYUsWVjjNigDMJQ8ssh7jUQETG4SK7ed/wiylow5T8aucljAW5lqxglFjvSnuHiZ2AJd+OU5dogWFGeZ32jsJ+uDC8K6KVODp5Hv35jhA6Sw86JskVrQRISzkxZhSFpR07oi2jHOpuUhkoCb0h1zDyUhIBZuwW09f4lffkuFDaP2nxwruZ4YgwZi4yHymInZg2a5z/Y6PKFu/GjsmysiDpslFRcWwVblaBc6aBWj73glDN/KyYTogm1PqFdRMJt1QJQWTuklntkqZBVrhZXW8zWDNos8KzTPG8mV9xVzRcw2ZEtc6udul0Taee4q3i8QpSwl3c7ny5AS/b8GoDXu12HWzgQTvXXxc3/8bK3Jc2z6nyOE+XEVq45t8O4so0NTKi/f44b+OB1YTJrSnW9Zasbs40ah/lrjg/7kdhP/rs7/UNWto+eoFeotcoQm/RCfqAztAQUTRHP9BP9CvAwWnwMfi0DO3srXIO0ZYF8V+Tez+X</latexit><latexit sha1_base64=\"7MpCRnJhGiU8gxhNPAXOXrcEg0w=\">AAADnnicbZLdahNBFMenWbU1fqUWvBFksAhehd1e1F4WBPFGrI1p02ZjmJ092wydj2VmNiYMe+lzeKtv4Kv4DL6Es9lIko0HBv5zfueLw0lyzowNw987reDO3Xu7e/fbDx4+evyks//0wqhCU+hTxZUeJMQAZxL6llkOg1wDEQmHy+T2bcUvp6ANU/KznecwEuRGsoxRYr1r3DmI7QQs+XI0drEWGGaUl+POYdgNF4a3RbQUh6fPoj/fXvw6Pxvvt0ycKloIkJZyYswwCnM7ckRbRjmU7bgwkBN6S25g6KUkAszILaYv8SvvSXGmtH/S4oV3PcMRYcxcJD5SEDsxTVY5/8eGhc1ORo7JvLAgad0oKzi2ClerwCnTQC2fe0GoZn5WTCdEE2r9wtqxhK9UCUFk6uJZ6eKqQZK5WVluMlgxaLLMs0TxtJpfcZdVXMN6RLHKLrbpdEWnnuKN4oMlpIS7QbPz1Rq8asLrNXi93bW3hnvNXH9d3PwbK3HnTZ5S5XE6riO0cNW/GcSVqWokRPv9cd7EPasJkxtTrOrVrKzONGoe5ba4OOpGYTf65O/1GNW2h56jl+g1itAbdIreozPURxTN0Xf0A/0McPAu+BB8rENbO8ucA7RhweAvESVAxA==</latexit><latexit sha1_base64=\"7MpCRnJhGiU8gxhNPAXOXrcEg0w=\">AAADnnicbZLdahNBFMenWbU1fqUWvBFksAhehd1e1F4WBPFGrI1p02ZjmJ092wydj2VmNiYMe+lzeKtv4Kv4DL6Es9lIko0HBv5zfueLw0lyzowNw987reDO3Xu7e/fbDx4+evyks//0wqhCU+hTxZUeJMQAZxL6llkOg1wDEQmHy+T2bcUvp6ANU/KznecwEuRGsoxRYr1r3DmI7QQs+XI0drEWGGaUl+POYdgNF4a3RbQUh6fPoj/fXvw6Pxvvt0ycKloIkJZyYswwCnM7ckRbRjmU7bgwkBN6S25g6KUkAszILaYv8SvvSXGmtH/S4oV3PcMRYcxcJD5SEDsxTVY5/8eGhc1ORo7JvLAgad0oKzi2ClerwCnTQC2fe0GoZn5WTCdEE2r9wtqxhK9UCUFk6uJZ6eKqQZK5WVluMlgxaLLMs0TxtJpfcZdVXMN6RLHKLrbpdEWnnuKN4oMlpIS7QbPz1Rq8asLrNXi93bW3hnvNXH9d3PwbK3HnTZ5S5XE6riO0cNW/GcSVqWokRPv9cd7EPasJkxtTrOrVrKzONGoe5ba4OOpGYTf65O/1GNW2h56jl+g1itAbdIreozPURxTN0Xf0A/0McPAu+BB8rENbO8ucA7RhweAvESVAxA==</latexit><latexit sha1_base64=\"vv4xnB8HI7o4PgV7wWFi4UdmEKk=\">AAADnnicbZJLb9NAEMe3MY8SXik9clkRIXGK7B4ox0pIVS+IQkjrNg7Wej1uVt2HtbsOiVb+LFzhI/FtWCdBSRxGWum/85uXRpOVnBkbhn8OOsGDh48eHz7pPn32/MXL3tGrK6MqTWFEFVc6zogBziSMLLMc4lIDERmH6+z+Y8OvZ6ANU/KbXZQwEeROsoJRYr0r7R0ndgqWfD9JXaIFhjnlddrrh4NwaXhfRGvRR2u7TI86JskVrQRISzkxZhyFpZ04oi2jHOpuUhkoCb0ndzD2UhIBZuKW09f4rffkuFDaP2nx0rud4YgwZiEyHymInZo2a5z/Y+PKFh8mjsmysiDpqlFRcWwVblaBc6aBWr7wglDN/KyYTokm1PqFdRMJP6gSgsjcJfPaJU2DrHDzut5lsGHQZoVnmeJ5M7/irmi4hu2IapNd7dPZhs48xTvF4zWkhLu43flmC9604e0WvN3vOtzCw3auvy5u/o2Vua9tnlPlcZ6uIrRwzb8dxJVpamRE+/1x3sZDqwmTO1Ns6q1Y3Zxp1D7KfXF1MojCQfQl7J+9Xx/sIXqN3qB3KEKn6AxdoEs0QhQt0E/0C/0OcHAefAo+r0I7B+ucY7RjQfwXxj49pQ==</latexit>\u2713Kexcl<latexit sha1_base64=\"tj079oxGL0tvEhaqG8UfH0l5SLs=\">AAADnnicbZJbaxNBFMenWS9tvKUWfPFlsAg+hV0f1MeCIIKI1Zh222xcZmfPNkPnsszMxoRhwW/iq34kP0O/hLNJJMnGAwP/Ob9z43CykjNjw/DPXie4dfvO3f2D7r37Dx4+6h0+PjOq0hSGVHGl44wY4EzC0DLLIS41EJFxOM+u3zb8fAraMCW/2nkJY0GuJCsYJda70t5RYidgybcPqUu0wDCjvE57x2E/XBjeFdFKHJ88iW5+IIRO08OOSXJFKwHSUk6MGUVhaceOaMsoh7qbVAZKQq/JFYy8lESAGbvF9DV+7j05LpT2T1q88G5mOCKMmYvMRwpiJ6bNGuf/2KiyxZuxY7KsLEi6bFRUHFuFm1XgnGmgls+9IFQzPyumE6IJtX5h3UTCd6qEIDJ3yax2SdMgK9ysrrcZrBm0WeFZpnjezK+4KxquYTOiWmdXu3S6plNP8VbxeAUp4S5ud77YgBdteLkBL3e7DjbwoJ3rr4ubf2Nl7kub51R5nKfLCC1c828HcWWaGhnRfn+ct/HAasLk1hTrektWN2catY9yV5y97EdhP/rs7/UVWto+eoqeoRcoQq/RCXqPTtEQUTRHP9Ev9DvAwbvgY/BpGdrZW+UcoS0L4r/jRD+w</latexit><latexit sha1_base64=\"fmKrGb0LxROBthB9UEsEm24r+EA=\">AAADnnicbZLdahNBFMenWbU1fqUWvBFksAhehd1eqJeFgggi1sa0abMxzM6ebYbOxzIzGxOGvfQ5vNU38FV8Bl/C2SSSzcYDA/85v/PF4SQ5Z8aG4e+dVnDr9p3dvbvte/cfPHzU2X98blShKfSp4koPEmKAMwl9yyyHQa6BiITDRXJzUvGLKWjDlPxs5zmMBLmWLGOUWO8adw5iOwFLvrwfu1gLDDPKy3HnMOyGC8PbIlqJw+Mn0Z9vz36dnY73WyZOFS0ESEs5MWYYhbkdOaItoxzKdlwYyAm9Idcw9FISAWbkFtOX+IX3pDhT2j9p8cJbz3BEGDMXiY8UxE5Mk1XO/7FhYbM3I8dkXliQdNkoKzi2ClerwCnTQC2fe0GoZn5WTCdEE2r9wtqxhK9UCUFk6uJZ6eKqQZK5WVluMlgzaLLMs0TxtJpfcZdVXEM9olhnF9t0uqZTT/FG8cEKUsLdoNn5sgYvm/CqBq+2u/ZquNfM9dfFzb+xEnfW5ClVHqfjZYQWrvo3g7gyVY2EaL8/zpu4ZzVhcmOKdb0lK6szjZpHuS3Oj7pR2I0++Xt9hZa2h56i5+glitBrdIzeoVPURxTN0Xf0A/0McPA2+BB8XIa2dlY5B2jDgsFfYO5A3Q==</latexit><latexit sha1_base64=\"fmKrGb0LxROBthB9UEsEm24r+EA=\">AAADnnicbZLdahNBFMenWbU1fqUWvBFksAhehd1eqJeFgggi1sa0abMxzM6ebYbOxzIzGxOGvfQ5vNU38FV8Bl/C2SSSzcYDA/85v/PF4SQ5Z8aG4e+dVnDr9p3dvbvte/cfPHzU2X98blShKfSp4koPEmKAMwl9yyyHQa6BiITDRXJzUvGLKWjDlPxs5zmMBLmWLGOUWO8adw5iOwFLvrwfu1gLDDPKy3HnMOyGC8PbIlqJw+Mn0Z9vz36dnY73WyZOFS0ESEs5MWYYhbkdOaItoxzKdlwYyAm9Idcw9FISAWbkFtOX+IX3pDhT2j9p8cJbz3BEGDMXiY8UxE5Mk1XO/7FhYbM3I8dkXliQdNkoKzi2ClerwCnTQC2fe0GoZn5WTCdEE2r9wtqxhK9UCUFk6uJZ6eKqQZK5WVluMlgzaLLMs0TxtJpfcZdVXEM9olhnF9t0uqZTT/FG8cEKUsLdoNn5sgYvm/CqBq+2u/ZquNfM9dfFzb+xEnfW5ClVHqfjZYQWrvo3g7gyVY2EaL8/zpu4ZzVhcmOKdb0lK6szjZpHuS3Oj7pR2I0++Xt9hZa2h56i5+glitBrdIzeoVPURxTN0Xf0A/0McPA2+BB8XIa2dlY5B2jDgsFfYO5A3Q==</latexit><latexit sha1_base64=\"4BFOsL05clzxtIs/Lp6AxvYs2EM=\">AAADnnicbZJbb9MwFMe9hssot4498mJRIfFUJTwwHichTZMQYlC6ZWtK5DgnqzVfItsprax8Fl7hI/FtcNqitilHsvT3+Z2bjk5WcmZsGP456AT37j94ePio+/jJ02fPe0cvLo2qNIURVVzpOCMGOJMwssxyiEsNRGQcrrK7Dw2/moE2TMlvdlHCRJBbyQpGifWutHec2ClY8v1j6hItMMwpr9NePxyES8P7IlqLPlrbRXrUMUmuaCVAWsqJMeMoLO3EEW0Z5VB3k8pASegduYWxl5IIMBO3nL7Gr70nx4XS/kmLl97tDEeEMQuR+UhB7NS0WeP8HxtXtng/cUyWlQVJV42KimOrcLMKnDMN1PKFF4Rq5mfFdEo0odYvrJtI+EGVEETmLpnXLmkaZIWb1/Uugw2DNis8yxTPm/kVd0XDNWxHVJvsap/ONnTmKd4pHq8hJdzF7c7XW/C6DW+24M1+1+EWHrZz/XVx82+szH1t85wqj/N0FaGFa/7tIK5MUyMj2u+P8zYeWk2Y3JliU2/F6uZMo/ZR7ovLt4MoHERfwv7pu/XBHqKX6BV6gyJ0gk7RObpAI0TRAv1Ev9DvAAdnwafg8yq0c7DOOUY7FsR/ARYWPb4=</latexit>fK\u2713K<latexit sha1_base64=\"Gx7V+LPC+gaLkbPTAdCgGshmQuY=\">AAADmnicbZJLb9NAEMe3MdASHk3hhOBgUSFximwOwLEqF1AuhTRt2ji11utxs+o+rN11aLTyJ+EKH6o3znwK1nFQHIeRVvrv/Oal0SQ5o9oEwd1Ox7t3/8Hu3sPuo8dPnu73Dp6daVkoAiMimVTjBGtgVMDIUMNgnCvAPGFwntx8qvj5HJSmUpyaRQ5Tjq8FzSjBxrni3n6UXQ1iG5kZGHw1KOPeYdAPluZvi3AlDo9e3P35jRA6iQ86OkolKTgIQxjWehIGuZlarAwlDMpuVGjIMbnB1zBxUmAOemqXk5f+G+dJ/Uwq94Txl95mhsVc6wVPXCTHZqbbrHL+j00Kk32cWirywoAgdaOsYL6RfrUGP6UKiGELJzBR1M3qkxlWmBi3rG4k4DuRnGOR2ui2tFHVIMnsbVluMlgzaLPMsUSytJpfMptVXEEzolhnF9t0vqZzR/2N4uMVJJjZcbvzRQNetOFlA15udx028LCd6y6L6X9jJfZbm6dEOpzGdYTitvq3g5jUVY0EK7c/xtp4aBSmYmOKdb2aldWZhu2j3BZn7/ph0A+/unt9j2rbQy/Ra/QWhegDOkKf0QkaIYIK9AP9RL+8V96x98Ub1KGdnVXOc7Rh3ulfNXk+9g==</latexit><latexit sha1_base64=\"5qJ8/fZ3engjjkGTeHbfNcMioxU=\">AAADmnicbZLNbtNAEMe3MdCSFprCCcHBokLiFNkcgGNVLqBcCmnapHFqrdfjZtX9sHbXodHKT8IVHqoHXoCnYJ0ExXEYaaX/zm++NJokZ1SbILjfaXkPHj7a3Xvc3j948vSwc/TsQstCERgQyaQaJlgDowIGhhoGw1wB5gmDy+T2U8UvZ6A0leLczHOYcHwjaEYJNs4Vdw6j7LoX28hMweDrXhl3joNusDB/W4QrcXzy4v7P79HB/ll81NJRKknBQRjCsNbjMMjNxGJlKGFQtqNCQ47JLb6BsZMCc9ATu5i89N84T+pnUrknjL/w1jMs5lrPeeIiOTZT3WSV839sXJjs48RSkRcGBFk2ygrmG+lXa/BTqoAYNncCE0XdrD6ZYoWJcctqRwK+E8k5FqmN7kobVQ2SzN6V5SaDNYMmyxxLJEur+SWzWcUV1COKdXaxTWdrOnPU3yg+XEGCmR02O49qcNSEVzV4td21X8P9Zq67LKb/jZXYb02eEulwGi8jFLfVvxnEpK5qJFi5/THWxH2jMBUbU6zrLVlZnWnYPMptcfGuGwbd8Ku71/doaXvoJXqN3qIQfUAn6DM6QwNEUIF+oJ/ol/fKO/W+eL1laGtnlfMcbZh3/hd5aD9j</latexit><latexit sha1_base64=\"5qJ8/fZ3engjjkGTeHbfNcMioxU=\">AAADmnicbZLNbtNAEMe3MdCSFprCCcHBokLiFNkcgGNVLqBcCmnapHFqrdfjZtX9sHbXodHKT8IVHqoHXoCnYJ0ExXEYaaX/zm++NJokZ1SbILjfaXkPHj7a3Xvc3j948vSwc/TsQstCERgQyaQaJlgDowIGhhoGw1wB5gmDy+T2U8UvZ6A0leLczHOYcHwjaEYJNs4Vdw6j7LoX28hMweDrXhl3joNusDB/W4QrcXzy4v7P79HB/ll81NJRKknBQRjCsNbjMMjNxGJlKGFQtqNCQ47JLb6BsZMCc9ATu5i89N84T+pnUrknjL/w1jMs5lrPeeIiOTZT3WSV839sXJjs48RSkRcGBFk2ygrmG+lXa/BTqoAYNncCE0XdrD6ZYoWJcctqRwK+E8k5FqmN7kobVQ2SzN6V5SaDNYMmyxxLJEur+SWzWcUV1COKdXaxTWdrOnPU3yg+XEGCmR02O49qcNSEVzV4td21X8P9Zq67LKb/jZXYb02eEulwGi8jFLfVvxnEpK5qJFi5/THWxH2jMBUbU6zrLVlZnWnYPMptcfGuGwbd8Ku71/doaXvoJXqN3qIQfUAn6DM6QwNEUIF+oJ/ol/fKO/W+eL1laGtnlfMcbZh3/hd5aD9j</latexit><latexit sha1_base64=\"fo5KiR9c19JAmR0wVTldwTVI2nU=\">AAADmnicbZJLb9NAEMe3MdASHk3hCAeLCIlTZHOAHiu4gHIpTdOmjVNrvR43q+7D2l2HRit/Eq7wofg2XSdGcRxGWum/85uXRpPkjGoTBH/3Ot6jx0/2D552nz1/8fKwd/TqQstCERgTyaSaJFgDowLGhhoGk1wB5gmDy+Tua8UvF6A0leLcLHOYcXwraEYJNs4V9w6j7GYY28jMweCbYRn3+sEgWJm/K8Ja9FFtp/FRR0epJAUHYQjDWk/DIDczi5WhhEHZjQoNOSZ3+BamTgrMQc/savLSf+88qZ9J5Z4w/srbzLCYa73kiYvk2Mx1m1XO/7FpYbLjmaUiLwwIsm6UFcw30q/W4KdUATFs6QQmirpZfTLHChPjltWNBPwkknMsUhvdlzaqGiSZvS/LbQYbBm2WOZZIllbzS2aziitoRhSb7GKXLjZ04ai/VXxSQ4KZnbQ7XzXgVRteN+D1btdRA4/aue6ymP43VmLP2jwl0uE0Xkcobqt/O4hJXdVIsHL7Y6yNR0ZhKram2NRbs7I607B9lLvi4uMgDAbhj6B/8qk+2AP0Br1DH1CIPqMT9A2dojEiqEC/0G/0x3vrffG+e8N1aGevznmNtsw7fwBO8zv6</latexit>f2\u27132<latexit sha1_base64=\"BVvO1YdpYTEOVHgbilQJFhA20BE=\">AAADmnicbZJLb9NAEMe3MY8SHk3hggQHiwiJU2TnAByr9gLiUkjTpo1Ta70eN6vuw9pdh0YrfxKu8JkQ34Z1HBTHYaSV/ju/eWk0Sc6oNkHwZ6/j3bv/4OH+o+7jJ0+fHfQOn59rWSgCYyKZVJMEa2BUwNhQw2CSK8A8YXCR3J5U/GIBSlMpzswyhxnHN4JmlGDjXHHvIMquh7GNzBwMvh6Wca8fDIKV+bsiXIv+0cv+bx8hdBofdnSUSlJwEIYwrPU0DHIzs1gZShiU3ajQkGNyi29g6qTAHPTMriYv/bfOk/qZVO4J46+8zQyLudZLnrhIjs1ct1nl/B+bFib7OLNU5IUBQepGWcF8I/1qDX5KFRDDlk5goqib1SdzrDAxblndSMB3IjnHIrXRXWmjqkGS2buy3GawYdBmmWOJZGk1v2Q2q7iCZkSxyS526WJDF476W8Una0gws5N258sGvGzDqwa82u06auBRO9ddFtP/xkrstzZPiXQ4jesIxW31bwcxqasaCVZuf4y18cgoTMXWFJt6NSurMw3bR7krzoeDMBiEX929vke17aNX6A16h0L0AR2hT+gUjRFBBfqBfqJf3mvv2PvsfalDO3vrnBdoy7yzvwBQPTo=</latexit><latexit sha1_base64=\"7pyYROFbxRTiMb2nS3NAdIWIOwc=\">AAADmnicbZJLb9NAEMe3MY8SHk3hggQHiwiJU2TnAFyQKriAuBTStGnj1Fqvx82q+7B216HRyp+Ea/lMiG/DOg6K4zDSSv+d37w0miRnVJsg+LPX8e7cvXd//0H34aPHTw56h09PtSwUgTGRTKpJgjUwKmBsqGEwyRVgnjA4S64/VfxsAUpTKU7MMocZx1eCZpRg41xx7yDKLoexjcwcDL4clnGvHwyClfm7IlyL/tHz/u/uh/z2OD7s6CiVpOAgDGFY62kY5GZmsTKUMCi7UaEhx+QaX8HUSYE56JldTV76r50n9TOp3BPGX3mbGRZzrZc8cZEcm7lus8r5PzYtTPZ+ZqnICwOC1I2ygvlG+tUa/JQqIIYtncBEUTerT+ZYYWLcsrqRgB9Eco5FaqOb0kZVgySzN2W5zWDDoM0yxxLJ0mp+yWxWcQXNiGKTXezSxYYuHPW3ik/WkGBmJ+3O5w143oYXDXix23XUwKN2rrsspv+NldjvbZ4S6XAa1xGK2+rfDmJSVzUSrNz+GGvjkVGYiq0pNvVqVlZnGraPclecDgdhMAi/uXt9i2rbRy/QK/QGhegdOkKf0TEaI4IK9BPdol/eS++j98X7Wod29tY5z9CWeSd/AacSPnU=</latexit><latexit sha1_base64=\"7pyYROFbxRTiMb2nS3NAdIWIOwc=\">AAADmnicbZJLb9NAEMe3MY8SHk3hggQHiwiJU2TnAFyQKriAuBTStGnj1Fqvx82q+7B216HRyp+Ea/lMiG/DOg6K4zDSSv+d37w0miRnVJsg+LPX8e7cvXd//0H34aPHTw56h09PtSwUgTGRTKpJgjUwKmBsqGEwyRVgnjA4S64/VfxsAUpTKU7MMocZx1eCZpRg41xx7yDKLoexjcwcDL4clnGvHwyClfm7IlyL/tHz/u/uh/z2OD7s6CiVpOAgDGFY62kY5GZmsTKUMCi7UaEhx+QaX8HUSYE56JldTV76r50n9TOp3BPGX3mbGRZzrZc8cZEcm7lus8r5PzYtTPZ+ZqnICwOC1I2ygvlG+tUa/JQqIIYtncBEUTerT+ZYYWLcsrqRgB9Eco5FaqOb0kZVgySzN2W5zWDDoM0yxxLJ0mp+yWxWcQXNiGKTXezSxYYuHPW3ik/WkGBmJ+3O5w143oYXDXix23XUwKN2rrsspv+NldjvbZ4S6XAa1xGK2+rfDmJSVzUSrNz+GGvjkVGYiq0pNvVqVlZnGraPclecDgdhMAi/uXt9i2rbRy/QK/QGhegdOkKf0TEaI4IK9BPdol/eS++j98X7Wod29tY5z9CWeSd/AacSPnU=</latexit><latexit sha1_base64=\"ZkZJ7HwRYMNb0oQzgwVdVg+OUJY=\">AAADmnicbZJLb9NAEMe3MdASHk3hCAeLCIlTZOcAPVZwAXFpSdOmjVNrvR43q+7D2l2HRit/Eq7wofg2XSdGcRxGWum/85uXRpPkjGoTBH/3Ot6jx0/2D552nz1/8fKwd/TqQstCERgTyaSaJFgDowLGhhoGk1wB5gmDy+TuS8UvF6A0leLcLHOYcXwraEYJNs4V9w6j7GYY28jMweCbYRn3+sEgWJm/K8Ja9FFtp/FRR0epJAUHYQjDWk/DIDczi5WhhEHZjQoNOSZ3+BamTgrMQc/savLSf+88qZ9J5Z4w/srbzLCYa73kiYvk2Mx1m1XO/7FpYbLjmaUiLwwIsm6UFcw30q/W4KdUATFs6QQmirpZfTLHChPjltWNBPwkknMsUhvdlzaqGiSZvS/LbQYbBm2WOZZIllbzS2aziitoRhSb7GKXLjZ04ai/VXxSQ4KZnbQ7XzXgVRteN+D1btdRA4/aue6ymP43VmJ/tHlKpMNpvI5Q3Fb/dhCTuqqRYOX2x1gbj4zCVGxNsam3ZmV1pmH7KHfFxXAQBoPwLOiffKwP9gC9Qe/QBxSiT+gEfUWnaIwIKtAv9Bv98d56n71v3vd1aGevznmNtsw7fwCwTDvI</latexit>f1\u27131<latexit sha1_base64=\"5CApFcHOIFrHTDJPxdBgpa4+jNw=\">AAADmnicbZJLb9NAEMe3MY8SHk3hggSHFRESp8juAThWcAFxKaRp08aptV6Pm1X3Ye2uQ6OVPwlX+EyIb8M6DkriMNJK/53fvDSatODM2DD8s9cJ7ty9d3//Qffho8dPDnqHT8+MKjWFEVVc6XFKDHAmYWSZ5TAuNBCRcjhPbz7W/HwO2jAlT+2igKkg15LljBLrXUnvIM6vosTFdgaWXEVV0uuHg3BpeFdEK9E/ft7/jRFCJ8lhx8SZoqUAaSknxkyisLBTR7RllEPVjUsDBaE35BomXkoiwEzdcvIKv/aeDOdK+yctXno3MxwRxixE6iMFsTPTZrXzf2xS2vz91DFZlBYkbRrlJcdW4XoNOGMaqOULLwjVzM+K6YxoQq1fVjeW8J0qIYjMXHxbubhukObutqq2GawZtFnuWap4Vs+vuMtrrmEzolxnl7t0vqZzT/FW8fEKUsLduN35YgNetOHlBrzc7TrcwMN2rr8sbv6NlbpvbZ5R5XGWNBFauPrfDuLK1DVSov3+OG/jodWEya0p1vUaVtVnGrWPclecHQ2icBB99ff6FjW2j16gV+gNitA7dIw+oRM0QhSV6Af6iX4FL4MPwefgSxPa2VvlPENbFpz+BfnpPTg=</latexit><latexit sha1_base64=\"ae4b/6UTZkw265Bunik0BonFqv0=\">AAADmnicbZJLb9NAEMe3MY8SHk3hggQHiwiJU2T3AFwqVeUC4lJI06aNU2u9Hjer7sPaXYdGK38SruUzIb4N6zgojsNIK/13fvPSaJKcUW2C4M9Ox7t3/8HD3Ufdx0+ePtvr7T8/07JQBEZEMqnGCdbAqICRoYbBOFeAecLgPLn5VPHzOShNpTg1ixymHF8LmlGCjXPFvb0ouwpjG5kZGHwVlnGvHwyCpfnbIlyJ/tHL/u/uYX53Eu93dJRKUnAQhjCs9SQMcjO1WBlKGJTdqNCQY3KDr2HipMAc9NQuJy/9t86T+plU7gnjL73NDIu51gueuEiOzUy3WeX8H5sUJvs4tVTkhQFB6kZZwXwj/WoNfkoVEMMWTmCiqJvVJzOsMDFuWd1IwA8iOccitdFtaaOqQZLZ27LcZLBm0GaZY4lkaTW/ZDaruIJmRLHOLrbpfE3njvobxccrSDCz43bniwa8aMPLBrzc7jps4GE7110W0//GSuz3Nk+JdDiN6wjFbfVvBzGpqxoJVm5/jLXx0ChMxcYU63o1K6szDdtHuS3ODgZhMAi/uXt9j2rbRa/QG/QOhegDOkKf0QkaIYIK9BPdoV/ea+/Y++J9rUM7O6ucF2jDvNO/oLo+cw==</latexit><latexit sha1_base64=\"ae4b/6UTZkw265Bunik0BonFqv0=\">AAADmnicbZJLb9NAEMe3MY8SHk3hggQHiwiJU2T3AFwqVeUC4lJI06aNU2u9Hjer7sPaXYdGK38SruUzIb4N6zgojsNIK/13fvPSaJKcUW2C4M9Ox7t3/8HD3Ufdx0+ePtvr7T8/07JQBEZEMqnGCdbAqICRoYbBOFeAecLgPLn5VPHzOShNpTg1ixymHF8LmlGCjXPFvb0ouwpjG5kZGHwVlnGvHwyCpfnbIlyJ/tHL/u/uYX53Eu93dJRKUnAQhjCs9SQMcjO1WBlKGJTdqNCQY3KDr2HipMAc9NQuJy/9t86T+plU7gnjL73NDIu51gueuEiOzUy3WeX8H5sUJvs4tVTkhQFB6kZZwXwj/WoNfkoVEMMWTmCiqJvVJzOsMDFuWd1IwA8iOccitdFtaaOqQZLZ27LcZLBm0GaZY4lkaTW/ZDaruIJmRLHOLrbpfE3njvobxccrSDCz43bniwa8aMPLBrzc7jps4GE7110W0//GSuz3Nk+JdDiN6wjFbfVvBzGpqxoJVm5/jLXx0ChMxcYU63o1K6szDdtHuS3ODgZhMAi/uXt9j2rbRa/QG/QOhegDOkKf0QkaIYIK9BPdoV/ea+/Y++J9rUM7O6ucF2jDvNO/oLo+cw==</latexit><latexit sha1_base64=\"QrHBFByEPXCN9qRvTYkdahrwFOs=\">AAADmnicbZJLT9tAEMeXuA+aPgjtsRysRpV6iuweaI+oXFr1QhsCgThY6/WYrNiHtbsORCt/Eq70Q/XbdJ0YxXE60kr/nd+8NJokZ1SbIPi70/GePH32fPdF9+Wr12/2evtvz7QsFIERkUyqcYI1MCpgZKhhMM4VYJ4wOE9ujit+PgelqRSnZpHDlONrQTNKsHGuuLcXZVdhbCMzA4OvwjLu9YNBsDR/W4S16KPaTuL9jo5SSQoOwhCGtZ6EQW6mFitDCYOyGxUackxu8DVMnBSYg57a5eSl/9F5Uj+Tyj1h/KW3mWEx13rBExfJsZnpNquc/2OTwmRfp5aKvDAgyKpRVjDfSL9ag59SBcSwhROYKOpm9ckMK0yMW1Y3EnBLJOdYpDa6K21UNUgye1eWmwzWDNoscyyRLK3ml8xmFVfQjCjW2cU2na/p3FF/o/i4hgQzO253vmjAiza8bMDL7a7DBh62c91lMf04VmJ/t3lKpMNpvIpQ3Fb/dhCTuqqRYOX2x1gbD43CVGxMsa63YmV1pmH7KLfF2edBGAzCX0H/6LA+2F30Hn1An1CIvqAj9B2doBEiqED36AH98Q68b94P7+cqtLNT57xDG+ad/gOp9DvG</latexit>e1(y)<latexit sha1_base64=\"0ONiBXpeCcQFypWW0U5C7rHw3EI=\">AAADkHicbZJLbxMxEMfdLNASXm2RuHBZUSGVS7TLAXprERfEqRDSpM2GyOudba34sbK9oZG1H4IrSIivxbfBmw3KxmEkS+P5/edha9KCUW2i6M9OJ7hz997u3v3ug4ePHj/ZPzi80LJUBAZEMqlGKdbAqICBoYbBqFCAecpgmM7e13w4B6WpFF/MooAJx9eC5pRg40LDBL7Gx4tX0/2jqBctLdx24pVzdPas+PUbIXQ+PejoJJOk5CAMYVjrcRwVZmKxMpQwqLpJqaHAZIavYexcgTnoiV3OW4UvXSQLc6ncESZcRtsZFnOtFzx1So7NjfZZHfwfG5cmP5lYKorSgCBNo7xkoZFh/fgwowqIYQvnYKKomzUkN1hhYtwXdRMB34jkHIvMJreVTeoGaW5vq2qTwZqBz3LHUsmyen7JbF5zBW1Fuc4ut+l8TeeOhhvFRytIMLMjv/NlC1768KoFr7a79lu47+e6fWL631ip/ezzjEiHs2mjUNzWd1/EpK5rpFi5/2PMx32jMBUbU6zrNayq3JrG/lJuOxeve3HUiz+5fX2DGttDz9ELdIxi9BadoQ/oHA0QQTP0Hf1AP4PD4CQ4Dd410s7OKucp2rDg4194vDm6</latexit><latexit sha1_base64=\"Qnv5yYiVE9fYANkM28gLKOWtyCI=\">AAADkHicbZJLb9NAEMe3MdASXm2RuHCxqJDKJbI5QG4t6qXqqRDSpI1DtF6P21X2Ye2uQ6OVP0RvCCS+F4IPwzoOiuMw0kqz8/vPY1cTZ4xqEwS/tlrevfsPtnceth89fvL02e7e/oWWuSLQJ5JJNYyxBkYF9A01DIaZAsxjBoN4elLywQyUplJ8NvMMxhxfC5pSgo0LDSL4Eh7O30x2D4JOsDB/0wmXzsHxi+zntz+/u+eTvZaOEklyDsIQhrUehUFmxhYrQwmDoh3lGjJMpvgaRs4VmIMe28W8hf/aRRI/lcodYfxFtJ5hMdd6zmOn5Njc6CYrg/9jo9yk3bGlIssNCFI1SnPmG+mXj/cTqoAYNncOJoq6WX1ygxUmxn1ROxLwlUjOsUhsdFvYqGwQp/a2KNYZrBg0WepYLFlSzi+ZTUuuoK7IV9n5Jp2t6MxRf634cAkJZnbY7HxZg5dNeFWDV5tdezXca+a6fWL631ix/dTkCZEOJ5NKobgt700Rk7qsEWPl/o+xJu4ZhalYm2JVr2JF4dY0bC7lpnPxthMGnfCj29d3qLId9BK9QocoRO/RMTpF56iPCJqiO/Qd/fD2va535H2opK2tZc5ztGbe2V8fcjuh</latexit><latexit sha1_base64=\"Qnv5yYiVE9fYANkM28gLKOWtyCI=\">AAADkHicbZJLb9NAEMe3MdASXm2RuHCxqJDKJbI5QG4t6qXqqRDSpI1DtF6P21X2Ye2uQ6OVP0RvCCS+F4IPwzoOiuMw0kqz8/vPY1cTZ4xqEwS/tlrevfsPtnceth89fvL02e7e/oWWuSLQJ5JJNYyxBkYF9A01DIaZAsxjBoN4elLywQyUplJ8NvMMxhxfC5pSgo0LDSL4Eh7O30x2D4JOsDB/0wmXzsHxi+zntz+/u+eTvZaOEklyDsIQhrUehUFmxhYrQwmDoh3lGjJMpvgaRs4VmIMe28W8hf/aRRI/lcodYfxFtJ5hMdd6zmOn5Njc6CYrg/9jo9yk3bGlIssNCFI1SnPmG+mXj/cTqoAYNncOJoq6WX1ygxUmxn1ROxLwlUjOsUhsdFvYqGwQp/a2KNYZrBg0WepYLFlSzi+ZTUuuoK7IV9n5Jp2t6MxRf634cAkJZnbY7HxZg5dNeFWDV5tdezXca+a6fWL631ix/dTkCZEOJ5NKobgt700Rk7qsEWPl/o+xJu4ZhalYm2JVr2JF4dY0bC7lpnPxthMGnfCj29d3qLId9BK9QocoRO/RMTpF56iPCJqiO/Qd/fD2va535H2opK2tZc5ztGbe2V8fcjuh</latexit><latexit sha1_base64=\"V3E+6xQWTj51FLMdFjAeVlbhcC0=\">AAADkHicbZJLbxMxEMfdLI8SXm05crGIkMol2uVQeqMVF8SpENKkzYbI651trfixsr1pI2s/BFf4ZHwbvMmibDaMZGk8v/88bE2Sc2ZsGP7Z6wQPHj56vP+k+/TZ8xcvDw6PLo0qNIUhVVzpcUIMcCZhaJnlMM41EJFwGCXzTxUfLUAbpuR3u8xhKsiNZBmjxPrQKIYf0fHy3eygF/bDleFdJ6qdHqrtYnbYMXGqaCFAWsqJMZMozO3UEW0Z5VB248JATuic3MDEu5IIMFO3mrfEb30kxZnS/kiLV9FmhiPCmKVIvFIQe2varAr+j00Km51OHZN5YUHSdaOs4NgqXD0ep0wDtXzpHUI187Nieks0odZ/UTeWcEeVEESmLr4vXVw1SDJ3X5bbDDYM2izzLFE8reZX3GUV19BUFJvsYpcuNnThKd4qPq4hJdyN252vGvCqDa8b8Hq366CBB+1cv0/c/Bsrcd/aPKXK43S2VmjhqntbxJWpaiRE+//jvI0HVhMmt6bY1FuzsvRrGrWXcte5fN+Pwn70NeydndQLu49eozfoGEXoAzpDn9EFGiKK5ugn+oV+B0fBafAxOF9LO3t1ziu0ZcGXv3W6N7Y=</latexit>e2(y)<latexit sha1_base64=\"/cZAQAaeVrRdWj644IPf8gj45NY=\">AAADkHicbZJLbxMxEMfdLI8SXm2RuHBZUSGVS7TbQ+mNIi6IUyGkSZsNke2dba34sbK9oZG1H4IrSIivxbfBmw3KZsNIlsbz+8/D1pCcM2Oj6M9OJ7hz99793Qfdh48eP3m6t39wYVShKQyo4kqPCDbAmYSBZZbDKNeABeEwJLP3FR/OQRum5Be7yGEi8LVkGaPY+tAwga/HR4vX073DqBctLdx24pVzePY8//UbIXQ+3e+YJFW0ECAt5diYcRzlduKwtoxyKLtJYSDHdIavYexdiQWYiVvOW4avfCQNM6X9kTZcRpsZDgtjFoJ4pcD2xrRZFfwfGxc2O504JvPCgqR1o6zgoVVh9fgwZRqo5QvvYKqZnzWkN1hjav0XdRMJ36gSAsvUJbelS6oGJHO3ZbnJYM2gzTLPiOJpNb/iLqu4hqaiWGcX23S+pnNPw43ioxWkmLtRu/NlA1624VUDXm137Tdwv53r94mbf2MR97nNU6o8Tqe1QgtX3dsirkxVg2Dt/4/zNu5bjZncmGJdr2Zl6dc0bi/ltnNx3IujXvzJ7+sJqm0XvUAv0RGK0Rt0hj6gczRAFM3Qd/QD/QwOgtPgbfCulnZ2VjnP0IYFH/8Ce+U5uw==</latexit><latexit sha1_base64=\"uJ1HR+xcZyqFOO3JxD03CahK7i0=\">AAADkHicbZJLb9NAEMe3MY8SXm2RuHCxqJDKJbJ7KLlRxAX1VAhp0sYhWq/H7Sr7sHbXodHKH6I31Ep8LwQfhnUcFMdhpJVm5/efx64mzhjVJgh+bbW8e/cfPNx+1H785Omz5zu7e2da5opAn0gm1TDGGhgV0DfUMBhmCjCPGQzi6ceSD2agNJXiq5lnMOb4UtCUEmxcaBDBt8OD+dvJzn7QCRbmbzrh0tk/fpn9/PHnd/d0stvSUSJJzkEYwrDWozDIzNhiZShhULSjXEOGyRRfwsi5AnPQY7uYt/DfuEjip1K5I4y/iNYzLOZaz3nslBybK91kZfB/bJSbtDu2VGS5AUGqRmnOfCP98vF+QhUQw+bOwURRN6tPrrDCxLgvakcCvhPJORaJja4LG5UN4tReF8U6gxWDJksdiyVLyvkls2nJFdQV+So736SzFZ056q8VHy4hwcwOm53Pa/C8CS9q8GKza6+Ge81ct09M/xsrtl+aPCHS4WRSKRS35b0pYlKXNWKs3P8x1sQ9ozAVa1Os6lWsKNyahs2l3HTODjth0Ak/u309QpVto1foNTpAIXqHjtEndIr6iKApukG36M7b87ree+9DJW1tLXNeoDXzTv4CIps7og==</latexit><latexit sha1_base64=\"uJ1HR+xcZyqFOO3JxD03CahK7i0=\">AAADkHicbZJLb9NAEMe3MY8SXm2RuHCxqJDKJbJ7KLlRxAX1VAhp0sYhWq/H7Sr7sHbXodHKH6I31Ep8LwQfhnUcFMdhpJVm5/efx64mzhjVJgh+bbW8e/cfPNx+1H785Omz5zu7e2da5opAn0gm1TDGGhgV0DfUMBhmCjCPGQzi6ceSD2agNJXiq5lnMOb4UtCUEmxcaBDBt8OD+dvJzn7QCRbmbzrh0tk/fpn9/PHnd/d0stvSUSJJzkEYwrDWozDIzNhiZShhULSjXEOGyRRfwsi5AnPQY7uYt/DfuEjip1K5I4y/iNYzLOZaz3nslBybK91kZfB/bJSbtDu2VGS5AUGqRmnOfCP98vF+QhUQw+bOwURRN6tPrrDCxLgvakcCvhPJORaJja4LG5UN4tReF8U6gxWDJksdiyVLyvkls2nJFdQV+So736SzFZ056q8VHy4hwcwOm53Pa/C8CS9q8GKza6+Ge81ct09M/xsrtl+aPCHS4WRSKRS35b0pYlKXNWKs3P8x1sQ9ozAVa1Os6lWsKNyahs2l3HTODjth0Ak/u309QpVto1foNTpAIXqHjtEndIr6iKApukG36M7b87ree+9DJW1tLXNeoDXzTv4CIps7og==</latexit><latexit sha1_base64=\"BA3P1cTIdsF5WVMLzcZjNWA5WmM=\">AAADkHicbZJLbxMxEMfdLNASXm05crGIkMol2u0BeqMVF8SpENKkzYbI651trfixsr2hkbUfgit8Mr4N3mRRNhtGsjSe338etibJOTM2DP/sdYIHDx/tHzzuPnn67PmLw6PjK6MKTWFIFVd6nBADnEkYWmY5jHMNRCQcRsn8Y8VHC9CGKfnNLnOYCnIrWcYosT40iuH76cny7eywF/bDleFdJ6qdHqrtcnbUMXGqaCFAWsqJMZMozO3UEW0Z5VB248JATuic3MLEu5IIMFO3mrfEb3wkxZnS/kiLV9FmhiPCmKVIvFIQe2farAr+j00Km51NHZN5YUHSdaOs4NgqXD0ep0wDtXzpHUI187Niekc0odZ/UTeW8IMqIYhMXXxfurhqkGTuviy3GWwYtFnmWaJ4Ws2vuMsqrqGpKDbZxS5dbOjCU7xVfFxDSrgbtztfN+B1G9404M1u10EDD9q5fp+4+TdW4r62eUqVx+lsrdDCVfe2iCtT1UiI9v/HeRsPrCZMbk2xqbdmZenXNGov5a5zddqPwn70Jeydv6sX9gC9Qq/RCYrQe3SOPqFLNEQUzdFP9Av9Do6Ds+BDcLGWdvbqnJdoy4LPfwF44ze3</latexit>eK(y)<latexit sha1_base64=\"zZSZ+/DGrM92+xH47Zr5ac1ENfA=\">AAADkHicbZJLbxMxEMfdLI8SXm2RuHCxqJDKJdrtAXqjiAuCSyGkSZsNkdc721rxY2V7QyNrPwRXkBBfi2+DNxuUzYaRLI3n95+HrUlyzowNwz87neDW7Tt3d+917z94+Ojx3v7BuVGFpjCgiis9SogBziQMLLMcRrkGIhIOw2T2ruLDOWjDlPxiFzlMBLmSLGOUWB8axvD149Hi5XTvMOyFS8PbTrRyDk+f5r9+I4TOpvsdE6eKFgKkpZwYM47C3E4c0ZZRDmU3LgzkhM7IFYy9K4kAM3HLeUv8wkdSnCntj7R4GW1mOCKMWYjEKwWx16bNquD/2Liw2cnEMZkXFiStG2UFx1bh6vE4ZRqo5QvvEKqZnxXTa6IJtf6LurGEb1QJQWTq4pvSxVWDJHM3ZbnJYM2gzTLPEsXTan7FXVZxDU1Fsc4utul8Teee4o3ioxWkhLtRu/NFA1604WUDXm537Tdwv53r94mbf2Ml7nObp1R5nE5rhRauurdFXJmqRkK0/z/O27hvNWFyY4p1vZqVpV/TqL2U2875cS8Ke9Env6+vUG276Bl6jo5QhF6jU/QenaEBomiGvqMf6GdwEJwEb4K3tbSzs8p5gjYs+PAXyuY51A==</latexit><latexit sha1_base64=\"UiSf8tl1AGan+BKFr9Dkq6/3J0s=\">AAADkHicbZJLb9NAEMe3MY8SXm2RuHCxWiGVS2T3ALlRxAXBpRDSpI1DtF6P21X2Ye2uQ6OVPwQ3BFK/VwUfhnWcKo7DSCvNzu8/j11NnDGqTRDcbLW8O3fv3d9+0H746PGTpzu7e6da5opAn0gm1TDGGhgV0DfUMBhmCjCPGQzi6fuSD2agNJXiq5lnMOb4QtCUEmxcaBDBt0+H81eTnYOgEyzM33TCpXNw/Dy7/vn3T/dkstvSUSJJzkEYwrDWozDIzNhiZShhULSjXEOGyRRfwMi5AnPQY7uYt/Bfukjip1K5I4y/iNYzLOZaz3nslBybS91kZfB/bJSbtDu2VGS5AUGqRmnOfCP98vF+QhUQw+bOwURRN6tPLrHCxLgvakcCvhPJORaJja4KG5UN4tReFcU6gxWDJksdiyVLyvkls2nJFdQV+So736SzFZ056q8VHy4hwcwOm53PavCsCc9r8Hyza6+Ge81ct09M344V2y9NnhDpcDKpFIrb8t4UManLGjFW7v8Ya+KeUZiKtSlW9SpWFG5Nw+ZSbjqnR50w6ISf3b6+RpVtoxdoHx2iEL1Bx+gDOkF9RNAU/UC/0G9vz+t6b713lbS1tcx5htbM+/gPcZw7uw==</latexit><latexit sha1_base64=\"UiSf8tl1AGan+BKFr9Dkq6/3J0s=\">AAADkHicbZJLb9NAEMe3MY8SXm2RuHCxWiGVS2T3ALlRxAXBpRDSpI1DtF6P21X2Ye2uQ6OVPwQ3BFK/VwUfhnWcKo7DSCvNzu8/j11NnDGqTRDcbLW8O3fv3d9+0H746PGTpzu7e6da5opAn0gm1TDGGhgV0DfUMBhmCjCPGQzi6fuSD2agNJXiq5lnMOb4QtCUEmxcaBDBt0+H81eTnYOgEyzM33TCpXNw/Dy7/vn3T/dkstvSUSJJzkEYwrDWozDIzNhiZShhULSjXEOGyRRfwMi5AnPQY7uYt/Bfukjip1K5I4y/iNYzLOZaz3nslBybS91kZfB/bJSbtDu2VGS5AUGqRmnOfCP98vF+QhUQw+bOwURRN6tPLrHCxLgvakcCvhPJORaJja4KG5UN4tReFcU6gxWDJksdiyVLyvkls2nJFdQV+So736SzFZ056q8VHy4hwcwOm53PavCsCc9r8Hyza6+Ge81ct09M344V2y9NnhDpcDKpFIrb8t4UManLGjFW7v8Ya+KeUZiKtSlW9SpWFG5Nw+ZSbjqnR50w6ISf3b6+RpVtoxdoHx2iEL1Bx+gDOkF9RNAU/UC/0G9vz+t6b713lbS1tcx5htbM+/gPcZw7uw==</latexit><latexit sha1_base64=\"2ZQq0xYDttpZLLv8Wa+7uXHOsJg=\">AAADkHicbZJLbxMxEMfdLNASXm05crGIkMol2uUAvdGKC4JLIaRJmw2R1zvbWvFjZXtDI2s/BFf4ZHwbvMmibDaMZGk8v/88bE2Sc2ZsGP7Z6wT37j/YP3jYffT4ydNnh0fHl0YVmsKQKq70OCEGOJMwtMxyGOcaiEg4jJL5h4qPFqANU/KbXeYwFeRGsoxRYn1oFMP3zyfL17PDXtgPV4Z3nah2eqi2i9lRx8SpooUAaSknxkyiMLdTR7RllEPZjQsDOaFzcgMT70oiwEzdat4Sv/KRFGdK+yMtXkWbGY4IY5Yi8UpB7K1psyr4PzYpbHY6dUzmhQVJ142ygmOrcPV4nDIN1PKldwjVzM+K6S3RhFr/Rd1Ywg+qhCAydfFd6eKqQZK5u7LcZrBh0GaZZ4niaTW/4i6ruIamothkF7t0saELT/FW8XENKeFu3O581YBXbXjdgNe7XQcNPGjn+n3i5t9Yifva5ilVHqeztUILV93bIq5MVSMh2v8f5208sJowuTXFpt6alaVf06i9lLvO5Zt+FPajL2Hv7G29sAfoBXqJTlCE3qEz9BFdoCGiaI5+ol/od3AcnAbvg/O1tLNX5zxHWxZ8+gvH5DfQ</latexit>y<latexit sha1_base64=\"YG+YhEP4k3bUV66vibrBzoc3glc=\">AAADinicbZJLbxMxEMfdLI8SCjRw5GJRIXGKdnvgIS6V4MCxJaRNm40i2zvbWvVjZXtDI2s/AeIGH45vgzcblM2GkSyN5/efh62hheDWxfGfvV507/6Dh/uP+o8Pnjx9djh4fm51aRiMmRbaTCixILiCseNOwKQwQCQVcEFvP9X8YgHGcq2+uWUBM0muFc85Iy6Ezpbzw6N4GK8M7zrJ2jk6GfwcoGCn80HPpplmpQTlmCDWTpO4cDNPjONMQNVPSwsFYbfkGqbBVUSCnfnVpBV+HSIZzrUJRzm8irYzPJHWLiUNSkncje2yOvg/Ni1d/n7muSpKB4o1jfJSYKdx/WyccQPMiWVwCDM8zIrZDTGEufA5/VTBd6alJCrz6V3l07oBzf1dVW0z2DDosjwwqkVWz6+Fz2tuoK0oN9nlLl1s6CJQvFV8soaMCD/pdr5swcsuvGrBq92uoxYedXPDJgn7byzqv3Z5xnTA2bxRGOnre1cktK1rUGLC/wnRxSNnCFdbU2zqNayqwpom3aXcdc6Ph0k8TM7Cvr5Fje2jl+gVeoMS9A6doC/oFI0RQ4B+oF/od3QQHUcfoo+NtLe3znmBtiz6/BcwNDbI</latexit><latexit sha1_base64=\"icsg/DrdlbjmAepD5G5/XZt+Cvc=\">AAADinicbZLNbhMxEMfdLNASCiRwg8uKColTtNsDUHGpBAeOLSFt2mwUeb2zrVV/rGxvaGTtEyBu8B68Bo/Ag3DHmw3KxmEkS+P5/efD1qQFo9pE0e+dTnDn7r3dvfvdB/sPHz3u9Z+caVkqAiMimVTjFGtgVMDIUMNgXCjAPGVwnt68r/n5HJSmUnw2iwKmHF8JmlOCjQudLma9g2gQLS3cduKVc3Dc/9b78+zXz5NZv6OTTJKSgzCEYa0ncVSYqcXKUMKg6ialhgKTG3wFE+cKzEFP7XLSKnzpIlmYS+WOMOEy2s6wmGu94KlTcmyutc/q4P/YpDT526mloigNCNI0yksWGhnWzw4zqoAYtnAOJoq6WUNyjRUmxn1ONxHwhUjOschsclvZpG6Q5va2qjYZrBn4LHcslSyr55fM5jVX0FaU6+xym87XdO5ouFF8vIIEMzv2O1+04IUPL1vwcrvrsIWHfq7bJKb/jZXaTz7PiHQ4mzUKxW1990VM6rpGipX7P8Z8PDQKU7Exxbpew6rKrWnsL+W2c3Y4iKNBfOr29TVqbA89Ry/QKxSjN+gYfUQnaIQIAvQVfUc/gv3gMDgK3jXSzs4q5ynasODDX7K2OU8=</latexit><latexit sha1_base64=\"icsg/DrdlbjmAepD5G5/XZt+Cvc=\">AAADinicbZLNbhMxEMfdLNASCiRwg8uKColTtNsDUHGpBAeOLSFt2mwUeb2zrVV/rGxvaGTtEyBu8B68Bo/Ag3DHmw3KxmEkS+P5/efD1qQFo9pE0e+dTnDn7r3dvfvdB/sPHz3u9Z+caVkqAiMimVTjFGtgVMDIUMNgXCjAPGVwnt68r/n5HJSmUnw2iwKmHF8JmlOCjQudLma9g2gQLS3cduKVc3Dc/9b78+zXz5NZv6OTTJKSgzCEYa0ncVSYqcXKUMKg6ialhgKTG3wFE+cKzEFP7XLSKnzpIlmYS+WOMOEy2s6wmGu94KlTcmyutc/q4P/YpDT526mloigNCNI0yksWGhnWzw4zqoAYtnAOJoq6WUNyjRUmxn1ONxHwhUjOschsclvZpG6Q5va2qjYZrBn4LHcslSyr55fM5jVX0FaU6+xym87XdO5ouFF8vIIEMzv2O1+04IUPL1vwcrvrsIWHfq7bJKb/jZXaTz7PiHQ4mzUKxW1990VM6rpGipX7P8Z8PDQKU7Exxbpew6rKrWnsL+W2c3Y4iKNBfOr29TVqbA89Ry/QKxSjN+gYfUQnaIQIAvQVfUc/gv3gMDgK3jXSzs4q5ynasODDX7K2OU8=</latexit><latexit sha1_base64=\"UFGn3rMe1QKPN7QvGLEGaaKQ+8k=\">AAADinicbZJLbxMxEMfdLI8SCrRw5GIRIXGKdnugIC6V4MCxJaRNm40i2zvbWvVjZXtDI2s/AVf4cHwbvMmibDaMZGk8v/88bA0tBLcujv/s9aIHDx893n/Sf3rw7PmLw6OXF1aXhsGYaaHNhBILgisYO+4ETAoDRFIBl/Tuc80vF2As1+q7WxYwk+RG8Zwz4kLofDk/HMTDeGV410kaZ4AaO5sf9WyaaVZKUI4JYu00iQs388Q4zgRU/bS0UBB2R25gGlxFJNiZX01a4bchkuFcm3CUw6toO8MTae1S0qCUxN3aLquD/2PT0uUfZp6ronSg2LpRXgrsNK6fjTNugDmxDA5hhodZMbslhjAXPqefKvjBtJREZT69r3xaN6C5v6+qbQYbBl2WB0a1yOr5tfB5zQ20FeUmu9yliw1dBIq3ik8ayIjwk27nqxa86sLrFrze7Tpq4VE3N2ySsP/Gov5bl2dMB5zN1wojfX3vioS2dQ1KTPg/Ibp45AzhamuKTb01q6qwpkl3KXedi+NhEg+T83hw+r5Z2H30Gr1B71CCTtAp+orO0BgxBOgn+oV+RwfRcfQx+rSW9vaanFdoy6IvfwFmQzXZ</latexit>GloVe.6B<latexit sha1_base64=\"YYYHCiFnuTL5brZigISSYw+jAAQ=\">AAADl3icbZJLbxMxEMfdLI8SXi2cEBeLCKmnaJdDy60VlaDHlpA0bTZEXu9sa9WPle1NG1n7PbjC5+DA1+CDcMebDUqyYSRL4/nN46/RJDlnxobh761WcO/+g4fbj9qPnzx99nxn98XAqEJT6FPFlR4mxABnEvqWWQ7DXAMRCYfz5Oa44udT0IYp+cXOchgLciVZxiixPvTVxVrgT1wNoLv/oZzsdMJuODe86UQLp3P4c+/PL4TQ6WS3ZeJU0UKAtJQTY0ZRmNuxI9oyyqFsx4WBnNAbcgUj70oiwIzdXHaJ3/pIijOl/ZMWz6OrFY4IY2Yi8ZmC2GvTZFXwf2xU2Oz92DGZFxYkrQdlBcdW4WoHOGUaqOUz7xCqmdeK6TXRhFq/qXYs4ZYqIYhMXXxXurgakGTurizXGSwZNFnmWaJ4WulX3GUV17CaUSyri006XdKpp3it+XABKeFu2Jx8sQIvmvByBV5uTu2t4F6z1p8VN/9kJe5zk6dUeZxO6gwtXPVvJnFlqh4J0X5/nDdxz2rC5JqKZb+aldWZRs2j3HQG77pR2I3Ows7RPqptG71Gb9AeitABOkIn6BT1EUUafUPf0Y/gVXAYfAxO6tTW1qLmJVqz4Owvtl89kA==</latexit><latexit sha1_base64=\"lrkntAxpwSesfDTwB134uh/gbmQ=\">AAADl3icbZJLbxMxEMfdLI8SXi2cEBeLCKlcot0eCrdWIEGPLSFp2myIvN7Z1qofK9sbGln7Gbhyhc/BiW/BB+GONxuUZMNIlsbzm8dfo0lyzowNw99breDW7Tt3t++17z94+Ojxzu6TgVGFptCniis9TIgBziT0LbMchrkGIhIOZ8n1u4qfTUEbpuQnO8thLMilZBmjxPrQZxdrgT9wNYDuwdtystMJu+Hc8KYTLZzO4c+9P7++xq9OJrstE6eKFgKkpZwYM4rC3I4d0ZZRDmU7LgzkhF6TSxh5VxIBZuzmskv80kdSnCntn7R4Hl2tcEQYMxOJzxTEXpkmq4L/Y6PCZm/Gjsm8sCBpPSgrOLYKVzvAKdNALZ95h1DNvFZMr4gm1PpNtWMJX6gSgsjUxTeli6sBSeZuynKdwZJBk2WeJYqnlX7FXVZxDasZxbK62KTTJZ16iteaDxeQEu6GzcnnK/C8CS9W4MXm1N4K7jVr/Vlx809W4j42eUqVx+mkztDCVf9mElem6pEQ7ffHeRP3rCZMrqlY9qtZWZ1p1DzKTWew343CbnQado4OUG3b6Dl6gfZQhF6jI3SMTlAfUaTRN/Qd/QieBYfB++C4Tm1tLWqeojULTv8C8EY+pg==</latexit><latexit sha1_base64=\"lrkntAxpwSesfDTwB134uh/gbmQ=\">AAADl3icbZJLbxMxEMfdLI8SXi2cEBeLCKlcot0eCrdWIEGPLSFp2myIvN7Z1qofK9sbGln7Gbhyhc/BiW/BB+GONxuUZMNIlsbzm8dfo0lyzowNw99breDW7Tt3t++17z94+Ojxzu6TgVGFptCniis9TIgBziT0LbMchrkGIhIOZ8n1u4qfTUEbpuQnO8thLMilZBmjxPrQZxdrgT9wNYDuwdtystMJu+Hc8KYTLZzO4c+9P7++xq9OJrstE6eKFgKkpZwYM4rC3I4d0ZZRDmU7LgzkhF6TSxh5VxIBZuzmskv80kdSnCntn7R4Hl2tcEQYMxOJzxTEXpkmq4L/Y6PCZm/Gjsm8sCBpPSgrOLYKVzvAKdNALZ95h1DNvFZMr4gm1PpNtWMJX6gSgsjUxTeli6sBSeZuynKdwZJBk2WeJYqnlX7FXVZxDasZxbK62KTTJZ16iteaDxeQEu6GzcnnK/C8CS9W4MXm1N4K7jVr/Vlx809W4j42eUqVx+mkztDCVf9mElem6pEQ7ffHeRP3rCZMrqlY9qtZWZ1p1DzKTWew343CbnQado4OUG3b6Dl6gfZQhF6jI3SMTlAfUaTRN/Qd/QieBYfB++C4Tm1tLWqeojULTv8C8EY+pg==</latexit><latexit sha1_base64=\"bF3W54KGXHSHjr+PeRuGAWNcnFQ=\">AAADl3icbZJLb9NAEMe3MY8SXi2cKi4rIiROkc2hcKMCifbYEpKmjUO0Xo/bVfdh7a5Do5W/R6/wrfg2XSdGcRxGWum/85uXRpPknBkbhn93OsGDh48e7z7pPn32/MXLvf1XI6MKTWFIFVd6nBADnEkYWmY5jHMNRCQczpObrxU/n4M2TMkfdpHDVJAryTJGifWuny7WAh9zNYL+4ZdyttcL++HS8LaIatFDtZ3O9jsmThUtBEhLOTFmEoW5nTqiLaMcym5cGMgJvSFXMPFSEgFm6pZjl/id96Q4U9o/afHS28xwRBizEImPFMRemzarnP9jk8Jmn6aOybywIOmqUVZwbBWudoBTpoFavvCCUM38rJheE02o9ZvqxhJ+USUEkamLb0sXVw2SzN2W5SaDNYM2yzxLFE+r+RV3WcU1NCOKdXaxTedrOvcUbxQf15AS7sbtzhcNeNGGlw14ud110MCDdq4/K27+jZW4722eUuVxOltFaOGqfzuIK1PVSIj2++O8jQdWEyY3pljXW7GyOtOofZTbYvShH4X96CzsHR3WB7uL3qC36D2K0Ed0hE7QKRoiijS6Q7/Rn+Ag+Bx8C05WoZ2dOuc12rDg7B46gjq4</latexit>GloVe.42B<latexit sha1_base64=\"3TNCOMbiD9hmAviWqyS1TzV4MY0=\">AAADmnicbZJLbxMxEMfdLI8SHk3hCAeLCKmnaLdCwI2qHABxKaRJ02ajyOudba36sbK9oZG1n4QrfAsOfA0+CHe82aAkG0ayNJ7fPP4aTZJzZmwY/t5pBbdu37m7e699/8HDR3ud/cdDowpNYUAVV3qUEAOcSRhYZjmMcg1EJBzOkut3FT+bgTZMyVM7z2EiyKVkGaPE+tC0s+diLfB7robQe3l4XE473bAXLgxvO9HS6b79efDnF0LoZLrfMnGqaCFAWsqJMeMozO3EEW0Z5VC248JATug1uYSxdyURYCZuobzEL3wkxZnS/kmLF9H1CkeEMXOR+ExB7JVpsir4PzYubPZm4pjMCwuS1oOygmOrcLUGnDIN1PK5dwjVzGvF9IpoQq1fVjuW8JUqIYhMXXxTurgakGTupiw3GawYNFnmWaJ4WulX3GUV17CeUayqi206W9GZp3ij+WgJKeFu1Jx8vgbPm/BiDV5sT+2v4X6z1l8WN/9kJe5Lk6dUeZxO6wwtXPVvJnFlqh4J0X5/nDdx32rC5IaKVb+aldWZRs2j3HaGh70o7EWfw+7RK1TbLnqKnqMDFKHX6Ah9QCdogCgq0Df0Hf0IngXHwcfgU53a2lnWPEEbFpz+BSMFPfs=</latexit><latexit sha1_base64=\"XQvrH8rpUWKwOHdCsRX/ZUzS3/M=\">AAADmnicbZLNbtNAEMe3MR8lfDSFIxwsIqRyiewKQW+tygEQl0KaNG0cRev1uF11P6zddWi08jPwAFzhLTjxFjwId9ZxUByHkVaand98/DWaOGNUmyD4vdXybt2+c3f7Xvv+g4ePdjq7j4da5orAgEgm1SjGGhgVMDDUMBhlCjCPGZzF129LfjYDpakUp2aewYTjS0FTSrBxoWlnx0aK+++YHELv1f5xMe10g16wMH/TCZdO9/Dn3p9fX6OXJ9Pdlo4SSXIOwhCGtR6HQWYmFitDCYOiHeUaMkyu8SWMnSswBz2xC+WF/8JFEj+Vyj1h/EW0XmEx13rOY5fJsbnSTVYG/8fGuUkPJpaKLDcgSDUozZlvpF+uwU+oAmLY3DmYKOq0+uQKK0yMW1Y7EvCFSM6xSGx0U9ioHBCn9qYo1hmsGDRZ6lgsWVLql8ymJVdQz8hX1fkmna3ozFF/rfloCQlmdtScfF6D5014UYMXm1P7Ndxv1rrLYvqfrNh+bvKESIeTaZWhuC3/zSQmddkjxsrtj7Em7huFqVhTsepXsaI807B5lJvOcL8XBr3wU9A9eo0q20ZP0XO0h0L0Bh2h9+gEDRBBOfqGvqMf3jPv2PvgfaxSW1vLmidozbzTv1zsPxE=</latexit><latexit sha1_base64=\"XQvrH8rpUWKwOHdCsRX/ZUzS3/M=\">AAADmnicbZLNbtNAEMe3MR8lfDSFIxwsIqRyiewKQW+tygEQl0KaNG0cRev1uF11P6zddWi08jPwAFzhLTjxFjwId9ZxUByHkVaand98/DWaOGNUmyD4vdXybt2+c3f7Xvv+g4ePdjq7j4da5orAgEgm1SjGGhgVMDDUMBhlCjCPGZzF129LfjYDpakUp2aewYTjS0FTSrBxoWlnx0aK+++YHELv1f5xMe10g16wMH/TCZdO9/Dn3p9fX6OXJ9Pdlo4SSXIOwhCGtR6HQWYmFitDCYOiHeUaMkyu8SWMnSswBz2xC+WF/8JFEj+Vyj1h/EW0XmEx13rOY5fJsbnSTVYG/8fGuUkPJpaKLDcgSDUozZlvpF+uwU+oAmLY3DmYKOq0+uQKK0yMW1Y7EvCFSM6xSGx0U9ioHBCn9qYo1hmsGDRZ6lgsWVLql8ymJVdQz8hX1fkmna3ozFF/rfloCQlmdtScfF6D5014UYMXm1P7Ndxv1rrLYvqfrNh+bvKESIeTaZWhuC3/zSQmddkjxsrtj7Em7huFqVhTsepXsaI807B5lJvOcL8XBr3wU9A9eo0q20ZP0XO0h0L0Bh2h9+gEDRBBOfqGvqMf3jPv2PvgfaxSW1vLmidozbzTv1zsPxE=</latexit><latexit sha1_base64=\"81W3eQ71ESO0iRsmI+OybVbe2vc=\">AAADmnicbZJLb9NAEMe3MY8SHk3hCIcVERKnyK4q4FiVAyAuhTRp2jiK1utxu+o+rN11aLTyJ+EKH4pvwzoxiuMw0kr/nd+8NJok58zYMPyz1wnu3X/wcP9R9/GTp88OeofPx0YVmsKIKq70JCEGOJMwssxymOQaiEg4XCS3Hyt+sQBtmJLndpnDTJBryTJGifWuee/AxVrgT1yNYXB8dFrOe/1wEK4M74qoFn1U29n8sGPiVNFCgLSUE2OmUZjbmSPaMsqh7MaFgZzQW3INUy8lEWBmbjV5id94T4ozpf2TFq+8zQxHhDFLkfhIQeyNabPK+T82LWz2YeaYzAsLkq4bZQXHVuFqDThlGqjlSy8I1czPiukN0YRav6xuLOEHVUIQmbr4rnRx1SDJ3F1ZbjPYMGizzLNE8bSaX3GXVVxDM6LYZBe7dLGhC0/xVvFJDSnhbtLufNmAl2141YBXu12HDTxs5/rL4ubfWIn73uYpVR6n83WEFq76t4O4MlWNhGi/P87beGg1YXJrik29NSurM43aR7krxkeDKBxE38L+ybv6YPfRS/QavUUReo9O0Gd0hkaIogL9RL/Q7+BVcBp8Cb6uQzt7dc4LtGXB+V+nGTsj</latexit>FastText<latexit sha1_base64=\"sVqH8YvI/mj2yRb3D0hV1tgzQm0=\">AAADl3icbZJLb9NAEMe3MY8SXi2cEJcVEVJPkc0BuLUSUtVjS5o2bRyi9XrcrroPa3cdEq38PbjC5+DA1+CDcGcdB8VxGGml2fnN46/RJDlnxobh751OcO/+g4e7j7qPnzx99nxv/8WFUYWmMKSKKz1KiAHOJAwtsxxGuQYiEg6Xyd2nil/OQBum5Lld5DAR5EayjFFifeiLi7XAx8TYc5jbcrrXC/vh0vC2E62c3uHPgz+/EEKn0/2OiVNFCwHSUk6MGUdhbieOaMsoh7IbFwZyQu/IDYy9K4kAM3FL2SV+6yMpzpT2T1q8jDYrHBHGLETiMwWxt6bNquD/2Liw2ceJYzIvLEhaD8oKjq3C1Q5wyjRQyxfeIVQzrxXTW6IJtX5T3VjCV6qEIDJ18bx0cTUgydy8LDcZrBm0WeZZonha6VfcZRXX0Mwo1tXFNp2t6cxTvNF8tIKUcDdqT75qwKs2vG7A6+2pgwYetGv9WXHzT1biPrd5SpXH6bTO0MJV/3YSV6bqkRDt98d5Gw+sJkxuqFj3q1lZnWnUPspt5+JdPwr70VnYO3qPattFr9EbdIAi9AEdoRN0ioaIIo2+oe/oR/AqOAyOg5M6tbOzqnmJNiw4+wviDT5A</latexit><latexit sha1_base64=\"xH4mhS2gc1imMnRteTB2c3bDpuY=\">AAADl3icbZJLbxMxEMfdLI8SXi2cEBeLCKlcol0OwK2VKlU9tqRp02ZD5PXOtlb9WNneNJG1n4ErV/gcnPgWfBDueLNBSTaMZGk8v3n8NZok58zYMPy91Qru3X/wcPtR+/GTp8+e7+y+ODeq0BT6VHGlBwkxwJmEvmWWwyDXQETC4SK5Paz4xQS0YUqe2VkOI0GuJcsYJdaHvrhYC3xEjD2DqS3HO52wG84NbzrRwuns/9z78+tr/O5kvNsycapoIUBayokxwyjM7cgRbRnlULbjwkBO6C25hqF3JRFgRm4uu8RvfSTFmdL+SYvn0dUKR4QxM5H4TEHsjWmyKvg/Nixs9mnkmMwLC5LWg7KCY6twtQOcMg3U8pl3CNXMa8X0hmhCrd9UO5ZwR5UQRKYunpYurgYkmZuW5TqDJYMmyzxLFE8r/Yq7rOIaVjOKZXWxSSdLOvEUrzUfLCAl3A2aky9X4GUTXq3Aq82pvRXca9b6s+Lmn6zEfW7ylCqP03GdoYWr/s0krkzVIyHa74/zJu5ZTZhcU7HsV7OyOtOoeZSbzvn7bhR2o9Owc/AB1baNXqM3aA9F6CM6QMfoBPURRRp9Q9/Rj+BVsB8cBcd1amtrUfMSrVlw+hccAz9W</latexit><latexit sha1_base64=\"xH4mhS2gc1imMnRteTB2c3bDpuY=\">AAADl3icbZJLbxMxEMfdLI8SXi2cEBeLCKlcol0OwK2VKlU9tqRp02ZD5PXOtlb9WNneNJG1n4ErV/gcnPgWfBDueLNBSTaMZGk8v3n8NZok58zYMPy91Qru3X/wcPtR+/GTp8+e7+y+ODeq0BT6VHGlBwkxwJmEvmWWwyDXQETC4SK5Paz4xQS0YUqe2VkOI0GuJcsYJdaHvrhYC3xEjD2DqS3HO52wG84NbzrRwuns/9z78+tr/O5kvNsycapoIUBayokxwyjM7cgRbRnlULbjwkBO6C25hqF3JRFgRm4uu8RvfSTFmdL+SYvn0dUKR4QxM5H4TEHsjWmyKvg/Nixs9mnkmMwLC5LWg7KCY6twtQOcMg3U8pl3CNXMa8X0hmhCrd9UO5ZwR5UQRKYunpYurgYkmZuW5TqDJYMmyzxLFE8r/Yq7rOIaVjOKZXWxSSdLOvEUrzUfLCAl3A2aky9X4GUTXq3Aq82pvRXca9b6s+Lmn6zEfW7ylCqP03GdoYWr/s0krkzVIyHa74/zJu5ZTZhcU7HsV7OyOtOoeZSbzvn7bhR2o9Owc/AB1baNXqM3aA9F6CM6QMfoBPURRRp9Q9/Rj+BVsB8cBcd1amtrUfMSrVlw+hccAz9W</latexit><latexit sha1_base64=\"o1gA43HEtbBE2CNf70FiaACoO3M=\">AAADl3icbZJLa9tAEMc3Vh+p+0raU+llqSn0ZKQe0t4aKIQckzpOnFiuWa1GyZJ9iN2Va7Poe+Safqt8m6xsFctyBxb+O795MUySc2ZsGD7sdIInT589333Rffnq9Zu3e/vvzo0qNIUhVVzpUUIMcCZhaJnlMMo1EJFwuEhuf1b8YgbaMCXP7CKHiSDXkmWMEutdv12sBT4ixp7B3JbTvV7YD5eGt0VUix6q7WS63zFxqmghQFrKiTHjKMztxBFtGeVQduPCQE7oLbmGsZeSCDATtxy7xJ+9J8WZ0v5Ji5feZoYjwpiFSHykIPbGtFnl/B8bFzb7PnFM5oUFSVeNsoJjq3C1A5wyDdTyhReEauZnxfSGaEKt31Q3lvCHKiGITF08L11cNUgyNy/LTQZrBm2WeZYonlbzK+6yimtoRhTr7GKbztZ05ineKD6qISXcjdqdLxvwsg2vGvBqu+uggQftXH9W3PwbK3G/2jylyuN0uorQwlX/dhBXpqqREO33x3kbD6wmTG5Msa63YmV1plH7KLfF+dd+FPaj07B3eFAf7C76iD6hLyhC39AhOkYnaIgo0ugO3aO/wYfgR3AUHK9COzt1znu0YcHpI2YwO2g=</latexit>\u2026\u2026\f\u02c6y = arg min\n\ny\u2208Y\n\ndcos(ek(y), f k\n\n\u03b8k (x)).\n\n(3)\n\nIn order to distinguish between in- and out-of-distribution examples, we consider the norms of the\npredicted embedding vectors. When the sum of all the norms is below a detection threshold \u03b1 we\ndenote the example as an out-of-distribution example, namely,\n\nK(cid:88)\n\nk=1\n\nK(cid:88)\n\nAnother, more successful procedure for decoding, is the soft decision, where we predict the label y\nwhich has the minimal distance to all of the K embedding vectors:\n\n(cid:107)f k\n\n\u03b8k (x)(cid:107)2\n\n2 < \u03b1.\n\n(4)\n\nThis is inspired by the discussion of [39, Section 4] and motivated empirically in Section 4.3.\n\nk=1\n\n4 Experiments\n\nIn this section, we present our experimental results. First, we describe our experimental setup.\nThen, we evaluate our model using in-distribution examples, and lastly, we evaluate our model using\nout-of-distribution examples from different domains. We implemented the code using PyTorch [35].\nIt will be available under www.github.com/MLSpeech/semantic_OOD.\n\n4.1 Experimental Setup\n\nRecall that each regression function f k\n\u03b8k is composed of a shared part and an exclusive part. In our\nsetting, we used state-of-the-art known architectures as the shared part, and three fully-connected\nlayers, with ReLU activation function between the \ufb01rst two, as the exclusive part.\nWe evaluated our approach on CIFAR-10, CIFAR-100 [18] and Google Speech Commands Dataset1,\nabbreviated here as GCommands. For CIFAR-10 and CIFAR-100 we trained ResNet-18 and ResNet34\nmodels [12], respectively, using stochastic gradient descent with momentum for 180 epochs. We\nused the standard normalization and data augmentation techniques. We used learning rate of 0.1,\nmomentum value of 0.9 and weight decay of 0.0005. During training we divided the learning rate\nby 5 after 60, 120 and 160 epochs. For the GCommands dataset, we trained LeNet model [22]\nusing Adam [16] for 20 epochs using batch size of 100 and a learning rate of 0.001. Similarly\nto [44] we extracted normalized spectrograms from the original waveforms where we zero-padded\nthe spectrograms to equalize their sizes at 160 \u00d7 101.\nFor our supervision, we fetched \ufb01ve word representations for each label. The \ufb01rst two word represen-\ntations were based on the Skip-Gram model [29] trained on Google News dataset and One Billion\nWords benchmark [5], respectively. The third and forth representations were based on GloVe [36],\nwhere the third one was trained using both Wikipedia corpus and Gigawords [34] dataset, the fourth\nwas trained using Common Crawl dataset. The last word representations were obtained using Fast-\nText [28] trained on Wikipedia corpus. We use the terms 1-embed, 3-embed, and 5-embed to specify\nthe number of embeddings we use as supervision, i.e., the number of predicted embedding vectors.\nOn 1-embed and 3-embed models we randomly pick 1 or 3 embeddings (respectively), out of the \ufb01ve\nembeddings.\nWe compared our results to a softmax classi\ufb01er (baseline) [13], ensemble of softmax classi\ufb01ers\n(ensemble) [19] and to [25] (ODIN). For the ensemble method, we followed a similar approach to\n[19, 24] where we randomly initialized each of the models. For ODIN, we followed the scheme\nproposed in by the authors where we did a grid search over the \u0001 and T values for each setting. In all\nof these models we optimized the cross-entropy loss function using the same architectures as in the\nproposed models.\n\n4.2\n\nIn-Distribution\n\nAccuracy\nIn this subsection, we evaluate the performance of our model and compare it to models\nbased on softmax. Similar to [25], we considered CIFAR-10, CIFAR-100 and GCommands datasets\n\n1https://research.googleblog.com/2017/08/launching-speech-commands-dataset.html\n\n4\n\n\fTable 1: Accuracy on in-distribution examples and semantical relevance of misclassi\ufb01cations.\n\nCIFAR-10\n\nDataset\nModel\nGCommands Baseline\n1-embed\n3-embed\n5-embed\nEnsemble\nBaseline\n1-embed\n3-embed\n5-embed\nEnsemble\nBaseline\n1-embed\n3-embed\n5-embed\nEnsemble\n\nCIFAR-100\n\nAccuracy Avg. WUP Avg. LCH Avg. Path\n\n90.3\n90.42\n91.04\n91.13\n90.9\n95.28\n95.11\n94.99\n95.04\n95.87\n79.14\n77.62\n78.31\n78.23\n81.38\n\n0.2562\n0.3279\n0.3215\n0.3095\n0.2206\n0.7342\n0.741\n0.7352\n0.7302\n0.733\n0.506\n0.51\n0.501\n0.5129\n0.5122\n\n1.07\n1.23\n1.22\n1.19\n0.96\n1.7\n1.73\n1.71\n1.69\n1.71\n1.38\n1.39\n1.38\n1.4\n1.4\n\n0.0937\n0.1204\n0.1184\n0.1141\n0.0748\n0.1594\n0.1633\n0.1609\n0.157\n0.1601\n0.1263\n0.1277\n0.1251\n0.1293\n0.1291\n\nas in-distribution examples. We report the accuracy for our models using K = 1, 3 or 5 word\nembeddings, and compare it to the baseline and to the ensemble of softmax classi\ufb01er. Results are\nsummarized in Table 1.\n\nSemantic Measure Word embeddings usually capture the semantic hierarchy between the\nwords [29], since the proposed models are trained with word embeddings as supervision, they\ncan capture such semantics. To measure the semantic quality of the model, we compute three se-\nmantic measures based on WordNet hierarchy: (i) Node-counting on the shortest path that connects\nthe senses in the is-a taxonomy; (ii) Wu-Palmer (WUP) [41], calculates the semantic relatedness by\nconsidering the depth of the two senses in the taxonomy; and (iii) Leacock-Chodorow (LCH) [21],\ncalculates relatedness by \ufb01nding the shortest path between two concepts and scaling that value by\ntwice the maximum depth of the hierarchy. Results suggest that on average our model produces labels\nwhich have slightly better semantic quality.\n\n4.3 Out-of-Distribution\n\nOut-of-Distribution Datasets For out-of-distribution examples, we followed a similar setting as\nin [25, 13] and evaluated our models on several different datasets. All visual models were trained on\nCIFAR-10 and tested on SVHN[32], LSUN[42] (resized to 32x32x3) and CIFAR-100; and trained on\nCIFAR-100 and were tested on SVHN, LSUN (resized to 32x32x3) and CIFAR-10. For the speech\nmodels, we split the dataset into two disjoint subsets, the \ufb01rst contains 7 classes2 (denote by SC-7)\nand the other one contains the remaining 23 classes (denote by SC-23). We trained our models using\nSC-23 and test them on SC-7.\n\nEvaluation We followed the same metrics used by [13, 25]: (i) False Positive Rate (FPR) at 95%\nTrue Positive Rate (TPR): the probability that an out-of-distribution example is misclassi\ufb01ed as\nin-distribution when the TPR is as high as 95%; (ii) Detection error: the misclassi\ufb01cation probability\nwhen TPR is 95%, where we assumed that both out- and in-distribution have an equal prior of\nappearing; (iii) Area Under the Receiver Operating Characteristic curve (AUROC); and (iv) Area\nUnder the Precision-Recall curve (AUPR) for in-distribution and out-of-distribution examples.\nFigure 2 presents the distribution of the L2-norm of the proposed method using 5 word embeddings\nand the maximum probability of the baseline model for CIFAR-100 (in-distribution) and SVHN\n(out-of-distribution). Both models were trained on CIFAR-100 and evaluated on CIFAR-100 test set\n(in-distribution) and SVHN (out-of-distribution).\n\nResults Table 2 summarizes the results for all the models. Results suggest that our method\noutperforms all the three methods in all but two settings: CIFAR-100 versus CIFAR-10 and CIFAR-\n\n2Composed of the following classes: bed, down, eight, \ufb01ve, four, wow, and zero.\n\n5\n\n\fFigure 2: Distribution of the L2-norm of the proposed model with 5 word embeddings (left) and of the\nmax probability of the baseline model (right). Both were evaluated on CIFAR-100 (in-distribution)\nand SVHN (out-of-distribution).\n\n100 versus LSUN. This can be explained by taking a closer look into the structure of the datasets\nand the predictions made by our model. For these two settings, the in- and out-of-distribution\ndatasets share common classes, and some of the classes of the in-distribution dataset appear in the\nout-of-distribution images. For example: the class bedroom in LSUN was detected as a bed, couch,\nand wardrobe in CIFAR-100 (56%, 18%, and 8% of the time, respectively); bridge of LSUN was\ndetected as a bridge, road, and sea in CIFAR-100 (26%, 13%, and 9%); and tower of LSUN was\ndetected as a skyscraper, castle, and rocket in CIFAR-100 (34%, 20%, and 6%), and there are many\nmore. Similarly, CIFAR-10 and CIFAR-100 contain shared classes such as truck. Recall that the\nproportion of this class is 10% in CIFAR-10 and 1% in CIFAR-100. Hence, when the model is trained\non CIFAR-100 and evaluated on CIFAR-10, it has 10% in-distribution examples a-priori. When\nthe model is trained on CIFAR-10 and evaluated on CIFAR-100, it has 1% in-distribution examples\na-priori.\n\n5 Adversarial Examples\n\nNext, we evaluated the performance of our approach for detecting adversarial examples [11]. Although\nit is not clear if adversarial examples can be considered as an out-of-distribution, we found that our\nmethod is very ef\ufb01cient in detecting these examples. Since our goal is detecting adversarial examples\nrather than suggesting a defense against them, we generated the adversarial examples in a black box\nsettings.\nWe compared our method to an ensemble of softmax classi\ufb01ers [19], both with K = 5 predictors, on\nthe ImageNet dataset [7]. Both models are based on DenseNet-121 [15], wherein our model the K\nregression functions are composed of three fully-connected layers with ReLU as described earlier.\nWe omitted from ImageNet these classes which do not have \u201coff-the-shelf\u201d word representations and\nwere left with 645 labels3.\nWe generated adversarial examples in a black box setting using a third model, namely VGG-19 [38]\nwith the fast gradient sign method and \u0001 = 0.007 [11], and measured the detection rate for each of\nthe models. Notice that our goal was not to be robust to correctly classify adversarial examples but\nrather to detect them.\nWe used the method of detecting out-of-distribution examples to detect adversarial examples, results\nare in Table 3. We further explored the inter-predictor agreements on the predicted class across the K\npredicted embeddings. The histogram of the differences between the maximum and the minimum\nrankings of the predicted label is presented in Figure 3. It can be observed that for our model the\ninter-predictor variability is much higher than that of the ensemble model. One explanation for\nthis behavior is the transferability of adversarial examples across softmax classi\ufb01ers, which can be\nreduced by using different supervisions.\n\n3In [8] the authors suggested to use random vectors as labels. We leave the exploration of this research\n\ndirection for future work.\n\n6\n\n0.00.10.20.30.40.50.60.70.8L2-norm0246810densitycifar100svhn\fTable 2: Results for in- and out-of-distribution detection for various settings. All values are in\npercentages. \u2191 indicates larger value is better, and \u2193 indicates lower value is better.\n\nIn-Distribution\n\nOut-of-\n\nDistribution Model\n\nSC-23\n(LeNet)\n\nSC-7\n\nCIFAR-10\n(ResNet18)\n\nSVHN\n\nCIFAR-10\n(ResNet18)\n\nLSUN\n\nCIFAR-10\n(ResNet18)\n\nCIFAR-100\n\nCIFAR-100\n(ResNet34)\n\nSVHN\n\nCIFAR-100\n(ResNet34)\n\nCIFAR-10\n\nCIFAR-100\n(ResNet34)\n\nLSUN\n\nBaseline\nODIN\n1-embed\n3-embed\n5-embed\nEnsemble\n\nBaseline\nODIN\n1-embed\n3-embed\n5-embed\nEnsemble\nBaseline\nODIN\n1-embed\n3-embed\n5-embed\nEnsemble\nBaseline\nODIN\n1-embed\n3-embed\n5-embed\nEnsemble\n\nBaseline\nODIN\n1-embed\n3-embed\n5-embed\nEnsemble\nBaseline\nODIN\n1-embed\n3-embed\n5-embed\nEnsemble\nBaseline\nODIN\n1-embed\n3-embed\n5-embed\nEnsemble\n\nFPR\n\n(95% TPR) \u2193 Detection\nError \u2193\n41.26\n38.01\n37.8\n36.24\n37.09\n38.85\n\n77.53\n71.02\n70.64\n67.5\n69.23\n72.73\n\n7.19\n4.95\n2.41\n4.92\n4.14\n6.01\n50.25\n41.8\n26.11\n29.2\n22.98\n46.16\n58.75\n54.85\n48.72\n50.9\n45.25\n56.14\n\n87.88\n76.64\n75.79\n74.74\n60.14\n85.92\n77.21\n74.15\n80.82\n78.17\n77.03\n73.57\n80.41\n79.88\n80.99\n81.08\n80.87\n79.53\n\n6.09\n4.97\n3.7\n4.95\n4.57\n5.5\n27.62\n23.39\n15.55\n17.09\n13.99\n25.58\n31.87\n29.92\n26.86\n27.95\n25.12\n30.57\n\n46.44\n40.82\n40.39\n39.87\n32.57\n45.46\n41.1\n39.57\n42.9\n41.77\n41.01\n39.28\n42.7\n42.44\n42.99\n43.03\n42.93\n42.26\n\nAUROC \u2191 AUPR-In \u2191 AUPR-Out \u2191\n\n82.74\n85.49\n85.85\n87.34\n86.93\n85.99\n\n97.2\n98.65\n99.48\n98.52\n99.1\n98.24\n91.28\n90.35\n95.37\n95.07\n96.05\n92.93\n87.76\n85.59\n89.18\n89.76\n91.23\n90.03\n\n74.11\n79.86\n81.82\n82.57\n87.42\n79.1\n79.18\n80.40\n75.99\n77.35\n77.7\n81.49\n78.02\n78.94\n76.41\n74.88\n76.08\n79.05\n\n94.33\n95.11\n95.21\n95.91\n95.74\n95.69\n\n96.35\n96.89\n98.77\n97.76\n98.3\n97.41\n91.58\n96.38\n95.81\n95.63\n96.72\n94.1\n86.73\n82.26\n88.91\n90.36\n91.86\n90.01\n\n63.6\n68.22\n71.52\n75.01\n77.95\n69.23\n80.71\n80.41\n74.27\n77.42\n77.23\n83.24\n79.25\n80.22\n75.08\n72.21\n75.3\n92.61\n\n50.44\n57.7\n58.08\n59.97\n58.97\n50.71\n\n98.05\n99.21\n99.79\n99.07\n99.55\n98.94\n89.3\n75.07\n94.85\n94.38\n94.86\n77.57\n85.83\n85.41\n88.39\n88.13\n89.63\n88.27\n\n84.17\n90.1\n90.1\n90.4\n93.56\n89.3\n75.22\n77.2\n73.01\n74.39\n74.61\n78.16\n73.34\n73.31\n74.02\n72.38\n72.67\n74.06\n\nWe labeled an input as an adversarial example unless all the predictors were in full agreement.\nWe calculated the detection rate of adversarial examples and the false rejection rate of legitimate\nexamples. The ensemble achieved 43.88% detection rate and 11.69% false rejection rate, while the\nembedding model achieved 62.04% detection rate and 15.16% false rejection rate. Although the\nensemble method achieves slightly better false rejection rate (3% improvement), the detection rate of\nour approach is signi\ufb01cantly better (18% improvement). To better qualify that, we \ufb01xed the false\nrejection rate in both methods to be 3%. In this setting, the ensemble reaches 15.41% detection rate\nwhile our model reaches 28.64% detection rate (13% improvement).\n\n6 Wrongly Classi\ufb01ed Examples\n\nRecall the embedding models were trained to minimize the cosine distance between the output vector\nand the word representation of the target label according to some embedding space. When we plot\nthe average L2-norm of the models\u2019 output vectors as a function of the epochs, we have noticed\n\n7\n\n\fFigure 3: Histogram of the differences between the max and the min rankings of the predicted label\nin our model (left) and the ensemble model (right) for original and adversarial examples.\n\nFigure 4: The average L2-norm for wrongly and correctly classi\ufb01ed examples as a function of training\nepochs (left). Density of the L2-norm for wrongly, correctly classi\ufb01ed, and out-of-distributions\nexamples (right).\n\nthat the norm of the wrongly classi\ufb01ed examples is signi\ufb01cantly smaller than those of correctly\nclassi\ufb01ed examples. These \ufb01nding goes in hand with the results in [39], which observed that lower\nrepresentation norms are negatively associated with the softmax predictions, as shown in the left\npanel of Figure 4. Similarly, we observe a similar behavior for out-of-distribution examples as shown\nin the right panel of Figure 4. These \ufb01ndings suggest that we can adjust the threshold \u03b1 accordingly.\nWe leave this further exploration for future work.\n\n7 Related Work\n\nThe problem of detecting out-of-distribution examples in low-dimensional space has been well-\nstudied, however those methods found to be unreliable for high-dimensional space [40]. Recently,\nout-of-distribution detectors based on deep models have been proposed. Several studies require\nenlarging or modifying the neural networks [23, 37, 3], and various approaches suggest to use the\noutput of a pre-trained model with some minor modi\ufb01cations [13, 25].\nThere has been a variety of works revolving around Bayesian inference approximations, which\napproximate the posterior distribution over the parameters of the neural network and use them to\nquantify predictive uncertainty [31, 27]. These Bayesian approximations often harder to implement\nand computationally slower to train compared to non-Bayesian approaches. The authors of [10]\nproposed to use Monte Carlo dropout to estimate uncertainty at test time as Bayesian approximation.\nMore recently, [19] introduced a non-Bayesian method, using an ensemble of classi\ufb01ers for predictive\nuncertainty estimation, which proved to be signi\ufb01cantly better than previous methods.\n\n8\n\n++050100150200250300350Epochs2468101214Avg L2-normcorrect exampleswrong examples\fTable 3: Results for in- and out-of-distribution detection, for ImageNet (in) and adversarial examples\n(out)\n\nModel\n\nEnsemble\n5-embed\n\nFPR\n\n(95% TPR) \u2193 Detection\nError \u2193\n31.15\n31.12\n\n57.3\n57.26\n\nAUROC \u2191 AUPR-In \u2191 AUPR-Out \u2191\n\n88.66\n89.58\n\n98.71\n98.64\n\n43.46\n47.2\n\n8 Discussion and Future Work\n\nIn this paper, we propose to use several semantic representations for each target label as supervision\nto the model in order to detect out-of-distribution examples, where the detection score is based on the\nL2-norm of the output representations.\nFor future work, we would like to further investigate the following: (i) we would like to explore better\ndecision strategy for detecting out-of-distribution examples; (ii) we would like to rigorously analyze\nthe notion of con\ufb01dence based on the L2-norm beyond [39]; (iii) we would like to inspect the relation\nbetween detecting wrongly classi\ufb01ed examples and adversarial examples to out-of-distribution\nexamples.\n\nReferences\n[1] Dario Amodei, Rishita Anubhai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro,\nJingdong Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, et al. Deep speech 2: End-\nto-end speech recognition in english and mandarin. In International Conference on Machine\nLearning, pages 173\u2013182, 2016.\n\n[2] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man\u00e9.\n\nConcrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016.\n\n[3] Jerone TA Andrews, Thomas Tanay, Edward J Morton, and Lewis D Grif\ufb01n. Transfer\n\nrepresentation-learning for anomaly detection. ICML, 2016.\n\n[4] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly\n\nlearning to align and translate. arXiv preprint arXiv:1409.0473, 2014.\n\n[5] Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and\nTony Robinson. One billion word benchmark for measuring progress in statistical language\nmodeling. arXiv preprint arXiv:1312.3005, 2013.\n\n[6] Moustapha M Cisse, Yossi Adi, Natalia Neverova, and Joseph Keshet. Houdini: Fooling deep\nstructured visual and speech recognition models with adversarial examples. In Advances in\nNeural Information Processing Systems, pages 6980\u20136990, 2017.\n\n[7] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.\n\nHierarchical Image Database. In CVPR09, 2009.\n\nImageNet: A Large-Scale\n\n[8] Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Tomas Mikolov, et al. De-\nvise: A deep visual-semantic embedding model. In Advances in neural information processing\nsystems, pages 2121\u20132129, 2013.\n\n[9] Tzeviya Fuchs and Joseph Keshet. Spoken term detection automatically adjusted for a given\n\nthreshold. IEEE Journal of Selected Topics in Signal Processing, 11(8):1310\u20131317, 2017.\n\n[10] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model\nuncertainty in deep learning. In international conference on machine learning, pages 1050\u20131059,\n2016.\n\n[11] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar-\n\nial examples. arXiv preprint arXiv:1412.6572, 2014.\n\n9\n\n\f[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image\nrecognition. In Proceedings of the IEEE conference on computer vision and pattern recognition,\npages 770\u2013778, 2016.\n\n[13] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassi\ufb01ed and out-of-distribution\n\nexamples in neural networks. arXiv preprint arXiv:1610.02136, 2016.\n\n[14] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.\n\narXiv preprint arXiv:1503.02531, 2015.\n\n[15] Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der Maaten. Densely connected\nconvolutional networks. In Proceedings of the IEEE conference on computer vision and pattern\nrecognition, volume 1, page 3, 2017.\n\n[16] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint\n\narXiv:1412.6980, 2014.\n\n[17] Felix Kreuk, Yossi Adi, Moustapha Cisse, and Joseph Keshet. Fooling end-to-end speaker\n\nveri\ufb01cation by adversarial examples. arXiv preprint arXiv:1801.03339, 2018.\n\n[18] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images.\n\n2009.\n\n[19] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable\npredictive uncertainty estimation using deep ensembles. In Advances in Neural Information\nProcessing Systems, pages 6405\u20136416, 2017.\n\n[20] Amos Lapidoth. A foundation in digital communication. Cambridge University Press, 2017.\n\n[21] Claudia Leacock and Martin Chodorow. Combining local context and wordnet similarity for\n\nword sense identi\ufb01cation. WordNet: An electronic lexical database, 49(2):265\u2013283, 1998.\n\n[22] Yann LeCun, L\u00e9on Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning\n\napplied to document recognition. Proceedings of the IEEE, 86(11):2278\u20132324, 1998.\n\n[23] Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training con\ufb01dence-calibrated classi\ufb01ers\n\nfor detecting out-of-distribution samples. arXiv preprint arXiv:1711.09325, 2017.\n\n[24] Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why\nm heads are better than one: Training a diverse ensemble of deep networks. arXiv preprint\narXiv:1511.06314, 2015.\n\n[25] Shiyu Liang, Yixuan Li, and R Srikant. Enhancing the reliability of out-of-distribution image\n\ndetection in neural networks.\n\n[26] Etai Littwin and Lior Wolf. The multiverse loss for robust transfer learning. In Proceedings of\n\nthe IEEE Conference on Computer Vision and Pattern Recognition, pages 3957\u20133966, 2016.\n\n[27] David JC MacKay. Bayesian methods for adaptive models. PhD thesis, California Institute of\n\nTechnology, 1992.\n\n[28] Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin.\nAdvances in pre-training distributed word representations. In Proceedings of the International\nConference on Language Resources and Evaluation (LREC 2018), 2018.\n\n[29] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre-\nsentations of words and phrases and their compositionality. In Advances in neural information\nprocessing systems, pages 3111\u20133119, 2013.\n\n[30] Einat Naaman, Yossi Adi, and Joseph Keshet. Learning similarity function for pronunciation\n\nvariations. In Proc. of Interspeech, 2017.\n\n[31] Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science &\n\nBusiness Media, 2012.\n\n10\n\n\f[32] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng.\nReading digits in natural images with unsupervised feature learning. In NIPS workshop on deep\nlearning and unsupervised feature learning, volume 2011, page 5, 2011.\n\n[33] Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High\ncon\ufb01dence predictions for unrecognizable images. In Proceedings of the IEEE Conference on\nComputer Vision and Pattern Recognition, pages 427\u2013436, 2015.\n\n[34] Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. English gigaword \ufb01fth\n\nedition, linguistic data consortium. Google Scholar, 2011.\n\n[35] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito,\nZeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in\npytorch. In NIPS-W, 2017.\n\n[36] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for\nword representation. In Proceedings of the 2014 conference on empirical methods in natural\nlanguage processing (EMNLP), pages 1532\u20131543, 2014.\n\n[37] Thomas Schlegl, Philipp Seeb\u00f6ck, Sebastian M Waldstein, Ursula Schmidt-Erfurth, and Georg\nLangs. Unsupervised anomaly detection with generative adversarial networks to guide marker\ndiscovery. In International Conference on Information Processing in Medical Imaging, pages\n146\u2013157. Springer, 2017.\n\n[38] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale\n\nimage recognition. arXiv preprint arXiv:1409.1556, 2014.\n\n[39] Yaniv Taigman, Ming Yang, Marc\u2019Aurelio Ranzato, and Lior Wolf. Web-scale training for\nface identi\ufb01cation. In Proceedings of the IEEE conference on computer vision and pattern\nrecognition, pages 2746\u20132754, 2015.\n\n[40] Lucas Theis, A\u00e4ron van den Oord, and Matthias Bethge. A note on the evaluation of generative\n\nmodels. ICLR, 2015.\n\n[41] Zhibiao Wu and Martha Palmer. Verbs semantics and lexical selection. In Proceedings of the\n32nd annual meeting on Association for Computational Linguistics, pages 133\u2013138. Association\nfor Computational Linguistics, 1994.\n\n[42] Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction\nof a large-scale image dataset using deep learning with humans in the loop. arXiv preprint\narXiv:1506.03365, 2015.\n\n[43] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding\n\ndeep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016.\n\n[44] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond\n\nempirical risk minimization. arXiv preprint arXiv:1710.09412, 2017.\n\n11\n\n\f", "award": [], "sourceid": 3678, "authors": [{"given_name": "Gabi", "family_name": "Shalev", "institution": "Dept. of Computer Science, Bar-Ilan University"}, {"given_name": "Yossi", "family_name": "Adi", "institution": "Bar Ilan University"}, {"given_name": "Joseph", "family_name": "Keshet", "institution": "Bar-Ilan University"}]}