{"title": "Altitude Training: Strong Bounds for Single-Layer Dropout", "book": "Advances in Neural Information Processing Systems", "page_first": 100, "page_last": 108, "abstract": "Dropout training, originally designed for deep neural networks, has been successful on high-dimensional single-layer natural language tasks. This paper proposes a theoretical explanation for this phenomenon: we show that, under a generative Poisson topic model with long documents, dropout training improves the exponent in the generalization bound for empirical risk minimization. Dropout achieves this gain much like a marathon runner who practices at altitude: once a classifier learns to perform reasonably well on training examples that have been artificially corrupted by dropout, it will do very well on the uncorrupted test set. We also show that, under similar conditions, dropout preserves the Bayes decision boundary and should therefore induce minimal bias in high dimensions.", "full_text": "Altitude Training:\n\nStrong Bounds for Single-Layer Dropout\n\nStefan Wager\u21e4, William Fithian\u21e4, Sida Wang\u2020, and Percy Liang\u21e4,\u2020\n\nDepartments of Statistics\u21e4 and Computer Science\u2020\nStanford University, Stanford, CA-94305, USA\n\n{swager, wfithian}@stanford.edu, {sidaw, pliang}@cs.stanford.edu\n\nAbstract\n\nDropout training, originally designed for deep neural networks, has been success-\nful on high-dimensional single-layer natural language tasks. This paper proposes\na theoretical explanation for this phenomenon: we show that, under a generative\nPoisson topic model with long documents, dropout training improves the exponent\nin the generalization bound for empirical risk minimization. Dropout achieves this\ngain much like a marathon runner who practices at altitude: once a classi\ufb01er learns\nto perform reasonably well on training examples that have been arti\ufb01cially cor-\nrupted by dropout, it will do very well on the uncorrupted test set. We also show\nthat, under similar conditions, dropout preserves the Bayes decision boundary and\nshould therefore induce minimal bias in high dimensions.\n\n1\n\nIntroduction\n\nDropout training [1] is an increasingly popular method for regularizing learning algorithms. Dropout\nis most commonly used for regularizing deep neural networks [2, 3, 4, 5], but it has also been found\nto improve the performance of logistic regression and other single-layer models for natural language\ntasks such as document classi\ufb01cation and named entity recognition [6, 7, 8]. For single-layer linear\nmodels, learning with dropout is equivalent to using \u201cblankout noise\u201d [9].\nThe goal of this paper is to gain a better theoretical understanding of why dropout regularization\nworks well for natural language tasks. We focus on the task of document classi\ufb01cation using linear\nclassi\ufb01ers where data comes from a generative Poisson topic model. In this setting, dropout effec-\ntively deletes random words from a document during training; this corruption makes the training\nexamples harder. A classi\ufb01er that is able to \ufb01t the training data will therefore receive an accuracy\nboost at test time on the much easier uncorrupted examples. An apt analogy is altitude training,\nwhere athletes practice in more dif\ufb01cult situations than they compete in. Importantly, our analysis\ndoes not rely on dropout merely creating more pseudo-examples for training, but rather on dropout\ncreating more challenging training examples. Somewhat paradoxically, we show that removing in-\nformation from training examples can induce a classi\ufb01er that performs better at test time.\n\nMain Result Consider training the zero-one loss empirical risk minimizer (ERM) using dropout,\nwhere each word is independently removed with probability  2 (0, 1). For a class of Poisson\ngenerative topic models, we show that dropout gives rise to what we call the altitude training phe-\nnomenon: dropout improves the excess risk of the ERM by multiplying the exponent in its decay\nrate by 1/(1  ). This improvement comes at the cost of an additive term of O(1/p), where \nis the average number of words per document. More formally, let h\u21e4 and \u02c6h0 be the expected and\n\nS. Wager and W. Fithian are supported by a B.C. and E.J. Eaves Stanford Graduate Fellowship and NSF\n\nVIGRE grant DMS\u20130502385 respectively.\n\n1\n\n\fempirical risk minimizers, respectively; let h\u21e4 and \u02c6h be the corresponding quantities for dropout\ntraining. Let Err(h) denote the error rate (on test examples) of h. In Section 4, we show that:\n\nErr\u21e3\u02c6h\u2318  Err (h\u21e4)\n|\n}\n\ndropout excess risk\n\n{z\n\n(1)\n\n= eOP0BB@\u21e3Err\u21e3\u02c6h0\u2318  Err (h\u21e4)\u2318\n}\n\nERM excess risk\n\n{z\n\n|\n\n1\n\n1 +\n\n1\np\n\n1CCA ,\n\nwhere eOP is a variant of big-O in probability notation that suppresses logarithmic factors. If  is\n\nlarge (we are classifying long documents rather than short snippets of text), dropout considerably\naccelerates the decay rate of excess risk. The bound (1) holds for \ufb01xed choices of . The constants\nin the bound worsen as  approaches 1, and so we cannot get zero excess risk by sending  to 1.\nOur result is modular in that it converts upper bounds on the ERM excess risk to upper bounds on\nthe dropout excess risk. For example, recall from classic VC theory that the ERM excess risk is\n\nexamples. With dropout  = 0.5, our result (1) directly implies that the dropout excess risk is\n\nThe intuition behind the proof of (1) is as follows: when  = 0.5, we essentially train on half\ndocuments and test on whole documents. By conditional independence properties of the generative\ntopic model, the classi\ufb01cation score is roughly Gaussian under a Berry-Esseen bound, and the error\nrate is governed by the tails of the Gaussian. Compared to half documents, the coef\ufb01cient of variation\n\neOP (pd/n), where d is the number of features (vocabulary size) and n is the number of training\neOP (d/n + 1/p).\nof the classi\ufb01cation score on whole documents (at test time) is scaled down by p1   compared to\nhalf documents (at training time), resulting in an exponential reduction in error. The additive penalty\nof 1/p stems from the Berry-Esseen approximation.\nNote that the bound (1) only controls the dropout excess risk. Even if dropout reduces the excess\nrisk, it may introduce a bias Err(h\u21e4)Err(h\u21e4), and thus (1) is useful only when this bias is small. In\nSection 5, we will show that the optimal Bayes decision boundary is not affected by dropout under\nthe Poisson topic model. Bias is thus negligible when the Bayes boundary is close to linear.\nIt is instructive to compare our generalization bound to that of Ng and Jordan [10], who showed that\nthe naive Bayes classi\ufb01er exploits a strong generative assumption\u2014conditional independence of the\n\nfeatures given the label\u2014to achieve an excess risk of OP (p(log d)/n). However, if the generative\nassumption is incorrect, then naive Bayes can have a large bias. Dropout enables us to cut excess risk\nwithout incurring as much bias. In fact, naive Bayes is closely related to logistic regression trained\nusing an extreme form of dropout with  ! 1. Training logistic regression with dropout rates from\nthe range  2 (0, 1) thus gives a family of classi\ufb01ers between unregularized logistic regression and\nnaive Bayes, allowing us to tune the bias-variance tradeoff.\n\nOther perspectives on dropout\nIn the general setting, dropout only improves generalization by\na multiplicative factor. McAllester [11] used the PAC-Bayes framework to prove a generalization\nbound for dropout that decays as 1  . Moreover, provided that  is not too close to 1, dropout\nbehaves similarly to an adaptive L2 regularizer with parameter /(1) [6, 12], and at least in linear\nregression such L2 regularization improves generalization error by a constant factor. In contrast, by\nleveraging the conditional independence assumptions of the topic model, we are able to improve the\nexponent in the rate of convergence of the empirical risk minimizer.\nIt is also possible to analyze dropout as an adaptive regularizer [6, 9, 13]: in comparison with L2\nregularization, dropout favors the use of rare features and encourages con\ufb01dent predictions. If we\nbelieve that good document classi\ufb01cation should produce con\ufb01dent predictions by understanding\nrare words with Poisson-like occurrence patterns, then the work on dropout as adaptive regulariza-\ntion and our generalization-based analysis are two complementary explanations for the success of\ndropout in natural language tasks.\n\n2 Dropout Training for Topic Models\nIn this section, we introduce binomial dropout, a form of dropout suitable for topic models, and the\nPoisson topic model, on which all our analyses will be based.\n\n2\n\n\fBinomial Dropout Suppose that we have a binary classi\ufb01cation problem1 with count features\nx(i) 2{ 0, 1, 2, . . .}d and labels y(i) 2{ 0, 1}. For example, x(i)\nis the number of times the j-th\nword in our dictionary appears in the i-th document, and y(i) is the label of the document. Our goal\nis to train a weight vector bw that classi\ufb01es new examples with features x via a linear decision rule\n\u02c6y = I{bw \u00b7 x > 0}. We start with the usual empirical risk minimizer:\n`\u21e3w; x(i), y(i)\u2318)\n\ndef= argminw2Rd( nXi=1\n\nfor some loss function ` (we will analyze the zero-one loss but use logistic loss in experiments [e.g.,\n10, 14, 15]). Binomial dropout trains on perturbed features \u02dcx(i) instead of the original features x(i):\n\n(2)\n\nj\n\nbw0\ndef= argminw( nXi=1\n\nbw\n\nEh`\u21e3w; \u02dcx(i), y(i)\u2318i) , where \u02dcx(i)\n\nj = Binom\u21e3x(i)\n\nj ; 1  \u2318 .\n\n(3)\n\nIn other words, during training, we randomly thin the j-th feature xj with binomial noise. If xj\ncounts the number of times the j-th word appears in the document, then replacing xj with \u02dcxj is\nequivalent to independently deleting each occurrence of word j with probability . Because we\nare only interested in the decision boundary, we do not scale down the weight vector obtained by\ndropout by a factor 1   as is often done [e.g., 1].\nBinomial dropout differs slightly from the usual de\ufb01nition of (blankout) dropout, which alters the\nfeature vector x by setting random coordinates to 0 [6, 9, 11, 12]. The reason we chose to study\nbinomial rather than blankout dropout is that Poisson random variables remain Poisson even after\nbinomial thinning; this fact lets us streamline our analysis. For rare words that appear once in the\ndocument, the two types of dropout are equivalent.\n\nj\n\nj according to x(i)\n\nA Generative Poisson Topic Model Throughout our analysis, we assume that the data is drawn\nfrom a Poisson topic model depicted in Figure 1a and de\ufb01ned as follows. Each document i is as-\nsigned a label y(i) according to some Bernoulli distribution. Then, given the label y(i), the document\ngets a topic \u2327 (i) 2 \u21e5 from a distribution \u21e2y(i). Given the topic \u2327 (i), for every word j in the vocabu-\nlary, we generate its frequency x(i)\nj 2 [0,1)\nis the expected number of times word j appears under topic \u2327. Note that k(\u2327 )k1 is the average\nlength of a document with topic \u2327. De\ufb01ne  def= min\u23272\u21e5 k(\u2327 )k1 to be the shortest average doc-\nument length across topics. If \u21e5 contains only two topics\u2014one for each class\u2014we get the naive\nBayes model. If \u21e5 is the (K  1)-dimensional simplex where (\u2327 ) is a \u2327-mixture over K basis\nvectors, we get the K-topic latent Dirichlet allocation [16].2\nNote that although our generalization result relies on a generative model, the actual learning algo-\nrithm is agnostic to it. Our analysis shows that dropout can take advantage of a generative structure\nwhile remaining a discriminative procedure. If we believed that a certain topic model held exactly\nand we knew the number of topics, we could try to \ufb01t the full generative model by EM. This,\nhowever, could make us vulnerable to model misspeci\ufb01cation. In contrast, dropout bene\ufb01ts from\ngenerative assumptions while remaining more robust to misspeci\ufb01cation.\n\n \u2327 (i) \u21e0 Poisson((\u2327 (i))\n\n), where (\u2327 )\n\nj\n\n3 Altitude Training: Linking the Dropout and Data-Generating Measures\n\nOur goal is to understand the behavior of a classi\ufb01er \u02c6h trained using dropout. During dropout,\nthe error of any classi\ufb01er h is characterized by two measures. In the end, we are interested in the\nusual generalization error (expected risk) of h where x is drawn from the underlying data-generating\nmeasure:\n\nErr (h) def= P [y 6= h(x)] .\n\n(4)\n\n1Dropout training is known to work well in practice for multi-class problems [8]. For simplicity, however,\n\nwe will restrict our theoretical analysis to a two-class setup.\n\n2 In topic modeling, the vertices of the simplex \u21e5 are \u201ctopics\u201d and \u2327 is a mixture of topics, whereas we call\n\n\u2327 itself a topic.\n\n3\n\n\fHowever, since dropout training works on the corrupted data \u02dcx (see (3)), in the limit of in\ufb01nite data,\nthe dropout estimator will converge to the minimizer of the generalization error with respect to the\ndropout measure over \u02dcx:\n\nErr (h) def= P [y 6= h(\u02dcx)] .\n\n(5)\n\nThe main dif\ufb01culty in analyzing the generalization of dropout is that classical theory tells us that\nthe generalization error with respect to the dropout measure will decrease as n ! 1, but we are\ninterested in the original measure. Thus, we need to bound Err in terms of Err. In this section, we\nshow that the error on the original measure is actually much smaller than the error on the dropout\nmeasure; we call this the altitude training phenomenon.\nUnder our generative model, the count features xj are conditionally independent given the topic\n\u2327. We thus focus on a single \ufb01xed topic \u2327 and establish the following theorem, which provides a\nper-topic analogue of (1). Section 4 will then use this theorem to obtain our main result.\nTheorem 1. Let h be a binary linear classi\ufb01er with weights w, and suppose that our features are\ndrawn from the Poisson generative model given topic \u2327. Let c\u2327 be the more likely label given \u2327:\n\nLet \u02dc\"\u2327 be the sub-optimal prediction rate in the dropout measure\n\nwhere \u02dcx(i) is an example thinned by binomial dropout (3), and P is taken over the data-generating\nprocess. Let \"\u2327 be the sub-optimal prediction rate in the original measure\n\n(6)\n\n(7)\n\n(8)\n\n(9)\n\nc\u2327\n\n\u02dc\"\u2327\n\ndef= arg max\nc2{0,1}\n\nPhy(i) = c \u2327 (i) = \u2327i .\ndef= PhInw \u00b7 \u02dcx(i) > 0o 6= c\u2327 \u2327 (i) = \u2327i ,\ndef= PhInw \u00b7 x(i) > 0o 6= c\u2327 \u2327 (i) = \u2327i .\n\"\u2327 = eO\u2713\u02dc\"\n\n\u2327 +p \u2327\u25c6 ,\n\n1\n1\n\n\"\u2327\n\nThen:\n\nwhere \u2327 = maxjw2\n\nj /Pd\n\nj=1 (\u2327 )\n\nj w2\n\nj , and the constants in the bound depend only on .\n\nTheorem 1 only provides us with a useful bound when the term \u2327 is small. Whenever the largest\nj , then p \u2327 scales as O(1/p), where  is the average\nw2\ndocument length. Thus, the bound (9) is most useful for long documents.\n\nj is not much larger than the average w2\n\nj=1 (\u2327 )\n\nj=1 (\u2327 )\n\nj w2\n\nj wj and 2\n\nA Heuristic Proof of Theorem 1. The proof of Theorem 1 is provided in the technical appendix.\nHere, we provide a heuristic argument for intuition. Given a \ufb01xed topic \u2327, suppose that it is optimal\n\nto predict c\u2327 = 1, so our test error is \"\u2327 = P\u21e5w \u00b7 x \uf8ff 0 \u2327\u21e4 . For long enough documents, by\nthe central limit theorem, the score s def= w \u00b7 x will be roughly Gaussian s \u21e0N \u00b5\u2327 , 2\n\u2327 , where\n\u00b5\u2327 =Pd\nj . This implies that \"\u2327 \u21e1 (\u00b5\u2327 /\u2327 ) , where  is the\ncumulative distribution function of the Gaussian. Now, let \u02dcs def= w \u00b7 \u02dcx be the score on a dropout\nsample. Clearly, E [\u02dcs] = (1  ) \u00b5\u2327 and Var [\u02dcs] = (1  ) 2\n\u2327 , because the variance of a Poisson\nrandom variable scales with its mean. Thus,\n\u2327\u25c6(1)\n\n\u02dc\"\u2327 \u21e1 \u2713p1  \n\n\u2327\u25c6 \u21e1 \u2713\n\n\u2327 =Pd\n\nFigure 1b illustrates the relationship between the two Gaussians. This explains the \ufb01rst term on the\nright-hand side of (9). The extra error term p \u2327 arises from a Berry-Esseen bound that approxi-\nmates Poisson mixtures by Gaussian random variables.\n\n\u21e1 \"(1)\n\n(10)\n\n\u00b5\u2327\n\n\u2327\n\n.\n\n\u00b5\u2327\n\n4 A Generalization Bound for Dropout\nBy setting up a bridge between the dropout measure and the original data-generating measure, The-\norem 1 provides a foundation for our analysis. It remains to translate this result into a statement\nabout the generalization error of dropout. For this, we need to make a few assumptions.\n\n4\n\n\f\u21e2\n\ny\n\n\n\n\u2327\n\nOriginal\nDropout\n\n5\n\n.\n\n0\n\n4\n\n.\n\n0\n\n.\n\n0\n\n3\n\ny\nt\ni\ns\nn\ne\nd\n\n2\n\n.\n\n0\n\n1\n\n.\n\n0\n\n0\n\n.\n\n0\n\nx\n\nJ\n\nI\n\n\u22122\n\n\u22121\n\n0\n\n1\n\n2\n\n3\n\nscore\n\n(a) Graphical representation of the Poisson topic model:\nGiven a document with label y, we draw a document\ntopic \u2327 from the multinomial distribution with proba-\nbilities \u21e2y. Then, we draw the words x from the topic\u2019s\nPoisson distribution with mean (\u2327 ). Boxes indicate re-\npeated observations, and greyed-out nodes are observed\nduring training.\n\n(b) For a \ufb01xed classi\ufb01er w, the probabilities of er-\nror on an example drawn from the original and\ndropout measures are governed by the tails of two\nGaussians (shaded). The dropout Gaussian has a\nlarger coef\ufb01cient of variation, which means the er-\nror on the original measure (test) is much smaller\nthan the error on the dropout measure (train) (10).\nIn this example, \u00b5 = 2.5,  = 1, and  = 0.5.\n\nFigure 1: (a) Graphical model. (b) The altitude training phenomenon.\n\nOur \ufb01rst assumption is fundamental: if the classi\ufb01cation signal is concentrated among just a few\nfeatures, then we cannot expect dropout training to do well. The second and third assumptions,\nwhich are more technical, guarantee that a classi\ufb01er can only do well overall if it does well on every\ntopic; this lets us apply Theorem 1. A more general analysis that relaxes Assumptions 2 and 3 may\nbe an interesting avenue for future work.\n\nAssumption 1: well-balanced weights First, we need to assume that all the signal is not concen-\ntrated in a few features. To make this intuition formal, we say a linear classi\ufb01er with weights w is\nwell-balanced if the following holds for each topic \u2327:\n\nj=1 (\u2327 )\n\nj\n\nj Pd\n\nj=1 (\u2327 )\n\nmaxjw2\nPd\n\nj w2\nj\n\n\uf8ff \uf8ff for some 0 <\uf8ff< 1.\n\n(11)\n\nFor example, suppose each word was either useful (|wj| = 1) or not (wj = 0); then \uf8ff is the inverse\nexpected fraction of words in a document that are useful. In Theorem 2 we restrict the ERM to\nwell-balanced classi\ufb01ers and assume that the expected risk minimizer h\u21e4 over all linear rules is also\nwell-balanced.\n\nAssumption 2: discrete topics Second, we assume that there are a \ufb01nite number T of topics, and\nthat the available topics are not too rare or ambiguous: the minimal probability of observing any\ntopic \u2327 is bounded below by\n\nand that each topic-conditional probability is bounded away from 1\n\n2 (random guessing):\n\nP [\u2327 ]  pmin > 0,\n\nPhy(i) = c \u2327 (i) = \u2327i \n\n1\n\n2  \u21b5> 0\n\nfor all topics \u2327 2{ 1, ..., T}. This assumption substantially simpli\ufb01es our arguments, allowing us\nto apply Theorem 1 to each topic separately without technical overhead.\n\nAssumption 3: distinct topics Finally, as an extension of Assumption 2, we require that the topics\nbe \u201cwell separated.\u201d First, de\ufb01ne Errmin = P[y(i) 6= c\u2327 (i)], where c\u2327 is the most likely label given\ntopic \u2327 (6); this is the error rate of the optimal decision rule that sees topic \u2327. We assume that the\nbest linear rule h\u21e4 satisfying (11) is almost as good as always guessing the best label c\u2327 under the\ndropout measure:\n\n(12)\n\n(13)\n\n(14)\n\nErr (h\u21e4) = Errmin +O\u2713 1\np\u25c6 ,\n\n5\n\n\fwhere, as usual,  is a lower bound on the average document length. If the dimension d is larger\nthan the number of topics T , this assumption is fairly weak: the condition (14) holds whenever the\nmatrix \u21e7 of topic centers has full rank, and the minimum singular value of \u21e7 is not too small (see\nProposition 6 in the Appendix for details). This assumption is satis\ufb01ed if the different topics can be\nseparated from each other with a large margin.\nUnder Assumptions 1\u20133 we can turn Theorem 1 into a statement about generalization error.\nTheorem 2. Suppose that our features x are drawn from the Poisson generative model (Figure 1a),\nand Assumptions 1\u20133 hold. De\ufb01ne the excess risks of the dropout classi\ufb01er \u02c6h on the dropout and\ndata-generating measures, respectively:\n\n\u02dc\u2318 def= Err\u21e3\u02c6h\u2318  Err (h\u21e4) and \u2318 def= Err\u21e3\u02c6h\u2318  Err (h\u21e4) .\n\nThen, the altitude training phenomenon applies:\n\n\u2318 = eO\u2713\u02dc\u2318\n\n1\n\n1 +\n\n1\n\np\u25c6 .\n\nThe above bound scales linearly in p1\n\nmin and \u21b51; the full dependence on  is shown in the appendix.\nIn a sense, Theorem 2 is a meta-generalization bound that allows us to transform generalization\nbounds with respect to the dropout measure (\u02dc\u2318) into ones on the data-generating measure (\u2318) in a\n\nmodular way. As a simple example, standard VC theory provides an \u02dc\u2318 = eOP (pd/n) bound which,\n\ntogether with Theorem 2, yields:\nCorollary 3. Under the same conditions as Theorem 2, the dropout classi\ufb01er \u02c6h achieves the fol-\nlowing excess risk:\n\nErr\u21e3\u02c6h\u2318  Err (h\u21e4) = eOP0@ r d\nn! 1\n\n1\n\n+\n\n1\n\np1A .\n\n(15)\n\n(16)\n\n(17)\n\nMore generally, we can often check that upper bounds for Err(\u02c6h)  Err(h\u21e4) also work as upper\nbounds for Err(\u02c6h)  Err(h\u21e4); this gives us the heuristic result from (1).\n5 The Bias of Dropout\nIn the previous section, we showed that under the Poisson topic model in Figure 1a, dropout can\nachieve a substantial cut in excess risk Err(\u02c6h) Err(h\u21e4). But to complete our picture of dropout\u2019s\nperformance, we must address the bias of dropout: Err(h\u21e4)  Err(h\u21e4).\nDropout can be viewed as importing \u201chints\u201d from a generative assumption about the data. Each ob-\nserved (x, y) pair (each labeled document) gives us information not only about the conditional class\nprobability at x, but also about the conditional class probabilities at numerous other hypothetical\nvalues \u02dcx representing shorter documents of the same class that did not occur. Intuitively, if these \u02dcx\nare actually good representatives of that class, the bias of dropout should be mild.\nFor our key result in this section, we will take the Poisson generative model from Figure 1a, but\nfurther assume that document length is independent of the topic. Under this assumption, we will\nshow that dropout preserves the Bayes decision boundary in the following sense:\nProposition 4. Let (x, y) be distributed according to the Poisson topic model of Figure 1a. Assume\nthat document length is independent of topic: k(\u2327 )k1 =  for all topics \u2327. Let \u02dcx be a binomial\ndropout sample of x with some dropout probability  2 (0, 1). Then, for every feature vector v 2 Rd,\nwe have:\n(18)\n\nP\u21e5y = 1 \u02dcx = v\u21e4 = P\u21e5y = 1 x = v\u21e4 .\n\nIf we had an in\ufb01nite amount of data (\u02dcx, y) corrupted under dropout, we would predict according to\n2}. The signi\ufb01cance of Proposition 4 is that this decision rule is identical to\nthe true Bayes decision boundary (without dropout). Therefore, the empirical risk minimizer of a\nsuf\ufb01ciently rich hypothesis class trained with dropout would incur very small bias.\n\nI{P\u21e5y = 1 \u02dcx = v\u21e4 > 1\n\n6\n\n\f0\n0\n3\n\n0\n0\n2\n\n2\nX\n\n0\n0\n1\n\n0\n\nLogReg Boundary\nDropout Boundary\nBayes Boundary\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf \u25cf\u25cf\n\u25cf\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf \u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf \u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf \u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf \u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf \u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\u25cf\u25cf\n\u25cf \u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf \u25cf\u25cf\u25cf\n\u25cf\u25cf\n\u25cf \u25cf\n\u25cf \u25cf\u25cf\n\u25cf \u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf \u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf \u25cf\n\u25cf \u25cf\u25cf\u25cf\n\u25cf\u25cf\n\u25cf \u25cf\u25cf\n\u25cf \u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf \u25cf\n\u25cf\u25cf\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf \u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf \u25cf\n\u25cf\u25cf \u25cf\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf \u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf \u25cf\n\u25cf \u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\u25cf \u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf \u25cf\u25cf\n\u25cf\n\u25cf\u25cf \u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf \u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf \u25cf\n\u25cf\u25cf\n\u25cf\u25cf\u25cf \u25cf\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf \u25cf\u25cf\u25cf\u25cf \u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf \u25cf\u25cf\n\u25cf\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf \u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf \u25cf\n\u25cf\u25cf\n\u25cf \u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf \u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf \u25cf\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf \u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf \u25cf\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf \u25cf\n\u25cf \u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf \u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf \u25cf\n\u25cf\u25cf\u25cf\n\u25cf \u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\u25cf\n\u25cf\n\u25cf\u25cf \u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf \u25cf\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf \u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf \u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf \u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf \u25cf\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf \u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf \u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf \u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf \u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf \u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\u25cf \u25cf\n\u25cf \u25cf \u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf \u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf \u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf \u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\u25cf\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf \u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf \u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf \u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf \u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\u25cf\n\n\u25cf\n\no Long Documents\n\nShort (Dropout) Documents\n\n0\n\n100\n\n200\n\n300\n\n400\n\n500\n\nX1\n\n 0 \n\n)\n\n%\n\n(\n \ne\nt\na\nR\n\n \nr\no\nr\nr\n\nE\n\n \nt\ns\ne\nT\n\n0\n2\n\n0\n1\n\n5\n\n2\n\n1\n\n5\n.\n0\n\nLR\n0.25\n0.5\n0.75\n0.9\n0.95\n0.99\nNB\n\n50\n\n100\n\n200\n\n500\n\n1000 2000\n\nn\n\n(a) Dropout ( = 0.75) with d = 2. For long doc-\numents (circles in the upper-right), logistic regres-\nsion focuses on capturing the small red cluster; the\nlarge red cluster has almost no in\ufb02uence. Dropout\n(dots in the lower-left) distributes in\ufb02uence more\nequally between the two red clusters.\n\n(b) Learning curves for the synthetic experiment.\nEach axis is plotted on a log scale. Here the dropout\nrate  ranges from 0 (logistic regression) to 1 (naive\nBayes) for multiple values of training set sizes n.\nAs n increases, less dropout is preferable, as the\nbias-variance tradeoff shifts.\n\nFigure 2: Behavior of binomial dropout in simulations. In the left panel, the circles are the original\ndata, while the dots are dropout-thinned examples. The Monte Carlo error is negligible.\n\nHowever, Proposition 4 does not guarantee that dropout incurs no bias when we \ufb01t a linear classi\ufb01er.\nIn general, the best linear approximation for classifying shorter documents is not necessarily the\nbest for classifying longer documents. As n ! 1, a linear classi\ufb01er trained on (x, y) pairs will\neventually outperform one trained on (\u02dcx, y) pairs.\n\nDropout for Logistic Regression To gain some more intuition about how dropout affects linear\nclassi\ufb01ers, we consider logistic regression. A similar phenomenon should also hold for the ERM,\nbut discussing this solution is more dif\ufb01cult since the ERM solution does not have have a simple\ncharacterization. The relationship between the 0-1 loss and convex surrogates has been studied\n\ni=1y(i)  \u02c6pi x(i), where\nby, e.g., [14, 15]. The score criterion for logistic regression is 0 = Pn\n\u02c6pi = (1 + ebw\u00b7x(i))1 are the \ufb01tted probabilities. Note that easily-classi\ufb01ed examples (where \u02c6pi is\n\nclose to y(i)) play almost no role in driving the \ufb01t. Dropout turns easy examples into hard examples,\ngiving more examples a chance to participate in learning a good classi\ufb01cation rule.\nFigure 2a illustrates dropout\u2019s tendency to spread in\ufb02uence more democratically for a simple classi-\n\ufb01cation problem with d = 2. The red class is a 99:1 mixture over two topics, one of which is much\nless common, but harder to classify, than the other. There is only one topic for the blue class. For\nlong documents (open circles in the top right), the infrequent, hard-to-classify red cluster dominates\nthe \ufb01t while the frequent, easy-to-classify red cluster is essentially ignored. For dropout documents\nwith  = 0.75 (small dots, lower left), both red clusters are relatively hard to classify, so the infre-\nquent one plays a less disproportionate role in driving the \ufb01t. As a result, the \ufb01t based on dropout is\nmore stable but misses the \ufb01ner structure near the decision boundary. Note that the solid gray curve,\nthe Bayes boundary, is unaffected by dropout, per Proposition 4. But, because it is nonlinear, we\nobtain a different linear approximation under dropout.\n\n6 Experiments and Discussion\nSynthetic Experiment Consider the following instance of the Poisson topic model: We choose\nthe document label uniformly at random: P\u21e5y(i) = 1\u21e4 = 1\n2. Given label 0, we choose topic \u2327 (i) = 0\ndeterministically; given label 1, we choose a real-valued topic \u2327 (i) \u21e0 Exp(3). The per-topic Poisson\nintensities (\u2327 ) are de\ufb01ned as follows:\n\u2713(\u2327 ) =8<:\n\n(1, . . . , 1  0, . . . , 0  0, . . . , 0)\n 0, . . . , 0\n| {z }\n| {z }\n\nThe \ufb01rst block of 7 independent words are indicative of label 0, the second block of 7 correlated\nwords are indicative of label 1, and the remaining 486 words are indicative of neither.\n\n \u2327, . . . , \u2327\n|\n{z\n}\n\n(\u2327 )\nj = 1000 \u00b7\n\nj\n\ne\u2713(\u2327 )\nj0=1 e\u2713(\u2327 )\n\nj0\n\nif \u2327 = 0,\notherwise,\n\nP500\n\n(0, . . . , 0\n\n486\n\n.\n\n(19)\n\n7\n\n7\n\n)\n\n7\n\n\fe\nt\na\nr\n \nr\no\nr\nr\n\nE\n\n0.6\n\n0.55\n\n0.5\n\n0.45\n\n0.4\n\n0.35\n\n0.3\n\n0.25\n\n0.2\n\n0.15\n\n0.1\n\n \n0\n\n \n\nLog.Reg.\nNaive Bayes\nDropout\u22120.8\nDropout\u22120.5\nDropout\u22120.2\n\n \n\nLog.Reg.\nNaive Bayes\nDropout\u22120.8\nDropout\u22120.5\nDropout\u22120.2\n\n0.26\n\n0.24\n\n0.22\n\n0.2\n\n0.18\n\n0.16\n\n0.14\n\n0.12\n\ne\nt\na\nr\n \nr\no\nr\nr\n\nE\n\n0.1\n\n0.2\n\n0.3\n\n0.4\n\n0.5\n\n0.6\n\n0.7\n\n0.8\n\nFraction of data used\n\n0.1\n\n \n0\n\n0.2\n\n0.6\n0.4\nFraction of data used\n\n0.8\n\n1\n\n(a) Polarity 2.0 dataset [17].\n\n(b) IMDB dataset [18].\n\nFigure 3: Experiments on sentiment classi\ufb01cation. More dropout is better relative to logistic regres-\nsion for small datasets and gradually worsens with more training data.\n\nWe train a model on training sets of various size n, and evaluate the resulting classi\ufb01ers\u2019 error rates\non a large test set. For dropout, we recalibrate the intercept on the training set. Figure 2b shows\nthe results. There is a clear bias-variance tradeoff, with logistic regression ( = 0) and naive Bayes\n( = 1) on the two ends of the spectrum. For moderate values of n, dropout improves performance,\nwith  = 0.95 (resulting in roughly 50-word documents) appearing nearly optimal for this example.\n\nSentiment Classi\ufb01cation We also examined the performance of dropout as a function of training\nset size on a document classi\ufb01cation task. Figure 3a shows results on the Polarity 2.0 task [17],\nwhere the goal is to classify positive versus negative movie reviews on IMDB. We divided the\ndataset into a training set of size 1,200 and a test set of size 800, and trained a bag-of-words logistic\nregression model with 50,922 features. This example exhibits the same behavior as our simulation.\nUsing a larger  results in a classi\ufb01er that converges faster at \ufb01rst, but then plateaus. We also\nran experiments on a larger IMDB dataset [18] with training and test sets of size 25,000 each and\napproximately 300,000 features. As Figure 3b shows, the results are similar, although the training\nset is not large enough for the learning curves to cross. When using the full training set, all but three\npairwise comparisons in Figure 3 are statistically signi\ufb01cant (p < 0.05 for McNemar\u2019s test).\n\nDropout and Generative Modeling Naive Bayes and empirical risk minimization represent two\ndivergent approaches to the classi\ufb01cation problem. ERM is guaranteed to \ufb01nd the best model as n !\n1 but can have suboptimal generalization error when n is not large relative to d. Conversely, naive\nBayes has very low generalization error, but suffers from asymptotic bias. In this paper, we showed\nthat dropout behaves as a link between ERM and naive Bayes, and can sometimes achieve a more\nfavorable bias-variance tradeoff. By training on randomly generated sub-documents rather than on\nwhole documents, dropout implicitly codi\ufb01es a generative assumption about the data, namely that\nexcerpts from a long document should have the same label as the original document (Proposition 4).\nLogistic regression with dropout appears to have an intriguing connection to the naive Bayes SVM\n[NBSVM, 19], which is a way of using naive Bayes generative assumptions to strengthen an SVM.\nIn a recent survey of bag-of-words classi\ufb01ers for document classi\ufb01cation, NBSVM and dropout often\nobtain state-of-the-art accuracies [e.g., 7]. This suggests that a good way to learn linear models for\ndocument classi\ufb01cation is to use discriminative models that borrow strength from an approximate\ngenerative assumption to cut their generalization error. Our analysis presents an interesting contrast\nto other work that directly combine generative and discriminative modeling by optimizing a hybrid\nlikelihood [20, 21, 22, 23, 24, 25]. Our approach is more guarded in that we only let the generative\nassumption speak through pseudo-examples.\n\nConclusion We have presented a theoretical analysis that explains how dropout training can be\nvery helpful under a Poisson topic model assumption. Speci\ufb01cally, by making training examples\narti\ufb01cially dif\ufb01cult, dropout improves the exponent in the generalization bound for ERM. We believe\nthat this work is just the \ufb01rst step in understanding the bene\ufb01ts of training with arti\ufb01cially corrupted\nfeatures, and we hope the tools we have developed can be extended to analyze other training schemes\nunder weaker data-generating assumptions.\n\n8\n\n\fReferences\n[1] Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov.\n\nImproving neural networks by preventing co-adaptation of feature detectors. arXiv:1207.0580, 2012.\n\n[2] Jimmy Ba and Brendan Frey. Adaptive dropout for training deep neural networks. In Advances in Neural\n\nInformation Processing Systems, 2013.\n\n[3] Ian J Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout\n\nnetworks. In Proceedings of the International Conference on Machine Learning, 2013.\n\n[4] Alex Krizhevsky, Ilya Sutskever, and Geoff Hinton.\n\nImagenet classi\ufb01cation with deep convolutional\n\nneural networks. In Advances in Neural Information Processing Systems, 2012.\n\n[5] Li Wan, Matthew Zeiler, Sixin Zhang, Yann L Cun, and Rob Fergus. Regularization of neural networks\n\nusing dropconnect. In Proceedings of the International Conference on Machine Learning, 2013.\n\n[6] Stefan Wager, Sida Wang, and Percy Liang. Dropout training as adaptive regularization. In Advances in\n\nNeural Information Processing Systems, 2013.\n\n[7] Sida I Wang and Christopher D Manning. Fast dropout training.\n\nConference on Machine Learning, 2013.\n\nIn Proceedings of the International\n\n[8] Sida I Wang, Mengqiu Wang, Stefan Wager, Percy Liang, and Christopher D Manning. Feature noising\n\nfor log-linear structured prediction. In Empirical Methods in Natural Language Processing, 2013.\n\n[9] Laurens van der Maaten, Minmin Chen, Stephen Tyree, and Kilian Q Weinberger. Learning with\n\nmarginalized corrupted features. In International Conference on Machine Learning, 2013.\n\n[10] Andrew Ng and Michael Jordan. On discriminative vs. generative classi\ufb01ers: A comparison of logistic\n\nregression and naive Bayes. Advances in Neural Information Processing Systems, 14, 2001.\n[11] David McAllester. A PAC-Bayesian tutorial with a dropout bound. arXiv:1307.2118, 2013.\n[12] Pierre Baldi and Peter Sadowski. The dropout learning algorithm. Arti\ufb01cial Intelligence, 210:78\u2013122,\n\n2014.\n\n[13] Amir Globerson and Sam Roweis. Nightmare at test time: robust learning by feature deletion. In Pro-\n\nceedings of the International Conference on Machine Learning, 2006.\n\n[14] Peter L Bartlett, Michael I Jordan, and Jon D McAuliffe. Convexity, classi\ufb01cation, and risk bounds.\n\nJournal of the American Statistical Association, 101(473):138\u2013156, 2006.\n\n[15] Tong Zhang. Statistical behavior and consistency of classi\ufb01cation methods based on convex risk mini-\n\nmization. Annals of Statistics, 32(1):56\u201385, 2004.\n\n[16] David M Blei, Andrew Y Ng, and Michael I Jordan. Latent Dirichlet allocation. Journal of Machine\n\nLearning Research, 3:993\u20131022, 2003.\n\n[17] Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summarization\n\nbased on minimum cuts. In Proceedings of the Association for Computational Linguistics, 2004.\n\n[18] Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts.\nLearning word vectors for sentiment analysis. In Proceedings of the Association for Computational Lin-\nguistics, 2011.\n\n[19] Sida Wang and Christopher D Manning. Baselines and bigrams: Simple, good sentiment and topic clas-\n\nsi\ufb01cation. In Proceedings of the Association for Computational Linguistics, 2012.\n\n[20] R. Raina, Y. Shen, A. Ng, and A. McCallum. Classi\ufb01cation with hybrid generative/discriminative models.\n\nIn Advances in Neural Information Processing Systems, 2004.\n\n[21] G. Bouchard and B. Triggs. The trade-off between generative and discriminative classi\ufb01ers. In Interna-\n\ntional Conference on Computational Statistics, 2004.\n\n[22] J. A. Lasserre, C. M. Bishop, and T. P. Minka. Principled hybrids of generative and discriminative models.\n\nIn Computer Vision and Pattern Recognition, 2006.\n\n[23] Guillaume Bouchard. Bias-variance tradeoff in hybrid generative-discriminative models. In International\n\nConference on Machine Learning and Applications. IEEE, 2007.\n\n[24] A. McCallum, C. Pal, G. Druck, and X. Wang. Multi-conditional learning: Generative/discriminative\ntraining for clustering and classi\ufb01cation. In Association for the Advancement of Arti\ufb01cial Intelligence,\n2006.\n\n[25] Percy Liang and Michael I Jordan. An asymptotic analysis of generative, discriminative, and pseudolike-\n\nlihood estimators. In Proceedings of the International Conference on Machine Learning, 2008.\n\n[26] Willliam Feller. An introduction to probability theory and its applications, volume 2. John Wiley & Sons,\n\n1971.\n\n[27] Olivier Bousquet, St\u00b4ephane Boucheron, and G\u00b4abor Lugosi. Introduction to statistical learning theory. In\n\nAdvanced Lectures on Machine Learning, pages 169\u2013207. Springer, 2004.\n\n9\n\n\f", "award": [], "sourceid": 94, "authors": [{"given_name": "Stefan", "family_name": "Wager", "institution": "Stanford University"}, {"given_name": "William", "family_name": "Fithian", "institution": "Stanford University"}, {"given_name": "Sida", "family_name": "Wang", "institution": "Stanford University"}, {"given_name": "Percy", "family_name": "Liang", "institution": "Stanford University"}]}