{"title": "Robustness of classifiers: from adversarial to random noise", "book": "Advances in Neural Information Processing Systems", "page_first": 1632, "page_last": 1640, "abstract": "Several recent works have shown that state-of-the-art classifiers are vulnerable to worst-case (i.e., adversarial) perturbations of the datapoints. On the other hand, it has been empirically observed that these same classifiers are relatively robust to random noise. In this paper, we propose to study a semi-random noise regime that generalizes both the random and worst-case noise regimes. We propose the first quantitative analysis of the robustness of nonlinear classifiers in this general noise regime. We establish precise theoretical bounds on the robustness of classifiers in this general regime, which depend on the curvature of the classifier's decision boundary. Our bounds confirm and quantify the empirical observations that classifiers satisfying curvature constraints are robust to random noise. Moreover, we quantify the robustness of classifiers in terms of the subspace dimension in the semi-random noise regime, and show that our bounds remarkably interpolate between the worst-case and random noise regimes. We perform experiments and show that the derived bounds provide very accurate estimates when applied to various state-of-the-art deep neural networks and datasets. This result suggests bounds on the curvature of the classifiers' decision boundaries that we support experimentally, and more generally offers important insights onto the geometry of high dimensional classification problems.", "full_text": "Robustness of classi\ufb01ers:\n\nfrom adversarial to random noise\n\nAlhussein Fawzi\u2217, Seyed-Mohsen Moosavi-Dezfooli\u2217, Pascal Frossard\n\n\u00c9cole Polytechnique F\u00e9d\u00e9rale de Lausanne\n\nLausanne, Switzerland\n\n{alhussein.fawzi, seyed.moosavi, pascal.frossard} at epfl.ch\n\nAbstract\n\nSeveral recent works have shown that state-of-the-art classi\ufb01ers are vulnerable to\nworst-case (i.e., adversarial) perturbations of the datapoints. On the other hand,\nit has been empirically observed that these same classi\ufb01ers are relatively robust\nto random noise. In this paper, we propose to study a semi-random noise regime\nthat generalizes both the random and worst-case noise regimes. We propose\nthe \ufb01rst quantitative analysis of the robustness of nonlinear classi\ufb01ers in this\ngeneral noise regime. We establish precise theoretical bounds on the robustness of\nclassi\ufb01ers in this general regime, which depend on the curvature of the classi\ufb01er\u2019s\ndecision boundary. Our bounds con\ufb01rm and quantify the empirical observations that\nclassi\ufb01ers satisfying curvature constraints are robust to random noise. Moreover,\nwe quantify the robustness of classi\ufb01ers in terms of the subspace dimension in\nthe semi-random noise regime, and show that our bounds remarkably interpolate\nbetween the worst-case and random noise regimes. We perform experiments and\nshow that the derived bounds provide very accurate estimates when applied to\nvarious state-of-the-art deep neural networks and datasets. This result suggests\nbounds on the curvature of the classi\ufb01ers\u2019 decision boundaries that we support\nexperimentally, and more generally offers important insights onto the geometry of\nhigh dimensional classi\ufb01cation problems.\n\n1\n\nIntroduction\n\nState-of-the-art classi\ufb01ers, especially deep networks, have shown impressive classi\ufb01cation perfor-\nmance on many challenging benchmarks in visual tasks [9] and speech processing [7]. An equally\nimportant property of a classi\ufb01er that is often overlooked is its robustness in noisy regimes, when\ndata samples are perturbed by noise. The robustness of a classi\ufb01er is especially fundamental when\nit is deployed in real-world, uncontrolled, and possibly hostile environments. In these cases, it\nis crucial that classi\ufb01ers exhibit good robustness properties. In other words, a suf\ufb01ciently small\nperturbation of a datapoint should ideally not result in altering the estimated label of a classi\ufb01er.\nState-of-the-art deep neural networks have recently been shown to be very unstable to worst-case\nperturbations of the data (or equivalently, adversarial perturbations) [17]. In particular, despite\nthe excellent classi\ufb01cation performances of these classi\ufb01ers, well-sought perturbations of the data\ncan easily cause misclassi\ufb01cation, since data points often lie very close to the decision boundary\nof the classi\ufb01er. Despite the importance of this result, the worst-case noise regime that is studied\nin [17] only represents a very speci\ufb01c type of noise. It furthermore requires the full knowledge of the\nclassi\ufb01cation model, which may be a hard assumption in practice.\nIn this paper, we precisely quantify the robustness of nonlinear classi\ufb01ers in two practical noise\nregimes, namely random and semi-random noise regimes. In the random noise regime, datapoints are\n\n\u2217The \ufb01rst two authors contributed equally to this work.\n\n30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain.\n\n\fperturbed by noise with random direction in the input space. The semi-random regime generalizes this\nmodel to random subspaces of arbitrary dimension, where a worst-case perturbation is sought within\nthe subspace. In both cases, we derive bounds that precisely describe the robustness of classi\ufb01ers in\nfunction of the curvature of the decision boundary. We summarize our contributions as follows:\n\n\u2022 In the random regime, we show that the robustness of classi\ufb01ers behaves as \u221ad times the\n\ndistance from the datapoint to the classi\ufb01cation boundary (where d denotes the dimension\nof the data) provided the curvature of the decision boundary is suf\ufb01ciently small. This\nresult highlights the blessing of dimensionality for classi\ufb01cation tasks, as it implies that\nrobustness to random noise in high dimensional classi\ufb01cation problems can be achieved,\neven at datapoints that are very close to the decision boundary.\n\nthat the robustness precisely behaves as(cid:112)d/m times the distance to boundary, with m the\n\n\u2022 This quanti\ufb01cation notably extends to the general semi-random regime, where we show\ndimension of the subspace. This result shows in particular that, even when m is chosen as a\nsmall fraction of the dimension d, it is still possible to \ufb01nd small perturbations that cause\ndata misclassi\ufb01cation.\n\n\u2022 We empirically show that our theoretical estimates are very accurately satis\ufb01ed by state-\nof-the-art deep neural networks on various sets of data. This in turn suggests quantitative\ninsights on the curvature of the decision boundary that we support experimentally through\nthe visualization and estimation on two-dimensional sections of the boundary.\n\nThe robustness of classi\ufb01ers to noise has been the subject of intense research. The robustness proper-\nties of SVM classi\ufb01ers have been studied in [19] for example, and robust optimization approaches for\nconstructing robust classi\ufb01ers have been proposed to minimize the worst possible empirical error\nunder noise disturbance [1, 10]. More recently, following the recent results on the instability of\ndeep neural networks to worst-case perturbations [17], several works have provided explanations of\nthe phenomenon [3, 5, 14, 18], and designed more robust networks [6, 8, 20, 13, 15, 12]. In [18],\nthe authors provide an interesting empirical analysis of the adversarial instability, and show that\nadversarial examples are not isolated points, but rather occupy dense regions of the pixel space. In\n[4], state-of-the-art classi\ufb01ers are shown to be vulnerable to geometrically constrained adversarial\nexamples. Our work differs from these works, as we provide a theoretical study of the robustness of\nclassi\ufb01ers to random and semi-random noise in terms of the robustness to adversarial noise. In [3], a\nformal relation between the robustness to random noise, and the worst-case robustness is established\nin the case of linear classi\ufb01ers. Our result therefore generalizes [3] in many aspects, as we study\ngeneral nonlinear classi\ufb01ers, and robustness to semi-random noise. Finally, it should be noted that\nthe authors in [5] conjecture that the \u201chigh linearity\u201d of classi\ufb01cation models explains their instability\nto adversarial perturbations. The objective and approach we follow here is however different, as we\nstudy theoretical relations between the robustness to random, semi-random and adversarial noise.\n\n2 De\ufb01nitions and notations\nLet f : Rd \u2192 RL be an L-class classi\ufb01er. Given a datapoint x0 \u2208 Rd, the estimated label is obtained\nby \u02c6k(x0) = argmaxk fk(x0), where fk(x) is the kth component of f (x) that corresponds to the kth\nclass. Let S be an arbitrary subspace of Rd of dimension m. Here, we are interested in quantifying the\nrobustness of f with respect to different noise regimes. To do so, we de\ufb01ne r\u2217\nS to be the perturbation\nin S of minimal norm that is required to change the estimated label of f at x0.2\n\n\u2217\nS (x0) = argmin\n\nr\n\nr\u2208S (cid:107)r(cid:107)2 s.t. \u02c6k(x0 + r) (cid:54)= \u02c6k(x0).\n\n(1)\n\nNote that r\u2217\n\nS (x0) can be equivalently written\n\nr\n\n\u2217\nS (x0) = argmin\nWhen S = Rd, r\u2217(x0) := r\u2217\nRd (x0) is the adversarial (or worst-case) perturbation de\ufb01ned in [17],\nwhich corresponds to the (unconstrained) perturbation of minimal norm that changes the label of the\n\nr\u2208S (cid:107)r(cid:107)2 s.t. \u2203k (cid:54)= \u02c6k(x0) : fk(x0 + r) \u2265 f\u02c6k(x0)(x0 + r).\n\n(2)\n\n2Perturbation vectors sending a datapoint exactly to the boundary are assumed to change the estimated label\n\nof the classi\ufb01er.\n\n2\n\n\fdatapoint x0. In other words, (cid:107)r\u2217(x0)(cid:107)2 corresponds to the minimal distance from x0 to the classi\ufb01er\nboundary. In the case where S \u2282 Rd, only perturbations along S are allowed. The robustness of f at\nx0 along S is naturally measured by the norm (cid:107)r\u2217\nS (x0)(cid:107)2. Different choices for S permit to study\nthe robustness of f in two different regimes:\n\n\u2022 Random noise regime: This corresponds to the case where S is a one-dimensional subspace\n(m = 1) with direction v, where v is a random vector sampled uniformly from the unit\nsphere Sd\u22121. Writing it explicitly, we study in this regime the robustness quantity de\ufb01ned\nby mint |t| s.t. \u2203k (cid:54)= \u02c6k(x0), fk(x0 + tv) \u2265 f\u02c6k(x0)(x0 + tv), where v is a vector sampled\nuniformly at random from the unit sphere Sd\u22121.\n\u2022 Semi-random noise regime: In this case, the subspace S is chosen randomly, but can be of\narbitrary dimension m.3 We use the semi-random terminology as the subspace is chosen\nrandomly, and the smallest vector that causes misclassi\ufb01cation is then sought in the subspace.\nIt should be noted that the random noise regime is a special case of the semi-random regime\nwith a subspace of dimension m = 1. We differentiate nevertheless between these two\nregimes for clarity.\n\nIn the remainder of the paper, the goal is to establish relations between the robustness in the random\nand semi-random regimes on the one hand, and the robustness to adversarial perturbations (cid:107)r\u2217(x0)(cid:107)2\non the other hand. We recall that the latter quantity captures the distance from x0 to the classi\ufb01er\nboundary, and is therefore a key quantity in the analysis of robustness.\nIn the following analysis, we \ufb01x x0 to be a datapoint classi\ufb01ed as \u02c6k(x0). To simplify the notation,\nwe remove the explicit dependence on x0 in our notations (e.g., we use r\u2217\nS (x0) and \u02c6k\ninstead of \u02c6k(x0)), and it should be implicitly understood that all our quantities pertain to the \ufb01xed\ndatapoint x0.\n\nS instead of r\u2217\n\n3 Robustness of af\ufb01ne classi\ufb01ers\nWe \ufb01rst assume that f is an af\ufb01ne classi\ufb01er, i.e., f (x) = W(cid:62)x + b for a given W = [w1 . . . wL]\nand b \u2208 RL.\nThe following result shows a precise relation between the robustness to semi-random noise, (cid:107)r\u2217\nS(cid:107)2\nand the robustness to adversarial perturbations, (cid:107)r\u2217\nTheorem 1. Let \u03b4 > 0, S be a random m-dimensional subspace of Rd, and f be a L-class af\ufb01ne\nclassi\ufb01er. Let\n\n(cid:107)2.\n\n(cid:33)\u22121\n\n,\n\n2 ln(1/\u03b4)\n\n(cid:32)\n(cid:18)\n\n1 + 2\n\nmax\n\n+\n\nm\n\nln(1/\u03b4)\n\n(cid:114)\n(cid:18)\n(1/e)\u03b42/m, 1 \u2212\n(cid:112)\n\n\u03b62(m, \u03b4) =\n\n\u03b61(m, \u03b4) =\n\n(cid:19)(cid:19)\u22121\nThe following inequalities hold between the robustness to semi-random noise (cid:107)r\u2217\n(cid:114)\nness to adversarial perturbations (cid:107)r\u2217\n(cid:107)2:\n\u2217\nd\nm(cid:107)r\n\n2(1 \u2212 \u03b42/m)\n\n\u2217\nS(cid:107)2 \u2264\n\n(cid:107)2 \u2264 (cid:107)r\n\nd\nm(cid:107)r\n\nm\n\n(cid:113)\n\n\u2217\n\n(cid:107)2,\n\n(cid:114)\n\n(cid:112)\n\n\u03b62(m, \u03b4)\n\n\u03b61(m, \u03b4)\n\n.\n\nS(cid:107)2, and the robust-\n\n(3)\n\n(4)\n\n(5)\n\nwith probability exceeding 1 \u2212 2(L + 1)\u03b4.\nThe proof can be found in the appendix. Our upper and lower bounds depend on the functions\n\u03b61(m, \u03b4) and \u03b62(m, \u03b4) that control the inequality constants (for m, \u03b4 \ufb01xed). It should be noted that\n\u03b61(m, \u03b4) and \u03b62(m, \u03b4) are independent of the data dimension d. Fig. 1 shows the plots of \u03b61(m, \u03b4)\nand \u03b62(m, \u03b4) as functions of m, for a \ufb01xed \u03b4. It should be noted that for suf\ufb01ciently large m, \u03b61(m, \u03b4)\nand \u03b62(m, \u03b4) are very close to 1 (e.g., \u03b61(m, \u03b4) and \u03b62(m, \u03b4) belong to the interval [0.8, 1.3] for\nm \u2265 250 in the settings of Fig. 1). The interval [\u03b61(m, \u03b4), \u03b62(m, \u03b4)] is however (unavoidably) larger\nwhen m = 1.\n3A random subspace is de\ufb01ned as the span of m independent vectors drawn uniformly at random from Sd\u22121.\n\n3\n\n\f(cid:107)2 by a factor of(cid:112)d/m. Specif-\n\nThe result in Theorem 1 shows that in the random and\nsemi-random noise regimes, the robustness to noise is\nprecisely related to (cid:107)r\u2217\nically, in the random noise regime (m = 1), the mag-\nnitude of the noise required to misclassify the datapoint\nbehaves as \u0398(\u221ad(cid:107)r\u2217\n(cid:107)2) with high probability, with con-\nstants in the interval [\u03b61(1, \u03b4), \u03b62(1, \u03b4)]. Our results there-\nfore show that, in high dimensional classi\ufb01cation set-\ntings, af\ufb01ne classi\ufb01ers can be robust to random noise,\neven if the datapoint lies very closely to the decision\nboundary (i.e., (cid:107)r\u2217\n(cid:107)2 is small). In the semi-random noise\nregime with m suf\ufb01ciently large (e.g., m \u2265 250), we have\n(cid:107)r\u2217\n(cid:107)2 with high probability, as the con-\nS(cid:107)2 \u2248\nstants \u03b61(m, \u03b4) \u2248 \u03b62(m, \u03b4) \u2248 1 for suf\ufb01ciently large m.\nOur bounds therefore \u201cinterpolate\u201d between the random\nnoise regime, which behaves as \u221ad(cid:107)r\u2217\n(cid:107)2. More importantly, the\nsquare root dependence is also notable here, as it shows that the semi-random robustness can remain\nsmall even in regimes where m is chosen to be a very small fraction of d. For example, choosing a\nsmall subspace of dimension m = 0.01d results in semi-random robustness of 10(cid:107)r\u2217\n(cid:107)2 with high\nprobability, which might still not be perceptible in complex visual tasks. Hence, for semi-random\nnoise that is mostly random and only mildly adversarial (i.e., the subspace dimension is small), af\ufb01ne\nclassi\ufb01ers remain vulnerable to such noise.\n\nFigure 1: \u03b61(m, \u03b4) and \u03b62(m, \u03b4) in func-\ntion of m [\u03b4 = 0.05] .\n\n(cid:107)2, and the worst-case noise (cid:107)r\u2217\n\n(cid:112)d/m(cid:107)r\u2217\n\n4 Robustness of general classi\ufb01ers\n\n4.1 Curvature of the decision boundary\n\nS(cid:107)2 and worst-case robustness (cid:107)r\u2217\n\nWe now consider the general case where f is a nonlinear classi\ufb01er. We derive relations between\nthe random and semi-random robustness (cid:107)r\u2217\n(cid:107)2 using properties\nof the classi\ufb01er\u2019s boundary. Let i and j be two arbitrary classes; we de\ufb01ne the pairwise boundary\nBi,j as the boundary of the binary classi\ufb01er where only classes i and j are considered. Formally, the\ndecision boundary is given by Bi,j := {x \u2208 Rd : fi(x)\u2212 fj(x) = 0}. The boundary Bi,j separates\nbetween two regions of Rd, namely Ri and Rj, where the estimated label of the binary classi\ufb01er is\nrespectively i and j.\nWe assume for the purpose of this analysis that the boundary Bi,j is smooth. We are now interested\nin the geometric properties of the boundary, namely its curvature. Many notions of curvature can\nbe de\ufb01ned on hypersurfaces [11]. In the simple case of a curve in a two-dimensional space, the\ncurvature is de\ufb01ned as the inverse of the radius of the so-called oscullating circle. One way to de\ufb01ne\ncurvature for high-dimensional hypersurfaces is by taking normal sections of the hypersurface, and\nmeasuring the curvature of the resulting planar curve (see Fig. 2). We however introduce a notion of\ncurvature that is speci\ufb01cally suited to the analysis of the decision boundary of a classi\ufb01er. Informally,\nour curvature captures the global bending of the decision boundary by inscribing balls in the regions\nseparated by the decision boundary. For a given p \u2208 Bi,j, we de\ufb01ne qi (cid:107) j(p) to be the radius of the\nlargest open ball included in the region Ri that intersects with Bi,j at p; i.e.,\nz\u2208Rd {(cid:107)z \u2212 p(cid:107)2 : B(z,(cid:107)z \u2212 p(cid:107)2) \u2286 Ri} ,\n\nqi (cid:107) j(p) = sup\n\n(6)\n\nwhere B(z,(cid:107)z \u2212 p(cid:107)2) is the open ball in Rd of center z and radius (cid:107)z \u2212 p(cid:107)2. An illustration\nof this quantity in two dimensions is provided in Fig. 2 (b). It is not hard to see that any ball\n\u2212 p(cid:107)2) centered in z\u2217 and included in Ri will have its tangent space at p coincide with\nB(z\u2217,(cid:107)z\u2217\nthe tangent of the decision boundary at the same point.\nIt should further be noted that the de\ufb01nition in Eq. (6) is not symmetric in i and j. We therefore\nde\ufb01ne the following symmetric quantity qi,j(p), where the worst-case ball inscribed in any of the\ntwo regions Ri and Rj is considered:\n\nqi,j(p) = min(cid:0)qi (cid:107) j(p), qj (cid:107) i(p)(cid:1) .\n\n4\n\nm0200400600800100010-210-1100101102103104\u03b61 (m\u03b4, )\u03b62\u03b4 (m, )\f(a)\n\n(b)\n\nFigure 2: (a) Normal section of the boundary Bi,j with respect to plane U = span(n, u), where n is\nthe normal to the boundary at p, and u is an arbitrary in the tangent space Tp(Bi,j). (b) Illustration\nof the quantities introduced for the de\ufb01nition of the curvature of the decision boundary.\n\nTo measure the global curvature, the worst-case radius is taken over all points on the decision\nboundary, i.e., q(Bi,j) = inf p\u2208Bi,j qi,j(p). The curvature \u03ba(Bi,j) is then de\ufb01ned as the inverse of\nthe worst-case radius: \u03ba(Bi,j) = 1/q(Bi,j ).\nIn the case of af\ufb01ne classi\ufb01ers, we have \u03ba(Bi,j) = 0, as it is possible to inscribe balls of in\ufb01nite\nradius inside each region of the space. When the classi\ufb01cation boundary is a union of (suf\ufb01ciently\ndistant) spheres with equal radius R, the curvature \u03ba(Bi,j) = 1/R. In general, the quantity \u03ba(Bi,j)\nprovides an intuitive way of describing the nonlinearity of the decision boundary by \ufb01tting balls\ninside the classi\ufb01cation regions.\n\n4.2 Robustness to random and semi-random noise\n\nWe now establish bounds on the robustness to random and semi-random noise in the binary classi\ufb01-\ncation case. Let x0 be a datapoint classi\ufb01ed as \u02c6k = \u02c6k(x0). We \ufb01rst study the binary classi\ufb01cation\nproblem, where only classes \u02c6k and k \u2208 {1, . . . , L}\\{\u02c6k} are considered. To simplify the notation,\nwe let Bk := Bk,\u02c6k be the decision boundary between classes k and \u02c6k. In the case of the binary\nclassi\ufb01cation problem where classes k and \u02c6k are considered, the semi-random perturbation de\ufb01ned in\nEq. (2) can be re-written as follows:\n\nrkS = argmin\n\nr\u2208S (cid:107)r(cid:107)2 s.t. fk(x0 + r) \u2265 f\u02c6k(x0 + r).\n\n(7)\nThe worst case perturbation (obtained with S = Rd) is denoted by rk. It should be noted that the\nglobal quantities r\u2217\nS and r\u2217 are obtained from rkS and rk by taking the vectors with minimum norm\nover all classes k.\nThe following result gives upper and lower bounds on the ratio (cid:107)rkS(cid:107)2\n(cid:107)rk(cid:107)2\nthe boundary separating class k and \u02c6k.\nTheorem 2. Let S be a random m-dimensional subspace of Rd. Let \u03ba := \u03ba(Bk). Assuming that the\ncurvature satis\ufb01es\n\nin function of the curvature of\n\nC\n\n\u03b62(m, \u03b4)(cid:107)rk(cid:107)2\n\nm\nd\n\n,\n\n(8)\n\nthe following inequality holds between the semi-random robustness (cid:107)rkS(cid:107)2 and the adversarial\nrobustness (cid:107)rk(cid:107)2:\n\n(cid:18)\n\n(cid:19)(cid:112)\n\n(cid:114)\n\n\u03ba \u2264\n\n(cid:114)\n\n(cid:18)\n1 \u2212 C1(cid:107)rk(cid:107)2\u03ba\u03b62\n\nd\nm\n\n(cid:19)(cid:112)\n\n\u03b61\n\nd\n\nm \u2264 (cid:107)rkS(cid:107)2\n(cid:107)rk(cid:107)2 \u2264\n\n1 + C2(cid:107)rk(cid:107)2\u03ba\u03b62\n\nd\nm\n\n\u03b62\n\nd\nm\n\n(9)\n\nwith probability larger than 1 \u2212 4\u03b4. We recall that \u03b61 = \u03b61(m, \u03b4) and \u03b62 = \u03b62(m, \u03b4) are de\ufb01ned in\nEq. (3, 4). The constants are C = 0.2, C1 = 0.625, C2 = 2.25.\n\nThe proof can be found in the appendix. This result shows that the bounds relating the robustness to\nrandom and semi-random noise to the worst-case robustness can be extended to nonlinear classi\ufb01ers,\n\n5\n\nUTpBjp\u03b3unR1R2p1B1,2p2q12(p1)q21(p2)\fprovided the curvature of the boundary \u03ba(Bk) is suf\ufb01ciently small. In the case of linear classi\ufb01ers,\nwe have \u03ba(Bk) = 0, and we recover the result for af\ufb01ne classi\ufb01ers from Theorem 1.\nTo extend this result to multi-class classi\ufb01cation, special care has to be taken. In particular, if k\ndenotes a class that has no boundary with class \u02c6k, (cid:107)rk(cid:107)2 can be very large and the previous curvature\ncondition is not satis\ufb01ed. It is therefore crucial to exclude such classes that have no boundary in\ncommon with class \u02c6k, or more generally, boundaries that are far from class \u02c6k. We de\ufb01ne the set A of\nexcluded classes k where (cid:107)rk(cid:107)2 is large\n\n(cid:112)\nA = {k : (cid:107)rk(cid:107)2 \u2265 1.45\n\n(cid:114)\n\n\u2217\n\nd\nm(cid:107)r\n\n\u03b62(m, \u03b4)\n\n(10)\nNote that A is independent of S, and depends only on d, m and \u03b4. Moreover, the constants in (10)\nwere chosen for simplicity of exposition.\nAssuming a curvature constraint only on the close enough classes, the following result establishes a\nS(cid:107)2 and (cid:107)r\u2217\nsimpli\ufb01ed relation between (cid:107)r\u2217\nCorollary 1. Let S be a random m-dimensional subspace of Rd. Assume that, for all k /\u2208 A, the\n(cid:114)\ncurvature condition in Eq. (8) holds. Then, we have\n\n(cid:114)\n\n(cid:107)2}.\n\n(cid:107)2.\n\n\u2217\n\nd\nm(cid:107)r\n\n(cid:107)2 \u2264 (cid:107)r\n\n\u2217\nS(cid:107)2 \u2264 1.45\n\n\u03b62(m, \u03b4)\n\n\u2217\n\nd\nm(cid:107)r\n\n(cid:107)2\n\n(11)\n\n(cid:112)\n\n0.875\n\n\u03b61(m, \u03b4)\n\n(cid:112)\n\nwith probability larger than 1 \u2212 4(L + 2)\u03b4.\nUnder the curvature condition in (8) on the boundaries between \u02c6k and classes in Ac, our result\n(cid:107)2 by a factor of(cid:112)d/m. In the random regime (m = 1), this factor\nshows that the robustness to random and semi-random noise exhibits the same behavior that has\nbeen observed earlier for linear classi\ufb01ers in Theorem 1. In particular, (cid:107)r\u2217\nS(cid:107)2 is precisely related to\nthe adversarial robustness (cid:107)r\u2217\nsemi-random, the factor is(cid:112)d/m and shows that robustness to semi-random noise might not be\nbecomes \u221ad, and shows that in high dimensional classi\ufb01cation problems, classi\ufb01ers with suf\ufb01ciently\n\ufb02at boundaries are much more robust to random noise than to adversarial noise. However, in the\n\nachieved even if m is chosen to be a tiny fraction of d. In other words, if a classi\ufb01er is highly\nvulnerable to adversarial perturbations, then it is also vulnerable to noise that is overwhelmingly\nrandom and only mildly adversarial.\nIt is important to note that the curvature condition in Corollary 1 is not an assumption on the curvature\nof the global decision boundary, but rather an assumption on the decision boundaries between pairs\nof classes. The distinction here is signi\ufb01cant, as junction points where two decision boundaries meet\nmight actually have a very large (or in\ufb01nite) curvature (even in linear classi\ufb01cation settings), and the\ncurvature condition in Corollary 1 typically does not hold for this global curvature de\ufb01nition. We\nrefer to our experimental section for a visualization of this phenomenon.\n5 Experiments\n\n1|D|\n\nx\u2208D\n\n(cid:80)\n\n(cid:107)r\u2217\nS (x)(cid:107)2\n(cid:107)r\u2217(x)(cid:107)2\n\n(cid:112)m/d\n\ncurvature property precisely behaves as(cid:112)d/m(cid:107)r\u2217(x)(cid:107)2. We \ufb01rst check the accuracy of these results\nnotes the test set. This quantity provides indication to the accuracy of our(cid:112)d/m(cid:107)r\u2217(x)(cid:107)2 estimate of\n\nWe now evaluate the robustness of different image classi\ufb01ers to random and semi-random pertur-\nbations, and assess the accuracy of our bounds on various datasets and state-of-the-art classi\ufb01ers.\nSpeci\ufb01cally, our theoretical results show that the robustness (cid:107)r\u2217\nS (x)(cid:107)2 of classi\ufb01ers satisfying the\nin different classi\ufb01cation settings. For a given classi\ufb01er f and subspace dimension m, we de\ufb01ne\n, where S is chosen randomly for each sample x and D de-\n\u03b2(f ; m) =\nthe robustness, and should ideally be equal to 1 (for suf\ufb01ciently large m). Since \u03b2 is a random quantity\n(because of S), we report both its mean and standard deviation for different networks in Table 1.\nIt should be noted that \ufb01nding (cid:107)r\u2217\n(cid:107)2 involves solving the optimization problem in (1).\nWe have used a similar approach to [13] to \ufb01nd subspace minimal perturbations. For each network,\nwe estimate the expectation by averaging \u03b2(f ; m) on 1000 random samples, with S also chosen\nrandomly for each sample. Observe that \u03b2 is suprisingly close to 1, even when m is a small fraction\nof d. This shows that our quantitative analysis provide very accurate estimates of the robustness to\nsemi-random noise. We visualize the robustness to random noise, semi-random noise (with m = 10)\n\nS(cid:107)2 and (cid:107)r\u2217\n\n6\n\n\fTable 1: \u03b2(f ; m) for different classi\ufb01ers f and different subspace dimensions m. The VGG-F and\nVGG-19 are respectively introduced in [2, 16].\n\nClassi\ufb01er\nLeNet (MNIST)\nLeNet (CIFAR-10)\nVGG-F (ImageNet)\nVGG-19 (ImageNet)\n\n1/4\n\n1/16\n\nm/d\n1/36\n\n1/64\n\n1/100\n\n1.00 \u00b1 0.06\n1.01 \u00b1 0.03\n1.00 \u00b1 0.01\n1.00 \u00b1 0.01\n\n1.01 \u00b1 0.12\n1.02 \u00b1 0.07\n1.02 \u00b1 0.02\n1.02 \u00b1 0.03\n\n1.03 \u00b1 0.20\n1.04 \u00b1 0.10\n1.03 \u00b1 0.04\n1.02 \u00b1 0.05\n\n1.01 \u00b1 0.26\n1.06 \u00b1 0.14\n1.03 \u00b1 0.05\n1.03 \u00b1 0.06\n\n1.05 \u00b1 0.34\n1.10 \u00b1 0.19\n1.04 \u00b1 0.06\n1.04 \u00b1 0.08\n\n(a)\n\n(b)\n\n(c)\n\n(d)\n\nFigure 3: (a) Original image classi\ufb01ed as \u201cCauli\ufb02ower\u201d. Fooling perturbations for VGG-F network:\n(b) Random noise, (c) Semi-random perturbation with m = 10, (d) Worst-case perturbation, all\nwrongly classi\ufb01ed as \u201cArtichoke\u201d.\n\nand worst-case perturbations on a sample image in Fig. 3. While random noise is clearly perceptible\n\ndue to the \u221ad \u2248 400 factor, semi-random noise becomes much less perceptible even with a relatively\n\n\u221a\nsmall value of m = 10, thanks to the 1/\nm factor that attenuates the required noise to misclassify\nthe datapoint. It should be noted that the robustness of neural networks to adversarial perturbations\nhas previously been observed empirically in [17], but we provide here a quantitative and generic\nexplanation for this phenomenon. The high accuracy of our bounds for different state-of-the-art\nclassi\ufb01ers, and different datasets suggest that the decision boundaries of these classi\ufb01ers have limited\ncurvature \u03ba(Bk), as this is a key assumption of our theoretical \ufb01ndings. To support the validity of this\ncurvature hypothesis in practice, we visualize two-dimensional sections of the classi\ufb01ers\u2019 boundary\nin Fig. 4 in three different settings. Note that we have opted here for a visualization strategy rather\nthan the numerical estimation of \u03ba(B), as the latter quantity is dif\ufb01cult to approximate in practice in\nhigh dimensional problems. In Fig. 4, x0 is chosen randomly from the test set for each data set, and\nthe decision boundaries are shown in the plane spanned by r\u2217 and r\u2217\nS, where S is a random direction\n(i.e., m = 1). Different colors on the boundary correspond to boundaries with different classes. It\ncan be observed that the curvature of the boundary is very small except at \u201cjunction\u201d points where\nthe boundary of two different classes intersect. Our curvature assumption, which only assumes a\nbound on the curvature of the decision boundary between pairs of classes \u02c6k(x0) and k (but not on the\nglobal decision boundary that contains junctions with high curvature) is therefore adequate to the\ndecision boundaries of state-of-the-art classi\ufb01ers according to Fig. 4. Interestingly, the assumption in\nCorollary 1 is satis\ufb01ed by taking \u03ba to be an empirical estimate of the curvature of the planar curves in\nFig. 4 (a) for the dimension of the subspace being a very small fraction of d; e.g., m = 10\u22123d. While\nnot re\ufb02ecting the curvature \u03ba(Bk) that drives the assumption of our theoretical analysis, this result\nstill seems to suggest that the curvature assumption holds in practice.\nWe now show a simple demonstration of the vulnerability of classi\ufb01ers to semi-random noise in Fig. 5,\nwhere a structured message is hidden in the image and causes data misclassi\ufb01cation. Speci\ufb01cally, we\nconsider S to be the span of random translated and scaled versions of words \u201cNIPS\u201d, \u201cSPAIN\u201d and\n\u201c2016\u201d in an image, such that (cid:98)d/m(cid:99) = 228. The resulting perturbations in the subspace are therefore\nlinear combinations of these words with different intensities.4 The perturbed image x0 + r\u2217\nS shown in\n\n4This example departs somehow from the theoretical framework of this paper, where random subspaces\nwere considered. However, this empirical example suggests that the theoretical \ufb01ndings in this paper seem to\napproximately hold when the subspace S have statistics that are close to a random subspace.\n\n7\n\n\f(a) VGG-F (ImageNet)\n\n(b) LeNet (CIFAR)\n\n(c) LeNet (MNIST)\n\nFigure 4: Boundaries of three classi\ufb01ers near randomly chosen samples. Axes are normalized by the\ncorresponding (cid:107)r\u2217\n(cid:107)2\u03ba.\nNote the difference in range between x and y axes. Note also that the range of horizontal axis in (c)\nis much smaller than the other two, hence the illustrated boundary is more curved.\n\n(cid:107)2 as our assumption in the theoretical bound depends on the product of (cid:107)r\u2217\n\n(a) Image of a \u201cPot\ufb02ower\u201d\n\n(b) Perturbation\n\n(c) Classi\ufb01ed as \u201cPineapple\u201d\n\nFigure 5: A fooling hidden message. S is the span of random translations and scales of the words\n\u201cNIPS\u201d, \u201cSPAIN\u201d, and \u201c2016\u201d.\n\nFig. 5 (c) is clearly indistinguishable from Fig. 5 (a). This shows that imperceptibly small structured\nmessages can be added to an image causing data misclassi\ufb01cation.\n\n6 Conclusion\n\nIn this work, we precisely characterized the robustness of classi\ufb01ers in a novel semi-random noise\nregime that generalizes the random noise regime. Speci\ufb01cally, our bounds relate the robustness\nin this regime to the robustness to adversarial perturbations. Our bounds depend on the curvature\nof the decision boundary, the data dimension, and the dimension of the subspace to which the\nperturbation belongs. Our results show, in particular, that when the decision boundary has a small\ncurvature, classi\ufb01ers are robust to random noise in high dimensional classi\ufb01cation problems (even if\nthe robustness to adversarial perturbations is relatively small). Moreover, for semi-random noise that\nis mostly random and only mildly adversarial (i.e., the subspace dimension is small), our results show\nthat state-of-the-art classi\ufb01ers remain vulnerable to such perturbations. To improve the robustness to\nsemi-random noise, our analysis encourages to impose geometric constraints on the curvature of the\ndecision boundary, as we have shown the existence of an intimate relation between the robustness of\nclassi\ufb01ers and the curvature of the decision boundary.\n\nAcknowledgments\n\nWe would like to thank the anonymous reviewers for their helpful comments. We thank Omar Fawzi\nand Louis Merlin for the fruitful discussions. We also gratefully acknowledge the support of NVIDIA\nCorporation with the donation of the Tesla K40 GPU used for this research. This work has been\npartly supported by the Hasler Foundation, Switzerland, in the framework of the CORA project.\n\n8\n\n-100-75-50-250255075100125150-2.5-2-1.5-1-0.500.511.522.5x0B2B1-150-100-50050100150200-2.502.557.51012.5x0B2B1-5-2.502.557.5-10.75-0.50.2500.250.5x0B1B2\fReferences\n[1] Caramanis, C., Mannor, S., and Xu, H. (2012). Robust optimization in machine learning. In Sra, S.,\n\nNowozin, S., and Wright, S. J., editors, Optimization for machine learning, chapter 14. Mit Press.\n\n[2] Chat\ufb01eld, K., Simonyan, K., Vedaldi, A., and Zisserman, A. (2014). Return of the devil in the details:\n\nDelving deep into convolutional nets. In British Machine Vision Conference.\n\n[3] Fawzi, A., Fawzi, O., and Frossard, P. (2015). Analysis of classi\ufb01ers\u2019 robustness to adversarial perturbations.\n\nCoRR, abs/1502.02590.\n\n[4] Fawzi, A. and Frossard, P. (2015). Manitest: Are classi\ufb01ers really invariant? In British Machine Vision\n\nConference (BMVC), pages 106.1\u2013106.13.\n\n[5] Goodfellow, I. J., Shlens, J., and Szegedy, C. (2015). Explaining and harnessing adversarial examples. In\n\nInternational Conference on Learning Representations (ICLR).\n\n[6] Gu, S. and Rigazio, L. (2014). Towards deep neural network architectures robust to adversarial examples.\n\narXiv preprint arXiv:1412.5068.\n\n[7] Hinton, G. E., Deng, L., Yu, D., Dahl, G. E., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P.,\nSainath, T. N., and Kingsbury, B. (2012). Deep neural networks for acoustic modeling in speech recognition:\nThe shared views of four research groups. IEEE Signal Process. Mag., 29(6):82\u201397.\n\n[8] Huang, R., Xu, B., Schuurmans, D., and Szepesv\u00e1ri, C. (2015). Learning with a strong adversary. CoRR,\n\nabs/1511.03034.\n\n[9] Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). Imagenet classi\ufb01cation with deep convolutional\n\nneural networks. In Advances in neural information processing systems (NIPS), pages 1097\u20131105.\n\n[10] Lanckriet, G., Ghaoui, L., Bhattacharyya, C., and Jordan, M. (2003). A robust minimax approach to\n\nclassi\ufb01cation. The Journal of Machine Learning Research, 3:555\u2013582.\n\n[11] Lee, J. M. (2009). Manifolds and differential geometry, volume 107. American Mathematical Society\n\nProvidence.\n\n[12] Luo, Y., Boix, X., Roig, G., Poggio, T., and Zhao, Q. (2015). Foveation-based mechanisms alleviate\n\nadversarial examples. arXiv preprint arXiv:1511.06292.\n\n[13] Moosavi-Dezfooli, S.-M., Fawzi, A., and Frossard, P. (2016). Deepfool: a simple and accurate method to\n\nfool deep neural networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).\n\n[14] Sabour, S., Cao, Y., Faghri, F., and Fleet, D. J. (2016). Adversarial manipulation of deep representations.\n\nIn International Conference on Learning Representations (ICLR).\n\n[15] Shaham, U., Yamada, Y., and Negahban, S. (2015). Understanding adversarial training: Increasing local\n\nstability of neural nets through robust optimization. arXiv preprint arXiv:1511.05432.\n\n[16] Simonyan, K. and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recogni-\n\ntion. In International Conference on Learning Representations (ICLR).\n\n[17] Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. (2014).\nIntriguing properties of neural networks. In International Conference on Learning Representations (ICLR).\n[18] Tabacof, P. and Valle, E. (2016). Exploring the space of adversarial images. IEEE International Joint\n\nConference on Neural Networks.\n\n[19] Xu, H., Caramanis, C., and Mannor, S. (2009). Robustness and regularization of support vector machines.\n\nThe Journal of Machine Learning Research, 10:1485\u20131510.\n\n[20] Zhao, Q. and Grif\ufb01n, L. D. (2016). Suppressing the unusual: towards robust cnns using symmetric\n\nactivation functions. arXiv preprint arXiv:1603.05145.\n\n9\n\n\f", "award": [], "sourceid": 887, "authors": [{"given_name": "Alhussein", "family_name": "Fawzi", "institution": "Ecole Polytechnique Federale de Lausanne (EPFL)"}, {"given_name": "Seyed-Mohsen", "family_name": "Moosavi-Dezfooli", "institution": "EPFL"}, {"given_name": "Pascal", "family_name": "Frossard", "institution": "EPFL"}]}