{"title": "Empirically Measuring Concentration: Fundamental Limits on Intrinsic Robustness", "book": "Advances in Neural Information Processing Systems", "page_first": 5209, "page_last": 5220, "abstract": "Many recent works have shown that adversarial examples that fool classifiers can be found by minimally perturbing a normal input. Recent theoretical results, starting with Gilmer et al. (2018b), show that if the inputs are drawn from a concentrated metric probability space, then adversarial examples with small perturbation are inevitable. A concentrated space has the property that any subset with \u2126(1) (e.g.,1/100) measure, according to the imposed distribution, has small distance to almost all (e.g.,  99/100) of the points in the space. It is not clear,  however,  whether these theoretical results apply to actual distributions such as images. This paper presents a method for empirically measuring and bounding the concentration of a concrete dataset which is proven to converge to the actual concentration. We use it to empirically estimate the intrinsic robustness to and L_2 and L_infinity perturbations of several image classification benchmarks. Code for our experiments is available at https://github.com/xiaozhanguva/Measure-Concentration.", "full_text": "Empirically Measuring Concentration:\n\nFundamental Limits on Intrinsic Robustness\n\nSaeed Mahloujifar\u2217, Xiao Zhang\u2217, Mohammad Mahmoody, and David Evans\n\nUniversity of Virginia\n\n[saeed, shawn, mohammad, evans]@virginia.edu\n\nAbstract\n\nMany recent works have shown that adversarial examples that fool classi\ufb01ers can be\nfound by minimally perturbing a normal input. Recent theoretical results, starting\nwith Gilmer et al. (2018b), show that if the inputs are drawn from a concentrated\nmetric probability space, then adversarial examples with small perturbation are\ninevitable. A concentrated space has the property that any subset with \u2126(1) (e.g.,\n1/100) measure, according to the imposed distribution, has small distance to almost\nall (e.g., 99/100) of the points in the space. It is not clear, however, whether\nthese theoretical results apply to actual distributions such as images. This paper\npresents a method for empirically measuring and bounding the concentration of a\nconcrete dataset which is proven to converge to the actual concentration. We use\nit to empirically estimate the intrinsic robustness to (cid:96)\u221e and (cid:96)2 perturbations of\nseveral image classi\ufb01cation benchmarks. Code for our experiments is available at\nhttps://github.com/xiaozhanguva/Measure-Concentration.\n\n1\n\nIntroduction\n\nDespite achieving exceptionally high accuracy on natural inputs, state-of-the-art machine learning\nmodels have been shown to be vulnerable to adversaries who use small perturbations to fool the\nclassi\ufb01er (Szegedy et al., 2014; Goodfellow et al., 2015). This phenomenon, known as adversarial\nexamples, has motivated numerous studies (Papernot et al., 2016; Madry et al., 2018; Biggio & Roli,\n2018; Gilmer et al., 2018a) to develop heuristic defenses that aim to improve classi\ufb01er robustness.\nHowever, most defense mechanisms have been quickly broken by adaptive attacks (Carlini & Wagner,\n2017; Athalye et al., 2018). Although certi\ufb01cation methods (Raghunathan et al., 2018; Wong &\nKolter, 2018; Sinha et al., 2018; Wong et al., 2018; Gowal et al., 2019; Wang et al., 2018; Zhang\net al., 2019) have been proposed aiming to end such arms race and continuous efforts have been\nmade to develop better robust models, both the robustness guarantees and ef\ufb01ciency achieved by\nstate-of-the-art robust classi\ufb01ers are far from satisfying.\nThis motivates a fundamental information-theoretic question: what are the inherent limitations\nof developing robust classi\ufb01ers? Several recent works (Gilmer et al., 2018b; Fawzi et al., 2018;\nMahloujifar et al., 2019; Shafahi et al., 2019; Bhagoji et al., 2019) have shown that under certain\nassumptions regarding the data distribution and the perturbation metric, adversarial examples are\ntheoretically inevitable. As a result, for a broad set of theoretically natural metric probability spaces\nof inputs, there is no classi\ufb01er for the data distribution that achieves adversarial robustness. For\nexample, Gilmer et al. (2018b) assumed that the input data are sampled uniformly from n-spheres and\nproved a model-independent theoretical bound connecting the risk to the average Euclidean distance\nto the \u201ccaps\u201d (i.e., round regions on a sphere). Mahloujifar et al. (2019) generalized this result to any\nconcentrated metric probability space of inputs and showed, for example, that if the inputs come from\n\n\u2217Equal contribution.\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fany Normal L\u00e9vy family (L\u00e9vy, 1951), any classi\ufb01er with a noticable test error will be vulnerable to\nsmall (i.e., sublinear in the typical norm of the inputs) perturbations.\nAlthough such theoretical \ufb01ndings seem discouraging to the goal of developing robust classi\ufb01ers,\nall these impossibility results depend on assumptions about data distributions that might not hold\nfor cases of interest. Our work develops a general method for testing properties of concrete datasets\nagainst these theoretical assumptions.\n\nContributions. Our work shrinks the gap between theoretical analyses of robustness of classi\ufb01cation\nfor theoretical data distributions and understanding the intrinsic robustness of actual datasets. Indeed,\nquantitative estimates of the intrinsic robustness2 of benchmark image datasets such as MNIST and\nCIFAR-10 can provide us with a better understanding of the threat of adversarial examples for natural\nimage distributions and may suggest promising directions for further improving classi\ufb01er robustness.\nOur main technical contribution is a general method to evaluate the concentration of a given input\ndistribution \u00b5 based on a set of data samples. We prove that by simultaneously increasing the sample\nsize m and a complexity parameter T , the concentration of the empirical measure converges to the\nactual concentration of \u00b5 (Section 3). Using this method, we perform experiments to demonstrate\nthe existence of robust error regions for benchmark datasets under both (cid:96)\u221e and (cid:96)2 perturbations\n(Section 4). Compared with state-of-the-art robustly trained models, our estimated intrinsic robustness\nshows that, for most settings, there exists a large gap between the robust error achieved by the best\ncurrent models and the theoretical limits implied by concentration. This suggests the concentration\nof measure is not the only reason behind the vulnerability of existing classi\ufb01ers to adversarial\nperturbations. Thus, either there is room for improving the robustness of image classi\ufb01ers (even with\nnon-zero classi\ufb01cation error) or a need for deeper understanding of the reasons for the gap between\nintrinsic robustness and the actual robustness achieved by robust models, at least for the datasets like\nthe image classi\ufb01cation benchmarks used in our experiments.\n\nRelated Work. We are aware of only one previous work that attempts to heuristically estimate these\nproperties. To extend their theoretical impossibility result to the practical distributions, Gilmer et al.\n(2018b) studied MNIST dataset to \ufb01nd a region that is somewhat robust in terms of the expected\n(cid:96)2 distance of other images from the region. In their setting, they showed the existence of a set of\nmeasure 0.01 with average (cid:96)2 distance 6.59 to all points. In comparison, our work is the \ufb01rst to\nprovide a general methodology to empirically estimate the concentration of measure with provable\nguarantees. Moreover, we are able to deal with (cid:96)\u221e, and worst-case bounded perturbations for\nmodeling adversarial risk, which is the most popular setting for research in adversarial examples.\nIn addition, another related concurrent work (Bhagoji et al., 2019) studied lower bounds on the\nadversarial risk using optimal transport on the metric probability space of instances. They also\nmeasure the optimal transport on the empirical distributions but do not characterize the relationship\nbetween the optimal transport of empirical datasets and the actual one of the underlying distributions.\nAnother related line of work estimated lower bounds on the concentration of measure of the underlying\ndistribution through simulating distributions by generative models. Fawzi et al. (2018) proved a lower\nbound on the concentration of the generated image distribution, assuming the underlying generative\nmodel has Gaussian latent space and small Lipschitz constant. Krusinga et al. (2019) estimated\nan upper bound on the density function of the distribution using generative model, then proved\nconcentration inequalities based on upper bounds on the density function. Our work is distinct from\nthese works, because we directly learn the concentration function instead of a lower bound, and we\nuse the actual data samples instead of samples generated from some trained generative model.\nThe work of Tsipras et al. (2019) studied the trade-off between robustness and accuracy. They show\nthat for some speci\ufb01c learning problems, achieving robustness and accuracy together is not possible.\nAt \ufb01rst glance, it might seem that this trade-off contradicts the existing lower bounds that come from\nconcentration of measure. However, there is no contradiction and what is proved there is with regard\nto a different de\ufb01nition of adversarial examples. The de\ufb01nition of adversarial examples used there\ncould diverge from our de\ufb01nition in some learning problems (see Diochnos et al. (2018)), but they\ncoincide in the cases that the ground truth function is robust to small perturbations.\n\n2See De\ufb01nition 2.2 for the formal de\ufb01nition of intrinsic robustness. The term robustness has been used with\ndifferent meanings in previous works (e.g., in Diochnos et al. (2018), it refers to the average distances to the\nerror region). However, all such uses refer to a desirable property of the classi\ufb01er in being resilient to adversarial\nperturbations, which is the case here as well. See Diochnos et al. (2018) for a taxonomy of different de\ufb01nitions.\n\n2\n\n\fof x are de\ufb01ned as (cid:107)x(cid:107)\u221e = maxi\u2208[n] |xi| and (cid:107)x(cid:107)2 = ((cid:80)\nmeasure with respect to a set S sampled from \u00b5 as \u02c6\u00b5S (A) = (cid:80)\n\nNotation. Lowercase boldface letters such as x are used to denote vectors, and [n] is used to\nrepresent {1, 2, . . . , n}. For any set A, let Pow(A), |A| and 1A(\u00b7) be the set of measurable subsets of\nA, cardinality and indicator function of A, respectively. For any x \u2208 Rn, the (cid:96)\u221e-norm and (cid:96)2-norm\ni )1/2 respectively. Let (X , \u00b5) be a\nprobability space and d : X \u00d7 X \u2192 R be some distance metric de\ufb01ned on X . De\ufb01ne the empirical\nx\u2208S 1A(x)/|S|,\u2200A \u2286 X . Let\nBall(x, \u0001) = {x(cid:48) \u2208 X : d(x(cid:48), x) \u2264 \u0001} be the ball around x with radius \u0001. For any subset A \u2286 X ,\nde\ufb01ne the \u0001-expansion A\u0001 = {x \u2208 X : \u2203 x(cid:48) \u2208 Ball(x, \u0001) \u2229 A}. The collection of the \u0001-expansions\nfor members of any G \u2286 Pow(X ) is de\ufb01ned and denoted as G\u0001 = {A\u0001 : A \u2208 G}.\n\ni\u2208[n] x2\n\n2 Robustness and Concentration of Measure\n\nIn this paper, we work with the following de\ufb01nition of adversarial risk:\nDe\ufb01nition 2.1 (Adversarial Risk). Let (X , \u00b5) be the probability space of instances and f\u2217 be the\nunderlying ground-truth. The adversarial risk of a classi\ufb01er f in metric d with strength \u0001 is de\ufb01ned as\n\n(cid:2)\u2203 x(cid:48) \u2208 Ball(x, \u0001) s.t. f (x(cid:48)) (cid:54)= f\u2217(x(cid:48))(cid:3).3\n\nAdvRisk\u0001(f, f\u2217) = Pr\nx\u2190\u00b5\n\nFor \u0001 = 0, which allows no perturbation, the notion of adversarial risk coincides with traditional risk.\nDe\ufb01nition 2.2 (Intrinsic Robustness). Consider the same setting as in De\ufb01nition 2.1. Let F be some\nfamily of classi\ufb01ers, then the intrinsic robustness is de\ufb01ned as the maximum adversarial robustness\nthat can be achieved within F, namely\n\nRob\u0001(F, f\u2217) = 1 \u2212 inf\nf\u2208F\n\n(cid:8)AdvRisk\u0001(f, f\u2217)(cid:9).\n\nIn this work, we specify F as the family of imperfect classi\ufb01ers that have risk at least \u03b1 \u2208 (0, 1).\nPrevious work shows a connection between concentration of measure and the intrinsic robustness\nwith respect to some families of classi\ufb01ers (Gilmer et al. (2018b); Fawzi et al. (2018); Mahloujifar\net al. (2019); Shafahi et al. (2019)). The concentration of measure on a metric probability space is\nde\ufb01ned by a concentration function as follows.\nDe\ufb01nition 2.3 (Concentration Function). Consider a metric probability space (X , \u00b5, d). Suppose\n\u0001 > 0 and \u03b1 \u2208 (0, 1) are given parameters, then the concentration function of the probability measure\n\u00b5 with respect to \u0001, \u03b1 is de\ufb01ned as\n\nh(\u00b5, \u03b1, \u0001) = inf\n\nE\u2208Pow(X )\n\n{\u00b5(E\u0001) : \u00b5(E) \u2265 \u03b1} .\n\nNote that the standard notion of concentration function (e.g., see Talagrand (1995)) is related to a\nspecial case of De\ufb01nition 2.3 by \ufb01xing \u03b1 = 1/2.\nGeneralizing the result of Gilmer et al. (2018b) about instances drawn from spheres, Mahloujifar\net al. (2019) showed that, in general, if the metric probability space of instances is concentrated, then\nany classi\ufb01er with 1% risk incurs large adversarial risk for small amount of perturbations.\nTheorem 2.4 (Mahloujifar et al. (2019)). Let (X , \u00b5) be the probability space of instances and f\u2217 be\nthe underlying ground-truth. For any classi\ufb01er f, we have\n\nAdvRisk\u0001(f, f\u2217) \u2265 h(\u00b5, Risk(f, f\u2217), \u0001).\n\nIn order for this theorem to be useful, we need to know the concentration function. The behavior of\nthis function is studied extensively for certain theoretical metric probability spaces (Ledoux, 2001;\nMilman & Schechtman, 1986). However, it is not known how to measure the concentration function\nfor arbitrary metric probability spaces. In this work, we provide a framework to (algorithmically)\nbound the concentration function from i.i.d. samples from a distribution. Namely, we want to solve\nthe following optimization task using our i.i.d. samples:\n\nminimize\nE\u2208Pow(X )\n\n\u00b5(E\u0001)\n\nsubject to \u00b5(E) \u2265 \u03b1.\n\n(1)\n\n3Note that bounding lp norm might be restrictive for the adversary (Gilmer et al., 2018a) and this de\ufb01nition\n\nonly covers a subset of possible adversaries.\n\n3\n\n\fWe aim to estimate the minimum possible adversarial risk, which captures the intrinsic robustness\nfor classi\ufb01cation in terms of the underlying distribution \u00b5, conditioned on the fact that the original\nrisk is at least \u03b1. Note that solving this optimization problem only shows the possibility of existence\nof an error region E with certain (small) expansion. This means that there could potentially exist a\nclassi\ufb01er with risk at least \u03b1 and adversarial risk equal to the solution of the optimization problem\nof (1). Actually \ufb01nding such an optimally robust classi\ufb01er (with error \u03b1) using a learning algorithm\nmight be a much more dif\ufb01cult task or even infeasible. We do not consider that problem in this work.\n\n3 Method for Measuring Concentration\n\nIn this section, we present a method to measure the concentration of measure on a metric probability\nspace using i.i.d. samples. To measure concentration, there are two main challenges:\n\n1. Measuring concentration appears to require knowledge of the density function of the distri-\n\nbution, but we only have a data set sampled from the distribution.\n\n2. Even with the density function, we have to \ufb01nd the best possible subset among all the subsets\n\nof the space, which seems infeasible.\n\nWe show how to overcome these challenges and \ufb01nd the actual concentration in the limit by \ufb01rst\nempirically simulating the distribution and then narrowing down our search space to a speci\ufb01c\ncollection of subsets. Our results show that for a carefully chosen family of sets, the set with\nminimum expansion can be approximated using polynomially many samples. On the other hand, the\nminimum expansion convergence to the actual concentration (without the limits on the sets) as the\ncomplexity of the collection goes to in\ufb01nity.\nBefore stating our main theorems, we introduce two useful de\ufb01nitions. The following de\ufb01nition\ncaptures the concentration function for a speci\ufb01c collection of subsets.\nDe\ufb01nition 3.1 (Concentration Function for a Collection of Subsets). Consider a metric probability\nspace (X , \u00b5, d). Let \u0001 > 0 and \u03b1 \u2208 (0, 1) be given parameters, then the concentration function of\nthe probability measure \u00b5 with respect to \u0001, \u03b1 and a collection of subsets G \u2286 Pow(X ) is de\ufb01ned as\n\nh(\u00b5, \u03b1, \u0001,G) = infE\u2208G {\u00b5(E\u0001) : \u00b5(E) \u2265 \u03b1} .\n\nWhen G = Pow(X ), we write h(\u00b5, \u03b1, \u0001) for simplicity.\nWe also need to de\ufb01ne the notion of complexity penalty for a collection of subsets. The complexity\npenalty for a collection of subsets captures the rate of the uniform convergence for the subsets in\nthat collection. One can get such uniform convergence rates using the VC dimension or Rademacher\ncomplexity of the collection.\nDe\ufb01nition 3.2 (Complexity Penalty). Let G \u2286 Pow(X ) be a collection of subsets of X . A function\n\u03c6 : N \u00d7 R \u2192 [0, 1] is a complexity penalty for G iff for any probability measure \u00b5 supported on X\nand any \u03b4 \u2208 [0, 1], we have\nPr\n\n[\u2203 E \u2208 G s.t. |\u00b5(E) \u2212 \u02c6\u00b5S(E)| \u2265 \u03b4] \u2264 \u03c6(m, \u03b4).\n\nS\u2190\u00b5m\n\nTheorem 3.3 shows how to overcome the challenge of measuring concentration from \ufb01nite samples,\nwhen the concentration is de\ufb01ned with respect to speci\ufb01c families of subsets. Namely, it shows that\nthe empirical concentration is close to the true concentration, if the underlying collection of subsets\nis not too complex. The proof of Theorem 3.3 is provided in Appendix A.1.\nTheorem 3.3 (Generalization of Concentration). Let (X , \u00b5, d) be a metric probability space and\nG \u2286 Pow(X ). For any \u03b4, \u03b1, \u0001 \u2208 [0, 1], we have\n\n[h(\u00b5, \u03b1\u2212 \u03b4, \u0001,G)\u2212 \u03b4 \u2264 h(\u02c6\u00b5S, \u03b1, \u0001,G) \u2264 h(\u00b5, \u03b1 + \u03b4, \u0001,G) + \u03b4] \u2265 1\u2212 2(cid:0)\u03c6(m, \u03b4) + \u03c6\u0001(m, \u03b4)(cid:1)\n\nPr\nS\u2190\u00b5m\nwhere \u03c6 and \u03c6\u0001 are complexity penalties for G and G\u0001 respectively.\nRemark 3.4. Theorem 3.3 shows that if we narrow down our search to a collection of subsets G\nsuch that both G and G\u0001 have small complexity penalty, then we can use the empirical distribution to\nmeasure concentration of measure for that speci\ufb01c collection. Note that the generalization bound of\n\n4\n\n\fTheorem 3.3 depends on complexity penalties for both G and G\u0001. Therefore, in order for this theorem\nto be useful, the collection G must be chosen in a careful way. For example, if G has bounded VC\ndimension, then G\u0001 might still have a very large VC dimension. Alternatively, G might denote the\ncollection of subsets that are decidable by a neural network of a certain size. In that case, even though\nthere are well known complexity penalties for such collections (see Neyshabur et al. (2017)), the\ncomplexity of their expansions is unknown. In fact, relating the complexity penalty for expansion of a\ncollection to that of the original collection is tightly related to generalization bounds in the adversarial\nsettings, which has also been the subject of several recent works (Cullina et al., 2018; Attias et al.,\n2019; Montasser et al., 2019; Yin et al., 2019; Raghunathan et al., 2019).\n\n(cid:9)\n\n(cid:8)\u03c6T(cid:9)\n\nT\u2208N and(cid:8)\u03c6T\n\nThe following theorem, proved in Appendix A.2, states that if we gradually increase the complexity of\nthe collection and the number of samples together, the empirical estimate of concentration converges\nto actual concentration, as long as several conditions hold. Theorem 3.5 and the techniques used in\nits proof are inspired by the work of Scott & Nowak (2006) on learning minimum volume sets.\nTheorem 3.5. Let {G(T )}T\u2208N be a family of subset collections de\ufb01ned over a space X . Let\n\u0001 are\ncomplexity penalties for G(T ) and G\u0001(T ) respectively, for some \u0001 \u2208 [0, 1]. Let {m(T )}T\u2208N and\n{\u03b4(T )}T\u2208N be two sequences such that m(T ) \u2208 N and \u03b4(T ) \u2208 [0, 1].\nConsider a sequence of datasets {ST}T\u2208N, where ST consists of m(T ) i.i.d. samples from a measure\n\u00b5 supported on X . Also let \u03b1 \u2208 [0, 1] be such that h is locally continuous w.r.t the second parameter\nat point (\u00b5, \u03b1, \u0001, Pow(X )). If all the following hold,\n\nT\u2208N be two families of complexity penalty functions such that \u03c6T and \u03c6T\n\n\u0001\n\n1. (cid:80)\u221e\n2. (cid:80)\u221e\n\nT =1 \u03c6T (m(T ), \u03b4(T )) < \u221e\n\u0001 (m(T ), \u03b4(T )) < \u221e\nT =1 \u03c6T\n\n3. limT\u2192\u221e \u03b4(T ) = 0\n4. limT\u2192\u221e h(\u00b5, \u03b1, \u0001,G(T )) = h(\u00b5, \u03b1, \u0001)\n\nthen with probability 1, we have limT\u2192\u221e h(\u02c6\u00b5ST , \u03b1, \u0001,G(T )) = h(\u00b5, \u03b1, \u0001).\nRemark 3.6. In Theorem 3.5, the \ufb01rst two conditions restrict the growth rate for the complexity of\nthe collections. Namely, we need the complexity penalties \u03c6T (m(T ), \u03b4(T )) and \u03c6T\n\u0001 (m(T ), \u03b4(T )) to\nrapidly approach 0 as T \u2192 \u221e, which means the complexity of G(T ) and G\u0001(T ) should grow at a\nslow rate. The third condition requires that our generalization error goes to zero as we increase T .\nNote that the complexity penalty is a decreasing function with respect to \u03b4, which means condition 3\nmakes achieving the \ufb01rst two conditions harder. However, since the complexity penalty is a function\nof both \u03b4 and sample size, we can still increase the sample size with a faster rate to satisfy the \ufb01rst two\nconditions. Finally, the fourth condition requires our approximation error goes to 0 as we increase T .\nNote that this condition holds for any family of collections of subsets that is a universal approximator\n(e.g., decision trees or neural networks). However, in order for our theorem to hold, we also need all\nthe other conditions. In particular, we cannot use decision trees or neural networks as our collection\nof subsets, because we do not know if there is a complexity penalty for them that satis\ufb01es condition 2.\n\n3.1 Special Case of (cid:96)\u221e\n\n(cid:110)Rn \\ \u222aT\n\nIn this subsection, we show how to instantiate Theorem 3.5 for the case of (cid:96)\u221e. Below, we introduce\na special collection of subsets characterized by the complement of a union of hyperrectangles:\nDe\ufb01nition 3.7 (Complement of union of hyperrectangles). For any positive integer T , the collection\nof subsets speci\ufb01ed by the complement of a union of T n-dimensional hyperrectangles is de\ufb01ned as\n\nCR(T, n) =\n\nwhere Rect(u, r) =(cid:8)x \u2208 X : \u2200j \u2208 [n],|xj \u2212 uj| \u2264 rj/2(cid:9) denotes the hyperrectangle centered at\n\nt=1Rect(u(t), r(t)) : \u2200t \u2208 [T ], (u(t), r(t)) \u2208 Rn \u00d7 Rn\u22650\n\nu with r representing the edge size vector. When n is free of context, we simply write CR(T ).\nRecall that our goal is to \ufb01nd a subset E \u2208 Rn such that E has measure at least \u03b1 and the \u0001\u221e-expansion\nof E under (cid:96)\u221e has the minimum measure. To achieve this goal, we approximate the distribution \u00b5\n\n,\n\n(cid:111)\n\n5\n\n\f(cid:110) \u222aT\n\n(cid:111)\n\n.\n\nwith an empirical distribution \u02c6\u00b5S, and limit our search to the special collection CR(T ) (though our\ngoal is to \ufb01nd the minimum concentration around arbitrary subsets). Namely, what we \ufb01nd is still an\nupper bound on the concentration function, and it is an upper bound that we know it converges the\nactual value in the limit. Our problem thus becomes the following optimization task:\n\nminimize\nE\u2208CR(T )\n\n\u02c6\u00b5S (E\u0001\u221e )\n\nsubject to \u02c6\u00b5S (E) \u2265 \u03b1.\n\n(2)\n\nThe following theorem provides the key to our empirical method by providing a convergence\nguarantee. It states that if we increase the number of rectangles and the number of samples together\nin a careful way, the solution to the problem using restricted sets converges to the true concentration.\nTheorem 3.8. Consider a nice metric probability space (Rn, \u00b5, (cid:96)\u221e). Let {ST}T\u2208N be a family\nof datasets such that for all T \u2208 N, ST contains at least T 4 i.i.d. samples from \u00b5. For any \u0001\u221e\nand \u03b1 \u2208 [0, 1], if h is locally continuous w.r.t the second parameter at point (\u00b5, \u03b1, \u0001\u221e), then with\nprobability 1 we get\n\nT\u2192\u221e h(\u02c6\u00b5ST , \u03b1, \u0001\u221e,CR(T )) = h(\u00b5, \u03b1, \u0001\u221e).\n\nlim\n\nNote that the size of ST is selected as T 4 to guarantee conditions 1 and 2 are satis\ufb01ed in Theorem\n3.5. In fact, we can tune the parameters more carefully to get T 2, instead of T 4, but the convergence\nwill be slower. See Appendix A.3 for the proof.\n\n3.2 Special Case of (cid:96)2\n\nThis subsection demonstrates how to apply Theorem 3.5 to the case of (cid:96)2. The following de\ufb01nition\nintroduces the collection of subsets characterized by a union of balls:\nDe\ufb01nition 3.9 (Union of Balls). For any positive integer T , the collection of subsets speci\ufb01ed by a\nunion of T n-dimensional balls is de\ufb01ned as\n\nB(T, n) =\n\nt=1 Ball(u(t), r(t)) : \u2200t \u2208 [T ], (u(t), r(t)) \u2208 Rn \u00d7 Rn\u22650\n\nWhen n is free of context, we simply write B(T ).\nBy restricting our search to the collection of a union of balls B(T ) and replacing the underlying\ndistribution \u00b5 with the empirical one \u02c6\u00b5S, our problem becomes the following optimization task\n\nminimize\nE\u2208B(T )\n\n\u02c6\u00b5S (E\u00012 )\n\nsubject to \u02c6\u00b5S (E) \u2265 \u03b1.\n\n(3)\n\nTheorem 3.10, proven in Appendix A.4, guarantees that if we increase the number of balls and\nsamples together in a careful way, the solution to the empirical problem (3) converges to the true\nconcentration.\nTheorem 3.10. Consider a nice metric probability space (Rn, \u00b5, (cid:96)2). Let {ST}T\u2208N be a family\nof datasets such that for all T \u2208 N, ST contains at least T 4 i.i.d. samples from \u00b5. For any \u00012\nand \u03b1 \u2208 [0, 1], if h is locally continuous w.r.t the second parameter at point (\u00b5, \u03b1, \u00012), then with\nprobability 1 we get\n\nT\u2192\u221e h(\u02c6\u00b5ST , \u03b1, \u00012,B(T )) = h(\u00b5, \u03b1, \u00012).\n\nlim\n\n4 Experiments\n\nIn this section, we provide heuristic methods to \ufb01nd the best possible error region, which covers at\nleast \u03b1 fraction of the samples and its expansion covers the least number of points, for both (cid:96)\u221e and (cid:96)2\nsettings. Speci\ufb01cally, we \ufb01rst introduce our algorithm, then evaluate our approach on two benchmark\nimage datasets: MNIST (LeCun et al., 2010) and CIFAR-10 (Krizhevsky & Hinton, 2009). Note that\nin our experiments we exactly use the collection of subsets as suggested by our theoretical results in\nthe previous section. However, that is not necessary and one might work with any subset collection to\nrun experiments, as long as they can estimate the measure of the sets and their expansion. We tried\nworking with other collection of subsets that we do not have theoretical support for (e.g. sets de\ufb01ned\nby a neural network) and observed a large generalization gap. This observation shows the importance\nof working with subset collections that we can theoretically control their generalization penalty.\n\n6\n\n\f(a) varying q\n\n(b) varying T\n\nFigure 1: (a) Plots of risk and adversarial risk w.r.t. the resulted error region using our method as q\nvaries (CIFAR-10, \u0001\u221e = 8/255, T = 30); (b) Plots of adversarial risk w.r.t. the resulted error region\nusing our method (best q) as T varies on MNIST (\u0001\u221e = 0.3) and CIFAR-10 (\u0001\u221e = 8/255).\n\n4.1 Experiments for (cid:96)\u221e\nTheorem 3.8 shows that the empirical concentration function h(\u02c6\u00b5S , \u03b1, \u0001\u221e,CR(T )) converges to the\nactual concentration h(\u00b5, \u03b1, \u0001\u221e) asymptotically, when T and |S| go to in\ufb01nity with |S| \u2265 T 4. Thus,\nto measure the concentration of \u00b5, it remains to solve the optimization problem (2).\n\nMethod. Although the collection of subsets is speci\ufb01ed using simple topology, solving (2) exactly is\nstill dif\ufb01cult, as the problem itself is combinatorial in nature. Borrowing techniques from clustering,\nwe propose an empirical method to search for desirable error region within CR(T ). Any error region\nE could be used to de\ufb01ne fE, i.e., fE (x) = f\u2217(x), if x /\u2208 E; fE (x) (cid:54)= f\u2217(x), if x \u2208 E. However,\n\ufb01nding a classi\ufb01er corresponding to fE using a learning algorithm might be a very dif\ufb01cult task. Here,\nwe \ufb01nd the optimally robust error region, not the corresponding classi\ufb01er. A desirable error region\nshould have small adversarial risk4, compared with all subsets in CR(T ) that have measure at least \u03b1.\nThe high-level intuition is that images from different classes are likely to be concentrated in separable\nregions, since it is generally believed that small perturbations preserve the ground-truth class at the\nsampled images. Therefore, if we cluster all the images into different clusters, a desired region with\nlow adversarial risk should exclude any image from the dense clusters, otherwise the expansion of\nsuch a region will quickly cover the whole cluster. In other words, a desirable subset within CR(T )\nshould be \u0001\u221e away (in (cid:96)\u221e norm) from all the dense image clusters, which motivates our method to\ncover the dense image clusters using hyperrectangles and treat the complement of them as error set.\nMore speci\ufb01cally, our algorithm (for pseudocode, see Algorithm 1 in Appendix B) starts by sorting\nall the training images in an ascending order based on the (cid:96)1-norm distance to the k-th nearest\nneighbour with k = 50, and then obtains T hyperrectangular image clusters by performing k-means\nclustering (Hartigan & Wong, 1979) on the top-q densest images, where the metric is chosen as (cid:96)1\nand the maximum iterations is set as 30. Finally, we perform a binary search over q \u2208 [0, 1], where\nwe set \u03b4bin = 0.005 as the stopping criteria, to obtain the best robust subset (lowest adversarial risk)\nin CR(T ) with empirical measure at least \u03b1.\nResults. We choose \u03b1 to re\ufb02ect the best accuracy achieved by state-of-the-art classi\ufb01ers, using\n\u03b1 = 0.01 and \u0001\u221e \u2208 {0.1, 0.2, 0.3, 0.4} for MNIST and selecting appropriate values to represent the\nbest typical results on the other datasets (see Table 1). Given the number of hyperrectangles, T , we\nobtain the resulting error region using the proposed algorithm on the training dataset, and tune T for\nthe minimum adversarial risk on the testing dataset.\nFigure 1 shows the learning curves regarding risk and adversarial risk for two speci\ufb01c experimental\nsettings (similar results are obtained under other experimental settings, see Appendix C.3). Figure\n1(a) suggests that as we increase the initial covered percentage q, both risk and adversarial risk of the\ncorresponding error region decrease. This supports our use of binary search on q in Algorithm 1. On\n\n4The adversarial risk of an error region E simply refers to the adversarial risk of fE.\n\n7\n\n0%20%40%60%80%100%initial covered percentage q0%20%40%60%80%100%risk / adversarial riskrisk (training)risk (testing)adversarial risk (training)adversarial risk (testing)020406080number of hyperrectangles T0%10%20%30%adversarial riskmnist trainingmnist testingcifar trainingcifar testing\fTable 1: Summary of the main results using our method for different settings with (cid:96)\u221e perturbations.\n\nDataset\n\n\u03b1\n\nMNIST\n\n0.01\n\nCIFAR-10\n\n0.05\n\n\u0001\u221e\n\n0.1\n0.2\n0.3\n0.4\n\n2/255\n4/255\n8/255\n16/255\n\nT\n\n5\n10\n10\n10\n\n10\n20\n40\n75\n\nBest q\n\n0.662\n0.660\n0.629\n0.598\n\n0.680\n0.688\n0.734\n0.719\n\n1.22 \u00b1 0.11\n1.12 \u00b1 0.13\n1.12 \u00b1 0.12\n1.15 \u00b1 0.09\n5.32 \u00b1 0.21\n5.59 \u00b1 0.25\n5.55 \u00b1 0.21\n5.16 \u00b1 0.25\n\n1.23 \u00b1 0.12\n1.11 \u00b1 0.10\n1.15 \u00b1 0.13\n1.21 \u00b1 0.09\n5.72 \u00b1 0.25\n6.05 \u00b1 0.40\n5.94 \u00b1 0.34\n5.28 \u00b1 0.23\n\nEmpirical Risk (%)\ntraining\ntesting\n\nEmpirical AdvRisk (%)\ntraining\ntesting\n\n3.65 \u00b1 0.29\n5.76 \u00b1 0.38\n7.34 \u00b1 0.38\n9.89 \u00b1 0.57\n7.29 \u00b1 0.20\n11.43 \u00b1 0.24\n13.69 \u00b1 0.19\n19.77 \u00b1 0.22\n\n3.64 \u00b1 0.30\n5.89 \u00b1 0.44\n7.24 \u00b1 0.38\n9.92 \u00b1 0.60\n8.13 \u00b1 0.26\n13.66 \u00b1 0.33\n18.13 \u00b1 0.30\n28.83 \u00b1 0.46\n\nTable 2: Comparisons between our method and the existing adversarially trained robust classi\ufb01ers\nunder different settings. We use the Risk and AdvRisk for robust training methods to denote the\nstandard test error and attack success rate reported in literature. The AdvRisk reported for our method\ncan be seen as an estimated lower bound of adversarial risk for existing classi\ufb01ers.\n\nDataset\n\nStrength (metric)\n\nMethod\n\nEmpirical Risk\n\nEmpirical AdvRisk\n\nMNIST\n\nMNIST\n\n\u0001\u221e = 0.3\n\n\u00012 = 1.5\n\nCIFAR-10\n\n\u0001\u221e = 8/255\n\nMadry et al. (2018)\n\nOurs (T = 10, \u03b1 = 0.012)\n\n1.20%\n\n1.35% \u00b1 0.08%\n\n10.70%\n\n8.28% \u00b1 0.22%\n\nSchott et al. (2019)\n\nOurs (T = 20, \u03b1 = 0.01)\n\nMadry et al. (2018)\n\nOurs (T = 40, \u03b1 = 0.127)\n\n1.00%\n1.08%\n\n20.00%\n2.12%\n\n12.70%\n\n14.22% \u00b1 0.46%\n\n52.96%\n\n29.21% \u00b1 0.35%\n\nthe other hand, as can be seen from Figure 1(b), over\ufb01tting with respect to adversarial risk becomes\nsigni\ufb01cant as we increase the number of hyperrectangles. According to the adversarial risk curve for\ntesting data, the optimal value of T is selected as T = 10 for MNIST (\u0001\u221e = 0.3) and T = 40 for\nCIFAR-10 (\u0001\u221e = 8/255).\nTable 1 summarizes the optimal parameters, the empirical risk and adversarial risk of the correspond-\ning error region on both training and testing datasets for each experimental setting (see Appendix C.1\nfor similar results on Fashion-MNIST and SVHN). Since the k-means algorithm does not guarantee\nglobal optimum, we repeat our method for 10 runs with random restarts in terms of the best param-\neters, then report both the mean and the standard deviation. Our experiments provide examples of\nrather robust error regions for real image datasets. For instance, in Table 1 we have a case where\nthe measure of the resulting error region increases from 5.94% to 18.13% after expansion with\n\u0001\u221e = 8/255 on CIFAR-10 dataset. This means that there could potentially be a classi\ufb01er with 5.94%\nrisk and 18.13% adversarial risk, but the-state-of-the-art robust classi\ufb01er has empirically-measured\nadversarial risk 52.96% (Madry et al., 2018).\nNoticing that the risk lower threshold \u03b1 = 0.05 is much lower than the empirical risk 12.70% of the\nadversarially-trained robust model reported in Madry et al. (2018), we further measure the empirical\nconcentration on MNIST and CIFAR-10 using our method with \u03b1 set to be the same as the reported\nstandard test error in Madry et al. (2018), which is demonstrated in Table 2. In particular, we show\nthat the gap between the attack success rate of Madry et al.\u2019s classi\ufb01er (10.70%) and our estimated\nbest-achievable adversarial risk (8.28%) is quite small on MNIST, suggesting that the robustness of\nMadry et al.\u2019s classi\ufb01er is actually close to the intrinsic robustness. In sharp contrast, the gap becomes\nsigni\ufb01cantly larger on CIFAR-10: 29.21% for our estimate, while 52.96% for the reported attack\nsuccess rate in Madry et al. (2018). Regardless of the difference, this gap cannot be explained by\nthe concentration of measure phenomenon, suggesting there may still be room for developing more\nrobust classi\ufb01ers, or that other inherent reasons impede learning a more robust classi\ufb01er.\n\n8\n\n\fTable 3: Comparisons between different methods for \ufb01nding robust error region with (cid:96)2 perturbations.\n\nDataset\n\n\u03b1\n\n\u00012\n\nMNIST\n\n0.01\n\n1.58\n3.16\n4.74\n\nGilmer et al. (2018b)\nRisk\n1.18%\n1.18%\n1.18%\n\nAdvRisk\n3.92%\n9.73%\n23.40%\n\nCIFAR-10\n\n0.05\n\n0.2453\n0.4905\n0.9810\n\n5.27%\n5.27%\n5.27%\n\n5.58%\n5.93%\n6.47%\n\nOur Method\nAdvRisk\nRisk\n2.19%\n1.07%\n1.02%\n4.15%\n1.07% 10.09%\n\n5.16%\n5.14%\n5.12%\n\n5.53%\n5.83%\n6.56%\n\nT\n\n20\n20\n20\n\n5\n5\n5\n\n4.2 Experiments for (cid:96)2\n\nFor (cid:96)2 adversaries, Theorem 3.10 guarantees the asymptotic convergence of the empirical concentra-\ntion function characterized by union of balls B(T ) towards the actual concentration. Thus, it remains\nto solve the corresponding optimization problem (3). Similar to (cid:96)\u221e, we propose an empirical method\nto search for desirable robust error regions under (cid:96)2 perturbations. From a high level, our algorithm\n(for pseudocode, see Algorithm 2 in Appendix B) places T balls in a sequential manner, and searches\nfor the best possible placement using a greedy approach at each time. Since enumerating all the\npossible ball centers is infeasible, we restrict the choice of the center to be the set of training data\npoints. Our method keeps two sets of indices: one for the initial coverage and one for the coverage\nafter expansion, and updates them when we \ufb01nd the optimal placement, i.e. the ball centered at some\ntraining data point that has the minimum expansion with respect to both sets.\nWe compare our empirical method for \ufb01nding robust error regions characterized by a union of balls\nwith the hyperplane-based approach (Gilmer et al., 2018b) on MNIST and CIFAR-10. In particular,\nthe risk threshold \u03b1 is set to be the same as the case of (cid:96)\u221e, and the adversarial strength \u00012 is chosen\nsuch that the volume of an (cid:96)2 ball with radius \u00012 is roughly the same as the (cid:96)\u221e ball with radius \u0001\u221e,\n\nusing the conversion rule \u00012 =(cid:112)n/\u03c0 \u00b7 \u0001\u221e as in Wong et al. (2018). Table 3 summarizes the optimal\n\nparameters, the testing risk and adversarial risk (see Appendix C.2 for more detailed results, including\nfor other datasets) of the trained error regions using different methods, where we tune the number of\nballs T for our method.\nOur results show that there exist rather robust (cid:96)2 error regions for real image datasets. For example,\nthe measure of the resulting error region using our method only increases by 0.69% (from 5.14% to\n5.83%) after expansion with \u00012 = 0.4905 on CIFAR-10. Compared with Gilmer et al. (2018b), our\nmethod is able to \ufb01nd regions with signi\ufb01cantly smaller adversarial risk (around half the adversarial\nrisk of regions found by their method) on MNIST, while attaining comparable error region robustness\non CIFAR-10. Nevertheless, the adversarial risk attained by state-of-the-art robust classi\ufb01ers against\n(cid:96)2 perturbations is much higher than these reported rates (see Table 2 for a comparison with the best\nrobust classi\ufb01er against (cid:96)2 perturbations proposed in Schott et al. (2019)).\n\n5 Conclusion\n\nTo understand whether theoretical results showing limits of intrinsic robustness for natural distribu-\ntions apply to concrete datasets, we developed a general framework to measure the concentration\nof an unknown distribution through its i.i.d. samples and a carefully-selected collection of subsets.\nOur experimental results suggest that the concentration of measure phenomenon is not the sole\nreason behind vulnerability of the existing classi\ufb01ers to adversarial examples. In other words, recent\nimpossibility results (Gilmer et al., 2018b; Fawzi et al., 2018; Mahloujifar et al., 2019; Shafahi et al.,\n2019) should not cause us to lose hope in the possibility of \ufb01nding more robust classi\ufb01ers.\n\nAcknowledgements. This work was partially funded by an award from the National Science Founda-\ntion SaTC program (Center for Trustworth Machine Learning, #1804603), an NSF CAREER award\n(CCF-1350939), and support from Baidu, Intel, and Amazon.\n\n9\n\n\fReferences\nAnish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of\nsecurity: Circumventing defenses to adversarial examples. In International Conference on Machine\nLearning, 2018.\n\nIdan Attias, Aryeh Kontorovich, and Yishay Mansour. Improved generalization bounds for robust\n\nlearning. In Algorithmic Learning Theory, 2019.\n\nArjun Nitin Bhagoji, Daniel Cullina, and Prateek Mittal. Lower bounds on adversarial robustness\n\nfrom optimal transport. In Advances in Neural Information Processing Systems, 2019.\n\nBattista Biggio and Fabio Roli. Wild patterns: Ten years after the rise of adversarial machine learning.\n\nPattern Recognition, 84:317\u2013331, 2018.\n\nNicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In IEEE\n\nSymposium on Security and Privacy, 2017.\n\nDaniel Cullina, Arjun Nitin Bhagoji, and Prateek Mittal. PAC-learning in the presence of adversaries.\n\nIn Advances in Neural Information Processing Systems, 2018.\n\nLuc Devroye, L\u00e1szl\u00f3 Gy\u00f6r\ufb01, and G\u00e1bor Lugosi. A Probabilistic Theory of Pattern Recognition.\n\nSpringer Science & Business Media, 2013.\n\nDimitrios Diochnos, Saeed Mahloujifar, and Mohammad Mahmoody. Adversarial risk and robust-\nness: General de\ufb01nitions and implications for the uniform distribution. In Advances in Neural\nInformation Processing Systems, 2018.\n\nDavid Eisenstat and Dana Angluin. The VC dimension of k-fold union. Information Processing\n\nLetters, 101(5):181\u2013184, 2007.\n\nAlhussein Fawzi, Hamza Fawzi, and Omar Fawzi. Adversarial vulnerability for any classi\ufb01er. In\n\nAdvances in Neural Information Processing Systems, 2018.\n\nJustin Gilmer, Ryan P Adams, Ian Goodfellow, David Andersen, and George E Dahl. Motivating the\n\nrules of the game for adversarial example research. arXiv preprint arXiv:1807.06732, 2018a.\n\nJustin Gilmer, Luke Metz, Fartash Faghri, Samuel S Schoenholz, Maithra Raghu, Martin Wattenberg,\n\nand Ian Goodfellow. Adversarial spheres. arXiv preprint arXiv:1801.02774, 2018b.\n\nIan Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial\n\nexamples. In International Conference on Learning Representations, 2015.\n\nSven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan\nUesato, Relja Arandjelovic, Timothy Mann, and Pushmeet Kohli. Scalable veri\ufb01ed training for\nprovably robust image classi\ufb01cation. In IEEE International Conference on Computer Vision\n(ICCV), 2019.\n\nJohn A Hartigan and Manchek A Wong. A K-means clustering algorithm. Journal of the Royal\n\nStatistical Society. Series C (Applied Statistics), 28(1):100\u2013108, 1979.\n\nAlex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images.\n\nTechnical report, University of Toronto, 2009.\n\nRyen Krusinga, Sohil Shah, Matthias Zwicker, Tom Goldstein, and David Jacobs. Understanding\nthe (un)interpretability of natural image distributions using generative models. arXiv preprint\narXiv:1901.01499, 2019.\n\nYann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. http://yann.lecun.\n\ncom/exdb/mnist, 2010.\n\nMichel Ledoux. The Concentration of Measure Phenomenon. Number 89 in Mathematical Surveys and\n\nMonographs. American Mathematical Society, 2001.\n\nPaul L\u00e9vy. Probl\u00e8mes concrets d\u2019analyse fonctionnelle, volume 6. Gauthier-Villars Paris, 1951.\n\n10\n\n\fAleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep\nlearning models resistant to adversarial attacks. In International Conference on Learning Representations,\n2018.\n\nSaeed Mahloujifar, Dimitrios I Diochnos, and Mohammad Mahmoody. The curse of concentration in robust\nlearning: Evasion and poisoning attacks from concentration of measure. In AAAI Conference on Arti\ufb01cial\nIntelligence, 2019.\n\nVitali D Milman and Gideon Schechtman. Asymptotic theory of \ufb01nite dimensional normed spaces. Springer-\n\nVerlag, 1986.\n\nOmar Montasser, Steve Hanneke, and Nathan Srebro. VC classes are adversarially robustly learnable, but only\n\nimproperly. Proceedings of Machine Learning Research, 99:1\u201319, 2019.\n\nYuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in\nnatural images with unsupervised feature learning. In NeurIPS Workshop on Deep Learning and Unsupervised\nFeature Learning, 2011.\n\nBehnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generalization in deep\n\nlearning. In Advances in Neural Information Processing Systems, 2017.\n\nStephen M Omohundro. Five balltree construction algorithms.\n\nBerkeley, 1989.\n\nInternational Computer Science Institute\n\nNicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to\nadversarial perturbations against deep neural networks. In IEEE Symposium on Security and Privacy, 2016.\n\nFabian Pedregosa, Ga\u00ebl Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel,\nMathieu Blondel, Andreas M\u00fcller, Joel Nothman, Gilles Louppe, Peter Prettenhofer, Ron Weiss, Vincent\nDubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and\n\u00c9douard Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research,\nOctober 2011.\n\nAditi Raghunathan, Jacob Steinhardt, and Percy Liang. Certi\ufb01ed defenses against adversarial examples. In\n\nInternational Conference on Learning Representations, 2018.\n\nAditi Raghunathan, Sang Michael Xie, Fanny Yang, John C Duchi, and Percy Liang. Adversarial training can\n\nhurt generalization. arXiv preprint arXiv:1906.06032, 2019.\n\nLukas Schott, Jonas Rauber, Matthias Bethge, and Wieland Brendel. Towards the \ufb01rst adversarially robust neural\n\nnetwork model on MNIST. In International Conference on Learning Representations, 2019.\n\nClayton D Scott and Robert D Nowak. Learning minimum volume sets. Journal of Machine Learning Research,\n\n7(Apr):665\u2013704, 2006.\n\nAli Shafahi, W. Ronny Huang, Christoph Studer, Soheil Feizi, and Tom Goldstein. Are adversarial examples\n\ninevitable? In International Conference on Learning Representations, 2019.\n\nAman Sinha, Hongseok Namkoong, and John Duchi. Certi\ufb01able distributional robustness with principled\n\nadversarial training. In International Conference on Learning Representations, 2018.\n\nChristian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob\nFergus. Intriguing properties of neural networks. In International Conference on Learning Representations,\n2014.\n\nMichel Talagrand. Concentration of measure and isoperimetric inequalities in product spaces. Publications\n\nMath\u00e9matiques de l\u2019Institut des Hautes Etudes Scienti\ufb01ques, 81(1):73\u2013205, 1995.\n\nDimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness\n\nmay be at odds with accuracy. In International Conference on Learning Representations, 2019.\n\nShiqi Wang, Yizheng Chen, Ahmed Abdou, and Suman Jana. MixTrain: Scalable training of formally robust\n\nneural networks. arXiv preprint arXiv:1811.02625, 2018.\n\nEric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial\n\npolytope. In International Conference on Machine Learning, 2018.\n\nEric Wong, Frank R Schmidt, Jan Hendrik Metzen, and Zico Kolter. Scaling provable adversarial defenses. In\n\nAdvances in Neural Information Processing Systems, 2018.\n\n11\n\n\fHan Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a novel image dataset for benchmarking machine\n\nlearning algorithms. arXiv preprint arXiv:1708.07747, 2017.\n\nDong Yin, Ramchandran Kannan, and Peter Bartlett. Rademacher complexity for adversarially robust general-\n\nization. In International Conference on Machine Learning, 2019.\n\nHuan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Duane Boning, and Cho-Jui Hsieh. Towards stable and\n\nef\ufb01cient training of veri\ufb01ably robust neural networks. arXiv preprint arXiv:1906.06316, 2019.\n\n12\n\n\f", "award": [], "sourceid": 2823, "authors": [{"given_name": "Saeed", "family_name": "Mahloujifar", "institution": "University of Virginia"}, {"given_name": "Xiao", "family_name": "Zhang", "institution": "University of Virginia"}, {"given_name": "Mohammad", "family_name": "Mahmoody", "institution": "University of Virginia"}, {"given_name": "David", "family_name": "Evans", "institution": "University of Virginia"}]}