{"title": "Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks", "book": "Advances in Neural Information Processing Systems", "page_first": 11427, "page_last": 11438, "abstract": "Tight estimation of the Lipschitz constant for deep neural networks (DNNs) is useful in many applications ranging from robustness certification of classifiers to stability analysis of closed-loop systems with reinforcement learning controllers. Existing methods in the literature for estimating the Lipschitz constant suffer from either lack of accuracy or poor scalability. In this paper, we present a convex optimization framework to compute guaranteed upper bounds on the Lipschitz constant of DNNs both accurately and efficiently. Our main idea is to interpret activation functions as gradients of convex potential functions. Hence, they satisfy certain properties that can be described by quadratic constraints. This particular description allows us to pose the Lipschitz constant estimation problem as a semidefinite program (SDP). The resulting SDP can be adapted to increase either the estimation accuracy (by capturing the interaction between activation functions of different layers) or scalability (by decomposition and parallel implementation).  We illustrate the utility of our approach with a variety of experiments on randomly generated networks and on classifiers trained on the MNIST and Iris datasets. In particular, we experimentally demonstrate that our Lipschitz bounds are the most accurate compared to those in the literature. We also study the impact of adversarial training methods on the Lipschitz bounds of the resulting classifiers and show that our bounds can be used to efficiently provide robustness guarantees.", "full_text": "Ef\ufb01cient and Accurate Estimation of Lipschitz\n\nConstants for Deep Neural Networks\n\nMahyar Fazlyab\nESE Department\n\nUniversity of Pennsylvania\n\nPhiladephia , PA 19104\n\nmahyarfa@seas.upenn.edu\n\nAlexander Robey\nESE Department\n\nUniversity of Pennsylvania\n\nPhiladephia , PA 19104\n\narobey1@seas.upenn.edu\n\nHamed Hassani\nESE Department\n\nUniversity of Pennsylvania\n\nPhiladephia , PA 19104\n\nhassani@seas.upenn.edu\n\nManfred Morari\nESE Department\n\nUniversity of Pennsylvania\n\nPhiladephia , PA 19104\n\nmorari@seas.upenn.edu\n\nGeorge J. Pappas\nESE Department\n\nUniversity of Pennsylvania\n\nPhiladephia , PA 19104\n\npappasg@seas.upenn.edu\n\nAbstract\n\nTight estimation of the Lipschitz constant for deep neural networks (DNNs) is\nuseful in many applications ranging from robustness certi\ufb01cation of classi\ufb01ers to\nstability analysis of closed-loop systems with reinforcement learning controllers.\nExisting methods in the literature for estimating the Lipschitz constant suffer from\neither lack of accuracy or poor scalability. In this paper, we present a convex\noptimization framework to compute guaranteed upper bounds on the Lipschitz\nconstant of DNNs both accurately and ef\ufb01ciently. Our main idea is to interpret\nactivation functions as gradients of convex potential functions. Hence, they satisfy\ncertain properties that can be described by quadratic constraints. This particu-\nlar description allows us to pose the Lipschitz constant estimation problem as a\nsemide\ufb01nite program (SDP). The resulting SDP can be adapted to increase either\nthe estimation accuracy (by capturing the interaction between activation functions\nof different layers) or scalability (by decomposition and parallel implementation).\nWe illustrate the utility of our approach with a variety of experiments on randomly\ngenerated networks and on classi\ufb01ers trained on the MNIST and Iris datasets. In\nparticular, we experimentally demonstrate that our Lipschitz bounds are the most\naccurate compared to those in the literature. We also study the impact of adversarial\ntraining methods on the Lipschitz bounds of the resulting classi\ufb01ers and show that\nour bounds can be used to ef\ufb01ciently provide robustness guarantees.\n\nIntroduction\n\n1\nA function f : Rn \u2192 Rm is globally Lipschitz continuous on X \u2286 Rn if there exists a nonnegative\nconstant L \u2265 0 such that\n(1)\nThe smallest such L is called the Lipschitz constant of f. The Lipschitz constant is the maximum\nratio between variations in the output space and variations in the input space of f and thus is a\nmeasure of sensitivity of the function with respect to input perturbations.\nWhen a function f is characterized by a deep neural network (DNN), tight bounds on its Lipschitz\nconstant can be extremely useful in a variety of applications. In classi\ufb01cation tasks, for instance,\nL can be used as a certi\ufb01cate of robustness of a neural network classi\ufb01er to adversarial attacks if\n\n(cid:107)f (x) \u2212 f (y)(cid:107) \u2264 L(cid:107)x \u2212 y(cid:107) for all x, y \u2208 X .\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fit is estimated tightly [34]. In deep reinforcement learning, tight bounds on the Lipschitz constant\nof a DNN-based controller can be directly used to analyze the stability of the closed-loop system.\nLipschitz regularity can also play a key role in derivation of generalization bounds [6]. In these\napplications and many others, it is essential to have tight bounds on the Lipschitz constant of DNNs.\nHowever, as DNNs have highly complex and non-linear structures, estimating the Lipschitz constant\nboth accurately and ef\ufb01ciently has remained a signi\ufb01cant challenge.\nOur contributions. In this paper we propose a novel convex programming framework to derive tight\nbounds on the global Lipschitz constant of deep feed-forward neural networks. Our framework yields\nsigni\ufb01cantly more accurate bounds compared to the state-of-the-art and lends itself to a distributed\nimplementation, leading to ef\ufb01cient computation of the bounds for large-scale networks.\nOur approach. We use the fact that all common nonlinear activation functions used in neural\nnetworks are gradients of convex functions; hence, as operators, they satisfy certain properties that\ncan be abstracted as quadratic constraints on their input-output values. This particular abstraction\nallows us to pose the Lipschitz estimation problem as a semide\ufb01nite program (SDP), which we\ncall LipSDP. A striking feature of LipSDP is its \ufb02exibility to span the trade-off between estimation\naccuracy and computational ef\ufb01ciency by adding or removing extra decision variables. In particular,\nfor a neural network with (cid:96) layers and a total of n hidden neurons, the number of decision variables\ncan vary from (cid:96) (least accurate but most scalable) to O(n2) (most accurate but least scalable). As\nsuch, we derive several distinct yet related formulations of LipSDP that span this trade-off. To scale\neach variant of LipSDP to larger networks, we also propose a distributed implementation.\nOur results. We illustrate our approach in a variety of experiments on both randomly generated\nnetworks as well as networks trained on the MNIST [23] and Iris [11] datasets. First, we show\nempirically that our Lipschitz bounds are the most accurate compared to all other existing methods of\nwhich we are aware. In particular, our experiments on neural networks trained for MNIST show that\nour bounds outperform all comparable methods; see Figure 2a for details. Furthermore, we investigate\nthe effect of two robust training procedures [24, 40] on the Lipschitz constant for networks trained\non the MNIST dataset. Our results suggest that robust training procedures signi\ufb01cantly decrease the\nLipschitz constant of the resulting classi\ufb01ers. Moreover, we use the Lipschitz bound for two robust\ntraining procedures to derive non-vacuous lower bounds on the minimum adversarial perturbation\nnecessary to change the classi\ufb01cation of any instance from the test set. For details, see Figure 3.\nRelated work. The problem of estimating the Lipschitz constant for neural networks has been\nstudied in several works. In [34], the authors estimate the global Lipschitz constant of DNNs by the\nproduct of Lipschitz constants of individual layers. This approach is scalable and general but yields\ntrivial bounds. In [10], the authors derive bounds on Lipschitz constants by treating the activation\nfunctions as non-expansive averaged operators. The resulting algorithm scales well with the number\nof hidden units per layer, but exponentially with the number of layers. In [37], the authors decompose\nthe weight matrices of a neural network via singular value decomposition and approximately solve a\nconvex maximization problem over the unit cube. Notably, estimating the Lipschitz constant using\nthe method in [37] is intractable even for small networks; indeed, the authors of [37] use a greedy\nalgorithm to compute a bound, which may underapproximate the Lipschitz constant. In [2], the\nmaximum spectral norm of the network Jacobian (taken over the data distribution) is used as an\nestimate of the true Lipschitz constant. Again, this approach is not guaranteed to be an upper bound\non the Lipschitz constant. Bounding Lipschitz constants for the speci\ufb01c case of convolutional neural\nnetworks (CNNs) has also been addressed in [5, 44, 6].\nUsing Lipschitz bounds in the context of adversarial robustness and safety veri\ufb01cation has also been\naddressed in several works [39, 31, 38]. In particular, in [39], the authors convert the robustness\nanalysis problem into a local Lipschitz constant estimation problem, where they estimate this local\nconstant by a set of independently and identically sampled local gradients. This algorithm is scalable\nbut is not guaranteed to provide upper bounds. In a similar work, the authors of [38] exploit the\npiece-wise linear structure of ReLU functions to estimate the local Lipschitz constant of neural\nnetworks. In [13], the authors use quadratic constraints and semide\ufb01nite programming to analyze\nlocal (point-wise) robustness of neural networks. In contrast, our Lipschitz bounds can be used as a\nglobal certi\ufb01cate of robustness and are agnostic to the choice of the test data.\n\n2\n\n\f1.1 Motivating applications\n\nWe now enumerate two applications that highlight the importance of estimating the Lipschitz constant\nof DNNs accurately and ef\ufb01ciently.\nRobustness certi\ufb01cation of classi\ufb01ers. In response to fragility of DNNs to adversarial attacks,\nthere has been considerable effort in recent years to improve the robustness of neural networks\nagainst adversarial attacks and input perturbations [16, 26, 43, 22, 24, 40]. In order to certify and/or\nimprove the robustness of neural networks, one must be able to bound the possible outputs of the\nneural network over a region of input space. This can be done either locally around a speci\ufb01c\ninput [7, 35, 15, 33, 12, 29, 30, 13, 40, 21, 41, 42], or globally by bounding the sensitivity of the\nfunction to input perturbations, i.e., the Lipschitz constant [19, 34, 28, 39]. Indeed, tight upper bounds\non the Lipschitz constant can be used to derive non-vacuous lower bounds on the magnitudes of\nperturbations necessary to change the decision of neural networks. Finally, an ef\ufb01cient computation\nof these bounds can be useful in either assessing robustness after training [29, 30, 13] or promoting\nrobustness during training [40, 36, 17]. In the experiments section, we explore this application in\ndepth.\nStability analysis of closed-loop systems with learning controllers. A central problem in learning-\nbased control is to provide stability or safety guarantees for a feedback control loop when a learning-\nenabled component, such as a deep neural network, is introduced in the loop [4, 8, 20]. The Lipschitz\nconstant of a neural network controller bounds its gain. Therefore a tight estimate can be useful for\ncertifying the stability of the closed-loop system.\nNotation. We denote the set of real n-dimensional vectors by Rn, the set of m \u00d7 n-dimensional\nmatrices by Rm\u00d7n, and the n-dimensional identity matrix by In. We denote by Sn, Sn\n+, and Sn\n++\nthe sets of n-by-n symmetric, positive semide\ufb01nite, and positive de\ufb01nite matrices, respectively. The\np-norm (p \u2265 1) is denoted by (cid:107) \u00b7 (cid:107)p : Rn \u2192 R+. The (cid:96)2-norm of a matrix W \u2208 Rm\u00d7n is the largest\nsingular value of W . We denote the i-th unit vector in Rn by ei. We write diag(a1, ..., an) for a\ndiagonal matrix whose diagonal entries starting in the upper left corner are a1,\u00b7\u00b7\u00b7 , an.\n2 LipSDP: Lipschitz certi\ufb01cates via semide\ufb01nite programming\n\nx0 = x,\n\nf (x) = W (cid:96)x(cid:96) + b(cid:96).\n\nxk+1 = \u03c6(W kxk + bk) for k = 0,\u00b7\u00b7\u00b7 , (cid:96) \u2212 1,\n\n2.1 Problem statement\nConsider an (cid:96)-layer feed-forward neural network f (x) : Rn0 \u2192 Rn(cid:96)+1 described by the following\nrecursive equations:\n(2)\nHere x \u2208 Rn0 is an input to the network and W k \u2208 Rnk+1\u00d7nk and bk \u2208 Rnk+1 are the weight matrix\nand bias vector for the k-th layer. The function \u03c6 is the concatenation of activation functions at each\nlayer, i.e., it is of the form \u03c6(x) = [\u03d5(x1) \u00b7\u00b7\u00b7 \u03d5(xn)](cid:62). In this paper, our goal is to \ufb01nd tight bounds\non the Lipschitz constant of the map x (cid:55)\u2192 f (x) in (cid:96)2-norm. More precisely, we wish to \ufb01nd the\nsmallest constant L2 \u2265 0 such that (cid:107)f (x) \u2212 f (y)(cid:107)2 \u2264 L2(cid:107)x \u2212 y(cid:107)2 for all x, y \u2208 Rn0.\nThe main source of dif\ufb01culty in solving this problem is the presence of the nonlinear activation\nfunctions. To combat this dif\ufb01culty, our main idea is to abstract these activation functions by a set of\nconstraints that they impose on their input and output values. Then any property (including Lipschitz\ncontinuity) that is satis\ufb01ed by our abstraction will also be satis\ufb01ed by the original network.\n\n2.2 Description of activation functions by quadratic constraints\n\nIn this section, we introduce several de\ufb01nitions and lemmas that characterize our abstraction of\nnonlinear activation functions. These results are crucial to the formulation of an SDP that can bound\nthe Lipschitz constants of networks in Section 2.3.\nDe\ufb01nition 1 (Slope-restricted non-linearity) A function \u03d5 : R \u2192 R is slope-restricted on [\u03b1, \u03b2]\nwhere 0 \u2264 \u03b1 < \u03b2 < \u221e if\n\n\u03b1 \u2264\n\n\u03d5(y) \u2212 \u03d5(x)\n\ny \u2212 x\n\n\u2264 \u03b2 \u2200x, y \u2208 R.\n\n3\n\n(3)\n\n\finequality, \u03d5(x) is one-sided Lipschitz with parameter \u03b2. Altogether, the preceding inequalities state\n\nThe inequality in (3) simply states that the slope of the chord connecting any two points on the curve\nof the function x (cid:55)\u2192 \u03d5(x) is at least \u03b1 and at most \u03b2 (see Figure 1). By multiplying all sides of (3) by\n(y\u2212x)2, we can write the slope restriction condition as \u03b1(y\u2212x)2 \u2264 (\u03d5(y)\u2212\u03d5(x))(y\u2212x) \u2264 \u03b2(y\u2212x)2.\nBy the left inequality, the operator \u03d5(x) is strongly monotone with parameter \u03b1 [32], or equivalently\nthe anti-derivative function(cid:82) \u03d5(x)dx is strongly convex with parameter \u03b1. By the right-hand side\nthat the anti-derivative function(cid:82) \u03d5(x)dx is \u03b1-strongly convex and \u03b2-smooth.\nNote that, except for special cases [2], all common activation functions used in deep learning satisfy\nthe slope restriction condition in (3) for some 0 \u2264 \u03b1 < \u03b2 < \u221e. For instance, the ReLU, tanh, and\nsigmoid activation functions are all slope restricted with \u03b1 = 0 and \u03b2 = 1. More details can be found\nin [13].\nDe\ufb01nition 2 (Incremental Quadratic Constraint [1]) A function \u03c6 : Rn \u2192 Rn satis\ufb01es the incre-\nmental quadratic constraint de\ufb01ned by Q \u2282 S2n if for any Q \u2208 Q and x, y \u2208 Rn,\n\n(cid:20)\n\u03c6(x) \u2212 \u03c6(y)(cid:21)(cid:62)\n\nx \u2212 y\n\nQ(cid:20)\n\u03c6(x) \u2212 \u03c6(y)(cid:21) \u2265 0.\n\nx \u2212 y\n\n(4)\n\nIn the above de\ufb01nition, Q is the set of all multiplier matrices that characterize \u03c6,\nthe softmax operator \u03c6(x) =\nand is a convex cone by de\ufb01nition.\n((cid:80)n\ni=1 exp(xi))\u22121[exp(x1)\u00b7\u00b7\u00b7 exp(xn)](cid:62) is the gradient of the convex function \u03c8(x) =\nlog((cid:80)n\ni=1 exp(xi)). This function is smooth and strongly convex with paramters \u03b1 = 0 and \u03b2 = 1\n[9]. For this class of functions, it is known that the gradient function \u03c6(x) = \u2207\u03c8(x) satis\ufb01es the\nquadratic inequality [25]\n\nAs an example,\n\nx \u2212 y\n\n(cid:20)\n\u03c6(x) \u2212 \u03c6(y)(cid:21)(cid:62)(cid:20) \u22122\u03b1\u03b2In\n\n\u22122In (cid:21)(cid:20)\nTherefore, the softmax operator satis\ufb01es the incremental quadratic constraint de\ufb01ned by Q = {\u03bbM |\n\u03bb \u2265 0}, where M the middle matrix in the above inequality.\nTo see the connection between incremental quadratic constraints and slope-restricted nonlinearities,\nnote that (3) can be equivalently written as the single inequality\n\n\u03c6(x) \u2212 \u03c6(y)(cid:21) \u2265 0.\n\n(\u03b1 + \u03b2)In\n\n(\u03b1 + \u03b2)In\n\nx \u2212 y\n\n(5)\n\n(\n\n\u03d5(y) \u2212 \u03d5(x)\n\ny \u2212 x\n\n\u2212 \u03b1)(\n\n\u03d5(y) \u2212 \u03d5(x)\n\ny \u2212 x\n\n\u2212 \u03b2) \u2264 0.\n\nMultiplying through by (y \u2212 x)2 and rearranging terms, we can write (6) as\n\u03d5(x) \u2212 \u03d5(y)(cid:21) \u2265 0,\n\n\u03d5(x) \u2212 \u03d5(y)(cid:21)(cid:62)(cid:20)\u22122\u03b1\u03b2 \u03b1 + \u03b2\n(cid:20)\n\u22122 (cid:21)(cid:20)\n\nx \u2212 y\n\nx \u2212 y\n\n\u03b1 + \u03b2\n\n(6)\n\n(7)\n\nwhich, in view of De\ufb01nition 2, is an incremental quadratic constraint for \u03d5. From this perspective,\nincremental quadratic constraints generalize the notion of slope-restricted nonlinearities to multi-\nvariable vector-valued nonlinearities.\nRepeated nonlinearities. Now consider the vector-valued function \u03c6(x) = [\u03d5(x1)\u00b7\u00b7\u00b7 \u03d5(xn)](cid:62)\nobtained by applying a slope-restricted function \u03d5 component-wise to a vector x \u2208 Rn. By exploiting\nthe fact that the same function \u03d5 is applied to each component, we can characterize \u03c6(x) by O(n2)\nincremental quadratic constraints. In the following lemma, we provide such a characterization.\nLemma 1 Suppose \u03d5 : R \u2192 R is slope-restricted on [\u03b1, \u03b2]. De\ufb01ne the set\n\nTn = {T \u2208 Sn | T =\n\nn(cid:88)i=1\n\n\u03bbiieie(cid:62)i + (cid:88)1\u2264i<j\u2264n\n\n\u03bbij(ei \u2212 ej)(ei \u2212 ej)(cid:62), \u03bbij \u2265 0}.\n\nThen for any T \u2208 Tn the vector-valued function \u03c6(x) = [\u03d5(x1)\u00b7\u00b7\u00b7 \u03d5(xn)](cid:62) : Rn \u2192 Rn satis\ufb01es\n\n(cid:20)\n\u03c6(x) \u2212 \u03c6(y)(cid:21)(cid:62)(cid:20) \u22122\u03b1\u03b2T\n\nx \u2212 y\n\n(\u03b1 + \u03b2)T\n\n(\u03b1 + \u03b2)T\n\n\u22122T (cid:21)(cid:20)\n\n\u03c6(x) \u2212 \u03c6(y)(cid:21) \u2265 0 for all x, y \u2208 Rn.\n\nx \u2212 y\n\n(8)\n\n(9)\n\n4\n\n\fFigure 1: An illustrative description of encoding activation functions by quadratic constraints.\n\nConcretely, this lemma captures the coupling between neurons in a neural network by taking advantage\nof two particular structures: (a) the same activation function is applied to each hidden neuron and\n(b) all activation functions are slope-restricted on the same interval [\u03b1, \u03b2]. In this way, we can write\nthe slope restriction condition in (1) for any pair of activation functions in a given neural network.\nA conic combination of these constraints would yield (9), where \u03bbij are the coef\ufb01cients of this\ncombination. See Figure 1 for an illustrative description.\nWe will see in the next section that the matrix T that parameterizes the multiplier matrix in (9) appears\nas a decision variable in an SDP, in which the objective is to \ufb01nd an admissible T that yields the\ntightest bound on the Lipschitz constant.\n\n2.3 LipSDP for single-layer neural network\n\nTo develop an optimization problem to estimate the Lipschitz constant of a fully-connected feed-\nforward neural network, the key insight is that the Lipschitz condition in (1) is in fact equivalent\nto an incremental quadratic constraint for the map x (cid:55)\u2192 f (x) characterized by the neural network.\nBy coupling this to the incremental quadratic constraints satis\ufb01ed by the cascade combination of\nthe activation functions [14], we can develop an SDP to minimize an upper bound on the Lipschitz\nconstant of f. This result is formally stated in the following theorem.\nTheorem 1 (Lipshitz certi\ufb01cates for single-layer neural networks) Consider a single-layer neu-\nral network described by f (x) = W 1\u03c6(W 0x + b0) + b1. Suppose \u03c6(x) : Rn \u2192 Rn =\n[\u03d5(x1)\u00b7\u00b7\u00b7 \u03d5(xn)], where \u03d5 is slope-restricted in the sector [\u03b1, \u03b2]. De\ufb01ne Tn as in (8). Suppose there\nexists a \u03c1 > 0 such that the matrix inequality\n\nM (\u03c1, T ) :=(cid:34)\u22122\u03b1\u03b2W 0(cid:62)T W 0 \u2212 \u03c1In0\n\n\u22122T + W 1(cid:62)W 1(cid:35) (cid:22) 0,\nholds for some T \u2208 Tn. Then (cid:107)f (x) \u2212 f (y)(cid:107)2 \u2264 \u221a\u03c1(cid:107)x \u2212 y(cid:107)2 for all x, y \u2208 Rn0.\nTheorem 1 provides us with a suf\ufb01cient condition for L2 = \u221a\u03c1 to be an upper bound on the Lipschitz\nconstant of f (x) = W 1\u03c6(W 0x + b0) + b1. In particular, we can \ufb01nd the tightest bound by solving\nthe following optimization problem:\n\n(\u03b1 + \u03b2)W 0(cid:62)T\n\n(\u03b1 + \u03b2)T W 0\n\n(10)\n\nminimize \u03c1\n\n(11)\nwhere the decision variables are (\u03c1, T ) \u2208 R+ \u00d7 Tn. Note that M (\u03c1, T ) is linear in \u03c1 and T and the\nset Tn is convex. Hence, (11) is an SDP, which can be solved numerically for its global minimum.\n2.4 LipSDP for multi-layer neural networks\n\nsubject to M (\u03c1, T ) (cid:22) 0\n\nand T \u2208 Tn,\n\nWe now consider the multi-layer case. Assuming that all the activation functions are the same, we\ncan write the neural network model in (2) compactly as\n\nBx = \u03c6(Ax + b)\n\nand\n\nf (x) = Cx + b(cid:96),\n\n(12)\n\n5\n\n-505-1-0.500.51(xj,'(xj))<latexit sha1_base64=\"7KVgF9Hepxz+AlzTMESMIJp61E0=\">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQUpSTe6LLhxWcFeoA1hMp20YyeTMDMp1lJ8FTcuFHHre7jzbZy0WWjrDwMf/zmHc+b3Y0alsu1vI7e2vrG5ld8u7Ozu7R+Yh0ctGSUCkyaOWCQ6PpKEUU6aiipGOrEgKPQZafuj67TeHhMhacTv1CQmbogGnAYUI6UtzzwplR+8+4veGIl4SFOuVEqeWbSr9lzWKjgZFCFTwzO/ev0IJyHhCjMkZdexY+VOkVAUMzIr9BJJYoRHaEC6GjkKiXSn8+tn1rl2+lYQCf24subu74kpCqWchL7uDJEayuVaav5X6yYquHKnlMeJIhwvFgUJs1RkpVFYfSoIVmyiAWFB9a0WHiKBsNKBFXQIzvKXV6FVqzqab2vFej2LIw+ncAZlcOAS6nADDWgChkd4hld4M56MF+Pd+Fi05oxs5hj+yPj8AXzPk+0=</latexit><latexit sha1_base64=\"7KVgF9Hepxz+AlzTMESMIJp61E0=\">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQUpSTe6LLhxWcFeoA1hMp20YyeTMDMp1lJ8FTcuFHHre7jzbZy0WWjrDwMf/zmHc+b3Y0alsu1vI7e2vrG5ld8u7Ozu7R+Yh0ctGSUCkyaOWCQ6PpKEUU6aiipGOrEgKPQZafuj67TeHhMhacTv1CQmbogGnAYUI6UtzzwplR+8+4veGIl4SFOuVEqeWbSr9lzWKjgZFCFTwzO/ev0IJyHhCjMkZdexY+VOkVAUMzIr9BJJYoRHaEC6GjkKiXSn8+tn1rl2+lYQCf24subu74kpCqWchL7uDJEayuVaav5X6yYquHKnlMeJIhwvFgUJs1RkpVFYfSoIVmyiAWFB9a0WHiKBsNKBFXQIzvKXV6FVqzqab2vFej2LIw+ncAZlcOAS6nADDWgChkd4hld4M56MF+Pd+Fi05oxs5hj+yPj8AXzPk+0=</latexit><latexit sha1_base64=\"7KVgF9Hepxz+AlzTMESMIJp61E0=\">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQUpSTe6LLhxWcFeoA1hMp20YyeTMDMp1lJ8FTcuFHHre7jzbZy0WWjrDwMf/zmHc+b3Y0alsu1vI7e2vrG5ld8u7Ozu7R+Yh0ctGSUCkyaOWCQ6PpKEUU6aiipGOrEgKPQZafuj67TeHhMhacTv1CQmbogGnAYUI6UtzzwplR+8+4veGIl4SFOuVEqeWbSr9lzWKjgZFCFTwzO/ev0IJyHhCjMkZdexY+VOkVAUMzIr9BJJYoRHaEC6GjkKiXSn8+tn1rl2+lYQCf24subu74kpCqWchL7uDJEayuVaav5X6yYquHKnlMeJIhwvFgUJs1RkpVFYfSoIVmyiAWFB9a0WHiKBsNKBFXQIzvKXV6FVqzqab2vFej2LIw+ncAZlcOAS6nADDWgChkd4hld4M56MF+Pd+Fi05oxs5hj+yPj8AXzPk+0=</latexit><latexit sha1_base64=\"7KVgF9Hepxz+AlzTMESMIJp61E0=\">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQUpSTe6LLhxWcFeoA1hMp20YyeTMDMp1lJ8FTcuFHHre7jzbZy0WWjrDwMf/zmHc+b3Y0alsu1vI7e2vrG5ld8u7Ozu7R+Yh0ctGSUCkyaOWCQ6PpKEUU6aiipGOrEgKPQZafuj67TeHhMhacTv1CQmbogGnAYUI6UtzzwplR+8+4veGIl4SFOuVEqeWbSr9lzWKjgZFCFTwzO/ev0IJyHhCjMkZdexY+VOkVAUMzIr9BJJYoRHaEC6GjkKiXSn8+tn1rl2+lYQCf24subu74kpCqWchL7uDJEayuVaav5X6yYquHKnlMeJIhwvFgUJs1RkpVFYfSoIVmyiAWFB9a0WHiKBsNKBFXQIzvKXV6FVqzqab2vFej2LIw+ncAZlcOAS6nADDWgChkd4hld4M56MF+Pd+Fi05oxs5hj+yPj8AXzPk+0=</latexit>(xi,'(xi))<latexit sha1_base64=\"GllpdvBtNW9LH6e2gGOX+rXlP0k=\">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQWRpBtdFty4rGAv0IYwmU7aoZNJmJkUayi+ihsXirj1Pdz5Nk7aLLT1h4GP/5zDOfP7MaNS2fa3UVhb39jcKm6Xdnb39g/Mw6O2jBKBSQtHLBJdH0nCKCctRRUj3VgQFPqMdPzxTVbvTIiQNOL3ahoTN0RDTgOKkdKWZ55Uqg8evehPkIhHNONareKZZfvSnstaBSeHMuRqeuZXfxDhJCRcYYak7Dl2rNwUCUUxI7NSP5EkRniMhqSnkaOQSDedXz+zzrUzsIJI6MeVNXd/T6QolHIa+rozRGokl2uZ+V+tl6jg2k0pjxNFOF4sChJmqcjKorAGVBCs2FQDwoLqWy08QgJhpQMr6RCc5S+vQrt+6Wi+q5cbjTyOIpzCGVTBgStowC00oQUYHuEZXuHNeDJejHfjY9FaMPKZY/gj4/MHebWT6w==</latexit><latexit sha1_base64=\"GllpdvBtNW9LH6e2gGOX+rXlP0k=\">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQWRpBtdFty4rGAv0IYwmU7aoZNJmJkUayi+ihsXirj1Pdz5Nk7aLLT1h4GP/5zDOfP7MaNS2fa3UVhb39jcKm6Xdnb39g/Mw6O2jBKBSQtHLBJdH0nCKCctRRUj3VgQFPqMdPzxTVbvTIiQNOL3ahoTN0RDTgOKkdKWZ55Uqg8evehPkIhHNONareKZZfvSnstaBSeHMuRqeuZXfxDhJCRcYYak7Dl2rNwUCUUxI7NSP5EkRniMhqSnkaOQSDedXz+zzrUzsIJI6MeVNXd/T6QolHIa+rozRGokl2uZ+V+tl6jg2k0pjxNFOF4sChJmqcjKorAGVBCs2FQDwoLqWy08QgJhpQMr6RCc5S+vQrt+6Wi+q5cbjTyOIpzCGVTBgStowC00oQUYHuEZXuHNeDJejHfjY9FaMPKZY/gj4/MHebWT6w==</latexit><latexit sha1_base64=\"GllpdvBtNW9LH6e2gGOX+rXlP0k=\">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQWRpBtdFty4rGAv0IYwmU7aoZNJmJkUayi+ihsXirj1Pdz5Nk7aLLT1h4GP/5zDOfP7MaNS2fa3UVhb39jcKm6Xdnb39g/Mw6O2jBKBSQtHLBJdH0nCKCctRRUj3VgQFPqMdPzxTVbvTIiQNOL3ahoTN0RDTgOKkdKWZ55Uqg8evehPkIhHNONareKZZfvSnstaBSeHMuRqeuZXfxDhJCRcYYak7Dl2rNwUCUUxI7NSP5EkRniMhqSnkaOQSDedXz+zzrUzsIJI6MeVNXd/T6QolHIa+rozRGokl2uZ+V+tl6jg2k0pjxNFOF4sChJmqcjKorAGVBCs2FQDwoLqWy08QgJhpQMr6RCc5S+vQrt+6Wi+q5cbjTyOIpzCGVTBgStowC00oQUYHuEZXuHNeDJejHfjY9FaMPKZY/gj4/MHebWT6w==</latexit><latexit sha1_base64=\"GllpdvBtNW9LH6e2gGOX+rXlP0k=\">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQWRpBtdFty4rGAv0IYwmU7aoZNJmJkUayi+ihsXirj1Pdz5Nk7aLLT1h4GP/5zDOfP7MaNS2fa3UVhb39jcKm6Xdnb39g/Mw6O2jBKBSQtHLBJdH0nCKCctRRUj3VgQFPqMdPzxTVbvTIiQNOL3ahoTN0RDTgOKkdKWZ55Uqg8evehPkIhHNONareKZZfvSnstaBSeHMuRqeuZXfxDhJCRcYYak7Dl2rNwUCUUxI7NSP5EkRniMhqSnkaOQSDedXz+zzrUzsIJI6MeVNXd/T6QolHIa+rozRGokl2uZ+V+tl6jg2k0pjxNFOF4sChJmqcjKorAGVBCs2FQDwoLqWy08QgJhpQMr6RCc5S+vQrt+6Wi+q5cbjTyOIpzCGVTBgStowC00oQUYHuEZXuHNeDJejHfjY9FaMPKZY/gj4/MHebWT6w==</latexit>\u21b5\uf8ff'(xj)'(xi)xjxi\uf8ff<latexit sha1_base64=\"bXpVOSwBhezioWcgI+XxN1XSrkk=\">AAACLHicbZDLSgMxFIYzXmu9jbp0E6yCLiwz3eiy0I3LCrYVOqWcSc/Y2MzFJFMsQx/Ija8iiAuLuPU5TNsBrz8EvvznHJLz+4ngSjvOxFpYXFpeWS2sFdc3Nre27Z3dpopTybDBYhHLax8UCh5hQ3Mt8DqRCKEvsOUPatN6a4hS8Ti60qMEOyHcRDzgDLSxunbt0AOR9IF6Au+o1wsksMwbgkz6/Pi+e3ty+nXhJ+PMWKeGxnm/jxoOu3bJKTsz0b/g5lAiuepd+9nrxSwNMdJMgFJt10l0JwOpORM4LnqpwgTYAG6wbTCCEFUnmy07pkfG6dEgluZEms7c7xMZhEqNQt90hqD76ndtav5Xa6c6OO9kPEpSjRGbPxSkguqYTpOjPS6RaTEyAExy81fK+mDy0ibfognB/b3yX2hWyq7hy0qpWs3jKJB9ckCOiUvOSJVckDppEEYeyBN5JRPr0Xqx3qz3eeuClc/skR+yPj4Be/+nuQ==</latexit><latexit sha1_base64=\"bXpVOSwBhezioWcgI+XxN1XSrkk=\">AAACLHicbZDLSgMxFIYzXmu9jbp0E6yCLiwz3eiy0I3LCrYVOqWcSc/Y2MzFJFMsQx/Ija8iiAuLuPU5TNsBrz8EvvznHJLz+4ngSjvOxFpYXFpeWS2sFdc3Nre27Z3dpopTybDBYhHLax8UCh5hQ3Mt8DqRCKEvsOUPatN6a4hS8Ti60qMEOyHcRDzgDLSxunbt0AOR9IF6Au+o1wsksMwbgkz6/Pi+e3ty+nXhJ+PMWKeGxnm/jxoOu3bJKTsz0b/g5lAiuepd+9nrxSwNMdJMgFJt10l0JwOpORM4LnqpwgTYAG6wbTCCEFUnmy07pkfG6dEgluZEms7c7xMZhEqNQt90hqD76ndtav5Xa6c6OO9kPEpSjRGbPxSkguqYTpOjPS6RaTEyAExy81fK+mDy0ibfognB/b3yX2hWyq7hy0qpWs3jKJB9ckCOiUvOSJVckDppEEYeyBN5JRPr0Xqx3qz3eeuClc/skR+yPj4Be/+nuQ==</latexit><latexit sha1_base64=\"bXpVOSwBhezioWcgI+XxN1XSrkk=\">AAACLHicbZDLSgMxFIYzXmu9jbp0E6yCLiwz3eiy0I3LCrYVOqWcSc/Y2MzFJFMsQx/Ija8iiAuLuPU5TNsBrz8EvvznHJLz+4ngSjvOxFpYXFpeWS2sFdc3Nre27Z3dpopTybDBYhHLax8UCh5hQ3Mt8DqRCKEvsOUPatN6a4hS8Ti60qMEOyHcRDzgDLSxunbt0AOR9IF6Au+o1wsksMwbgkz6/Pi+e3ty+nXhJ+PMWKeGxnm/jxoOu3bJKTsz0b/g5lAiuepd+9nrxSwNMdJMgFJt10l0JwOpORM4LnqpwgTYAG6wbTCCEFUnmy07pkfG6dEgluZEms7c7xMZhEqNQt90hqD76ndtav5Xa6c6OO9kPEpSjRGbPxSkguqYTpOjPS6RaTEyAExy81fK+mDy0ibfognB/b3yX2hWyq7hy0qpWs3jKJB9ckCOiUvOSJVckDppEEYeyBN5JRPr0Xqx3qz3eeuClc/skR+yPj4Be/+nuQ==</latexit><latexit sha1_base64=\"bXpVOSwBhezioWcgI+XxN1XSrkk=\">AAACLHicbZDLSgMxFIYzXmu9jbp0E6yCLiwz3eiy0I3LCrYVOqWcSc/Y2MzFJFMsQx/Ija8iiAuLuPU5TNsBrz8EvvznHJLz+4ngSjvOxFpYXFpeWS2sFdc3Nre27Z3dpopTybDBYhHLax8UCh5hQ3Mt8DqRCKEvsOUPatN6a4hS8Ti60qMEOyHcRDzgDLSxunbt0AOR9IF6Au+o1wsksMwbgkz6/Pi+e3ty+nXhJ+PMWKeGxnm/jxoOu3bJKTsz0b/g5lAiuepd+9nrxSwNMdJMgFJt10l0JwOpORM4LnqpwgTYAG6wbTCCEFUnmy07pkfG6dEgluZEms7c7xMZhEqNQt90hqD76ndtav5Xa6c6OO9kPEpSjRGbPxSkguqYTpOjPS6RaTEyAExy81fK+mDy0ibfognB/b3yX2hWyq7hy0qpWs3jKJB9ckCOiUvOSJVckDppEEYeyBN5JRPr0Xqx3qz3eeuClc/skR+yPj4Be/+nuQ==</latexit>xi<latexit sha1_base64=\"pmg+gnVv+HU4HLbjrH6Vq/vhhsQ=\">AAAB7HicbZA9TwJBEIbn8AvxC7W02QgmVuSORkoSG0tM5COBC9lb9mDD3t5ld85ILvwGGwuNsfUH2flvXOAKBd9kkyfvzGRn3iCRwqDrfjuFre2d3b3ifung8Oj4pHx61jFxqhlvs1jGuhdQw6VQvI0CJe8lmtMokLwbTG8X9e4j10bE6gFnCfcjOlYiFIyitdrVp6GoDssVt+YuRTbBy6ECuVrD8tdgFLM04gqZpMb0PTdBP6MaBZN8XhqkhieUTemY9y0qGnHjZ8tl5+TKOiMSxto+hWTp/p7IaGTMLApsZ0RxYtZrC/O/Wj/FsOFnQiUpcsVWH4WpJBiTxeVkJDRnKGcWKNPC7krYhGrK0OZTsiF46ydvQqde8yzf1yvNRh5HES7gEq7Bgxtowh20oA0MBDzDK7w5ynlx3p2PVWvByWfO4Y+czx8VwY4o</latexit><latexit sha1_base64=\"pmg+gnVv+HU4HLbjrH6Vq/vhhsQ=\">AAAB7HicbZA9TwJBEIbn8AvxC7W02QgmVuSORkoSG0tM5COBC9lb9mDD3t5ld85ILvwGGwuNsfUH2flvXOAKBd9kkyfvzGRn3iCRwqDrfjuFre2d3b3ifung8Oj4pHx61jFxqhlvs1jGuhdQw6VQvI0CJe8lmtMokLwbTG8X9e4j10bE6gFnCfcjOlYiFIyitdrVp6GoDssVt+YuRTbBy6ECuVrD8tdgFLM04gqZpMb0PTdBP6MaBZN8XhqkhieUTemY9y0qGnHjZ8tl5+TKOiMSxto+hWTp/p7IaGTMLApsZ0RxYtZrC/O/Wj/FsOFnQiUpcsVWH4WpJBiTxeVkJDRnKGcWKNPC7krYhGrK0OZTsiF46ydvQqde8yzf1yvNRh5HES7gEq7Bgxtowh20oA0MBDzDK7w5ynlx3p2PVWvByWfO4Y+czx8VwY4o</latexit><latexit sha1_base64=\"pmg+gnVv+HU4HLbjrH6Vq/vhhsQ=\">AAAB7HicbZA9TwJBEIbn8AvxC7W02QgmVuSORkoSG0tM5COBC9lb9mDD3t5ld85ILvwGGwuNsfUH2flvXOAKBd9kkyfvzGRn3iCRwqDrfjuFre2d3b3ifung8Oj4pHx61jFxqhlvs1jGuhdQw6VQvI0CJe8lmtMokLwbTG8X9e4j10bE6gFnCfcjOlYiFIyitdrVp6GoDssVt+YuRTbBy6ECuVrD8tdgFLM04gqZpMb0PTdBP6MaBZN8XhqkhieUTemY9y0qGnHjZ8tl5+TKOiMSxto+hWTp/p7IaGTMLApsZ0RxYtZrC/O/Wj/FsOFnQiUpcsVWH4WpJBiTxeVkJDRnKGcWKNPC7krYhGrK0OZTsiF46ydvQqde8yzf1yvNRh5HES7gEq7Bgxtowh20oA0MBDzDK7w5ynlx3p2PVWvByWfO4Y+czx8VwY4o</latexit><latexit sha1_base64=\"pmg+gnVv+HU4HLbjrH6Vq/vhhsQ=\">AAAB7HicbZA9TwJBEIbn8AvxC7W02QgmVuSORkoSG0tM5COBC9lb9mDD3t5ld85ILvwGGwuNsfUH2flvXOAKBd9kkyfvzGRn3iCRwqDrfjuFre2d3b3ifung8Oj4pHx61jFxqhlvs1jGuhdQw6VQvI0CJe8lmtMokLwbTG8X9e4j10bE6gFnCfcjOlYiFIyitdrVp6GoDssVt+YuRTbBy6ECuVrD8tdgFLM04gqZpMb0PTdBP6MaBZN8XhqkhieUTemY9y0qGnHjZ8tl5+TKOiMSxto+hWTp/p7IaGTMLApsZ0RxYtZrC/O/Wj/FsOFnQiUpcsVWH4WpJBiTxeVkJDRnKGcWKNPC7krYhGrK0OZTsiF46ydvQqde8yzf1yvNRh5HES7gEq7Bgxtowh20oA0MBDzDK7w5ynlx3p2PVWvByWfO4Y+czx8VwY4o</latexit>'(xi)<latexit sha1_base64=\"2RSP5OM1rKvlHPSZBvlymYbyN3E=\">AAAB9XicbZDLTgIxFIbP4A3xhrp00wgmuCEzbHRJ4sYlJnJJYCSd0oGGTjtpOyiZ8B5uXGiMW9/FnW9jgVko+CdNvvznnJzTP4g508Z1v53cxubW9k5+t7C3f3B4VDw+aWmZKEKbRHKpOgHWlDNBm4YZTjuxojgKOG0H45t5vT2hSjMp7s00pn6Eh4KFjGBjrYdyb4JVPGKVpz67LPeLJbfqLoTWwcugBJka/eJXbyBJElFhCMdadz03Nn6KlWGE01mhl2gaYzLGQ9q1KHBEtZ8urp6hC+sMUCiVfcKghft7IsWR1tMosJ0RNiO9Wpub/9W6iQmv/ZSJODFUkOWiMOHISDSPAA2YosTwqQVMFLO3IjLCChNjgyrYELzVL69Dq1b1LN/VSvV6FkcezuAcKuDBFdThFhrQBAIKnuEV3pxH58V5dz6WrTknmzmFP3I+fwBh5JHB</latexit><latexit sha1_base64=\"2RSP5OM1rKvlHPSZBvlymYbyN3E=\">AAAB9XicbZDLTgIxFIbP4A3xhrp00wgmuCEzbHRJ4sYlJnJJYCSd0oGGTjtpOyiZ8B5uXGiMW9/FnW9jgVko+CdNvvznnJzTP4g508Z1v53cxubW9k5+t7C3f3B4VDw+aWmZKEKbRHKpOgHWlDNBm4YZTjuxojgKOG0H45t5vT2hSjMp7s00pn6Eh4KFjGBjrYdyb4JVPGKVpz67LPeLJbfqLoTWwcugBJka/eJXbyBJElFhCMdadz03Nn6KlWGE01mhl2gaYzLGQ9q1KHBEtZ8urp6hC+sMUCiVfcKghft7IsWR1tMosJ0RNiO9Wpub/9W6iQmv/ZSJODFUkOWiMOHISDSPAA2YosTwqQVMFLO3IjLCChNjgyrYELzVL69Dq1b1LN/VSvV6FkcezuAcKuDBFdThFhrQBAIKnuEV3pxH58V5dz6WrTknmzmFP3I+fwBh5JHB</latexit><latexit sha1_base64=\"2RSP5OM1rKvlHPSZBvlymYbyN3E=\">AAAB9XicbZDLTgIxFIbP4A3xhrp00wgmuCEzbHRJ4sYlJnJJYCSd0oGGTjtpOyiZ8B5uXGiMW9/FnW9jgVko+CdNvvznnJzTP4g508Z1v53cxubW9k5+t7C3f3B4VDw+aWmZKEKbRHKpOgHWlDNBm4YZTjuxojgKOG0H45t5vT2hSjMp7s00pn6Eh4KFjGBjrYdyb4JVPGKVpz67LPeLJbfqLoTWwcugBJka/eJXbyBJElFhCMdadz03Nn6KlWGE01mhl2gaYzLGQ9q1KHBEtZ8urp6hC+sMUCiVfcKghft7IsWR1tMosJ0RNiO9Wpub/9W6iQmv/ZSJODFUkOWiMOHISDSPAA2YosTwqQVMFLO3IjLCChNjgyrYELzVL69Dq1b1LN/VSvV6FkcezuAcKuDBFdThFhrQBAIKnuEV3pxH58V5dz6WrTknmzmFP3I+fwBh5JHB</latexit><latexit sha1_base64=\"2RSP5OM1rKvlHPSZBvlymYbyN3E=\">AAAB9XicbZDLTgIxFIbP4A3xhrp00wgmuCEzbHRJ4sYlJnJJYCSd0oGGTjtpOyiZ8B5uXGiMW9/FnW9jgVko+CdNvvznnJzTP4g508Z1v53cxubW9k5+t7C3f3B4VDw+aWmZKEKbRHKpOgHWlDNBm4YZTjuxojgKOG0H45t5vT2hSjMp7s00pn6Eh4KFjGBjrYdyb4JVPGKVpz67LPeLJbfqLoTWwcugBJka/eJXbyBJElFhCMdadz03Nn6KlWGE01mhl2gaYzLGQ9q1KHBEtZ8urp6hC+sMUCiVfcKghft7IsWR1tMosJ0RNiO9Wpub/9W6iQmv/ZSJODFUkOWiMOHISDSPAA2YosTwqQVMFLO3IjLCChNjgyrYELzVL69Dq1b1LN/VSvV6FkcezuAcKuDBFdThFhrQBAIKnuEV3pxH58V5dz6WrTknmzmFP3I+fwBh5JHB</latexit>'(yi)<latexit sha1_base64=\"RG1iAtaTSYwavALhRg+orYL4LFU=\">AAAB9XicbZDLTgIxFIbPeEW8oS7dNIIJbsgMG12SuHGJiVwSGEmndKCh02naDmYy4T3cuNAYt76LO9/GArNQ8E+afPnPOTmnfyA508Z1v52Nza3tnd3CXnH/4PDouHRy2tZxoghtkZjHqhtgTTkTtGWY4bQrFcVRwGknmNzO650pVZrF4sGkkvoRHgkWMoKNtR4r/SlWcsyq6YBdVQalsltzF0Lr4OVQhlzNQemrP4xJElFhCMda9zxXGj/DyjDC6azYTzSVmEzwiPYsChxR7WeLq2fo0jpDFMbKPmHQwv09keFI6zQKbGeEzViv1ubmf7VeYsIbP2NCJoYKslwUJhyZGM0jQEOmKDE8tYCJYvZWRMZYYWJsUEUbgrf65XVo12ue5ft6udHI4yjAOVxAFTy4hgbcQRNaQEDBM7zCm/PkvDjvzseydcPJZ87gj5zPH2NskcI=</latexit><latexit sha1_base64=\"RG1iAtaTSYwavALhRg+orYL4LFU=\">AAAB9XicbZDLTgIxFIbPeEW8oS7dNIIJbsgMG12SuHGJiVwSGEmndKCh02naDmYy4T3cuNAYt76LO9/GArNQ8E+afPnPOTmnfyA508Z1v52Nza3tnd3CXnH/4PDouHRy2tZxoghtkZjHqhtgTTkTtGWY4bQrFcVRwGknmNzO650pVZrF4sGkkvoRHgkWMoKNtR4r/SlWcsyq6YBdVQalsltzF0Lr4OVQhlzNQemrP4xJElFhCMda9zxXGj/DyjDC6azYTzSVmEzwiPYsChxR7WeLq2fo0jpDFMbKPmHQwv09keFI6zQKbGeEzViv1ubmf7VeYsIbP2NCJoYKslwUJhyZGM0jQEOmKDE8tYCJYvZWRMZYYWJsUEUbgrf65XVo12ue5ft6udHI4yjAOVxAFTy4hgbcQRNaQEDBM7zCm/PkvDjvzseydcPJZ87gj5zPH2NskcI=</latexit><latexit sha1_base64=\"RG1iAtaTSYwavALhRg+orYL4LFU=\">AAAB9XicbZDLTgIxFIbPeEW8oS7dNIIJbsgMG12SuHGJiVwSGEmndKCh02naDmYy4T3cuNAYt76LO9/GArNQ8E+afPnPOTmnfyA508Z1v52Nza3tnd3CXnH/4PDouHRy2tZxoghtkZjHqhtgTTkTtGWY4bQrFcVRwGknmNzO650pVZrF4sGkkvoRHgkWMoKNtR4r/SlWcsyq6YBdVQalsltzF0Lr4OVQhlzNQemrP4xJElFhCMda9zxXGj/DyjDC6azYTzSVmEzwiPYsChxR7WeLq2fo0jpDFMbKPmHQwv09keFI6zQKbGeEzViv1ubmf7VeYsIbP2NCJoYKslwUJhyZGM0jQEOmKDE8tYCJYvZWRMZYYWJsUEUbgrf65XVo12ue5ft6udHI4yjAOVxAFTy4hgbcQRNaQEDBM7zCm/PkvDjvzseydcPJZ87gj5zPH2NskcI=</latexit><latexit sha1_base64=\"RG1iAtaTSYwavALhRg+orYL4LFU=\">AAAB9XicbZDLTgIxFIbPeEW8oS7dNIIJbsgMG12SuHGJiVwSGEmndKCh02naDmYy4T3cuNAYt76LO9/GArNQ8E+afPnPOTmnfyA508Z1v52Nza3tnd3CXnH/4PDouHRy2tZxoghtkZjHqhtgTTkTtGWY4bQrFcVRwGknmNzO650pVZrF4sGkkvoRHgkWMoKNtR4r/SlWcsyq6YBdVQalsltzF0Lr4OVQhlzNQemrP4xJElFhCMda9zxXGj/DyjDC6azYTzSVmEzwiPYsChxR7WeLq2fo0jpDFMbKPmHQwv09keFI6zQKbGeEzViv1ubmf7VeYsIbP2NCJoYKslwUJhyZGM0jQEOmKDE8tYCJYvZWRMZYYWJsUEUbgrf65XVo12ue5ft6udHI4yjAOVxAFTy4hgbcQRNaQEDBM7zCm/PkvDjvzseydcPJZ87gj5zPH2NskcI=</latexit>yi<latexit sha1_base64=\"WmgSYaT4KZFZh7wSYgzHlBn+laY=\">AAAB7HicbZBNT8JAEIan+IX4hXr0shFMPJGWix5JvHjExAIJNGS7bGHDdtvsTk1Iw2/w4kFjvPqDvPlvXKAHBd9kkyfvzGRn3jCVwqDrfjulre2d3b3yfuXg8Oj4pHp61jFJphn3WSIT3Qup4VIo7qNAyXup5jQOJe+G07tFvfvEtRGJesRZyoOYjpWIBKNoLb8+G4r6sFpzG+5SZBO8AmpQqD2sfg1GCctirpBJakzfc1MMcqpRMMnnlUFmeErZlI5536KiMTdBvlx2Tq6sMyJRou1TSJbu74mcxsbM4tB2xhQnZr22MP+r9TOMboNcqDRDrtjqoyiTBBOyuJyMhOYM5cwCZVrYXQmbUE0Z2nwqNgRv/eRN6DQbnuWHZq3VKuIowwVcwjV4cAMtuIc2+MBAwDO8wpujnBfn3flYtZacYuYc/sj5/AEZsI4x</latexit><latexit sha1_base64=\"WmgSYaT4KZFZh7wSYgzHlBn+laY=\">AAAB7HicbZBNT8JAEIan+IX4hXr0shFMPJGWix5JvHjExAIJNGS7bGHDdtvsTk1Iw2/w4kFjvPqDvPlvXKAHBd9kkyfvzGRn3jCVwqDrfjulre2d3b3yfuXg8Oj4pHp61jFJphn3WSIT3Qup4VIo7qNAyXup5jQOJe+G07tFvfvEtRGJesRZyoOYjpWIBKNoLb8+G4r6sFpzG+5SZBO8AmpQqD2sfg1GCctirpBJakzfc1MMcqpRMMnnlUFmeErZlI5536KiMTdBvlx2Tq6sMyJRou1TSJbu74mcxsbM4tB2xhQnZr22MP+r9TOMboNcqDRDrtjqoyiTBBOyuJyMhOYM5cwCZVrYXQmbUE0Z2nwqNgRv/eRN6DQbnuWHZq3VKuIowwVcwjV4cAMtuIc2+MBAwDO8wpujnBfn3flYtZacYuYc/sj5/AEZsI4x</latexit><latexit sha1_base64=\"WmgSYaT4KZFZh7wSYgzHlBn+laY=\">AAAB7HicbZBNT8JAEIan+IX4hXr0shFMPJGWix5JvHjExAIJNGS7bGHDdtvsTk1Iw2/w4kFjvPqDvPlvXKAHBd9kkyfvzGRn3jCVwqDrfjulre2d3b3yfuXg8Oj4pHp61jFJphn3WSIT3Qup4VIo7qNAyXup5jQOJe+G07tFvfvEtRGJesRZyoOYjpWIBKNoLb8+G4r6sFpzG+5SZBO8AmpQqD2sfg1GCctirpBJakzfc1MMcqpRMMnnlUFmeErZlI5536KiMTdBvlx2Tq6sMyJRou1TSJbu74mcxsbM4tB2xhQnZr22MP+r9TOMboNcqDRDrtjqoyiTBBOyuJyMhOYM5cwCZVrYXQmbUE0Z2nwqNgRv/eRN6DQbnuWHZq3VKuIowwVcwjV4cAMtuIc2+MBAwDO8wpujnBfn3flYtZacYuYc/sj5/AEZsI4x</latexit><latexit sha1_base64=\"WmgSYaT4KZFZh7wSYgzHlBn+laY=\">AAAB7HicbZBNT8JAEIan+IX4hXr0shFMPJGWix5JvHjExAIJNGS7bGHDdtvsTk1Iw2/w4kFjvPqDvPlvXKAHBd9kkyfvzGRn3jCVwqDrfjulre2d3b3yfuXg8Oj4pHp61jFJphn3WSIT3Qup4VIo7qNAyXup5jQOJe+G07tFvfvEtRGJesRZyoOYjpWIBKNoLb8+G4r6sFpzG+5SZBO8AmpQqD2sfg1GCctirpBJakzfc1MMcqpRMMnnlUFmeErZlI5536KiMTdBvlx2Tq6sMyJRou1TSJbu74mcxsbM4tB2xhQnZr22MP+r9TOMboNcqDRDrtjqoyiTBBOyuJyMhOYM5cwCZVrYXQmbUE0Z2nwqNgRv/eRN6DQbnuWHZq3VKuIowwVcwjV4cAMtuIc2+MBAwDO8wpujnBfn3flYtZacYuYc/sj5/AEZsI4x</latexit>-505-1-0.500.51(xi,'(xi))<latexit sha1_base64=\"GllpdvBtNW9LH6e2gGOX+rXlP0k=\">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQWRpBtdFty4rGAv0IYwmU7aoZNJmJkUayi+ihsXirj1Pdz5Nk7aLLT1h4GP/5zDOfP7MaNS2fa3UVhb39jcKm6Xdnb39g/Mw6O2jBKBSQtHLBJdH0nCKCctRRUj3VgQFPqMdPzxTVbvTIiQNOL3ahoTN0RDTgOKkdKWZ55Uqg8evehPkIhHNONareKZZfvSnstaBSeHMuRqeuZXfxDhJCRcYYak7Dl2rNwUCUUxI7NSP5EkRniMhqSnkaOQSDedXz+zzrUzsIJI6MeVNXd/T6QolHIa+rozRGokl2uZ+V+tl6jg2k0pjxNFOF4sChJmqcjKorAGVBCs2FQDwoLqWy08QgJhpQMr6RCc5S+vQrt+6Wi+q5cbjTyOIpzCGVTBgStowC00oQUYHuEZXuHNeDJejHfjY9FaMPKZY/gj4/MHebWT6w==</latexit><latexit sha1_base64=\"GllpdvBtNW9LH6e2gGOX+rXlP0k=\">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQWRpBtdFty4rGAv0IYwmU7aoZNJmJkUayi+ihsXirj1Pdz5Nk7aLLT1h4GP/5zDOfP7MaNS2fa3UVhb39jcKm6Xdnb39g/Mw6O2jBKBSQtHLBJdH0nCKCctRRUj3VgQFPqMdPzxTVbvTIiQNOL3ahoTN0RDTgOKkdKWZ55Uqg8evehPkIhHNONareKZZfvSnstaBSeHMuRqeuZXfxDhJCRcYYak7Dl2rNwUCUUxI7NSP5EkRniMhqSnkaOQSDedXz+zzrUzsIJI6MeVNXd/T6QolHIa+rozRGokl2uZ+V+tl6jg2k0pjxNFOF4sChJmqcjKorAGVBCs2FQDwoLqWy08QgJhpQMr6RCc5S+vQrt+6Wi+q5cbjTyOIpzCGVTBgStowC00oQUYHuEZXuHNeDJejHfjY9FaMPKZY/gj4/MHebWT6w==</latexit><latexit sha1_base64=\"GllpdvBtNW9LH6e2gGOX+rXlP0k=\">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQWRpBtdFty4rGAv0IYwmU7aoZNJmJkUayi+ihsXirj1Pdz5Nk7aLLT1h4GP/5zDOfP7MaNS2fa3UVhb39jcKm6Xdnb39g/Mw6O2jBKBSQtHLBJdH0nCKCctRRUj3VgQFPqMdPzxTVbvTIiQNOL3ahoTN0RDTgOKkdKWZ55Uqg8evehPkIhHNONareKZZfvSnstaBSeHMuRqeuZXfxDhJCRcYYak7Dl2rNwUCUUxI7NSP5EkRniMhqSnkaOQSDedXz+zzrUzsIJI6MeVNXd/T6QolHIa+rozRGokl2uZ+V+tl6jg2k0pjxNFOF4sChJmqcjKorAGVBCs2FQDwoLqWy08QgJhpQMr6RCc5S+vQrt+6Wi+q5cbjTyOIpzCGVTBgStowC00oQUYHuEZXuHNeDJejHfjY9FaMPKZY/gj4/MHebWT6w==</latexit><latexit sha1_base64=\"GllpdvBtNW9LH6e2gGOX+rXlP0k=\">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQWRpBtdFty4rGAv0IYwmU7aoZNJmJkUayi+ihsXirj1Pdz5Nk7aLLT1h4GP/5zDOfP7MaNS2fa3UVhb39jcKm6Xdnb39g/Mw6O2jBKBSQtHLBJdH0nCKCctRRUj3VgQFPqMdPzxTVbvTIiQNOL3ahoTN0RDTgOKkdKWZ55Uqg8evehPkIhHNONareKZZfvSnstaBSeHMuRqeuZXfxDhJCRcYYak7Dl2rNwUCUUxI7NSP5EkRniMhqSnkaOQSDedXz+zzrUzsIJI6MeVNXd/T6QolHIa+rozRGokl2uZ+V+tl6jg2k0pjxNFOF4sChJmqcjKorAGVBCs2FQDwoLqWy08QgJhpQMr6RCc5S+vQrt+6Wi+q5cbjTyOIpzCGVTBgStowC00oQUYHuEZXuHNeDJejHfjY9FaMPKZY/gj4/MHebWT6w==</latexit>(yi,'(yi))<latexit sha1_base64=\"OwRczwPm0a80lXMROfDF4JnscN8=\">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQWRpBtdScGNywr2Am0Ik+mkHTqZhJlJIZbiq7hxoYhb38Odb+OkzUJbfxj4+M85nDO/HzMqlW1/G4W19Y3NreJ2aWd3b//APDxqyygRmLRwxCLR9ZEkjHLSUlQx0o0FQaHPSMcf32b1zoQISSP+oNKYuCEachpQjJS2PPOkUk09etGfIBGPaMa1WsUzy/alPZe1Ck4OZcjV9Myv/iDCSUi4wgxJ2XPsWLlTJBTFjMxK/USSGOExGpKeRo5CIt3p/PqZda6dgRVEQj+urLn7e2KKQinT0NedIVIjuVzLzP9qvUQF1+6U8jhRhOPFoiBhloqsLAprQAXBiqUaEBZU32rhERIIKx1YSYfgLH95Fdr1S0fzfb3cuMnjKMIpnEEVHLiCBtxBE1qA4RGe4RXejCfjxXg3PhatBSOfOYY/Mj5/AHw5k+s=</latexit><latexit sha1_base64=\"OwRczwPm0a80lXMROfDF4JnscN8=\">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQWRpBtdScGNywr2Am0Ik+mkHTqZhJlJIZbiq7hxoYhb38Odb+OkzUJbfxj4+M85nDO/HzMqlW1/G4W19Y3NreJ2aWd3b//APDxqyygRmLRwxCLR9ZEkjHLSUlQx0o0FQaHPSMcf32b1zoQISSP+oNKYuCEachpQjJS2PPOkUk09etGfIBGPaMa1WsUzy/alPZe1Ck4OZcjV9Myv/iDCSUi4wgxJ2XPsWLlTJBTFjMxK/USSGOExGpKeRo5CIt3p/PqZda6dgRVEQj+urLn7e2KKQinT0NedIVIjuVzLzP9qvUQF1+6U8jhRhOPFoiBhloqsLAprQAXBiqUaEBZU32rhERIIKx1YSYfgLH95Fdr1S0fzfb3cuMnjKMIpnEEVHLiCBtxBE1qA4RGe4RXejCfjxXg3PhatBSOfOYY/Mj5/AHw5k+s=</latexit><latexit sha1_base64=\"OwRczwPm0a80lXMROfDF4JnscN8=\">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQWRpBtdScGNywr2Am0Ik+mkHTqZhJlJIZbiq7hxoYhb38Odb+OkzUJbfxj4+M85nDO/HzMqlW1/G4W19Y3NreJ2aWd3b//APDxqyygRmLRwxCLR9ZEkjHLSUlQx0o0FQaHPSMcf32b1zoQISSP+oNKYuCEachpQjJS2PPOkUk09etGfIBGPaMa1WsUzy/alPZe1Ck4OZcjV9Myv/iDCSUi4wgxJ2XPsWLlTJBTFjMxK/USSGOExGpKeRo5CIt3p/PqZda6dgRVEQj+urLn7e2KKQinT0NedIVIjuVzLzP9qvUQF1+6U8jhRhOPFoiBhloqsLAprQAXBiqUaEBZU32rhERIIKx1YSYfgLH95Fdr1S0fzfb3cuMnjKMIpnEEVHLiCBtxBE1qA4RGe4RXejCfjxXg3PhatBSOfOYY/Mj5/AHw5k+s=</latexit><latexit sha1_base64=\"OwRczwPm0a80lXMROfDF4JnscN8=\">AAAB/XicbZDLSsNAFIZP6q3WW7zs3ARboQWRpBtdScGNywr2Am0Ik+mkHTqZhJlJIZbiq7hxoYhb38Odb+OkzUJbfxj4+M85nDO/HzMqlW1/G4W19Y3NreJ2aWd3b//APDxqyygRmLRwxCLR9ZEkjHLSUlQx0o0FQaHPSMcf32b1zoQISSP+oNKYuCEachpQjJS2PPOkUk09etGfIBGPaMa1WsUzy/alPZe1Ck4OZcjV9Myv/iDCSUi4wgxJ2XPsWLlTJBTFjMxK/USSGOExGpKeRo5CIt3p/PqZda6dgRVEQj+urLn7e2KKQinT0NedIVIjuVzLzP9qvUQF1+6U8jhRhOPFoiBhloqsLAprQAXBiqUaEBZU32rhERIIKx1YSYfgLH95Fdr1S0fzfb3cuMnjKMIpnEEVHLiCBtxBE1qA4RGe4RXejCfjxXg3PhatBSOfOYY/Mj5/AHw5k+s=</latexit>\u21b5\uf8ff'(yi)'(xi)yixi\uf8ff<latexit sha1_base64=\"71vqPKjY+TK9gITBZZq4AVlX2HI=\">AAACLHicbZBNS8NAEIY3flu/qh69LFZBD0rSix4LXjxWsLbQlDLZTpqlmw93N8US8oO8+FcE8WARr/4OtzWCtg4sPPPODLPzeongStv22FpYXFpeWV1bL21sbm3vlHf37lScSoYNFotYtjxQKHiEDc21wFYiEUJPYNMbXE3qzSFKxePoVo8S7ITQj7jPGWgjdctXRy6IJADqCrynbs+XwDJ3CDIJ+Mmoy0/PfpIHk+SZkc4M5UW/hxqOuuWKfW5Pg86DU0CFFFHvll/cXszSECPNBCjVduxEdzKQmjOBeclNFSbABtDHtsEIQlSdbHpsTo+N0qN+LM2LNJ2qvycyCJUahZ7pDEEHarY2Ef+rtVPtX3YyHiWpxoh9L/JTQXVMJ87RHpfItBgZACa5+StlARi/tPG3ZExwZk+eh7vquWP4plqp1Qo71sgBOSQnxCEXpEauSZ00CCOP5Jm8kbH1ZL1a79bHd+uCVczskz9hfX4BfAOnuQ==</latexit><latexit sha1_base64=\"71vqPKjY+TK9gITBZZq4AVlX2HI=\">AAACLHicbZBNS8NAEIY3flu/qh69LFZBD0rSix4LXjxWsLbQlDLZTpqlmw93N8US8oO8+FcE8WARr/4OtzWCtg4sPPPODLPzeongStv22FpYXFpeWV1bL21sbm3vlHf37lScSoYNFotYtjxQKHiEDc21wFYiEUJPYNMbXE3qzSFKxePoVo8S7ITQj7jPGWgjdctXRy6IJADqCrynbs+XwDJ3CDIJ+Mmoy0/PfpIHk+SZkc4M5UW/hxqOuuWKfW5Pg86DU0CFFFHvll/cXszSECPNBCjVduxEdzKQmjOBeclNFSbABtDHtsEIQlSdbHpsTo+N0qN+LM2LNJ2qvycyCJUahZ7pDEEHarY2Ef+rtVPtX3YyHiWpxoh9L/JTQXVMJ87RHpfItBgZACa5+StlARi/tPG3ZExwZk+eh7vquWP4plqp1Qo71sgBOSQnxCEXpEauSZ00CCOP5Jm8kbH1ZL1a79bHd+uCVczskz9hfX4BfAOnuQ==</latexit><latexit sha1_base64=\"71vqPKjY+TK9gITBZZq4AVlX2HI=\">AAACLHicbZBNS8NAEIY3flu/qh69LFZBD0rSix4LXjxWsLbQlDLZTpqlmw93N8US8oO8+FcE8WARr/4OtzWCtg4sPPPODLPzeongStv22FpYXFpeWV1bL21sbm3vlHf37lScSoYNFotYtjxQKHiEDc21wFYiEUJPYNMbXE3qzSFKxePoVo8S7ITQj7jPGWgjdctXRy6IJADqCrynbs+XwDJ3CDIJ+Mmoy0/PfpIHk+SZkc4M5UW/hxqOuuWKfW5Pg86DU0CFFFHvll/cXszSECPNBCjVduxEdzKQmjOBeclNFSbABtDHtsEIQlSdbHpsTo+N0qN+LM2LNJ2qvycyCJUahZ7pDEEHarY2Ef+rtVPtX3YyHiWpxoh9L/JTQXVMJ87RHpfItBgZACa5+StlARi/tPG3ZExwZk+eh7vquWP4plqp1Qo71sgBOSQnxCEXpEauSZ00CCOP5Jm8kbH1ZL1a79bHd+uCVczskz9hfX4BfAOnuQ==</latexit><latexit sha1_base64=\"71vqPKjY+TK9gITBZZq4AVlX2HI=\">AAACLHicbZBNS8NAEIY3flu/qh69LFZBD0rSix4LXjxWsLbQlDLZTpqlmw93N8US8oO8+FcE8WARr/4OtzWCtg4sPPPODLPzeongStv22FpYXFpeWV1bL21sbm3vlHf37lScSoYNFotYtjxQKHiEDc21wFYiEUJPYNMbXE3qzSFKxePoVo8S7ITQj7jPGWgjdctXRy6IJADqCrynbs+XwDJ3CDIJ+Mmoy0/PfpIHk+SZkc4M5UW/hxqOuuWKfW5Pg86DU0CFFFHvll/cXszSECPNBCjVduxEdzKQmjOBeclNFSbABtDHtsEIQlSdbHpsTo+N0qN+LM2LNJ2qvycyCJUahZ7pDEEHarY2Ef+rtVPtX3YyHiWpxoh9L/JTQXVMJ87RHpfItBgZACa5+StlARi/tPG3ZExwZk+eh7vquWP4plqp1Qo71sgBOSQnxCEXpEauSZ00CCOP5Jm8kbH1ZL1a79bHd+uCVczskz9hfX4BfAOnuQ==</latexit>xj<latexit sha1_base64=\"Qmcs1an4x+++o9Cy9ZfnlIj1FDM=\">AAAB7HicbZA9TwJBEIbn/ET8Qi1tNoKJFbmjkZLExhITD0iAkL1lDlb29i67e0Zy4TfYWGiMrT/Izn/jAlco+CabPHlnJjvzBong2rjut7OxubW9s1vYK+4fHB4dl05OWzpOFUOfxSJWnYBqFFyib7gR2EkU0igQ2A4mN/N6+xGV5rG8N9ME+xEdSR5yRo21/MrT4KEyKJXdqrsQWQcvhzLkag5KX71hzNIIpWGCat313MT0M6oMZwJnxV6qMaFsQkfYtShphLqfLZadkUvrDEkYK/ukIQv390RGI62nUWA7I2rGerU2N/+rdVMT1vsZl0lqULLlR2EqiInJ/HIy5AqZEVMLlCludyVsTBVlxuZTtCF4qyevQ6tW9Szf1cqNeh5HAc7hAq7Ag2towC00wQcGHJ7hFd4c6bw4787HsnXDyWfO4I+czx8XRo4p</latexit><latexit sha1_base64=\"Qmcs1an4x+++o9Cy9ZfnlIj1FDM=\">AAAB7HicbZA9TwJBEIbn/ET8Qi1tNoKJFbmjkZLExhITD0iAkL1lDlb29i67e0Zy4TfYWGiMrT/Izn/jAlco+CabPHlnJjvzBong2rjut7OxubW9s1vYK+4fHB4dl05OWzpOFUOfxSJWnYBqFFyib7gR2EkU0igQ2A4mN/N6+xGV5rG8N9ME+xEdSR5yRo21/MrT4KEyKJXdqrsQWQcvhzLkag5KX71hzNIIpWGCat313MT0M6oMZwJnxV6qMaFsQkfYtShphLqfLZadkUvrDEkYK/ukIQv390RGI62nUWA7I2rGerU2N/+rdVMT1vsZl0lqULLlR2EqiInJ/HIy5AqZEVMLlCludyVsTBVlxuZTtCF4qyevQ6tW9Szf1cqNeh5HAc7hAq7Ag2towC00wQcGHJ7hFd4c6bw4787HsnXDyWfO4I+czx8XRo4p</latexit><latexit sha1_base64=\"Qmcs1an4x+++o9Cy9ZfnlIj1FDM=\">AAAB7HicbZA9TwJBEIbn/ET8Qi1tNoKJFbmjkZLExhITD0iAkL1lDlb29i67e0Zy4TfYWGiMrT/Izn/jAlco+CabPHlnJjvzBong2rjut7OxubW9s1vYK+4fHB4dl05OWzpOFUOfxSJWnYBqFFyib7gR2EkU0igQ2A4mN/N6+xGV5rG8N9ME+xEdSR5yRo21/MrT4KEyKJXdqrsQWQcvhzLkag5KX71hzNIIpWGCat313MT0M6oMZwJnxV6qMaFsQkfYtShphLqfLZadkUvrDEkYK/ukIQv390RGI62nUWA7I2rGerU2N/+rdVMT1vsZl0lqULLlR2EqiInJ/HIy5AqZEVMLlCludyVsTBVlxuZTtCF4qyevQ6tW9Szf1cqNeh5HAc7hAq7Ag2towC00wQcGHJ7hFd4c6bw4787HsnXDyWfO4I+czx8XRo4p</latexit><latexit sha1_base64=\"Qmcs1an4x+++o9Cy9ZfnlIj1FDM=\">AAAB7HicbZA9TwJBEIbn/ET8Qi1tNoKJFbmjkZLExhITD0iAkL1lDlb29i67e0Zy4TfYWGiMrT/Izn/jAlco+CabPHlnJjvzBong2rjut7OxubW9s1vYK+4fHB4dl05OWzpOFUOfxSJWnYBqFFyib7gR2EkU0igQ2A4mN/N6+xGV5rG8N9ME+xEdSR5yRo21/MrT4KEyKJXdqrsQWQcvhzLkag5KX71hzNIIpWGCat313MT0M6oMZwJnxV6qMaFsQkfYtShphLqfLZadkUvrDEkYK/ukIQv390RGI62nUWA7I2rGerU2N/+rdVMT1vsZl0lqULLlR2EqiInJ/HIy5AqZEVMLlCludyVsTBVlxuZTtCF4qyevQ6tW9Szf1cqNeh5HAc7hAq7Ag2towC00wQcGHJ7hFd4c6bw4787HsnXDyWfO4I+czx8XRo4p</latexit>yj<latexit sha1_base64=\"GXeTI3hCKBcAvNjSYFW2ns3xRoM=\">AAAB7HicbZDNTgIxFIVv8Q/xD3XpphFMXJEZNrIkceMSEwdIYEI6pQOVTmfSdkwmE57BjQuNcesDufNtLDALBU/S5Mu596b3niARXBvH+Ualre2d3b3yfuXg8Oj4pHp61tVxqijzaCxi1Q+IZoJL5hluBOsnipEoEKwXzG4X9d4TU5rH8sFkCfMjMpE85JQYa3n1bPRYH1VrTsNZCm+CW0ANCnVG1a/hOKZpxKShgmg9cJ3E+DlRhlPB5pVhqllC6IxM2MCiJBHTfr5cdo6vrDPGYazskwYv3d8TOYm0zqLAdkbETPV6bWH+VxukJmz5OZdJapikq4/CVGAT48XleMwVo0ZkFghV3O6K6ZQoQo3Np2JDcNdP3oRus+Favm/W2q0ijjJcwCVcgws30IY76IAHFDg8wyu8IYle0Dv6WLWWUDFzDn+EPn8AGM2OKg==</latexit><latexit sha1_base64=\"GXeTI3hCKBcAvNjSYFW2ns3xRoM=\">AAAB7HicbZDNTgIxFIVv8Q/xD3XpphFMXJEZNrIkceMSEwdIYEI6pQOVTmfSdkwmE57BjQuNcesDufNtLDALBU/S5Mu596b3niARXBvH+Ualre2d3b3yfuXg8Oj4pHp61tVxqijzaCxi1Q+IZoJL5hluBOsnipEoEKwXzG4X9d4TU5rH8sFkCfMjMpE85JQYa3n1bPRYH1VrTsNZCm+CW0ANCnVG1a/hOKZpxKShgmg9cJ3E+DlRhlPB5pVhqllC6IxM2MCiJBHTfr5cdo6vrDPGYazskwYv3d8TOYm0zqLAdkbETPV6bWH+VxukJmz5OZdJapikq4/CVGAT48XleMwVo0ZkFghV3O6K6ZQoQo3Np2JDcNdP3oRus+Favm/W2q0ijjJcwCVcgws30IY76IAHFDg8wyu8IYle0Dv6WLWWUDFzDn+EPn8AGM2OKg==</latexit><latexit sha1_base64=\"GXeTI3hCKBcAvNjSYFW2ns3xRoM=\">AAAB7HicbZDNTgIxFIVv8Q/xD3XpphFMXJEZNrIkceMSEwdIYEI6pQOVTmfSdkwmE57BjQuNcesDufNtLDALBU/S5Mu596b3niARXBvH+Ualre2d3b3yfuXg8Oj4pHp61tVxqijzaCxi1Q+IZoJL5hluBOsnipEoEKwXzG4X9d4TU5rH8sFkCfMjMpE85JQYa3n1bPRYH1VrTsNZCm+CW0ANCnVG1a/hOKZpxKShgmg9cJ3E+DlRhlPB5pVhqllC6IxM2MCiJBHTfr5cdo6vrDPGYazskwYv3d8TOYm0zqLAdkbETPV6bWH+VxukJmz5OZdJapikq4/CVGAT48XleMwVo0ZkFghV3O6K6ZQoQo3Np2JDcNdP3oRus+Favm/W2q0ijjJcwCVcgws30IY76IAHFDg8wyu8IYle0Dv6WLWWUDFzDn+EPn8AGM2OKg==</latexit><latexit sha1_base64=\"GXeTI3hCKBcAvNjSYFW2ns3xRoM=\">AAAB7HicbZDNTgIxFIVv8Q/xD3XpphFMXJEZNrIkceMSEwdIYEI6pQOVTmfSdkwmE57BjQuNcesDufNtLDALBU/S5Mu596b3niARXBvH+Ualre2d3b3yfuXg8Oj4pHp61tVxqijzaCxi1Q+IZoJL5hluBOsnipEoEKwXzG4X9d4TU5rH8sFkCfMjMpE85JQYa3n1bPRYH1VrTsNZCm+CW0ANCnVG1a/hOKZpxKShgmg9cJ3E+DlRhlPB5pVhqllC6IxM2MCiJBHTfr5cdo6vrDPGYazskwYv3d8TOYm0zqLAdkbETPV6bWH+VxukJmz5OZdJapikq4/CVGAT48XleMwVo0ZkFghV3O6K6ZQoQo3Np2JDcNdP3oRus+Favm/W2q0ijjJcwCVcgws30IY76IAHFDg8wyu8IYle0Dv6WLWWUDFzDn+EPn8AGM2OKg==</latexit>'(xj)<latexit sha1_base64=\"oI6G4J0ZwbR7XFQ0Ze3vRM+PxDI=\">AAAB9XicbZDLTsJAFIZP8YZ4Q126mQgmuCEtG1mSuHGJiVwSqGQ6TGFkOm1mpihpeA83LjTGre/izrdxCl0o+CeTfPnPOTlnfi/iTGnb/rZyG5tb2zv53cLe/sHhUfH4pK3CWBLaIiEPZdfDinImaEszzWk3khQHHqcdb3Kd1jtTKhULxZ2eRdQN8EgwnxGsjXVf7k+xjMas8jR4uCwPiiW7ai+E1sHJoASZmoPiV38YkjigQhOOleo5dqTdBEvNCKfzQj9WNMJkgke0Z1DggCo3WVw9RxfGGSI/lOYJjRbu74kEB0rNAs90BliP1WotNf+r9WLt192EiSjWVJDlIj/mSIcojQANmaRE85kBTCQztyIyxhITbYIqmBCc1S+vQ7tWdQzf1kqNehZHHs7gHCrgwBU04Aaa0AICEp7hFd6sR+vFerc+lq05K5s5hT+yPn8AYQKRug==</latexit><latexit sha1_base64=\"oI6G4J0ZwbR7XFQ0Ze3vRM+PxDI=\">AAAB9XicbZDLTsJAFIZP8YZ4Q126mQgmuCEtG1mSuHGJiVwSqGQ6TGFkOm1mpihpeA83LjTGre/izrdxCl0o+CeTfPnPOTlnfi/iTGnb/rZyG5tb2zv53cLe/sHhUfH4pK3CWBLaIiEPZdfDinImaEszzWk3khQHHqcdb3Kd1jtTKhULxZ2eRdQN8EgwnxGsjXVf7k+xjMas8jR4uCwPiiW7ai+E1sHJoASZmoPiV38YkjigQhOOleo5dqTdBEvNCKfzQj9WNMJkgke0Z1DggCo3WVw9RxfGGSI/lOYJjRbu74kEB0rNAs90BliP1WotNf+r9WLt192EiSjWVJDlIj/mSIcojQANmaRE85kBTCQztyIyxhITbYIqmBCc1S+vQ7tWdQzf1kqNehZHHs7gHCrgwBU04Aaa0AICEp7hFd6sR+vFerc+lq05K5s5hT+yPn8AYQKRug==</latexit><latexit sha1_base64=\"oI6G4J0ZwbR7XFQ0Ze3vRM+PxDI=\">AAAB9XicbZDLTsJAFIZP8YZ4Q126mQgmuCEtG1mSuHGJiVwSqGQ6TGFkOm1mpihpeA83LjTGre/izrdxCl0o+CeTfPnPOTlnfi/iTGnb/rZyG5tb2zv53cLe/sHhUfH4pK3CWBLaIiEPZdfDinImaEszzWk3khQHHqcdb3Kd1jtTKhULxZ2eRdQN8EgwnxGsjXVf7k+xjMas8jR4uCwPiiW7ai+E1sHJoASZmoPiV38YkjigQhOOleo5dqTdBEvNCKfzQj9WNMJkgke0Z1DggCo3WVw9RxfGGSI/lOYJjRbu74kEB0rNAs90BliP1WotNf+r9WLt192EiSjWVJDlIj/mSIcojQANmaRE85kBTCQztyIyxhITbYIqmBCc1S+vQ7tWdQzf1kqNehZHHs7gHCrgwBU04Aaa0AICEp7hFd6sR+vFerc+lq05K5s5hT+yPn8AYQKRug==</latexit><latexit sha1_base64=\"oI6G4J0ZwbR7XFQ0Ze3vRM+PxDI=\">AAAB9XicbZDLTsJAFIZP8YZ4Q126mQgmuCEtG1mSuHGJiVwSqGQ6TGFkOm1mpihpeA83LjTGre/izrdxCl0o+CeTfPnPOTlnfi/iTGnb/rZyG5tb2zv53cLe/sHhUfH4pK3CWBLaIiEPZdfDinImaEszzWk3khQHHqcdb3Kd1jtTKhULxZ2eRdQN8EgwnxGsjXVf7k+xjMas8jR4uCwPiiW7ai+E1sHJoASZmoPiV38YkjigQhOOleo5dqTdBEvNCKfzQj9WNMJkgke0Z1DggCo3WVw9RxfGGSI/lOYJjRbu74kEB0rNAs90BliP1WotNf+r9WLt192EiSjWVJDlIj/mSIcojQANmaRE85kBTCQztyIyxhITbYIqmBCc1S+vQ7tWdQzf1kqNehZHHs7gHCrgwBU04Aaa0AICEp7hFd6sR+vFerc+lq05K5s5hT+yPn8AYQKRug==</latexit>'(yj)<latexit sha1_base64=\"I7R4a32uvuL7gDjagUmI6UlCRsU=\">AAAB9XicbZDLTsJAFIZP8YZ4Q126mQgmuCEtG1iSuHGJiVwSqGQ6TGFkOm1mppim4T3cuNAYt76LO9/GAbpQ8E8m+fKfc3LO/F7EmdK2/W3ltrZ3dvfy+4WDw6Pjk+LpWUeFsSS0TUIeyp6HFeVM0LZmmtNeJCkOPE673vRmUe/OqFQsFPc6iagb4LFgPiNYG+uhPJhhGU1YJRk+XpeHxZJdtZdCm+BkUIJMrWHxazAKSRxQoQnHSvUdO9JuiqVmhNN5YRArGmEyxWPaNyhwQJWbLq+eoyvjjJAfSvOERkv390SKA6WSwDOdAdYTtV5bmP/V+rH2G27KRBRrKshqkR9zpEO0iACNmKRE88QAJpKZWxGZYImJNkEVTAjO+pc3oVOrOobvaqVmI4sjDxdwCRVwoA5NuIUWtIGAhGd4hTfryXqx3q2PVWvOymbO4Y+szx9iipG7</latexit><latexit sha1_base64=\"I7R4a32uvuL7gDjagUmI6UlCRsU=\">AAAB9XicbZDLTsJAFIZP8YZ4Q126mQgmuCEtG1iSuHGJiVwSqGQ6TGFkOm1mppim4T3cuNAYt76LO9/GAbpQ8E8m+fKfc3LO/F7EmdK2/W3ltrZ3dvfy+4WDw6Pjk+LpWUeFsSS0TUIeyp6HFeVM0LZmmtNeJCkOPE673vRmUe/OqFQsFPc6iagb4LFgPiNYG+uhPJhhGU1YJRk+XpeHxZJdtZdCm+BkUIJMrWHxazAKSRxQoQnHSvUdO9JuiqVmhNN5YRArGmEyxWPaNyhwQJWbLq+eoyvjjJAfSvOERkv390SKA6WSwDOdAdYTtV5bmP/V+rH2G27KRBRrKshqkR9zpEO0iACNmKRE88QAJpKZWxGZYImJNkEVTAjO+pc3oVOrOobvaqVmI4sjDxdwCRVwoA5NuIUWtIGAhGd4hTfryXqx3q2PVWvOymbO4Y+szx9iipG7</latexit><latexit sha1_base64=\"I7R4a32uvuL7gDjagUmI6UlCRsU=\">AAAB9XicbZDLTsJAFIZP8YZ4Q126mQgmuCEtG1iSuHGJiVwSqGQ6TGFkOm1mppim4T3cuNAYt76LO9/GAbpQ8E8m+fKfc3LO/F7EmdK2/W3ltrZ3dvfy+4WDw6Pjk+LpWUeFsSS0TUIeyp6HFeVM0LZmmtNeJCkOPE673vRmUe/OqFQsFPc6iagb4LFgPiNYG+uhPJhhGU1YJRk+XpeHxZJdtZdCm+BkUIJMrWHxazAKSRxQoQnHSvUdO9JuiqVmhNN5YRArGmEyxWPaNyhwQJWbLq+eoyvjjJAfSvOERkv390SKA6WSwDOdAdYTtV5bmP/V+rH2G27KRBRrKshqkR9zpEO0iACNmKRE88QAJpKZWxGZYImJNkEVTAjO+pc3oVOrOobvaqVmI4sjDxdwCRVwoA5NuIUWtIGAhGd4hTfryXqx3q2PVWvOymbO4Y+szx9iipG7</latexit><latexit sha1_base64=\"I7R4a32uvuL7gDjagUmI6UlCRsU=\">AAAB9XicbZDLTsJAFIZP8YZ4Q126mQgmuCEtG1iSuHGJiVwSqGQ6TGFkOm1mppim4T3cuNAYt76LO9/GAbpQ8E8m+fKfc3LO/F7EmdK2/W3ltrZ3dvfy+4WDw6Pjk+LpWUeFsSS0TUIeyp6HFeVM0LZmmtNeJCkOPE673vRmUe/OqFQsFPc6iagb4LFgPiNYG+uhPJhhGU1YJRk+XpeHxZJdtZdCm+BkUIJMrWHxazAKSRxQoQnHSvUdO9JuiqVmhNN5YRArGmEyxWPaNyhwQJWbLq+eoyvjjJAfSvOERkv390SKA6WSwDOdAdYTtV5bmP/V+rH2G27KRBRrKshqkR9zpEO0iACNmKRE88QAJpKZWxGZYImJNkEVTAjO+pc3oVOrOobvaqVmI4sjDxdwCRVwoA5NuIUWtIGAhGd4hTfryXqx3q2PVWvOymbO4Y+szx9iipG7</latexit>nonnegative combination(x)=264'(x1)...'(xn)375<latexit sha1_base64=\"guxvl0q2bU+xVoPoliEsRkzbL0M=\">AAACNXicbZDLSgMxFIYzXmu9VV26CVbBbmSmG7sRCm5cuKhgL9CUksmctsFMZkgypWXoS7nxPVzpwoUibn0F04uirQcCH/9/Djnn92PBtXHdZ2dpeWV1bT2zkd3c2t7Zze3t13SUKAZVFolINXyqQXAJVcONgEasgIa+gLp/dzn2631Qmkfy1gxjaIW0K3mHM2qs1M5dH5O4x08HhQviQ5fL1A+pUXwwIn2qJk7bK2BCMOkHkdFT+nZkgYAMfkaO27m8e+ZOCi+CN4M8mlWlnXskQcSSEKRhgmrd9NzYtFKqDGcCRlmSaIgpu6NdaFqUNATdSidXj/CJVQLciZR90uCJ+nsipaHWw9C3nXbBnp73xuJ/XjMxnVIr5TJODEg2/aiTCGwiPI4QB1wBM2JogTLF7a6Y9aiizNigszYEb/7kRagVzzzLN8V8uTSLI4MO0RE6RR46R2V0hSqoihi6R0/oFb05D86L8+58TFuXnNnMAfpTzucX/a+riw==</latexit><latexit sha1_base64=\"guxvl0q2bU+xVoPoliEsRkzbL0M=\">AAACNXicbZDLSgMxFIYzXmu9VV26CVbBbmSmG7sRCm5cuKhgL9CUksmctsFMZkgypWXoS7nxPVzpwoUibn0F04uirQcCH/9/Djnn92PBtXHdZ2dpeWV1bT2zkd3c2t7Zze3t13SUKAZVFolINXyqQXAJVcONgEasgIa+gLp/dzn2631Qmkfy1gxjaIW0K3mHM2qs1M5dH5O4x08HhQviQ5fL1A+pUXwwIn2qJk7bK2BCMOkHkdFT+nZkgYAMfkaO27m8e+ZOCi+CN4M8mlWlnXskQcSSEKRhgmrd9NzYtFKqDGcCRlmSaIgpu6NdaFqUNATdSidXj/CJVQLciZR90uCJ+nsipaHWw9C3nXbBnp73xuJ/XjMxnVIr5TJODEg2/aiTCGwiPI4QB1wBM2JogTLF7a6Y9aiizNigszYEb/7kRagVzzzLN8V8uTSLI4MO0RE6RR46R2V0hSqoihi6R0/oFb05D86L8+58TFuXnNnMAfpTzucX/a+riw==</latexit><latexit sha1_base64=\"guxvl0q2bU+xVoPoliEsRkzbL0M=\">AAACNXicbZDLSgMxFIYzXmu9VV26CVbBbmSmG7sRCm5cuKhgL9CUksmctsFMZkgypWXoS7nxPVzpwoUibn0F04uirQcCH/9/Djnn92PBtXHdZ2dpeWV1bT2zkd3c2t7Zze3t13SUKAZVFolINXyqQXAJVcONgEasgIa+gLp/dzn2631Qmkfy1gxjaIW0K3mHM2qs1M5dH5O4x08HhQviQ5fL1A+pUXwwIn2qJk7bK2BCMOkHkdFT+nZkgYAMfkaO27m8e+ZOCi+CN4M8mlWlnXskQcSSEKRhgmrd9NzYtFKqDGcCRlmSaIgpu6NdaFqUNATdSidXj/CJVQLciZR90uCJ+nsipaHWw9C3nXbBnp73xuJ/XjMxnVIr5TJODEg2/aiTCGwiPI4QB1wBM2JogTLF7a6Y9aiizNigszYEb/7kRagVzzzLN8V8uTSLI4MO0RE6RR46R2V0hSqoihi6R0/oFb05D86L8+58TFuXnNnMAfpTzucX/a+riw==</latexit><latexit sha1_base64=\"guxvl0q2bU+xVoPoliEsRkzbL0M=\">AAACNXicbZDLSgMxFIYzXmu9VV26CVbBbmSmG7sRCm5cuKhgL9CUksmctsFMZkgypWXoS7nxPVzpwoUibn0F04uirQcCH/9/Djnn92PBtXHdZ2dpeWV1bT2zkd3c2t7Zze3t13SUKAZVFolINXyqQXAJVcONgEasgIa+gLp/dzn2631Qmkfy1gxjaIW0K3mHM2qs1M5dH5O4x08HhQviQ5fL1A+pUXwwIn2qJk7bK2BCMOkHkdFT+nZkgYAMfkaO27m8e+ZOCi+CN4M8mlWlnXskQcSSEKRhgmrd9NzYtFKqDGcCRlmSaIgpu6NdaFqUNATdSidXj/CJVQLciZR90uCJ+nsipaHWw9C3nXbBnp73xuJ/XjMxnVIr5TJODEg2/aiTCGwiPI4QB1wBM2JogTLF7a6Y9aiizNigszYEb/7kRagVzzzLN8V8uTSLI4MO0RE6RR46R2V0hSqoihi6R0/oFb05D86L8+58TFuXnNnMAfpTzucX/a+riw==</latexit>\uf8ffxy(x)(y)>\uf8ff2\u21b5T(\u21b5+)T(\u21b5+)T2T\uf8ffxy(x)(y)0<latexit sha1_base64=\"xE4TKIfpLj8g4gjosp4+9y8sOeo=\">AAACtHiclVHBThsxEPVuaUtDW9Jy7MUiFCWqgnZzaDkicekRpIQgxSGddSaJhdfr2t4qq1W+sLfe+Jt6NxGCwIWRLD+9N288nkm0FNZF0V0Qvtp5/ebt7rvG3vsPH/ebnz5f2Sw3HAc8k5m5TsCiFAoHTjiJ19ogpInEYXJ7XunDP2isyFTfFRrHKcyVmAkOzlOT5t8jluBcqDJJwRmxXC27BWWMMr0Q7WWnW99Fh6Ga3qfcMJdpuuWj3R4DqRdQCQ5onx7T9pr5VjOdflV3mzr2vv6j6i/vh7I5/qbR0aTZik6iOuhTEG9Ai2ziYtL8x6YZz1NUjkuwdhRH2o1LME5wiasGyy1q4Lcwx5GHClK047Ie+op+9cyUzjLjj3K0Zh86SkitLdLEZ/o+F3Zbq8jntFHuZqfjUiidO1R8/dAsl9RltNognQqD3MnCA+BG+F4pX4AB7vyeG34I8faXn4Kr3kns8WWvdXa6Gccu+UIOSZvE5Ac5Iz/JBRkQHsTBMPgVQPg9ZCEPcZ0aBhvPAXkUofoPmdvT4g==</latexit><latexit sha1_base64=\"xE4TKIfpLj8g4gjosp4+9y8sOeo=\">AAACtHiclVHBThsxEPVuaUtDW9Jy7MUiFCWqgnZzaDkicekRpIQgxSGddSaJhdfr2t4qq1W+sLfe+Jt6NxGCwIWRLD+9N288nkm0FNZF0V0Qvtp5/ebt7rvG3vsPH/ebnz5f2Sw3HAc8k5m5TsCiFAoHTjiJ19ogpInEYXJ7XunDP2isyFTfFRrHKcyVmAkOzlOT5t8jluBcqDJJwRmxXC27BWWMMr0Q7WWnW99Fh6Ga3qfcMJdpuuWj3R4DqRdQCQ5onx7T9pr5VjOdflV3mzr2vv6j6i/vh7I5/qbR0aTZik6iOuhTEG9Ai2ziYtL8x6YZz1NUjkuwdhRH2o1LME5wiasGyy1q4Lcwx5GHClK047Ie+op+9cyUzjLjj3K0Zh86SkitLdLEZ/o+F3Zbq8jntFHuZqfjUiidO1R8/dAsl9RltNognQqD3MnCA+BG+F4pX4AB7vyeG34I8faXn4Kr3kns8WWvdXa6Gccu+UIOSZvE5Ac5Iz/JBRkQHsTBMPgVQPg9ZCEPcZ0aBhvPAXkUofoPmdvT4g==</latexit><latexit sha1_base64=\"xE4TKIfpLj8g4gjosp4+9y8sOeo=\">AAACtHiclVHBThsxEPVuaUtDW9Jy7MUiFCWqgnZzaDkicekRpIQgxSGddSaJhdfr2t4qq1W+sLfe+Jt6NxGCwIWRLD+9N288nkm0FNZF0V0Qvtp5/ebt7rvG3vsPH/ebnz5f2Sw3HAc8k5m5TsCiFAoHTjiJ19ogpInEYXJ7XunDP2isyFTfFRrHKcyVmAkOzlOT5t8jluBcqDJJwRmxXC27BWWMMr0Q7WWnW99Fh6Ga3qfcMJdpuuWj3R4DqRdQCQ5onx7T9pr5VjOdflV3mzr2vv6j6i/vh7I5/qbR0aTZik6iOuhTEG9Ai2ziYtL8x6YZz1NUjkuwdhRH2o1LME5wiasGyy1q4Lcwx5GHClK047Ie+op+9cyUzjLjj3K0Zh86SkitLdLEZ/o+F3Zbq8jntFHuZqfjUiidO1R8/dAsl9RltNognQqD3MnCA+BG+F4pX4AB7vyeG34I8faXn4Kr3kns8WWvdXa6Gccu+UIOSZvE5Ac5Iz/JBRkQHsTBMPgVQPg9ZCEPcZ0aBhvPAXkUofoPmdvT4g==</latexit><latexit sha1_base64=\"xE4TKIfpLj8g4gjosp4+9y8sOeo=\">AAACtHiclVHBThsxEPVuaUtDW9Jy7MUiFCWqgnZzaDkicekRpIQgxSGddSaJhdfr2t4qq1W+sLfe+Jt6NxGCwIWRLD+9N288nkm0FNZF0V0Qvtp5/ebt7rvG3vsPH/ebnz5f2Sw3HAc8k5m5TsCiFAoHTjiJ19ogpInEYXJ7XunDP2isyFTfFRrHKcyVmAkOzlOT5t8jluBcqDJJwRmxXC27BWWMMr0Q7WWnW99Fh6Ga3qfcMJdpuuWj3R4DqRdQCQ5onx7T9pr5VjOdflV3mzr2vv6j6i/vh7I5/qbR0aTZik6iOuhTEG9Ai2ziYtL8x6YZz1NUjkuwdhRH2o1LME5wiasGyy1q4Lcwx5GHClK047Ie+op+9cyUzjLjj3K0Zh86SkitLdLEZ/o+F3Zbq8jntFHuZqfjUiidO1R8/dAsl9RltNognQqD3MnCA+BG+F4pX4AB7vyeG34I8faXn4Kr3kns8WWvdXa6Gccu+UIOSZvE5Ac5Iz/JBRkQHsTBMPgVQPg9ZCEPcZ0aBhvPAXkUofoPmdvT4g==</latexit>\fwhere x = [x0(cid:62) x1(cid:62) \u00b7\u00b7\u00b7 x(cid:96)(cid:62)](cid:62) is the concatenation of the input and the activation values, and the\nmatrices b, A, B and C are given by [13]\n\nW 0\n0\n0 W 1\n...\n...\n\n0\n\n0\n\n. . .\n\n0\nIn2\n...\n\n0\n\n. . .\n. . .\n...\n. . .\n\n0\n0\n\n...\nIn(cid:96)\n\n\uf8f9\uf8fa\uf8fa\uf8fb ,\n\n(13)\n\nA =\uf8ee\uf8ef\uf8ef\uf8f0\nC =(cid:2)0\n\n...\n\n...\n\n0\n0\n\n0\n0\n\n0\n0\n\n. . .\n. . .\n...\n. . . W (cid:96)\u22121\n\n\uf8f9\uf8fa\uf8fa\uf8fb , B =\uf8ee\uf8ef\uf8ef\uf8f0\nIn1\n0\n...\n...\nb(cid:96)\u22121(cid:62)(cid:105)(cid:62) .\n0 W (cid:96)(cid:3) , b =(cid:104)b0(cid:62) \u00b7\u00b7\u00b7\n\n0\n\n0\n\n0\n\nThe particular representation in (12) facilitates the extension of LipSDP to multiple layers, as stated\nin the following theorem.\nTheorem 2 (Lipschitz certi\ufb01cates for multi-layer neural networks) Consider an (cid:96)-layer fully\nconnected neural network described by (2). Let n = (cid:80)(cid:96)\nk=1 nk be the total number of hidden\nneurons and suppose the activation functions are slope-restricted in the sector [\u03b1, \u03b2]. De\ufb01ne Tn as in\n(8). De\ufb01ne A and B as in (13). Consider the matrix inequality\nB(cid:21) +\uf8ee\uf8ef\uf8ef\uf8f0\n\u2212\u03c1In0\n\u22122T (cid:21)(cid:20)A\nM (\u03c1, T ) =(cid:20)A\nIf (14) is satis\ufb01ed for some (\u03c1, T ) \u2208 R+ \u00d7 Tn, then ||f (x) \u2212 f (y)||2 \u2264 \u221a\u03c1||x \u2212 y||2, \u2200x, y \u2208 Rn0.\nIn a similar way to the single-layer case, we can \ufb01nd the best bound on the Lipschitz constant by\nsolving the SDP in (11) with M (\u03c1, T ) de\ufb01ned as in (14).\nRemark 1 We have only considered the (cid:96)2 norm in our exposition. By using the inequality (cid:107)x(cid:107)p \u2264\np\u2212 1\n\nB(cid:21)(cid:62)(cid:20) \u22122\u03b1\u03b2T\n\n\uf8f9\uf8fa\uf8fa\uf8fb (cid:22) 0.\n\n. . .\n. . .\n...\n. . .\n\n(W (cid:96))(cid:62)W (cid:96)\n\n(\u03b1 + \u03b2)T\n\n(\u03b1 + \u03b2)T\n\n0\n0\n...\n0\n\n(14)\n\n0\n...\n0\n\n0\n0\n...\n\nq (cid:107)x(cid:107)q, the (cid:96)2-Lipschitz bound implies\n\nn\n\n1\n\nn\u2212( 1\n\np\u2212 1\n\n2 )(cid:107)f (y) \u2212 f (x)(cid:107)p \u2264 (cid:107)f (y) \u2212 f (x)(cid:107)2 \u2264 L2(cid:107)y \u2212 x(cid:107)2 \u2264 n\n\n1\n\n2\u2212 1\n\nq L2(cid:107)y \u2212 x(cid:107)q,\n\nor, equivalently, (cid:107)f (y) \u2212 f (x)(cid:107)p \u2264 n\nq L2 is a Lipschitz constant of f\nwhen (cid:96)q and (cid:96)p norms are used in the input and output spaces, respectively. We can also extend our\nframework to accommodate quadratic norms (cid:107)x(cid:107)P = \u221ax(cid:62)P x, where P \u2208 Sn\n++.\n\nq L2(cid:107)y \u2212 x(cid:107)q. Hence, n\n\n1\n\np\u2212 1\n\n1\n\np\u2212 1\n\n2.5 Variants of LipSDP: reconciling accuracy and ef\ufb01ciency\nIn LipSDP, there are O(n2) decision variables \u03bbij, 1 \u2264 i, j \u2264 n (\u03bbij = \u03bbji), where n is the total\nnumber of hidden neurons. For i (cid:54)= j, the variable \u03bbij couples the i-th and j-th hidden neuron.\nFor i = j, the variable \u03bbii constrains the input-output of the i-th activation function individually.\nUsing all these decision variables would provide the tightest convex relaxation in our formulation.\nHowever, solving this SDP with all the decision variables included is impractical for large networks.\nNevertheless, we can consider a hierarchy of relaxations of LipSDP by removing a subset of the\ndecision variables. Below, we give a brief description of the ef\ufb01ciency and accuracy of each variant.\nThroughout, we let n be the total number of neurons and (cid:96) the number of hidden layers.\n\n1. LipSDP-Network imposes constraints on all possible pairs of activation functions and has\n\nO(n2) decision variables. It is the least scalable but the most accurate method.\n\n2. LipSDP-Neuron ignores the cross coupling constraints among different neurons and has\nO(n) decision variables. It is more scalable and less accurate than LipSDP-Network. For\nthis case, we have T = diag(\u03bb11,\u00b7\u00b7\u00b7 , \u03bbnn).\nables.\nT = blkdiag(\u03bb1In1,\u00b7\u00b7\u00b7 , \u03bb(cid:96)In(cid:96)).\n\n3. LipSDP-Layer considers only one constraint per layer, resulting in O((cid:96)) decision vari-\nIt is the most scalable and least accurate method. For this variant, we have\n\nParallel implementation by splitting. The Lipschitz constant of the composition of two or more\nfunctions can be bounded by the product of the Lipschitz constants of the individual functions. By\n\n6\n\n\fn\n500\n1000\n1500\n2000\n2500\n3000\n\nLipSDP-\nNeuron\n\n5.22\n27.91\n82.12\n200.88\n376.07\n734.63\n\nLipSDP-\n\nLayer\n2.85\n17.88\n58.61\n146.09\n245.94\n473.25\n\n(cid:96)\n5\n10\n50\n100\n200\n500\n\nLipSDP-\nNeuron\n20.33\n32.18\n87.45\n135.85\n221.2\n707.56\n\nLipSDP-\n\nLayer\n3.41\n7.06\n25.88\n40.39\n64.90\n216.49\n\nTable 1: Computation time in seconds for evaluating\nLipschitz bounds of one-hidden-layer neural networks\nwith a varying number of hidden units. A plot showing\nthe Lipschitz constant for each network tested in this\ntable has been provided in the Appendix.\n\nTable 2: Computation time in seconds for comput-\ning Lipschitz bounds of (cid:96)-hidden-layer neural net-\nworks with 100 activation functions per layer. For\nLipSDP-Neuron and LipSDP-Layer, we split each\nnetwork up into 5-layer sub-networks.\n\nsplitting a neural network up into small sub-networks, one can \ufb01rst bound the Lipschitz constant of\neach sub-network and then multiply these constants together to obtain a Lipschitz constant for the\nentire network. Because sub-networks do not share weights, it is possible to compute the Lipschitz\nconstants for each sub-network in parallel. This greatly improves the scalability of each variant of\nLipSDP with respect to the total number of activation functions in the network. We remark that\nthis parallelization is not exclusive to our method. However, among all methods that can split the\ncomputations across layers, our method yields more accurate bounds per split.\n\n3 Experiments\n\nIn this section we describe several experiments that highlight the key aspects of this work. In\nparticular, we show empirically that our bounds are much tighter than any comparable method, we\nstudy the impact of robust training on our Lipschitz bounds, and we analyze the scalability of our\nmethods.\nExperimental setup. For our experiments we used MATLAB, the CVX toolbox [18] and MOSEK\n[3] on a 9-core CPU with 16GB of RAM to solve the SDPs. All classi\ufb01ers trained on MNIST used\nan 80-20 train-test split.\nTraining procedures. Several training procedures have recently been proposed to improve the\nrobustness of neural network classi\ufb01ers. Two prominent procedures are the LP-based method in [40]\nand projected gradient descent (PGD) based method in [24]. We refer to these training methods as\nLP-Train and PGD-Train, respectively. Both procedures take as input a parameter \u0001 that de\ufb01nes the\n(cid:96)\u221e perturbation of the training data points.\nBaselines. Throughout the experiments, we will often show comparisons to the naive upper bound\n. We are aware of only two methods\nthat bound the Lipschitz constant and can scale to fully-connected networks with more than two\nhidden layers; these methods are [10], which we will refer to as CPLip, and [37], which is called\nSeqLip. We compare the Lipschitz bounds obtained by LipSDP-Neuron, LipSDP-Layer, CPLip,\nand SeqLip in Figure 2a. It is evident from this \ufb01gure that the bounds from LipSDP-Neuron are\ntighter than CPLip and SeqLip.\nTo demonstrate the scalability of the LipSDP formulations, we split a 100-hidden layer neural\nnetwork into sub-networks with six hidden layers each and computed the Lipschitz bounds using\nLipSDP-Neuron and LipSDP-Layer. The results are shown in Figure 2b. Furthermore, in Tables\n1 and 2, we show the computation time for scaling the LipSDP methods in the number of hidden\nunits per layer and in the number of layers. In particular, the largest network we tested in Table 2 had\n50,000 hidden neurons; SDPLip-Neuron took approximately 12 minutes to \ufb01nd a Lipschitz bound,\nand SDPLip-Layer took approximately 4 minutes.\nTo evaluate SDPLip-Network, we coupled random pairs of hidden neurons in a one-hidden-layer\nnetwork and plotted the computation time and Lipschitz bound found by SDPLip-Network as we\nincreased the number of paired neurons. Our results show that as the number of coupled neurons\n\non the Lipschitz constant given by L2, upper =(cid:81)(cid:96)\n\ni=0(cid:12)(cid:12)(cid:12)(cid:12)W i(cid:12)(cid:12)(cid:12)(cid:12)2\n\n7\n\n\f(a) Comparison of Lipschitz\nbounds found by various methods\nfor \ufb01ve-hidden-layer networks\ntrained on MNIST with the Adam\noptimizer. Each network had a test\naccuracy above 97%.\nFigure 2: Comparison of the accuracy LipSDP methods to other methods that compute the Lipschitz constant\nand scalability analysis of all three SeqLip methods.\n\n(b) Lipschitz bounds obtained by\nsplitting a 100-layer network into\nsub-networks. Each sub-network\nhad six layers, and the weights\nwere generated randomly by sam-\npling from a normal distribution.\n\n(c) LipSDP-Network Lipschitz\nbounds and computation time for a\none-hidden-layer network with 100\nneurons. The weights for this net-\nwork were obtained by sampling\nfrom a normal distribution.\n\n(a) Lipschitz bounds for a one-hidden-layer neural net-\nworks trained on the MNIST dataset with the Adam\noptimizer and LP-Train and PGD-Train for two val-\nues of the robustness parameter \u0001. Each network\nreached an accuracy of 95% or higher.\n\n(b) Histograms showing the local robustness (in (cid:96)\u221e\nnorm) around each correctly-classi\ufb01ed test instance\nfrom the MNIST dataset. The neural networks had\nthree hidden layers with 100, 50, 20 neurons, respec-\ntively. All classi\ufb01ers had a test accuracy of 97%. We\nused Remark 1 to convert the norm from (cid:96)2 to (cid:96)\u221e.\n\nFigure 3: Analysis of impact of robust training on the Lipschitz constant and the distance to misclassi\ufb01cation\nfor networks trained on MNIST\n\nincreases, the computation time increases quadratically. This shows that while this method is the most\naccurate of the three proposed LipSDP methods, it is intractable for even modestly large networks.\nImpact of robust training. In Figure 3, we empirically demonstrate that the Lipschitz bound of a\nneural network is directly related to the robustness of the corresponding classi\ufb01er. This \ufb01gure shows\nthat LP-train and PGD-Train networks achieve lower Lipschitz bounds than standard training\nprocedures. Figure 3a indicates that robust training procedures yield lower Lipschitz constants than\nnetworks trained with standard training procedures such as the Adam optimizer. Figure 3b shows\nthe utility of sharply estimating the Lipschitz constant; a lower value of L2 guarantees that a neural\nnetwork is more locally robust to input perturbations; see Proposition 1 in the Appendix.\nIn the same vein, Figure 4 shows the impact of varying the robustness parameter \u0001 used in LP-Train\nand PGD-Train on the test accuracy of networks trained for a \ufb01xed number of epochs and the\ncorresponding Lipschitz constants. In essence, these results quantify how much robustness a \ufb01xed\nclassi\ufb01er can handle before accuracy plummets. Interestingly, the drops in accuracy as \u0001 increases\ncoincide with corresponding drops in the Lipschitz constant for both LP-Train and PGD-Train.\nRobustness for different activation functions. The framework proposed in this work allows us\nto examine the impact of using different activation functions on the Lipschitz constant of neural\n\n8\n\n\fFigure 4: Trade-off between accuracy and Lipschitz\nconstant for different values of the robustness parame-\nter used for LP-Train and PGD-Train. All networks\nhad one hidden layer with 50 hidden neurons.\n\nFigure 5: Lipschitz constants for topologically identi-\ncal three-hidden-layer networks with ReLU and leaky\nReLU activation functions. All classi\ufb01ers were trained\nuntil they reached 97% test accuracy.\n\nnetworks. We trained two sets of neural networks on the MNIST dataset. The \ufb01rst set used ReLU\nactivation functions, while the second set used leaky ReLU activations. Figure 5 shows empirically\nthat the networks with the leaky ReLU activation function have larger Lipschitz constants than\nnetworks of the same architecture with the ReLU activation function.\n\n4 Conclusions and future work\n\nIn this paper, we proposed a hierarchy of semide\ufb01nite programs to derive tight upper bounds on\nthe Lipschitz constant of feed-forward fully-connected neural networks. Some comments are in\norder. First, our framework can be directly used to certify convolutional neural networks (CNNs)\nby unrolling them to a large fully-connected neural network. This conversion implicitly handles the\npadding and stride hyper parameters. Since the max function is convex, we can describe the max\npooling operation using incremental quadratic constraints without additional assumptions. Therefore,\nin principle, LipSDP is applicable to CNNs. A future direction is to exploit the special structure of\nCNNs in the resulting SDP. Second, we only considered one application of Lipschitz bounds in depth\n(robustness certi\ufb01cation). Having an accurate upper bound on the Lipschitz constant can be useful\nin domains beyond robustness analysis, such as stability analysis of feedback systems with control\npolicies updated by deep reinforcement learning. Furthermore, Lipschitz bounds can be utilized\nduring training as a heuristic to promote out-of-sample generalization [36]. We intend to pursue these\napplications for future work.\n\nReferences\n[1] Beh\u00e7et A\u00e7\u0131kme\u00b8se and Martin Corless. Observers for systems with nonlinearities satisfying\n\nincremental quadratic constraints. Automatica, 47(7):1339\u20131348, 2011.\n\n[2] Cem Anil, James Lucas, and Roger Grosse. Sorting out Lipschitz function approximation. In\nKamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International\nConference on Machine Learning, volume 97 of Proceedings of Machine Learning Research,\npages 291\u2013301, Long Beach, California, USA, 09\u201315 Jun 2019. PMLR.\n\n[3] MOSEK ApS. The MOSEK optimization toolbox for MATLAB manual. Version 8.1., 2017.\n\n[4] Anil Aswani, Humberto Gonzalez, S Shankar Sastry, and Claire Tomlin. Provably safe and\n\nrobust learning-based model predictive control. Automatica, 49(5):1216\u20131226, 2013.\n\n[5] Radu Balan, Maneesh Singh, and Dongmian Zou. Lipschitz properties for deep convolutional\n\nnetworks. arXiv preprint arXiv:1701.05217, 2017.\n\n9\n\n\f[6] Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds\nfor neural networks. In Advances in Neural Information Processing Systems, pages 6240\u20136249,\n2017.\n\n[7] Osbert Bastani, Yani Ioannou, Leonidas Lampropoulos, Dimitrios Vytiniotis, Aditya Nori, and\nAntonio Criminisi. Measuring neural net robustness with constraints. In Advances in neural\ninformation processing systems, pages 2613\u20132621, 2016.\n\n[8] Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. Safe model-based\nreinforcement learning with stability guarantees. In Advances in neural information processing\nsystems, pages 908\u2013918, 2017.\n\n[9] Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press,\n\n2004.\n\n[10] Patrick L. Combettes and Jean-Christophe Pesquet. Lipschitz certi\ufb01cates for neural network\n\nstructures driven by averaged activation operators. arXiv preprint arXiv:1903.01014, 2019.\n\n[11] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.\n\n[12] Souradeep Dutta, Susmit Jha, Sriram Sankaranarayanan, and Ashish Tiwari. Output range\nanalysis for deep feedforward neural networks. In NASA Formal Methods Symposium, pages\n121\u2013138. Springer, 2018.\n\n[13] Mahyar Fazlyab, Manfred Morari, and George J Pappas. Safety veri\ufb01cation and robustness\nanalysis of neural networks via quadratic constraints and semide\ufb01nite programming. arXiv\npreprint arXiv:1903.01287, 2019.\n\n[14] Mahyar Fazlyab, Alejandro Ribeiro, Manfred Morari, and Victor M Preciado. Analysis of\noptimization algorithms via integral quadratic constraints: Nonstrongly convex problems. SIAM\nJournal on Optimization, 28(3):2654\u20132689, 2018.\n\n[15] Timon Gehr, Matthew Mirman, Dana Drachsler-Cohen, Petar Tsankov, Swarat Chaudhuri,\nand Martin Vechev. Ai2: Safety and robustness certi\ufb01cation of neural networks with abstract\ninterpretation. In 2018 IEEE Symposium on Security and Privacy (SP), pages 3\u201318. IEEE, 2018.\n\n[16] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar-\n\nial examples (2014). arXiv preprint arXiv:1412.6572.\n\n[17] Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael Cree. Regularisation of neural\n\nnetworks by enforcing lipschitz continuity. arXiv preprint arXiv:1804.04368, 2018.\n\n[18] Michael Grant, Stephen Boyd, and Yinyu Ye. Cvx: Matlab software for disciplined convex\n\nprogramming, 2008.\n\n[19] Todd Huster, Cho-Yu Jason Chiang, and Ritu Chadha. Limitations of the lipschitz constant as a\ndefense against adversarial examples. In Joint European Conference on Machine Learning and\nKnowledge Discovery in Databases, pages 16\u201329. Springer, 2018.\n\n[20] Ming Jin and Javad Lavaei. Stability-certi\ufb01ed reinforcement learning: A control-theoretic\n\nperspective. arXiv preprint arXiv:1810.11505, 2018.\n\n[21] Matt Jordan, Justin Lewis, and Alexandros G Dimakis. Provable certi\ufb01cates for adversarial\n\nexamples: Fitting a ball in the union of polytopes. arXiv preprint arXiv:1903.08778, 2019.\n\n[22] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale.\n\narXiv preprint arXiv:1611.01236, 2016.\n\n[23] Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/,\n\n1998.\n\n[24] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.\nTowards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083,\n2017.\n\n10\n\n\f[25] Yurii Nesterov. Introductory lectures on convex optimization: A basic course, volume 87.\n\nSpringer Science & Business Media, 2013.\n\n[26] Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as\na defense to adversarial perturbations against deep neural networks. In 2016 IEEE Symposium\non Security and Privacy (SP), pages 582\u2013597. IEEE, 2016.\n\n[27] Jonathan Peck, Joris Roels, Bart Goossens, and Yvan Saeys. Lower bounds on the robustness\nto adversarial perturbations. In Advances in Neural Information Processing Systems, pages\n804\u2013813, 2017.\n\n[28] Haifeng Qian and Mark N Wegman. L2-nonexpansive neural networks. arXiv preprint\n\narXiv:1802.07896, 2018.\n\n[29] Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. Certi\ufb01ed defenses against adversarial\n\nexamples. arXiv preprint arXiv:1801.09344, 2018.\n\n[30] Aditi Raghunathan, Jacob Steinhardt, and Percy S Liang. Semide\ufb01nite relaxations for certifying\nrobustness to adversarial examples. In Advances in Neural Information Processing Systems,\npages 10900\u201310910, 2018.\n\n[31] Wenjie Ruan, Xiaowei Huang, and Marta Kwiatkowska. Reachability analysis of deep neural\n\nnetworks with provable guarantees. arXiv preprint arXiv:1805.02242, 2018.\n\n[32] Ernest K Ryu and Stephen Boyd. Primer on monotone operator methods. Appl. Comput. Math,\n\n15(1):3\u201343, 2016.\n\n[33] Gagandeep Singh, Timon Gehr, Matthew Mirman, Markus P\u00fcschel, and Martin Vechev. Fast\nand effective robustness certi\ufb01cation. In Advances in Neural Information Processing Systems,\npages 10802\u201310813, 2018.\n\n[34] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel-\nlow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199,\n2013.\n\n[35] Vincent Tjeng, Kai Xiao, and Russ Tedrake. Evaluating robustness of neural networks with\n\nmixed integer programming. arXiv preprint arXiv:1711.07356, 2017.\n\n[36] Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama. Lipschitz-margin training: Scalable\nIn Advances in Neural\n\ncerti\ufb01cation of perturbation invariance for deep neural networks.\nInformation Processing Systems, pages 6541\u20136550, 2018.\n\n[37] Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: analysis and\nef\ufb01cient estimation. In Advances in Neural Information Processing Systems, pages 3835\u20133844,\n2018.\n\n[38] Tsui-Wei Weng, Huan Zhang, Hongge Chen, Zhao Song, Cho-Jui Hsieh, Duane Boning,\nInderjit S Dhillon, and Luca Daniel. Towards fast computation of certi\ufb01ed robustness for relu\nnetworks. arXiv preprint arXiv:1804.09699, 2018.\n\n[39] Tsui-Wei Weng, Huan Zhang, Pin-Yu Chen, Jinfeng Yi, Dong Su, Yupeng Gao, Cho-Jui Hsieh,\nand Luca Daniel. Evaluating the robustness of neural networks: An extreme value theory\napproach. arXiv preprint arXiv:1801.10578, 2018.\n\n[40] Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex\nouter adversarial polytope. In Jennifer Dy and Andreas Krause, editors, Proceedings of the\n35th International Conference on Machine Learning, volume 80 of Proceedings of Machine\nLearning Research, pages 5286\u20135295, Stockholmsm\u00e4ssan, Stockholm Sweden, 10\u201315 Jul 2018.\nPMLR.\n\n[41] Eric Wong, Frank Schmidt, Jan Hendrik Metzen, and J Zico Kolter. Scaling provable adversarial\n\ndefenses. In Advances in Neural Information Processing Systems, pages 8400\u20138409, 2018.\n\n11\n\n\f[42] Huan Zhang, Tsui-Wei Weng, Pin-Yu Chen, Cho-Jui Hsieh, and Luca Daniel. Ef\ufb01cient neural\nIn Advances in Neural\n\nnetwork robustness certi\ufb01cation with general activation functions.\nInformation Processing Systems, pages 4939\u20134948, 2018.\n\n[43] Stephan Zheng, Yang Song, Thomas Leung, and Ian Goodfellow. Improving the robustness of\ndeep neural networks via stability training. In Proceedings of the ieee conference on computer\nvision and pattern recognition, pages 4480\u20134488, 2016.\n\n[44] Dongmian Zou, Radu Balan, and Maneesh Singh. On lipschitz bounds of general convolutional\n\nneural networks. arXiv preprint arXiv:1808.01415, 2018.\n\n12\n\n\f", "award": [], "sourceid": 6086, "authors": [{"given_name": "Mahyar", "family_name": "Fazlyab", "institution": "University of Pennsylvania"}, {"given_name": "Alexander", "family_name": "Robey", "institution": "University of Pennsylvania"}, {"given_name": "Hamed", "family_name": "Hassani", "institution": "UPenn"}, {"given_name": "Manfred", "family_name": "Morari", "institution": "University of Pennsylvania"}, {"given_name": "George", "family_name": "Pappas", "institution": "University of Pennsylvania"}]}