{"title": "Learning ReLUs via Gradient Descent", "book": "Advances in Neural Information Processing Systems", "page_first": 2007, "page_last": 2017, "abstract": "In this paper we study the problem of learning Rectified Linear Units (ReLUs) which are functions of the form $\\vct{x}\\mapsto \\max(0,\\langle \\vct{w},\\vct{x}\\rangle)$ with $\\vct{w}\\in\\R^d$ denoting the weight vector.  We study this problem in the high-dimensional regime where the number of observations are fewer than the dimension of the weight vector. We assume that the weight vector belongs to some closed set (convex or nonconvex) which captures known side-information about its structure. We focus on the realizable model where the inputs are chosen i.i.d.~from a Gaussian distribution and the labels are generated according to a planted weight vector. We show that projected gradient descent, when initialized at $\\vct{0}$, converges at a linear rate to the planted model with a number of samples that is optimal up to numerical constants. Our results on the dynamics of convergence of these very shallow neural nets may provide some insights towards understanding the dynamics of deeper architectures.", "full_text": "Learning ReLUs via Gradient Descent\n\nMing Hsieh Department of Electrical Engineering\n\nUniversity of Southern California\n\nMahdi Soltanolkotabi\n\nLos Angeles, CA\n\nsoltanol@usc.edu\n\nAbstract\n\nwhich are functions of the form x\u0015 max(0,\u001bw, x\u001b) with w\u2208 Rd denoting the\n\nIn this paper we study the problem of learning Recti\ufb01ed Linear Units (ReLUs)\n\nweight vector. We study this problem in the high-dimensional regime where the\nnumber of observations are fewer than the dimension of the weight vector. We\nassume that the weight vector belongs to some closed set (convex or nonconvex)\nwhich captures known side-information about its structure. We focus on the\nrealizable model where the inputs are chosen i.i.d. from a Gaussian distribution\nand the labels are generated according to a planted weight vector. We show that\nprojected gradient descent, when initialized at 0, converges at a linear rate to the\nplanted model with a number of samples that is optimal up to numerical constants.\nOur results on the dynamics of convergence of these very shallow neural nets may\nprovide some insights towards understanding the dynamics of deeper architectures.\n\n1\n\nIntroduction\n\nNonlinear data-\ufb01tting problems are fundamental to many supervised learning tasks in signal process-\n\ning and machine learning. Given training data consisting of n pairs of input features xi\u2208 Rd and\ndesired outputs yi\u2208 R we wish to infer a function that best explains the training data. In this paper\nwe focus on \ufb01tting Recti\ufb01ed Linear Units (ReLUs) to the data which are functions \u03c6w\u2236 Rd\u2192 R of\n\nthe form\n\n\u03c6w(x)= max(0,\u001bw, x\u001b) .\n(max(0,\u001bw, xi\u001b)\u2212 yi)2\n\nsubject to R(w)\u2264 R,\n\nw\u2208Rd L(w)\u2236= 1\n\nmin\n\nnQ\ni=1\n\nA natural approach to \ufb01tting ReLUs to data is via minimizing the least-squares mis\ufb01t aggregated over\nthe data. This optimization problem takes the form\n\n(1.1)\n\nn\n\nwithR\u2236 Rd\u2192 R denoting a regularization function that encodes prior information on the weight\n\nvector.\nFitting nonlinear models such as ReLUs have a rich history in statistics and learning theory [12]\nwith interesting new developments emerging [6] (we shall discuss all these results in greater detail in\nSection 5). Most recently, nonlinear data \ufb01tting problems in the form of neural networks (a.k.a. deep\nlearning) have emerged as powerful tools for automatically extracting interpretable and actionable\ninformation from raw forms of data, leading to striking breakthroughs in a multitude of applications\n[13, 15, 4]. In these and many other empirical domains it is common to use local search heuristics\nsuch as gradient or stochastic gradient descent for nonlinear data \ufb01tting. These local search heuristics\nare surprisingly effective on real or randomly generated data. However, despite their empirical success\nthe reasons for their effectiveness remains mysterious.\n\n31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.\n\n\fFocusing on \ufb01tting ReLUs, a-priori it is completely unclear why local search heuristics such as\ngradient descent should converge for problems of the form (1.1), as not only the regularization\nfunction maybe nonconvex but also the loss function! Ef\ufb01cient \ufb01tting of ReLUs in this high-\ndimensional setting poses new challenges: When are the iterates able to escape local optima and\nsaddle points and converge to global optima? How many samples do we need? How does the number\nof samples depend on the a-priori prior knowledge available about the weights? What regularizer\nis best suited to utilizing a particular form of prior knowledge? How many passes (or iterations) of\nthe algorithm is required to get to an accurate solution? At the heart of answering these questions is\nthe ability to predict convergence behavior/rate of (non)convex constrained optimization algorithms.\nIn this paper we build up on a new framework developed in the context of phase retrieval [21] for\nanalyzing nonconvex optimization problems to address such challenges.\n\n2 Precise measures for statistical resources\n\n\u2217+ h)\u2264R(w\n\n\u2217)\u0002.\n\n\u2217)=\u0002h\u2236 R(w\n\nWe wish to characterize the rates of convergence for the projected gradient updates (3.2) as a function\nof the number of samples, the available prior knowledge and the choice of the regularizer. To make\nthese connections precise and quantitative we need a few de\ufb01nitions. Naturally the required number\n\nproperties of the weight vector w. For example, if we know that the weight vector is approximately\nsparse, naturally using an (cid:96)1 norm for the regularizer is superior to using an (cid:96)2 regularizer. To quantify\nthis capability we \ufb01rst need a couple of standard de\ufb01nitions which we adapt from [17, 18, 21].\n\nof samples for reliable data \ufb01tting depends on how well the regularization functionR can capture the\nDe\ufb01nition 2.1 (Descent set and cone) The set of descent of a functionR at a point w\u2217 is de\ufb01ned as\nDR(w\nThe cone of descent is de\ufb01ned as a closed coneCR(w\u2217) that contains the descent set, i.e.DR(w\u2217)\u2282\nCR(w\u2217). The tangent cone is the conic hull of the descent set. That is, the smallest closed cone\nCR(w\u2217) obeyingDR(w\u2217)\u2282CR(w\u2217).\nWe note that the capability of the regularizerR in capturing the properties of the unknown weight\nvector w\u2217 depends on the size of the descent coneCR(w\u2217). The smaller this cone is the more suited\nthe functionR is at capturing the properties of w\u2217. To quantify the size of this set we shall use the\nDe\ufb01nition 2.2 (Gaussian width) The Gaussian width of a setC\u2208 Rd is de\ufb01ned as:\nwhere the expectation is taken over g\u223cN(0, Ip). Throughout we useBd~Sd\u22121 to denote the the unit\nWe now have all the de\ufb01nitions in place to quantify the capability of the functionR in capturing the\nproperties of the unknown parameter w\u2217. This naturally leads us to the de\ufb01nition of the minimum\nDe\ufb01nition 2.3 (minimal number of samples) LetCR(w\u2217) be a cone of descent ofR at w\u2217. We\nde\ufb01ne the minimal sample function asM(R, w\nWe shall often use the short hand n0=M(R, w\u2217) with the dependence onR, w\u2217 implied.\n\n\u03c9(C)\u2236= Eg[sup\n\nz\u2208C\u001bg, z\u001b],\n\n\u2217)= \u03c92(CR(w\n\n\u2217)\u2229Bd).\n\nrequired number of samples.\n\nnotion of mean width.\n\nball/sphere of Rd.\n\nWe note that n0 is exactly the minimum number of samples required for structured signal recovery\nfrom linear measurements when using convex regularizers [3, 1]. Speci\ufb01cally, the optimization\nproblem\n\n(yr\u2212\u001bxi, w\n\n\u2217\u001b)2\n\nsubject to R(w)\u2264R(w\n\n\u2217),\n\nnQ\ni=1\n\n(2.1)\n\n2\n\n\fsucceeds at recovering an unknown weight vector w\u2217 with high probability from n observations of\nthe form yi=\u001bai, w\u2217\u001b if and only if n\u2265 n0.1 While this result is only known to be true for convex\ncompared to linear observations) we can not hope to recover the weight vector from n\u2264 n0 when\n\nregularization functions we believe that n0 also characterizes the minimal number of samples even for\nnonconvex regularizers in (2.1). See [17] for some results in the nonconvex case as well as the role\nthis quantity plays in the computational complexity of projected gradient schemes for linear inverse\nproblems. Given that with nonlinear samples we have less information (we loose some information\n\nusing (1.1). Therefore, we can use n0 as a lower-bound on the minimum number of observations\nrequired for projected gradient descent iterations (3.2) to succeed at \ufb01nding the right model.\n\n3 Theoretical results for learning ReLUs\n\nA simple heuristic for optimizing (1.1) is to use gradient descent. One challenging aspect of the\nabove loss function is that it is not differentiable and it is not clear how to run projected gradient\ndescent. However, this does not pose a fundamental challenge as the loss function is differentiable\nexcept for isolated points and we can use the notion of generalized gradients to de\ufb01ne the gradient at\na non-differentiable point as one of the limit points of the gradient in a local neighborhood of the\nnon-differentiable point. For the loss in (1.1) the generalized gradient takes the form\n\n\u2207L(w)\u2236= 1\n\nn\n\n(ReLU(\u001bw, xi\u001b)\u2212 yi)(1+ sgn(\u001bw, xi\u001b)) xi.\nnQ\ni=1\nw\u03c4+1=PK(w\u03c4\u2212 \u00b5\u03c4\u2207L(w\u03c4)) ,\n\nTherefore, projected gradient descent takes the form\n\n(3.2)\n\nEuclidean projection onto this set.\n\nwhere \u00b5\u03c4 is the step size andK={w\u2208 Rd\u2236R(w)\u2264 R} is the constraint set withPK denoting the\nTheorem 3.1 Let w\u2217 \u2208 Rd be an arbitrary weight vector andR\u2236 Rd \u2192 R be a proper function\n(convex or nonconvex). Suppose the feature vectors xi \u2208 Rd are i.i.d. Gaussian random vectors\ndistributed asN(0, I) with the corresponding labels given by\nyi= max(0,\u001bxi, w\n\u2217\u001b) .\nTo estimate w\u2217, we start from the initial point w0= 0 and apply the Projected Gradient Descent\nw\u03c4+1=PK(w\u03c4\u2212 \u00b5\u03c4\u2207L(w\u03c4)) ,\nwithK\u2236={w\u2208 Rd\u2236 R(w)\u2264R(w\u2217)} and\u2207L de\ufb01ned via (3.1). Also set the learning parameter\nsequence to \u00b50= 2 and \u00b5\u03c4 = 1 for all \u03c4= 1, 2, . . . and let n0=M(R, w\u2217), per De\ufb01nition 2.3, be\nholds for a \ufb01xed numerical constant c. Then there is an event of probability at least 1\u2212 9e\u2212\u03b3n such\n\nour lower bound on the number of observations. Also assume\n\n(PGD) updates of the form\n\n(3.4)\n\n(3.3)\n\nthat on this event the updates (3.3) obey\n\n(3.1)\n\nn> cn0,\n\u2217\u0001(cid:96)2\u2264\u0003 1\n\n\u0001w\u03c4\u2212 w\n\n2\u0003\u03c4\u0001w\n\n\u2217\u0001(cid:96)2\n\n.\n\n(3.5)\n\nHere \u03b3 is a \ufb01xed numerical constant.\n\nThe \ufb01rst interesting and perhaps surprising aspect of this result is its generality: it applies not only to\nconvex regularization functions but also nonconvex ones! As we mentioned earlier the optimization\nproblem in (1.1) is not known to be tractable even for convex regularizers. Despite the nonconvexity\nof both the objective and regularizer, the theorem above shows that with a near minimal number\n\nmore precise characterization is \u03c6\u22121(\u03c92(CR(w\u2217)\u2229Bd))\u2248 \u03c92(CR(w\u2217)\u2229Bd) where \u03c6(t)=\u221a\n\n1We would like to note that n0 only approximately characterizes the minimum number of samples required. A\nt.\n\n2\u0001 \u2248\u221a\n2 \u0001\n\u0393\u0001 t+1\n\u0393\u0001 t\n\nHowever, since our results have unspeci\ufb01ed constants we avoid this more accurate characterization.\n\n2\n\n3\n\n\faverage behavior over 100 trials. None bold color depict the estimation error of some sample trials.\n\nnumber of iterations \u03c4. The plots are for two different observations models: 1) ReLU observations of\n\nFigure 1: Estimation error (\u0001w\u03c4\u2212 w\u2217\u0001(cid:96)2) obtained via running PGD iterates as a function of the\nthe form y=ReLU(Xw\u2217) and 2) linear observations of the form y= Xw\u2217. The bold colors depict\nof data samples, projected gradient descent provably learns the original weight vector w\u2217 without\nachieve a relative error of \u0001 the total number of iterations is on the order ofO(log(1~\u0001)). Thus the\noverall computational complexity is on the order ofO(nd log(1~\u0001)) (in general the cost is the total\none matrix-vector multiplication which takesO(nd) time.\n\nnumber of iterations multiplied by the cost of applying the feature matrix X and its transpose). As\na result, the computational complexity is also now optimal in terms of dependence on the matrix\ndimensions. Indeed, for a dense matrix even verifying that a good solution has been achieved requires\n\ngetting trapped in any local optima.\nAnother interesting aspect of the above result is that the convergence rate is linear. Therefore, to\n\n4 Numerical experiments\n\nIn this section we carry out a simple numerical experiment to corroborate our theoretical results. For\n\nthis purpose we generate a unit norm sparse vector w\u2217\u2208 Rd of dimension d= 1000 containing s=\nd~50 non-zero entries. We also generate a random feature matrix X\u2208 Rn\u00d7d with n=\u00e68s log(d~s)\u00e6\nand containing i.i.d.N(0, 1) entries. We now take two sets of observations of size n from \u03b8\u2217:\nWe apply the projected gradient iterations to both observation models starting from w0= 0. For the\n\n\u2022 ReLU observations: the response vector is equal to y=ReLU(Xw\u2217).\n\u2022 Linear observations: the response is y= Xw\u2217.\n\nReLU observations we use the step size discussed in Theorem 3.1. For the linear model we apply\nprojected gradient descent updates of the form\n\nw\u03c4+1=PK\u0003w\u03c4\u2212 1\n\nX T(Xw\u03c4\u2212 y)\u0003 .\n\nIn both cases we use the regularizerR(w)=\u0001w\u0001(cid:96)0 so that the projection only keeps the top s\n(\u0001w\u03c4\u2212 w\u2217\u0001(cid:96)2) is depicted as a function of the number of iterations \u03c4. The bold colors depict average\n\nentries of the vector (a.k.a. iterative hard thresholding). In Figure 1 the resulting estimation errors\n\nbehavior over 100 trials. The estimation error of some sample trials are also depicted in none bold\n\nn\n\n4\n\n0510152000.20.40.60.81Estimation errorReLU samplesLinear samples\fcolors. This plot clearly show that PGD iterates applied to ReLU observations converge quickly\nto the ground truth. This \ufb01gure also clearly demonstrates that the behavior of the PGD iterates\napplied to both models are similar, further corroborating the results of Theorem 3.1. We note that\n\nthe sample complexity used in this simulation is 8s log(n~s) which is a constant factor away from\nn0\u221d s log(n~s) con\ufb01rming our assertion that the required sample complexity is a constant factor\n\naway from n0 (as predicted by Theorem 3.1).\n\n5 Discussions and prior art\n\nThere is a large body of work on learning nonlinear models. A particular class of such problems\nthat have been studied are the so called idealized Single Index Models (SIMs) [9, 10]. In these\n\nproblems the inputs are labeled examples{(xi, yi)}n\ni=1\u2208 Rd\u00d7 R which are guaranteed to satisfy\nyi= f(\u001bw, xi\u001b) for some w\u2208 Rd and nondecreasing (Lipchitz continuous) f\u2236 R\u2192 R. The goal in\n\nthis problem is to \ufb01nd a (nearly) accurate such f and w. An interesting polynomial-time algorithm\ncalled the Isotron exists for this problem [12, 11]. In principle, this approach can also be used to\n\ufb01t ReLUs. However, these results differ from ours in term of both assumptions and results. On the\none had, the assumptions are slightly more restrictive as they require bounded features xi, outputs\nyi and weights. On the other hand, these result hold for much more general distributions and more\ngeneral models than the realizable model studied in this paper. These results also do not apply in the\nhigh dimensional regime where the number of observations is signi\ufb01cantly smaller than the number\nof parameters (see [5] for some results in this direction). In the realizable case, the Isotron result\n\nrequireO( 1\n\u0001) iterations to achieve \u0001 error in objective value. In comparison, our results guarantee\nconvergence to a solution with relative error \u0001 (\u0001w\u03c4\u2212 w\u2217\u0001(cid:96)2~\u0001w\u2217\u0001(cid:96)2\u2264 \u0001) after log(1~\u0001) iterations.\nan accuracy of \u0001 the algorithm runs in poly(1~\u0001) time. In comparison, as mentioned earlier our result\nrquires log(1~\u0001) iterations for reliable parameter estimation. We note however we study the problem\nthat with two hidden unites (k= 2) there are no critical points for con\ufb01gurations where both weight\n\nin different settings and a direct comparison is not possible between the two results.\nWe would like to note that there is an interesting growing literature on learning shallow neural\nnetworks with a single hidden layer with i.i.d. inputs, and under a realizable model (i.e. the labels are\ngenerated from a network with planted weights) [23, 2, 25]. For isotropic Gaussian inputs, [23] shows\n\nFocusing on the speci\ufb01c case of ReLU functions, an interesting recent result [6] shows that reliable\nlearning of ReLUs is possible under very general but bounded distributional assumptions. To achieve\n\nvectors fall into (or outside) the cone of ground truth weights. With the same assumptions, [2] proves\nthat for a single-hidden ReLU network with a single non-overlapping convolutional \ufb01lter, all local\nminimizers of the population loss are global; they also give counter-examples in the overlapping case\nand prove the problem is NP-hard when inputs are not Gaussian. [25] studies general single-hidden\nlayer networks and shows that a version of gradient descent which uses a fresh batch of samples in\neach iteration converges to the planted model. This holds using an initialization obtained via a tensor\ndecomposition method. Our approach and convergence results differ from this literature in a variety\nof different ways. First, we focus on zero hidden layers with a regularization term. Some of this\nliterature focuses on one-hidden layers without (or with speci\ufb01c) regularization. Second, unlike some\nof these results such as [2, 14], we study the optimization properties of the empirical function, not its\nexpected value. Third, we initialize at zero in lieu of sophisticated initialization schemes. Finally,\nour framework does not require a fresh batch of samples per new gradient iteration as in [25]. We\nalso note that several publications study the effect of over-parametrization on the training of neural\nnetworks without any regularization [19, 8, 16, 22]. Therefore, the global optima are not unique\nand hence the solutions may not generalize. In comparison we study the problem with an arbitrary\nregularization which allows for a unique global optima.\n\n6 Proofs\n\n6.1 Preliminaries\n\nIn this section we gather some useful results on concentration of stochastic processes which will be\ncrucial in our proofs. These results are mostly adapted from [21]. We begin with a lemma which is a\ndirect consequence of Gordon\u2019s escape from the mesh lemma [7].\n\n5\n\n\f\u03b42\n\nn\n\n,\n\n(cid:96)2\n\n,\n\n2\n\n,\n\n\u03b42\n\n1\n\n,\n\n360 n.\n\n1440 n.\n\n\u0004 1\n\nn\n\nRd. Also assume that\n\nWe also need a generalization of the above lemma stated below.\n\nWe next state a generalization of Gordon\u2019s escape through the mesh lemma also from [21].\n\n2\u03b4\u2212 1\u0004 ,\n(cid:96)2\u0004\u2264 \u03b4\u0001h\u00012\n\n\u03b4\u2212 1\u0004 ,\nh\u0004\u2264 \u03b4\u0001u\u0001(cid:96)2\u0001h\u0001(cid:96)2\n\nLemma 6.1 AssumeC\u2282 Rd is a cone and Sd\u22121 is the unit sphere of Rd. Also assume that\n\u03c92(C\u2229 Sd\u22121)\nn\u2265 max\u000420\nfor a \ufb01xed numerical constant c. Then for all h\u2208C\n\u0004 1\n(\u001bxi, h\u001b)2\u2212\u0001h\u00012\nnQ\ni=1\nholds with probability at least 1\u2212 2e\u2212 \u03b42\nLemma 6.2 ([21]) AssumeC\u2282 Rd is a cone (not necessarily convex) and Sd\u22121 is the unit sphere of\n\u03c92(C\u2229 Sd\u22121)\nn\u2265 max\u000480\nfor a \ufb01xed numerical constant c. Then for all u, h\u2208C\n\u001bxi, u\u001b\u001bxi, h\u001b\u2212 u\nnQ\n\u2217\ni=1\nholds with probability at least 1\u2212 6e\u2212 \u03b42\nLemma 6.3 ([21]) Let s\u2208 Rd be \ufb01xed vector with nonzero entries and construct the diagonal matrix\nS= diag(s). Also, let X \u2208 Rn\u00d7d have i.i.d.N(0, 1) entries. Furthermore, assumeT \u2282 Rd and\nde\ufb01ne bd(s)= E[\u0001Sg\u0001(cid:96)2], where g\u2208 Rd is distributed asN(0, In). Also, de\ufb01ne\nThen for all u\u2208T\nholds with probability at least 1\u2212 6e\nCorollary 6.4 Let s\u2208 Rd be \ufb01xed vector with nonzero entries and assumeT \u2282Bd. Furthermore,\n\u0001s\u00012\n(cid:96)2\u2265 max\u000420\u0001s\u00012\nThen for all u\u2208T ,\nRRRRRRRRRRR\u2211n\ni(\u001bxi, u\u001b)2\ni=1 s2\n\u0001s\u00012\nholds with probability at least 1\u2212 6e\n1440\u0001s\u00012\n\u2212 \u03b42\nIn this section we shall prove Theorem 3.1. Throughout, we use the shorthandC to denote the descent\ncone ofR at w\u2217, i.e.C=CR(w\u2217). We begin by analyzing the \ufb01rst iteration. Using w0= 0 we have\n\n\u0001\u0001SAu\u0001(cid:96)2\u2212 bd(s)\u0001u\u0001(cid:96)2\u0001\u2264\u0001s\u0001(cid:96)\u221e \u03c9(T)+ \u03b7,\n\n(cid:96)\u221e \u03c92(T)\n\u2212\u0001u\u00012\n\n\u03b42\n\n3\n\n2\u03b4\u2212 1\u0004 .\nRRRRRRRRRRR\u2264 \u03b4,\n\n\u03c3(T)\u2236= max\n\nv\u2208T \u0001v\u0001(cid:96)2\n\n6.2 Convergence proof (Proof of Theorem 3.1)\n\nThe previous lemma leads to the following Corollary.\n\n8\u0001s\u00012\n\n\u03b72\n\n(cid:96)\u221e \u03c32(T)\n\nassume\n\n(cid:96)2\n\n(cid:96)2 .\n\n,\n\n(cid:96)2\n\nw1\u2236=PK(w0\u2212 \u00b50\u2207L(w0))=PK\u0004 2\n\nnQ\ni=1\n\nyixi\u0004=PK\u0004 2\n\nn\n\nnQ\ni=1\n\nn\n\nReLU(\u001bxi, w\n\n\u2217\u001b)xi\u0004 .\n\n\u2212\n\n.\n\n.\n\n6\n\n\fn\n\n(cid:96)2\n\nsup\n\n(cid:96)2\n\n(cid:96)2\n\n(6.1)\n\n\u2217\u001b,\n\n2\nn\n\nn\n\n(cid:96)2\n\n(cid:96)2\n\n(6.3)\n\n(6.4)\n\n(cid:96)2\n\nn\n\nn\n\nn\n\n2 we have\n\nX T X\u2212 I\u0003 w\n\nWe use the argument of [21][Page 25, inequality (7.34)] which shows that\n\nWe proceed by bounding the \ufb01rst term in the above equality. To this aim we decompose u in the\n\n\u2217\u0004 .\n\u2217\u001b)xi\u2212 w\nReLU(\u001bxi, w\nuT\u0004 2\n\u2217\u0001(cid:96)2\u2264 2\u22c5\n\u0001w1\u2212 w\nnQ\nu\u2208C\u2229Bd\ni=1\nUsing ReLU(z)= z+z\n\u001bxi, w\n\u2217+ 1\nX T X\u2212 I\u0003 w\n\u2217\u001b= uT\u0003 1\n\u2217\u001b)\u001bxi, u\u001b\u2212\u001bu, w\nReLU(\u001bxi, w\n\u2217\u001b\u001bxi, u\u001b. (6.2)\nnQ\nnQ\ni=1\ni=1\ndirection parallel/perpendicular to that of w\u2217 and arrive at\nn\u001bX\u00ef\u00efI\u2212 w\u2217(w\u2217)T\n\u00ef\u0017 u, Xw\n\u2217=(uT w\u2217)\n\u2217+ 1\n(w\n\u2217)T\u0003 1\nuT\u0003 1\nX T X\u2212 I\u0003 w\n\u0001w\u2217\u00012\n\u0001w\u2217\u00012\n\u2217)\u00ef\u00ef\u0001g\u00012\nn \u2212 1\u00ef\u0017+\u0001w\u2217\u0001(cid:96)2\u221an\naT\u00ef\u00efI\u2212 w\u2217(w\u2217)T\n\u00ef\u0017 u,\n\u223c(uT w\n\u0001w\u2217\u00012\nRRRRRRRRRRR\nRRRRRRRRRRR+\u0001w\u2217\u0001(cid:96)2\u221an\n\u0001g\u00012\n\u00ef\u0017 u,\naT\u00ef\u00efI\u2212 w\u2217(w\u2217)T\nn \u2212 1\n\u2217\u0001(cid:96)2\n\u2264\u0001w\n\u0001w\u2217\u00012\nu\u2208C\u2229Bd\nwith g\u2208 Rn and a\u2208 Rd are independent random Gaussian random vectors distributed asN(0, Id)\nandN(0, In). By concentration of Chi-squared random variables\n\u0002\u0001g\u00012\n(cid:96)2~n\u2212 1\u0002\u2264 \u2206,\nholds with probability at least 1\u2212 2e\u2212n \u22062\n\u00ef\u0017 u\u2264 1\u221an\u0001\u03c9\u0001C\u2229Bd\u0001+ \u03b7\u0001 ,\naT\u00ef\u00efI\u2212 w\u2217(w\u2217)T\n1\u221an\n\u0001w\u2217\u00012\n6 and (6.5) with \u03b7= \u03b4\nholds with probability at least 1\u2212 e\u2212 \u03b72\n2 . Plugging (6.4) with \u2206= \u03b4\n\u03b42 \u03c92\u0001C\u2229Bd\u0001, then\n(6.3), as long as n\u2265 36\nuT\u0003 1\n\u2217\u2264 \u03b4\n2\u0001w\nX T X\u2212 I\u0003 w\n\u2217\u0001(cid:96)2\nholds with probability at least 1\u2212 3e\u2212n \u03b42\nparallel/perpendicular to that of w\u2217 and arrive at\n\u2217\u001b\u001bxi, u\u001b\u0004=RRRRRRRRRRR(uT w\n\u001bxi, w\u2217\u001b\u001bxi, w\u2217\u001b\n\u0004 1\n+ 1\n\u2217) 1\nnQ\nnQ\n\u0001w\u2217\u00012\ni=1\ni=1\nRRRRRRRRRRR+\u0004 1\nRRRRRRRRRRR 1\n\u001bxi, w\u2217\u001b\u001bxi, w\u2217\u001b\n\u2264\u0001w\n\u2217\u0001(cid:96)2\nnQ\n\u0001w\u2217\u00012\ni=1\nwith u\u22a5=\u0003I\u2212 w\n\u0003 u. Now note that\n\u0001\u001bxi,w\n\u2217\u001b\u0001\u001bxi,w\n\u2217\u001b\n\u2217(w\n\u2217)T\n\u0001w\u2217\u00012\n\u0001w\u2217\u00012\n\u0005\u001bxi, w\u2217\u001b\u001bxi, w\u2217\u001b\n\u2264 c,\n\u0005\n\u0001w\u2217\u00012\nRRRRRRRRRRR\u2264 t,\nRRRRRRRRRRR 1\n\u001bxi, w\u2217\u001b\u001bxi, w\u2217\u001b\nnQ\n\u0001w\u2217\u00012\ni=1\n\n\u2217\u001b\u001bxi, u\u22a5\u001bRRRRRRRRRRR ,\n\u001bxi, w\nnQ\ni=1\n\u2217\u001b\u001bxi, u\u22a5\u001b\u0004 . (6.7)\n\u001bxi, w\nnQ\ni=1\n\nWe now focus on bounding the second term in (6.2). To this aim we decompose u in the direction\n\nwith \ufb01xed numerical constant. Thus by Bernstein\u2019s type inequality ([24][Proposition 5.16])\n\n(6.5)\n\n\u221an into\n\n\u001bxi, w\n\nis sub-exponential and\n\nu\u2208C\u2229Bd\n\nsup\n\n(cid:96)2\n\n(cid:96)2\n\n(cid:96)2\n\n,\n\n(6.6)\n\n8 . Also,\n\n(cid:96)2\n\nn\n\n288 .\n\nn\n\nn\n\n(cid:96)2\n\n\u03c81\n\nn\n\n(cid:96)2\n\n6\n\n(6.8)\n\nsup\n\nn\n\nn\n\nn\n\n(cid:96)2\n\n7\n\n\f1\nn\n\nn\n\n(cid:96)2\n\nsup\n\n8 and\n\nFurthermore, 1\n\n\u001bxi, w\n\n\u001bxi, w\n\n2 . Combining the last two inequalities we conclude that\n\n\u001bxi, w\u2217\u001b2 1\u221an\u001bg, u\u22a5\u001b.\nnQ\ni=1\n, holds with probability at least 1\u2212 2e\u2212n \u22062\n\nholds with probability at least 1\u2212 2e\u2212\u03b3n min\u0001t2,t\u0001 with \u03b3 a \ufb01xed numerical constant.. Also note that\n\n\u2217\u001b\u001bxi, u\u22a5\u001b\u223c\u00bf``(cid:192) 1\nnQ\ni=1\ni=1\u001bxi, w\u2217\u001b2\u2264(1+ \u2206)\u0001w\u2217\u00012\nn\u2211n\n\u001bg, u\u22a5\u001b\u2264(2\u03c9\u0001C\u2229 Sd\u22121\u0001+ \u03b7),\nu\u2208C\u2229Sd\u22121\nholds with probability at least 1\u2212 e\u2212 \u03b72\n\u2217\u001b\u001bxi, u\u22a5\u001b\u0004\u2264\u221a\n(2\u03c9\u0001C\u2229 Sd\u22121\u0001+ \u03b7)\n\u221an\n1+ \u2206\nholds with probability at least 1\u2212 2e\u2212n \u22062\n8 \u2212 e\u2212 \u03b72\n2 . Plugging (6.8) and (6.9) with t= \u03b4\n\u03b7= \u03b4\n\u221a\n\u2217\u001b\u001bxi, u\u001b\u0004\u2264 \u03b4\n\u001bxi, w\n2\u0001w\n8 as long as n\u2265 288 \u03c92\u0001C\u2229Sd\u22121\u0001\n\u2217\u0004\u2264 2\u03b4\u0001w\n\u2217\u001b)xi\u2212 w\n\nholds with probability at least 1\u2212 3e\u2212\u03b3n\u03b42\u2212 2e\u2212 n\nand (6.10) into (6.1) we conclude that for \u03b4= 7~400\n\u0001w1\u2212 w\n\u2217\u0001(cid:96)2\nReLU(\u001bxi, w\nholds with probability at least 1\u2212 8e\u2212\u03b3n as long as n\u2265 c\u03c92\u0001C\u2229 Sd\u22121\u0001 for a \ufb01xed numerical constant\n\n\u0004 1\nnQ\ni=1\n\u221an into (6.7)\n\nnQ\ni=1\nuT\u0004 2\n\n\u2217\u0001(cid:96)2\u2264 7\n200\u0001w\n\n6, \u2206= 1, and\n\n\u2217\u0001(cid:96)2\u2264 2\u22c5\n\nu\u2208C\u2229Bd\n\nsup\n\n. Thus pluggin (6.6)\n\n\u2217\u0001(cid:96)2\n\n\u2217\u0001(cid:96)2\n\n\u0001w\n\nnQ\ni=1\n\n\u0004 1\n\n(6.10)\n\n(6.9)\n\n6\n\n2\n\n,\n\n\u03b42\n\nn\n\nn\n\nn\n\n,\n\n,\n\nprojected gradient descent update w\u03c4+1=PK(w\u03c4\u2212\u2207L(w\u03c4)), the error h\u03c4= w\u03c4\u2212 w\u2217 obeys\n\nTo prove Theorem 3.1 we use [21][Page 25, inequality (7.34)] which shows that if we apply the\n\n200\n\n.\n\n(6.11)\n\n(6.13)\n\nWe will instead prove that the following stronger result holds for all u\u2208C\u2229Bn and w\u2208 E(\u0001)\n\n.\n\n(6.12)\n\n\u2217\u0001(cid:96)2\u0001 with \u0001= 7\n\u2217(h\u03c4\u2212\u2207L(w\u03c4)) .\n\u2217\u0001(cid:96)2\n4\u0001w\u03c4\u2212 w\n\u2217\u0001(cid:96)2\n\n.\n\nc. To introduce our general convergence analysis we begin by de\ufb01ning\n\nu\n\nu\n\nsup\n\nsup\n\nu\u2208C\u2229Bn\n\n\u2217\u0001(cid:96)2\u2264 \u0001\u0001w\n\nTo complete the convergence analysis it is then suf\ufb01cient to prove\n\nE(\u0001)=\u0001w\u2208 Rd\u2236R(w)\u2264R(w\n\u0001h\u03c4+1\u0001(cid:96)2=\u0001w\u03c4+1\u2212 w\n\n\u2217), \u0001w\u2212 w\n\u2217\u0001(cid:96)2\u2264 2\u22c5\nu\u2208C\u2229Bn\n\u2217(h\u03c4\u2212\u2207L(w\u03c4))\u2264 1\n4\u0001h\u03c4\u0001(cid:96)2= 1\n4\u0001w\u2212 w\n\u2217\u2212\u2207L(w))\u2264 1\n\u2217(w\u2212 w\n(6.13). To this aim note that ReLU(\u001bxi, w\u001b)= \u001bxi,w\u001b+\u001bxi,w\u001b\n\u001b\u2207L(w), u\u001b= 1\nsgn(\u001bxi, w\n\u2217\u001b\u001bxi, u\u001b+ 1\n\u001bxi, w\u2212 w\nnQ\nnQ\ni=1\ni=1\n(sgn(\u001bxi, w\u001b)\u2212 sgn(\u001bxi, w\n\u2217\u001b))\u001bxi, w\u2212 w\n+ 1\nnQ\ni=1\n\u2217\u001b))(sgn(\u001bxi, w\n(1\u2212 sgn(\u001bxi, w\n+ 1\nnQ\ni=1\n\npaper [20] for more detailed derivation of the identity below)\n\n2n\n\nu\n\nn\n\nn\n\nn\n\n2\n\n\u2217\u001b)\u001bxi, w\u2212 w\n\u2217\u001b\u001bxi, u\u001b\n\u2217\u001b\u001bxi, u\u001b\n\u2217\u001b)\u2212 sgn(\u001bxi, w\u001b))\u001bxi, w\n\n\u2217\u001b\u001bxi, u\u001b\n\nThe equation (6.13) above implies (6.12) which when combined with (6.11) proves the convergence\nresult of the Theorem (speci\ufb01cally equation (3.5)). The rest of this section is dedicated to proving\n. Thus (see the extended version of this\n\n8\n\n\f1\nn\n\nn\n\nn\n\nnQ\ni=1\n\nn\n\nn\n\n(cid:96)2\n\n2n\n\n\u2217\u001b))\n\n+ 1\n+ 1\n\nn\n\nn\n\nthe form (see the proof in the extended version of this paper [20] for more detailed derivation)\n\n\u2217\u001b)\u001bxi, h\u001b\u001bxi, u\u001b,\n\u2217\u001b)) sgn(\u001bxi, w\n\u2217\u001b))(1\u2212 sgn(\u001bxi, w\u001b)sgn(\u001bxi, w\n\n(6.14)\nWe now proceed by stating bounds on each of the four terms in (6.14). The detailed derivation of\nthese bounds appear in the the extended version of this paper [20].\n\nNow de\ufb01ning h= w\u2212 w\u2217 we conclude that\u001bu, w\u2212 w\u2217\u2212\u2207L(w)\u001b=\u001bu, h\u2212\u2207L(w)\u001b is equal to\nsgn(\u001bxi, w\nXX T\u0003 h\u2212 1\n\u001bu, h\u2212\u2207L(w)\u001b=uT\u0003I\u2212 1\n+\u001bh, w\u2217\u001b\n(1\u2212 sgn(\u001bxi, w\u001b)sgn(\u001bxi, w\n\u2217\u001b)\u001bxi, h\u001b\u001bxi, u\u001b,\nnQ\n\u0001w\u2217\u00012\ni=1\n+ sgn(\u001bxi, w\u001b)\n(1\u2212 sgn(\u001bxi, w\nnQ\ni=1\n\u2217\u001b\u001bxi, u\u001b.\n\u001bxi, w\nNow de\ufb01ne h\u22a5= h\u2212(hT w\u2217)~(\u0001w\u2217\u00012\n(cid:96)2)w\u2217. Using this we can rewrite the previous expression in\nXX T\u0003 h\u2212 1\n\u001bu, w\u2212 w\n\u2217\u2212\u2207L(w)\u001b=uT\u0003I\u2212 1\nsgn(\u001bxi, w\n\u2217\u001b)\u001bxi, h\u001b\u001bxi, u\u001b,\nnQ\ni=1\n\u2217\u001b)\u001bxi, h\u22a5\u001b\u001bxi, u\u001b,\n(1\u2212 sgn(\u001bxi, w\u001b)sgn(\u001bxi, w\n\u2217\u001b)) sgn(\u001bxi, w\nnQ\ni=1\n\u2217\u001b))+\u001bh, w\u2217\u001b\n\u0004sgn(\u001bxi, w\u001b)\n\u0004\n(1\u2212 sgn(\u001bxi, w\nnQ\n\u0001w\u2217\u00012\ni=1\n\u2217\u001b))\u001bxi, w\n(1\u2212 sgn(\u001bxi, w\u001b)sgn(\u001bxi, w\n\u2217\u001b\u001bxi, u\u001b\nLemma 6.5 Assume the setup of Theorem 3.1. Then as long as n\u2265 cn0, we have\n\u2217\u0003I\u2212 1\nX\u0003 h\u2264 \u03b4\u0001h\u0001(cid:96)2\n\u2217\u001b)\u001bxi, h\u001b\u001bxi, u\u001b\u2264 \u03b4\u0001h\u0001(cid:96)2\nsgn(\u001bxi, w\n1+ \u03b4\u00ef\u00ef\u03b4+\u0004\n\u221a\n\u2217\u001b)\u001bxi, h\u22a5\u001b\u001bxi, u\u001b\u22642\n\u2217\u001b)) sgn(\u001bxi, w\n\u2217\u001b))+\u001bh, w\u2217\u001b\n\u0004\n(1\u2212 sgn(\u001bxi, w\n\u0001w\u2217\u00012\n\u00ef\u00ef\u03b4+\u0004\n\u221a\n1+ \u03b4\n\u2217\u001b\u001bxi, u\u001b\u2264 4\n\u2217\u001b))\u001bxi, w\n(1\u2212 \u0001)2\nholds for all u\u2208C\u2229 Sd\u22121 and w\u2208 E(\u0001) with probability at least 1\u2212 9e\u2212\u03b3n.\n2(1\u2212 \u0001)2\u0004\u00ef\u00ef\u03b4+\u0004\n\u0001\u00ef\u0017\u00ef\u0017\u0001w\u2212 w\n\u2217\u0001(cid:96)2\nholds for all u\u2208C\u2229 Sd\u22121 and w\u2208 E(\u0001) with probability at least 1\u2212 16e\u2212\u03b3\u03b42n\u2212(n+ 10)e\u2212\u03b3n. Using\n4\u0001w\u2212 w\u2217\u0001(cid:96)2\nthis inequality with \u03b4= 10\u22124 and \u0001= 7~200 we conclude that\u001bu, w\u2212 w\u2217\u2212\u2207L(w)\u001b\u2264 1\nholds for all u\u2208C\u2229 Sd\u22121 and w\u2208 E(\u0001) with high probability.\n\nnQ\ni=1\n(1\u2212 sgn(\u001bxi, w\u001b)sgn(\u001bxi, w\n\u0004 sgn(\u001bxi, w\u001b)\nnQ\ni=1\n(1\u2212 sgn(\u001bxi, w\u001b)sgn(\u001bxi, w\n\n\u2217\u2212\u2207L(w)\u001b\u2264 2\u00ef\u00ef\u03b4+\u221a\n\nCombining (6.15), (6.16), (6.17), and (6.18) we conclude that\n\n\u001bu, w\u2212 w\n\n1+ \u03b4\u00041+\n\n(6.16)\n\n(6.17)\n\n\u0001\u00ef\u0017\u0001h\u0001(cid:96)2\n\u0001\u00ef\u0017\u0001h\u0001(cid:96)2\n\n(6.18)\n\nnQ\ni=1\n\n1\nn\n\n\u2217\n\nX\n\nn\n\n2\n\nu\n\n\u2212 1\n\nn\n\n21\n20\n\n21\n20\n\n,\n\n1\nn\n\n2\n\n,\n\n,\n\n(cid:96)2\n\n21\n20\n\n(cid:96)2\n\n(6.15)\n\n,\n\n,\n\n,\n\nAcknowledgements\n\nThis work was done in part while the author was visiting the Simon\u2019s Institute for the Theory of\nComputing. M.S. would like to thank Adam Klivans and Matus Telgarsky for discussions related to\n[6] and the Isotron algorithm.\n\n9\n\n\fReferences\n[1] D. Amelunxen, M. Lotz, M. B. McCoy, and J. A. Tropp. Living on the edge: Phase transitions\n\nin convex programs with random data. Information and Inference, 2014.\n\n[2] A. Brutzkus and A. Globerson. Globally optimal gradient descent for a convnet with gaussian\n\ninputs. International Conference on Machine Learning (ICML), 2017.\n\n[3] V. Chandrasekaran, B. Recht, P. A. Parrilo, and A. S. Willsky. The convex geometry of linear\n\ninverse problems. Foundations of Computational Mathematics, 12(6):805\u2013849, 2012.\n\n[4] R. Collobert and J. Weston. A uni\ufb01ed architecture for natural language processing: Deep\nneural networks with multitask learning. In Proceedings of the 25th international conference\non Machine learning, pages 160\u2013167. ACM, 2008.\n\n[5] R. Ganti, N. Rao, R. M. Willett, and R. Nowak. Learning single index models in high dimensions.\n\narXiv preprint arXiv:1506.08910, 2015.\n\n[6] S. Goel, V. Kanade, A. Klivans, and J. Thaler. Reliably learning the ReLU in polynomial time.\n\narXiv preprint arXiv:1611.10258, 2016.\n\n[7] Y. Gordon. On Milman\u2019s inequality and random subspaces which escape through a mesh in Rn.\n\nSpringer, 1988.\n\n[8] B. D. Haeffele and R. Vidal. Global optimality in tensor factorization, deep learning, and\n\nbeyond. arXiv preprint arXiv:1506.07540, 2015.\n\n[9] J. L. Horowitz and W. Hardle. Direct semiparametric estimation of single-index models with\ndiscrete covariates. Journal of the American Statistical Association, 91(436):1632\u20131640, 1996.\n\n[10] H. Ichimura. Semiparametric least squares (SLS) and weighted SLS estimation of single-index\n\nmodels. Journal of Econometrics, 58(1-2):71\u2013120, 1993.\n\n[11] S. M. Kakade, V. Kanade, O. Shamir, and A. Kalai. Ef\ufb01cient learning of generalized linear and\nsingle index models with isotonic regression. In Advances in Neural Information Processing\nSystems, pages 927\u2013935, 2011.\n\n[12] A. T. Kalai and R. Sastry. The isotron algorithm: High-dimensional isotonic regression. In\n\nCOLT, 2009.\n\n[13] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classi\ufb01cation with deep convolutional\nneural networks. In Advances in neural information processing systems, pages 1097\u20131105,\n2012.\n\n[14] Y. Li and Y. Yuan. Convergence analysis of two-layer neural networks with ReLU activation.\n\narXiv preprint arXiv:1705.09886, 2017.\n\n[15] A. Mohamed, G. E. Dahl, and G. Hinton. Acoustic modeling using deep belief networks. IEEE\n\nTransactions on Audio, Speech, and Language Processing, 20(1):14\u201322, 2012.\n\n[16] Quynh Nguyen and Matthias Hein. The loss surface of deep and wide neural networks. arXiv\n\npreprint arXiv:1704.08045, 2017.\n\n[17] S. Oymak, B. Recht, and M. Soltanolkotabi. Sharp time\u2013data tradeoffs for linear inverse\n\nproblems. arXiv preprint arXiv:1507.04793, 2015.\n\n[18] S. Oymak and M. Soltanolkotabi. Fast and reliable parameter estimation from nonlinear\n\nobservations. arXiv preprint arXiv:1610.07108, 2016.\n\n[19] T. Poston, C-N. Lee, Y. Choie, and Y. Kwon. Local minima and back propagation. In Neural\nNetworks, 1991., IJCNN-91-Seattle International Joint Conference on, volume 2, pages 173\u2013176.\nIEEE, 1991.\n\n[20] M. Soltanolkotabi. Learning ReLUs via gradient descent. arXiv preprint arXiv:1705.04591,\n\n2017.\n\n10\n\n\f[21] M. Soltanolkotabi. Structured signal recovery from quadratic measurements: Breaking sample\n\ncomplexity barriers via nonconvex optimization. arXiv preprint arXiv:1702.06175, 2017.\n\n[22] M. Soltanolkotabi, A. Javanmard, and J. D. Lee. Theoretical insights into the optimization\n\nlandscape of over-parameterized shallow neural networks. 07 2017.\n\n[23] Y. Tian. An analytical formula of population gradient for two-layered relu network and its\napplications in convergence and critical point analysis. International Conference on Machine\nLearning (ICML), 2017.\n\n[24] R. Vershynin. Introduction to the non-asymptotic analysis of random matrices. arXiv preprint\n\narXiv:1011.3027, 2010.\n\n[25] K. Zhong, Z. Song, P. Jain, P. L. Bartlett, and I. S. Dhillon. Recovery guarantees for one-hidden-\n\nlayer neural networks. arXiv preprint arXiv:1706.03175, 2017.\n\n11\n\n\f", "award": [], "sourceid": 1233, "authors": [{"given_name": "Mahdi", "family_name": "Soltanolkotabi", "institution": "University of Southern california"}]}