{"title": "Nonlinear Acceleration of Stochastic Algorithms", "book": "Advances in Neural Information Processing Systems", "page_first": 3982, "page_last": 3991, "abstract": "Extrapolation methods use the last few iterates of an optimization algorithm to produce a better estimate of the optimum. They were shown to achieve optimal convergence rates in a deterministic setting using simple gradient iterates. Here, we study extrapolation methods in a stochastic setting, where the iterates are produced by either a simple or an accelerated stochastic gradient algorithm. We first derive convergence bounds for arbitrary, potentially biased  perturbations, then produce asymptotic bounds using the ratio between the variance of the noise and the accuracy of the current point. Finally, we apply this acceleration technique to stochastic algorithms such as SGD, SAGA, SVRG and Katyusha in different settings, and show significant performance gains.", "full_text": "Nonlinear Acceleration of Stochastic Algorithms\n\nDamien Scieur\nINRIA, ENS,\n\nParis France\n\nFrancis Bach\nINRIA, ENS,\n\nParis France\n\nAlexandre d\u2019Aspremont\n\nCNRS, ENS,\n\nParis France\n\nPSL Research University,\n\nPSL Research University,\n\nPSL Research University,\n\ndamien.scieur@inria.fr\n\nfrancis.bach@inria.fr\n\naspremon@ens.fr\n\nAbstract\n\nExtrapolation methods use the last few iterates of an optimization algorithm to\nproduce a better estimate of the optimum. They were shown to achieve optimal\nconvergence rates in a deterministic setting using simple gradient iterates. Here,\nwe study extrapolation methods in a stochastic setting, where the iterates are\nproduced by either a simple or an accelerated stochastic gradient algorithm. We\n\ufb01rst derive convergence bounds for arbitrary, potentially biased perturbations, then\nproduce asymptotic bounds using the ratio between the variance of the noise and\nthe accuracy of the current point. Finally, we apply this acceleration technique\nto stochastic algorithms such as SGD, SAGA, SVRG and Katyusha in different\nsettings, and show signi\ufb01cant performance gains.\n\n1\n\nIntroduction\n\nWe focus on the problem\n\nmin\nx\u2208Rd\n\nf (x)\n\n(1)\n\nwhere f is a L-smooth and \u00b5-strongly convex function with respect to the Euclidean norm, i.e.,\n\n\u00b5\n2\n\n(cid:107)y \u2212 x(cid:107)2 \u2264 f (y) \u2212 f (x) \u2212 \u2207f (x)T (y \u2212 x) \u2264 L\n2\n\n(cid:107)y \u2212 x(cid:107)2.\n\n\u2207\u03b5f (x) = \u2207f (x) + \u03b5,\n\nWe consider a stochastic \ufb01rst-order oracle, which gives a noisy estimate of the gradient of f (x), with\n(2)\nwhere \u03b5 is a noise term with bounded variance. This is the case for example when f is a sum of\nstrongly convex functions, and we only have access to the gradient of one randomly selected function.\nStochastic optimization (2) is typically challenging as classical algorithms are not convergent (for\nexample, gradient descent or Nesterov\u2019s accelerated gradient). Even the averaged version of stochastic\ngradient descent with constant step size does not converge to the solution of (1), but to another point\nwhose proximity to the real minimizer depends of the step size [Nedi\u00b4c and Bertsekas, 2001; Moulines\nand Bach, 2011].\nWhen f is a \ufb01nite sum of N functions, then algorithms such as SAG [Schmidt et al., 2013], SAGA\n[Defazio et al., 2014], SDCA [Shalev-Shwartz and Zhang, 2013] and SVRG [Johnson and Zhang,\n2013] accelerate convergence using a variance reduction technique akin to control variate in Monte-\nCarlo methods. Their rate of convergence depends on 1 \u2212 \u00b5/L and thus does not exhibit an\n\naccelerated rate on par with the deterministic setting (in 1 \u2212(cid:112)\u00b5/L). Recently a generic acceleration\n\nalgorithm called Catalyst [Lin et al., 2015], based on the proximal point method improved this rate\nof convergence, but the practical performances highly depends on the input parameters. On the\nother hand, recent papers, for example [Shalev-Shwartz and Zhang, 2014] (Accelerated SDCA) and\n\n31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.\n\n\f[Allen-Zhu, 2016] (Katyusha), propose algorithms with accelerated convergence rates, if the strong\nconvexity parameter is given.\nWhen f is a quadratic function then averaged SGD converges, but the rate of decay of initial conditions\nis very slow. Recently, some results have focused on accelerated versions of SGD for quadratic\noptimization, showing that with a two step recursion it is possible to enjoy both the optimal rate for\nthe bias and variance terms [Flammarion and Bach, 2015], given an estimate of the ratio between the\ndistance to the solution and the variance of \u03b5.\nA novel generic acceleration technique was recently proposed by Scieur et al. [2016] in the determin-\nistic setting. This uses iterates from a slow algorithm to extrapolate estimates of the solution with\nasymptotically optimal convergence rate. Moreover, this rate is reached without prior knowledge of\nthe strong convexity constant, whose online estimation is still a challenge (even in the deterministic\ncase [Fercoq and Qu, 2016]) but required if one wants to obtain optimal rates of convergence.\nConvergence bounds are derived by Scieur et al. [2016], tracking the difference between the determin-\nistic \ufb01rst-order oracle of (1) and iterates from a linearized model. The main contribution of this paper\nis to extend the analysis to arbitrary perturbations, including stochastic ones, and to present numerical\nresults when this acceleration method is used to speed up stochastic optimization algorithms.\nIn Section 2 we recall the extrapolation algorithm, and quickly summarize its main convergence\nbounds in Section 3. In Section 4, we consider a stochastic oracle and analyze its asymptotic\nconvergence in Section 5. Finally, in Section 6 we describe numerical experiments which con\ufb01rm the\ntheoretical bounds and show the practical ef\ufb01ciency of this acceleration.\n\n2 Regularized Nonlinear Acceleration\n\nConsider the optimization problem\n\nmin\nx\u2208Rd\n\nf (x)\n\nwhere f is a L\u2212smooth and \u00b5\u2212strongly convex function [Nesterov, 2013]. Applying the \ufb01xed-step\ngradient method to this problem yields the following iterates\n\u2207f (\u02dcxt).\n\n(3)\n\n\u02dcxt+1 = \u02dcxt \u2212 1\nL\n\nLet x\u2217 be the unique optimal point, this algorithm is proved to converge with\n\n(cid:107)\u02dcxt \u2212 x\u2217(cid:107) \u2264 (1 \u2212 \u03ba)t(cid:107)\u02dcx0 \u2212 x\u2217(cid:107)\n\n(4)\nwhere (cid:107) \u00b7 (cid:107) stands for the (cid:96)2 norm and \u03ba = \u00b5/L \u2208 [0, 1[ is the (inverse of the) condition number of f\n[Nesterov, 2013]. Using a two-step recurrence, the accelerated gradient descent by Nesterov [2013]\nachieves the improved convergence rate\n\n(cid:16)\n\n\u03ba)t(cid:107)\u02dcx0 \u2212 x\u2217(cid:107)(cid:17)\n\n(1 \u2212 \u221a\n\n(cid:107)\u02dcxt \u2212 x\u2217(cid:107) \u2264 O\n\n.\n\n(5)\n\nIndeed, (5) converges faster than (4) but the accelerated algorithm requires the knowledge of \u00b5 and L.\nExtrapolation techniques however obtain a similar convergence rate, but do not need estimates of the\nparameters \u00b5 and L. The idea is based on the comparison between the process followed by \u02dcxi with a\nlinearized model around the optimum (obtained by the \ufb01rst-order approximation of \u2207f (x)), written\n\n(cid:17)\n\n+\u22072f (x\u2217)(xt \u2212 x\u2217)\n\n,\n\nx0 = \u02dcx0.\n\nxt+1 = xt \u2212 1\nL\n\nwhich can be rewritten as\n\n(cid:16)\u2207f (x\u2217)\n(cid:124) (cid:123)(cid:122) (cid:125)\n\n=0\n\nxt+1 \u2212 x\u2217 = (I \u2212 \u22072f (x\u2217)/L)(xt \u2212 x\u2217),\n\n(6)\nA better estimate of the optimum in (6) can be obtained by forming a linear combination of the\niterates (see [Anderson, 1965; Cabay and Jackson, 1976; Me\u0161ina, 1977]), with\n\nx0 = \u02dcx0.\n\n(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:28) (cid:107)xt \u2212 x\u2217(cid:107),\n\n(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) t(cid:88)\n\ni=0\n\ncixi \u2212 x\u2217\n\n2\n\n\ffor some speci\ufb01c ci (either data driven, or derived from Chebyshev polynomials). These procedures\nwere limited to quadratic functions only, i.e. when \u02dcxi = xi but this was recently extended to generic\nconvex problems by Scieur et al. [2016] and we brie\ufb02y recall these results below.\nTo simplify the notations, we write\n\n(7)\nto be one step of algorithm g. We have that g is differentiable, Lipchitz-continuous with constant\n(1 \u2212 \u03ba) < 1, g(x\u2217) = x\u2217 and g(cid:48)(x\u2217) is symmetric. For example, the gradient method (3) matches\nexactly this de\ufb01nition with g(x) = x \u2212 \u2207f (x)/L. Running k steps of (7) produces a sequence\n{\u02dcx0, ..., \u02dcxk}, which we extrapolate using Algorithm 1 from Scieur et al. [2016].\n\n\u02dcxt+1 = g(\u02dcxt)\n\nAlgorithm 1 Regularized Nonlinear Acceleration (RNA)\nInput: Iterates \u02dcx0, \u02dcx1, ..., \u02dcxk+1 \u2208 Rd produced by (7), and a regularization parameter \u03bb > 0.\n1: Compute \u02dcR = [\u02dcr0, ..., \u02dcrk], where \u02dcri = \u02dcxi+1 \u2212 \u02dcxi is the ith residue.\n2: Solve\n\nOutput: Approximation of x\u2217 computed as(cid:80)k\n\nor equivalently solve ( \u02dcRT \u02dcR + \u03bbI)z = 1 then set \u02dcc\u03bb = z/1T z.\n\ni=0 \u02dcc\u03bb\n\ni \u02dcxi\n\n\u02dcc\u03bb = argmin\ncT 1=1\n\n(cid:107) \u02dcRc(cid:107)2 + \u03bb(cid:107)c(cid:107)2,\n\nFor a good choice of \u03bb, the output of Algorithm (1) is a much better estimate of the optimum than\n\u02dcxk+1 (or any other points of the sequence). Using a simple grid search on a few values of \u03bb is usually\nsuf\ufb01cient to improve convergence (see [Scieur et al., 2016] for more details).\n\n3 Convergence of Regularized Nonlinear Acceleration\n\nWe quickly summarize the argument behind the convergence of Algorithm (1). The theoretical bound\ncompares \u02dcxi to the iterates produced by the linearized model\nxt+1 = x\u2217 + \u2207g(x\u2217)(xt \u2212 x\u2217),\n\nx0 = \u02dcx0.\n\n(8)\n\nThis sequence is useful to extend the convergence results to the nonlinear case, using sensivity\nanalysis. We write c\u03bb the coef\ufb01cients computed by Algorithm (1) from the \u201clinearized\u201d sequence\n{x0, ..., xk+1} and the error term can be decomposed into three parts,\n\n(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) k(cid:88)\n\ni=0\n\ni \u02dcxi \u2212 x\u2217\n\u02dcc\u03bb\n\n(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) \u2264\n\n(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) k(cid:88)\n(cid:124)\n\ni=0\n\n(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)\n(cid:125)\n\n(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) k(cid:88)\n(cid:124)\n\ni=0\n\n(cid:16)\n\n(cid:17)\ni \u2212 c\u03bb\n(cid:123)(cid:122)\n\u02dcc\u03bb\n\ni\n\nStability\n\n+\n\ni xi \u2212 x\u2217\n(cid:123)(cid:122)\nc\u03bb\n\nAcceleration\n\n(xi \u2212 x\u2217)\n\n+\n\n\u02dcxi \u2212 xi\n\n\u02dcc\u03bb\ni\n\n.\n\n(9)\n\n(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)\n(cid:125)\n\n(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) k(cid:88)\n(cid:124)\n\ni=0\n\n(cid:16)\n(cid:123)(cid:122)\n\nNonlinearity\n\n(cid:17)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)\n(cid:125)\n\nScieur et al. [2016] show that convergence is guaranteed as long as the errors (\u02dcxi \u2212 x\u2217) and (xi \u2212 \u02dcxi)\nconverge to zero fast enough, which ensures a good rate of decay for the regularization parameter\n\u03bb, leading to an asymptotic rate equivalent to the accelerated rate in (5). In this section, we will use\nresults from Scieur et al. [2016] to bound each individual term, but in this paper we improve the \ufb01nal\nconvergence result.\nThe stability term (in \u02dcc\u03bb \u2212 c\u03bb) is bounded using the perturbation matrix\n\nP (cid:44) RT R \u2212 \u02dcRT \u02dcR,\n\nwhere R and \u02dcR are the matrices of residuals,\nR (cid:44) [r0...rk]\n\u02dcR (cid:44) [\u02dcr0...\u02dcrk]\n\nrt = xt+1 \u2212 xt,\n\u02dcrt = \u02dcxt+1 \u2212 \u02dcxt.\n\nThe proofs of the following propositions were obtained by Scieur et al. [2016].\n\n(10)\n\n(11)\n(12)\n\n3\n\n\fProposition 3.1 (Stability). Let \u2206c\u03bb = \u02dcc\u03bb \u2212 c\u03bb be the gap between the coef\ufb01cients computed\nby Algorithm (1) using the sequences {\u02dcxi} and {xi} with regularization parameter \u03bb. Let P =\nRT R \u2212 \u02dcRT \u02dcR be de\ufb01ned in (10), (11) and (12). Then\n\n(cid:107)\u2206c\u03bb(cid:107) \u2264 (cid:107)P(cid:107)\n\n\u03bb (cid:107)c\u03bb(cid:107).\n\nThis implies that the stability term is bounded by\n\ni=0 \u2206c\u03bb\n\ni (xi \u2212 x\u2217)(cid:107) \u2264 (cid:107)P(cid:107)\n\n\u03bb (cid:107)c\u03bb(cid:107) O((cid:107)x0 \u2212 x\u2217(cid:107)).\n\n(cid:107)(cid:80)k\n\nThe term Nonlinearity is bounded by the norm of the coef\ufb01cients \u02dcc\u03bb (controlled thanks to the\nregularization parameter) times the norm of the noise matrix\n\nE = [x0 \u2212 \u02dcx0, x1 \u2212 \u02dcx1, ..., xk \u2212 \u02dcxk].\n\n(15)\nProposition 3.2 (Nonlinearity). Let \u02dcc\u03bb be computed by Algorithm 1 using the sequence\n{\u02dcx0, ..., \u02dcxk+1} with regularization parameter \u03bb and \u02dcR be de\ufb01ned in (12). The norm of \u02dcc\u03bb is\nbounded by\n\n(13)\n\n(14)\n\n(16)\n\n(17)\n\nThis bounds the nonlinearity term because\n\n(cid:107)\u02dcc\u03bb(cid:107) \u2264(cid:113)(cid:107) \u02dcR(cid:107)2+\u03bb\n(cid:13)(cid:13)(cid:13)(cid:80)k\n(cid:13)(cid:13)(cid:13) \u2264\n\ni (\u02dcxi \u2212 xi)\n\n(k+1)\u03bb \u2264 1\u221a\n(cid:113)\n\ni=0 \u02dcc\u03bb\n\n(cid:113)\n\n1 +\n\n(cid:107) \u02dcR(cid:107)2\n\u03bb .\n\nk+1\n\n1 +\n\n(cid:107) \u02dcR(cid:107)2\n\n\u03bb\n\n(cid:107)E(cid:107)\u221a\n\nk+1\n\n,\n\nwhere E is de\ufb01ned in (15).\nThese two propositions show that the regularization in Algorithm 1 limits the impact of the noise: the\nhigher \u03bb is, the smaller these terms are. It remains to control the acceleration term. For small \u03bb, this\nterm decreases as fast as the accelerated rate (5), as shown in the following proposition.\nProposition 3.3 (Acceleration). Let Pk be the subspace of real polynomials of degree at most k and\nS\u03ba(k, \u03b1) be the solution of the Regularized Chebychev Polynomial problem,\n\nS\u03ba(k, \u03b1) (cid:44) min\np\u2208Pk\n\nmax\n\nx\u2208[0,1\u2212\u03ba]\n\np2(x) + \u03b1(cid:107)p(cid:107)2\n\ns.t. p(1) = 1.\n\nLet \u00af\u03bb (cid:44)\n\n\u03bb\n\n(cid:107)x0\u2212x\u2217(cid:107)2 be the normalized value of \u03bb. The acceleration term is bounded by\n\n(cid:13)(cid:13)(cid:13)(cid:80)k\n\ni xi \u2212 x\u2217(cid:13)(cid:13)(cid:13) \u2264 1\n\n\u03ba\n\ni=0 c\u03bb\n\n(cid:112)S\u03ba(k, \u00af\u03bb)(cid:107)x0 \u2212 x\u2217(cid:107)2 \u2212 \u03bb(cid:107)c\u03bb(cid:107)2.\n\n(18)\n\n(19)\n\nWe also get the following corollary, which will be useful for the asymptotic analysis of the rate of\nconvergence of Algorithm 1.\nCorollary 3.4. If \u03bb \u2192 0, the bound (19) becomes\n\n(cid:13)(cid:13)(cid:13)(cid:80)k\n\ni=0 c\u03bb\n\n(cid:16) 1\u2212\u221a\n\ni xi \u2212 x\u2217(cid:13)(cid:13)(cid:13) \u2264 1\n(cid:112)S\u03ba(k, 0)(cid:107)x0\u2212x\u2217(cid:107). The exact value of(cid:112)S\u03ba(k, 0) is obtained\n\n(cid:17)k (cid:107)x0 \u2212 x\u2217(cid:107).\n\n\u221a\n\n\u03ba\n\u03ba\n\n1+\n\n\u03ba\n\nProof. When \u03bb = 0, (19) becomes 1\n\u03ba\nby using the coef\ufb01cients of a re-scaled Chebyshev polynomial, derived by Golub and Varga [1961];\nScieur et al. [2016], and is equal to 1\u2212\u221a\n\u221a\n\u03ba .\n\n1+\n\n\u03ba\n\nThese last results controlling stability, nonlinearity and acceleration are proved by Scieur et al. [2016].\nWe now re\ufb01ne the \ufb01nal step of Scieur et al. [2016] to produce a global bound on the error that will\nallow us to extend these results to the stochastic setting in the next sections.\nTheorem 3.5. If Algorithm 1 is applied to the sequence \u02dcxi with regularization parameter \u03bb, it\nconverges with rate\n\n(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) k(cid:88)\n\ni=0\n\n(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) \u2264 (cid:107)x0 \u2212 x\u2217(cid:107)S\n\n\u02dcc\u03bb\ni \u02dcxi\n\n(cid:114)\n\n1\n2\n\n\u03ba (k, \u00af\u03bb)\n\n(cid:115)\n\n(cid:107) \u02dcR(cid:107)2\n\u03bb\n\n.\n\n1 +\n\n(20)\n\n1\n\u03ba2 +\n\nO((cid:107)x \u2212 x\u2217(cid:107)2)(cid:107)P(cid:107)2\n\n\u03bb3\n\n(cid:107)E(cid:107)\u221a\n\nk + 1\n\n+\n\n4\n\n\fProof. The proof is inspired by Scieur et al. [2016] and is straightforward. We can bound (9) using\n(14) (Stability), (17) (Nonlinearity) and (19) (Acceleration). It remains to maximize over the value\nof (cid:107)c\u03bb(cid:107) using the result of Proposition A.2.\n\nThis last bound is not very explicit, in particular because of the regularized Chebyshev term S\u03ba(k, \u00af\u03bb).\nThe solution is well known when \u00af\u03bb = 0 since it corresponds exactly to the rescaled Chebyshev\npolynomial [Golub and Varga, 1961], but as far as we know there is no known result about its\nregularized version, thus making the \"\ufb01nite-step\" version hard to analyze. However, an asymptotic\nanalysis simpli\ufb01es it considerably. The next new proposition shows that when x0 is close to x\u2217, then\nextrapolation converges as fast as in (5) in some cases.\nProposition 3.6. Assume (cid:107) \u02dcR(cid:107) = O((cid:107)x0\u2212 x\u2217(cid:107)), (cid:107)E(cid:107) = O((cid:107)x0\u2212 x\u2217(cid:107)2) and (cid:107)P(cid:107) = O((cid:107)x0\u2212 x\u2217(cid:107)3).\nIf we chose \u03bb = O((cid:107)x0 \u2212 x\u2217(cid:107)s) with s \u2208 [2, 8\n\n3 ] then the bound (20) becomes\ni \u02dcxi(cid:107)\n\n(cid:18) 1 \u2212 \u221a\n\n(cid:107)(cid:80)k\ni=0 \u02dcc\u03bb\n(cid:107)x0 \u2212 x\u2217(cid:107) \u2264 1\n\n(cid:19)k\n\n1 +\n\n\u221a\n\n\u03ba\n\u03ba\n\n\u03ba\n\n.\n\nlim(cid:107)x0\u2212x\u2217(cid:107)\u21920\n\nProof. (Sketch) The proof is based on the fact that \u03bb decreases slowly enough to ensure that the\n(cid:107)x0\u2212x\u2217(cid:107)2 \u2192 0.\nStability and Nonlinearity terms vanish over time, but fast enough to have \u00af\u03bb =\nThen it remains to bound S\u03ba(k, 0) with Corollary 3.4. The complete proof can be found in the\nSupplementary materials.\n\n\u03bb\n\nNote: The assumptions are satis\ufb01ed if we apply the gradient method on a twice differentiable,\nsmooth and strongly convex function with Lipchitz-continuous Hessian [Scieur et al., 2016].\nThe ef\ufb01ciency of Algorithm 1 is thus ensured by two conditions. First, we need to be able to bound\n(cid:107) \u02dcR(cid:107), (cid:107)P(cid:107) and (cid:107)E(cid:107) by decreasing quantities. Second, we have to \ufb01nd a proper rate of decay for \u03bb\nand \u00af\u03bb such that the stability and nonlinearity terms go to zero when perturbations also go to zero. If\nthese two conditions are met, then the accelerated rate in Proposition 3.6 holds.\n\n4 Nonlinear and Noisy Updates\n\nIn (7) we de\ufb01ned g(x) to be non linear, which generates a sequence \u02dcxi. We now consider noisy\niterates\n\n\u02dcxt+1 = g(\u02dcxt) + \u03b7t+1,\n\nwhere \u03b7t is a stochastic noise. To simplify notations, we write (21) as\n\n\u02dcxt+1 = x\u2217 + G(\u02dcxt \u2212 x\u2217) + \u03b5t+1,\n\n(21)\n\n(22)\n\nwhere \u03b5t is a stochastic noise (potentially correlated with the iterates xi) with bounded mean \u03bdt,\n(cid:107)\u03bdt(cid:107) \u2264 \u03bd and bounded covariance \u03a3t (cid:22) (\u03c32/d)I. We also assume 0I (cid:22) G (cid:22) (1 \u2212 \u03ba)I and G is\nsymmetric. For example, (22) can be linked to (21) if we set \u03b5t = \u03b7t + O((cid:107)\u02dcxt \u2212 x\u2217(cid:107)2), which\ncorresponds to the combination of the noise \u03b7t+1 with the Taylor remainder of g(x) around x\u2217, i.e.,\n\n\u02dcxt+1 = g(\u02dcxt) + \u03b7t+1 = g(x\u2217)\n=x\u2217\n\n(cid:124)(cid:123)(cid:122)(cid:125)\n\n+\u2207g(x\u2217)\n\n(cid:124) (cid:123)(cid:122) (cid:125)\n\n=G\n\n(cid:124)\n\n(cid:123)(cid:122)\n\n=\u0001t+1\n\n(\u02dcxt \u2212 x\u2217) + O((cid:107)\u02dcxt \u2212 x\u2217(cid:107)) + \u03b7t+1\n\n.\n\n(cid:125)\n\nThe recursion (22) is also valid when we apply the stochastic gradient method with \ufb01xed step size h\nto the quadratic problem\n\nminx\n\n1\n\n2(cid:107)Ax \u2212 b(cid:107)2.\n\nThis corresponds to (22) with G = I \u2212 hAT A and mean \u03bd = 0. For the theoretical results, we will\ncompare \u02dcxt with their noiseless counterpart to control convergence,\n\nxt+1 = x\u2217 + G(xt \u2212 x\u2217),\n\nx0 = \u02dcx0.\n\n(23)\n\n5\n\n\f5 Convergence Analysis when Accelerating Stochastic Algorithms\n\nWe will control convergence in expectation. Bound (9) now becomes\n\n(cid:34)(cid:13)(cid:13)(cid:13) k(cid:88)\n\ni=0\n\nE\n\ni \u02dcxi \u2212 x\u2217(cid:13)(cid:13)(cid:13)(cid:35)\n\n\u02dcc\u03bb\n\n\u2264(cid:13)(cid:13)(cid:13) k(cid:88)\n\ni=0\n\ni xi \u2212 x\u2217(cid:13)(cid:13)(cid:13) + O((cid:107)x0 \u2212 x\u2217(cid:107))E(cid:104)(cid:107)\u2206c\u03bb(cid:107)(cid:105)\n\nc\u03bb\n\n+ E(cid:104)(cid:107)\u02dcc\u03bb(cid:107)(cid:107)E(cid:107)(cid:105)\n\n.\n\n(24)\n\nWe now need to enforce bounds (14), (17) and (19) in expectation. The proofs of the two next\npropositions are in the supplementary material. For simplicity, we will omit all constants in what\nfollows.\nProposition 5.1. Consider the sequences xi and \u02dcxi generated by (21) and (23). Then,\n\nE[(cid:107) \u02dcR(cid:107)] \u2264 O((cid:107)x0 \u2212 x\u2217(cid:107)) + O(\u03bd + \u03c3),\nE[(cid:107)E(cid:107)] \u2264 O(\u03bd + \u03c3),\nE[(cid:107)P(cid:107)] \u2264 O((\u03c3 + \u03bd)(cid:107)x0 \u2212 x\u2217(cid:107)) + O((\u03bd + \u03c3)2).\n\n(25)\n(26)\n(27)\n\nWe de\ufb01ne the following stochastic condition number\n\u03c4 (cid:44) \u03bd + \u03c3\n\n(cid:107)x0 \u2212 x\u2217(cid:107) .\n\nThe Proposition 5.2 gives the result when injecting these bounds in (24).\nProposition 5.2. The accuracy of extrapolation Algorithm 1 applied to the sequence {\u02dcx0, ..., \u02dcxk}\ngenerated by (21) is bounded by\n\n(cid:32)(cid:114)\n\n(cid:33)(cid:33)\n\nE(cid:104)(cid:107)(cid:80)k\n\ni \u02dcxi \u2212 x\u2217(cid:107)(cid:105)\n\n(cid:32)\n\ni=0 \u02dcc\u03bb\n(cid:107)x0 \u2212 x\u2217(cid:107)\n\n(cid:114)\n\n\u2264\n\nS\u03ba(k, \u00af\u03bb)\n\n1\n\u03ba2 +\n\nO(\u03c4 2(1 + \u03c4 )2)\n\n\u00af\u03bb3\n\n+ O\n\n\u03c4 2 +\n\n\u03c4 2(1 + \u03c4 2)\n\n\u00af\u03bb\n\n. (28)\n\nConsider a situation where \u03c4 is small, e.g. when using stochastic gradient descent with \ufb01xed step-size,\nwith x0 far from x\u2217. The following proposition details the dependence between \u00af\u03bb and \u03c4 ensuring the\nupper convergence bound remains stable when \u03c4 goes to zero.\nProposition 5.3. When \u03c4 \u2192 0, if \u00af\u03bb = \u0398(\u03c4 s) with s \u2208]0, 2\n\n3 [, we have the accelerated rate\n\n\u221a\nMoreover, if \u03bb \u2192 \u221e, we recover the averaged gradient,\n\n1+\n\n\u03ba\n\n\u03ba\n\u03ba\n\ni=0 \u02dcc\u03bb\n\n(cid:16) 1\u2212\u221a\ni \u02dcxi \u2212 x\u2217(cid:107)(cid:3) \u2264 1\ni \u02dcxi \u2212 x\u2217(cid:107)(cid:3) = E(cid:104)(cid:13)(cid:13)(cid:13) 1\n\n(cid:17)k (cid:107)x0 \u2212 x\u2217(cid:107).\nE(cid:2)(cid:107)(cid:80)k\ni=0 \u02dcxi \u2212 x\u2217(cid:13)(cid:13)(cid:13)(cid:105)\nE(cid:2)(cid:107)(cid:80)k\n(cid:80)k\n(cid:113) 1\ni \u02dcxi \u2212 x\u2217(cid:13)(cid:13)(cid:13)(cid:105) \u2264 (cid:107)x0 \u2212 x\u2217(cid:107)S\u03ba(k, \u03c4 s)\n\ni=0 \u02dcc\u03bb\n\nk+1\n\nE(cid:104)(cid:13)(cid:13)(cid:13)(cid:80)k\n\ni=0 \u02dcc\u03bb\n\nProof. Let \u00af\u03bb = \u0398(\u03c4 s), using (28) we have\n\n(29)\n\n.\n\n+(cid:107)x0 \u2212 x\u2217(cid:107)O((cid:112)\u03c4 2 + \u03c4 2\u22123s(1 + \u03c4 2)).\n\n\u03ba2 O(\u03c4 2\u22123s(1 + \u03c4 )2)\n\nBecause s \u2208]0, 2\nexactly (29). If \u03bb \u2192 \u221e, we have also\n\n3 [, means 2 \u2212 3s > 0, thus lim\u03c4\u21920 \u03c4 2\u22123s = 0. The limits when \u03c4 \u2192 0 is thus\n\nlim\n\u03bb\u2192\u221e \u02dcc\u03bb = lim\n\n\u03bb\u2192\u221e argminc:1T c=1 (cid:107) \u02dcRc(cid:107) + \u03bb(cid:107)c(cid:107)2 = argminc:1T c=1 (cid:107)c(cid:107)2 = 1\n\nk+1\n\nwhich yields the desired result.\n\nProposition 5.3 shows that Algorithm 1 is thus asymptotically optimal provided \u03bb is well chosen\nbecause it recovers the accelerated rate for smooth and strongly convex functions when the perturba-\ntions goes to zero. Moreover, we recover Proposition 3.6 when \u0001t is the Taylor remainder, i.e. with\n\u03bd = O((cid:107)x0 \u2212 x\u2217(cid:107)2) and \u03c3 = 0, which matches the deterministic results.\nAlgorithm 1 is particularly ef\ufb01cient when combined with a restart scheme [Scieur et al., 2016].\nFrom a theoretical point of view, the acceleration peak arises for small values of k. Empirically, the\n\n6\n\n\fimprovement is usually more important at the beginning, i.e. when k is small. Finally, the algorithmic\ncomplexity is O(k2d), which is linear in the problem dimension when k remains bounded.\nThe bene\ufb01ts of extrapolation are limited in a regime where the noise dominates. However, when\n\u03c4 is relatively small then we can expect a signi\ufb01cant speedup. This condition is satis\ufb01ed in many\ncases, for example at the initial phase of the stochastic gradient descent or when optimizing a sum of\nfunctions with variance reduction techniques, such as SAGA or SVRG.\n\n6 Numerical Experiments\n\n6.1 Stochastic gradient descent\n\nWe want to solve the least-squares problem\n\nmin\nx\u2208Rd\n\nF (x) =\n\n(cid:107)Ax \u2212 b(cid:107)2,\n\n1\n2\n\nwhere AT A satis\ufb01es \u00b5I (cid:22) (AT A) (cid:22) LI. To solve this problem, we have access to the stochastic\n\ufb01rst-order oracle\n\n\u2207\u03b5F (x) = \u2207F (x) + \u03b5,\n\nd I. We will compare several methods.\n\nwhere \u03b5 is a zero-mean noise of covariance matrix \u03a3 (cid:22) \u03c32\nL\u2207\u03b5F (xt).\n\n\u2022 SGD. Fixed step-size, xt+1 = xt \u2212 1\n\u2022 Averaged SGD. Iterate xk is the mean of the k \ufb01rst iterations of SGD.\n\u2022 AccSGD. The optimal two-step algorithm in Flammarion and Bach [2015], with optimal\n\u2022 RNA+SGD. The regularized nonlinear acceleration Algorithm 1 applied to a sequence of k\n\nparameters (this implies (cid:107)x0 \u2212 x\u2217(cid:107) and \u03c3 are known exactly).\niterates of SGD, with k = 10 and \u03bb = (cid:107) \u02dcRT \u02dcR(cid:107)/10\u22126.\n\nBy Proposition 5.2, we know that RNA+SGD will not converge to arbitrary precision because the\nnoise is additive with a non-vanishing variance. However, Proposition 5.3 predicts an improvement\nof the convergence at the beginning of the process. We illustrate this behavior in Figure 1. We\nclearly see that at the beginning, the performance of RNA+SGD is comparable to that of the optimal\naccelerated algorithm. However, because of the restart strategy, in the regime where the level of\nnoise becomes more important the acceleration becomes less effective and \ufb01nally the convergence\nstalls, as for SGD. Of course, for practical purposes, the \ufb01rst regime is the most important because it\neffectively minimizes the generalization error [D\u00e9fossez and Bach, 2015; Jain et al., 2016].\n\n1\n\ni=1\n\n6.2 Finite sums of functions\n\nWe focus on the composite problem minx\u2208Rd F (x) =(cid:80)N\n\nN fi(x) + \u00b5\n\n2(cid:107)x(cid:107)2, where fi are convex\nand L-smooth functions and \u00b5 is the regularization parameter. We will use classical methods for\nminimizing F (x) such as SGD (with \ufb01xed step size), SAGA [Defazio et al., 2014], SVRG [Johnson\nand Zhang, 2013], and also the accelerated algorithm Katyusha [Allen-Zhu, 2016]. We will compare\ntheir performance with and without the (potential) acceleration provided by Algorithm 1 with restart\nafter k data passes. The parameter \u03bb is found by a grid search of size k, the size of the input sequence,\nbut it adds only one data pass at each extrapolation. Actually, the grid search can be faster if we\napproximate F (x) with fewer samples, but we choose to present Algorithm 1 in its simplest version.\nWe set k = 10 for all the experiments.\nIn order to balance the complexity of the extrapolation algorithm and the optimization method we wait\nseveral data queries before adding the current point (the \u201csnapshot\u201d) of the method to the sequence.\nIndeed, the extrapolation algorithm has a complexity of O(k2d) + O(N ) (computing the coef\ufb01cients\n\u02dcc\u03bb and the grid search over \u03bb). If we wait at least O(N ) updates, then the extrapolation method is of\nthe same order of complexity as the optimization algorithm.\n\n\u2022 SGD. We add the current point after N data queries (i.e. one epoch) and k snapshots of\n\nSGD cost kN data queries.\n\n7\n\n\fLeft: \u03c3 = 10, \u03ba = 10\u22122. Center: \u03c3 = 1000, \u03ba = 10\u22122. Right: \u03c3 = 1000, \u03ba = 10\u22126.\n\nLeft: \u03c3 = 10, \u03ba = 1/d. Center: \u03c3 = 100, \u03ba = 1/d. Right: \u03c3 = 1000, \u03ba = 1/d.\n\nFigure 1: Comparison of performance between SGD, averaged SGD, Accelerated SGD [Flammarion\nand Bach, 2015] and RNA+SGD. We tested the performance on a matrix AT A of size d = 500, with\n(top) random eigenvalues between \u03ba and 1 and (bottom) decaying eigenvalues from 1 to 1/d. We\nstart at (cid:107)x0 \u2212 x\u2217(cid:107) = 104, where x0 and x\u2217 are generated randomly.\n\n\u2022 SAGA. We compute the gradient table exactly, then we add a new point after N queries,\nand k snapshots of SAGA cost (k + 1)N queries. Since we optimize a sum of quadratic or\nlogistic losses, we used the version of SAGA which stores O(N ) scalars.\n\n\u2022 SVRG. We compute the gradient exactly, then perform N queries (the inner-loop of SVRG),\n\nand k snapshots of SVRG cost 2kN queries.\n\n\u2022 Katyusha. We compute the gradient exactly, then perform 4N gradient calls (the inner-loop\n\nof Katyusha), and k snapshots of Katyusha cost 3kN queries.\n\nWe compare these various methods for solving least-squares regression and logistic regression\non several datasets (Table 1), with several condition numbers \u03ba: well (\u03ba = 100/N), moderately\n(\u03ba = 1/N) and badly (\u03ba = 1/100N) conditioned. In this section, we present the numerical results\non Sid (Sido0 dataset, where N = 12678 and d = 4932) with bad conditioning, see Figure 2. The\nother experiments are highlighted in the supplementary material.\nIn Figure 2, we clearly see that both SGD and RNA+SGD do not converge. This is mainly due to\nthe fact that we do not average the points. In any case, except for quadratic problems, the averaged\nversion of SGD does not converge to the minimum of F with arbitrary precision.\nWe also notice that Algorithm 1 is unable to accelerate Katyusha. This issue was already raised\nby Scieur et al. [2016]: when the algorithm has a momentum term (like Nesterov\u2019s method), the\nunderlying dynamical system is harder to extrapolate, in particular because the matrix presents in the\nlinearized version of such systems is not symmetric.\nBecause the iterates of SAGA and SVRG have low variance, their accelerated version converges\nfaster to the optimum, and their performance are then comparable to Katyusha. In our experiments,\nKatyusha was faster than RNA+SAGA only once, when solving a least square problem on Sido0\n\n8\n\nPSfragreplacementsSGDAve.SGDAcc.SGDRNA+SGD10010210410-2100102PSfragreplacementsSGDAve.SGDAcc.SGDRNA+SGDIterationf(x)\u2212f(x\u2217)100102104100102PSfragreplacementsSGDAve.SGDAcc.SGDRNA+SGDIterationf(x)\u2212f(x\u2217)100102104101102103104PSfragreplacementsSGDAve.SGDAcc.SGDRNA+SGDIterationf(x)\u2212f(x\u2217)10010210410-2100102PSfragreplacementsSGDAve.SGDAcc.SGDRNA+SGDIterationf(x)\u2212f(x\u2217)f(x)\u2212f(x\u2217)100102104100102PSfragreplacementsSGDAve.SGDAcc.SGDRNA+SGDIterationf(x)\u2212f(x\u2217)f(x)\u2212f(x\u2217)100102104100101102103PSfragreplacementsSGDAve.SGDAcc.SGDRNA+SGDIterationf(x)\u2212f(x\u2217)f(x)\u2212f(x\u2217)\fFigure 2: Optimization of quadratic loss (Top) and logistic loss (Bottom) with several algorithms,\nusing the Sid dataset with bad conditioning. The experiments are done in Matlab. Left: Error vs\nepoch number. Right: Error vs time.\n\nwith a bad condition number. Recall however that the acceleration Algorithm 1 does not require the\nspeci\ufb01cation of the strong convexity parameter, unlike Katyusha.\n\nAcknowledgments\n\nThe authors would like to acknowledge support from a starting grant from the European Research\nCouncil (ERC project SIPA), from the European Union\u2019s Seventh Framework Programme (FP7-\nPEOPLE-2013-ITN) under grant agreement number 607290 SpaRTaN, as well as support from the\nchaire \u00c9conomie des nouvelles donn\u00e9es with the data science joint research initiative with the fonds\nAXA pour la recherche and a gift from Soci\u00e9t\u00e9 G\u00e9n\u00e9rale Cross Asset Quantitative Research.\n\n9\n\n020040010-1010-5PSfragreplacementsf(x)\u2212f(x\u2217)Epoch05010015020010-1010-5PSfragreplacementsf(x)\u2212f(x\u2217)EpochTime(sec)020040010-1010-5PSfragreplacementsf(x)\u2212f(x\u2217)EpochTime(sec)f(x)\u2212f(x\u2217)Epoch010020030010-1010-5PSfragreplacementsf(x)\u2212f(x\u2217)EpochTime(sec)f(x)\u2212f(x\u2217)EpochTime(sec)PSfragreplacementsf(x)\u2212f(x\u2217)EpochTime(sec)f(x)\u2212f(x\u2217)EpochTime(sec)SAGASGDSVRGKatyushaRNA+SAGARNA+SGDRNA+SVRGRNA+Kat.\fReferences\nAllen-Zhu, Z. [2016], \u2018Katyusha: The \ufb01rst direct acceleration of stochastic gradient methods\u2019, arXiv preprint\n\narXiv:1603.05953 .\n\nAnderson, D. G. [1965], \u2018Iterative procedures for nonlinear integral equations\u2019, Journal of the ACM (JACM)\n\n12(4), 547\u2013560.\n\nCabay, S. and Jackson, L. [1976], \u2018A polynomial extrapolation method for \ufb01nding limits and antilimits of vector\n\nsequences\u2019, SIAM Journal on Numerical Analysis 13(5), 734\u2013752.\n\nDefazio, A., Bach, F. and Lacoste-Julien, S. [2014], Saga: A fast incremental gradient method with support\nfor non-strongly convex composite objectives, in \u2018Advances in Neural Information Processing Systems\u2019,\npp. 1646\u20131654.\n\nD\u00e9fossez, A. and Bach, F. [2015], Averaged least-mean-squares: Bias-variance trade-offs and optimal sampling\n\ndistributions, in \u2018Arti\ufb01cial Intelligence and Statistics\u2019, pp. 205\u2013213.\n\nFercoq, O. and Qu, Z. [2016], \u2018Restarting accelerated gradient methods with a rough strong convexity estimate\u2019,\n\narXiv preprint arXiv:1609.07358 .\n\nFlammarion, N. and Bach, F. [2015], From averaging to acceleration, there is only a step-size, in \u2018Conference on\n\nLearning Theory\u2019, pp. 658\u2013695.\n\nGolub, G. H. and Varga, R. S. [1961], \u2018Chebyshev semi-iterative methods, successive overrelaxation iterative\n\nmethods, and second order richardson iterative methods\u2019, Numerische Mathematik 3(1), 147\u2013156.\n\nJain, P., Kakade, S. M., Kidambi, R., Netrapalli, P. and Sidford, A. [2016], \u2018Parallelizing stochastic approximation\n\nthrough mini-batching and tail-averaging\u2019, arXiv preprint arXiv:1610.03774 .\n\nJohnson, R. and Zhang, T. [2013], Accelerating stochastic gradient descent using predictive variance reduction,\n\nin \u2018Advances in Neural Information Processing Systems\u2019, pp. 315\u2013323.\n\nLin, H., Mairal, J. and Harchaoui, Z. [2015], A universal catalyst for \ufb01rst-order optimization, in \u2018Advances in\n\nNeural Information Processing Systems\u2019, pp. 3384\u20133392.\n\nMe\u0161ina, M. [1977], \u2018Convergence acceleration for the iterative solution of the equations x= ax+ f\u2019, Computer\n\nMethods in Applied Mechanics and Engineering 10(2), 165\u2013173.\n\nMoulines, E. and Bach, F. R. [2011], Non-asymptotic analysis of stochastic approximation algorithms for\n\nmachine learning, in \u2018Advances in Neural Information Processing Systems\u2019, pp. 451\u2013459.\n\nNedi\u00b4c, A. and Bertsekas, D. [2001], Convergence rate of incremental subgradient algorithms, in \u2018Stochastic\n\noptimization: algorithms and applications\u2019, Springer, pp. 223\u2013264.\n\nNesterov, Y. [2013], Introductory lectures on convex optimization: A basic course, Vol. 87, Springer Science &\n\nBusiness Media.\n\nSchmidt, M., Le Roux, N. and Bach, F. [2013], \u2018Minimizing \ufb01nite sums with the stochastic average gradient\u2019,\n\nMathematical Programming pp. 1\u201330.\n\nScieur, D., d\u2019Aspremont, A. and Bach, F. [2016], Regularized nonlinear acceleration, in \u2018Advances In Neural\n\nInformation Processing Systems\u2019, pp. 712\u2013720.\n\nShalev-Shwartz, S. and Zhang, T. [2013], \u2018Stochastic dual coordinate ascent methods for regularized loss\n\nminimization\u2019, Journal of Machine Learning Research 14(Feb), 567\u2013599.\n\nShalev-Shwartz, S. and Zhang, T. [2014], Accelerated proximal stochastic dual coordinate ascent for regularized\n\nloss minimization., in \u2018ICML\u2019, pp. 64\u201372.\n\n10\n\n\f", "award": [], "sourceid": 2134, "authors": [{"given_name": "Damien", "family_name": "Scieur", "institution": "INRIA - ENS"}, {"given_name": "Francis", "family_name": "Bach", "institution": "Inria"}, {"given_name": "Alexandre", "family_name": "d'Aspremont", "institution": "CNRS - Ecole Normale Sup\u00e9rieure"}]}