Improved Convergence in High Probability of Clipped Gradient Methods with Heavy Tailed Noise

Part of Advances in Neural Information Processing Systems 36 (NeurIPS 2023) Main Conference Track

Authors

Ta Duy Nguyen, Thien H Nguyen, Alina Ene, Huy Nguyen

Abstract

In this work, we study the convergence in high probability of clipped gradient methods when the noise distribution has heavy tails, i.e., with bounded $p$th moments, for some $1