arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2610.10527

Decentralized SGD under Heavy-Tailed Noise: Optimal Convergence Rates and the Role of Gradient Clipping

Aleksandar Armacki·Haoyuan Cai·Ali H. Sayed
View original source ↗Read source document ↗Export metadata ↓

Abstract

Heavy-tailed noise has been widely observed in modern machine learning, motivating the use of methods like gradient clipping and normalization. While these methods are well understood in centralized settings, much less is known in decentralized ones, where applying a nonlinearity to local gradients affects both optimization and consensus. Recent works on decentralized non-convex optimization have studied both clipping and normalization under heavy-tailed noise, with clipping yielding suboptimal rates and normalization needing local momentum or mini-batches to converge. This raises the question: can a baseline decentralized method using a nonlinearity achieve optimal convergence rates under heavy-tailed noise? We answer affirmatively with clipped decentralized SGD ($\mathtt{DSGD}$). For smooth non-convex costs under bounded $p$-th moment noise, $p \in (1,2]$, we show that clipped $\mathtt{DSGD}$ achieves order-optimal rates both with high probability and in expectation. Moreover, we establish a linear speed-up in the number of agents, which, to our knowledge, has not been shown for decentralized methods with clipping. The key technical ingredient is a sharp analysis of the consensus gap that exploits the structure of clipping, relegating network effects to higher-order terms. Our results highlight an important distinction between clipping and normalization in decentralized settings: while normalized $\mathtt{DSGD}$ can fail to converge, clipping retains magnitude information, enabling $\mathtt{DSGD}$ to be convergent and order-optimal. Numerical experiments validate our theory.

Explore related subjects

math.OCcs.LGcs.MA

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Aleksandar Armacki, Haoyuan Cai, Ali H. Sayed. 2026-10-07. Decentralized SGD under Heavy-Tailed Noise: Optimal Convergence Rates and the Role of Gradient Clipping. https://arxiv.org/abs/2610.10527

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Perturbed Iterate SGD for Lipschitz Continuous Loss Functions with Numerical Error and Adaptive Step Sizes

Motivated by neural network training in finite-precision arithmetic environments, this work studies the convergence of perturbed iterate SGD using adaptive step sizes in an environment with numerical error. Considering a general stochastic Lipschitz continuous loss function, an asymptotic convergence result to a Clarke stationary point is proven as well as the non-asymptotic convergence to an approximate stationary point in expectation. It is assumed that only an approximation of the loss function's stochastic gradient can be computed, in addition to error in computing the SGD step itself.

math.OC↗
arXiv2026

Controllability Allocation Scores for Targeted Network Intervention

We introduce the controllability allocation score (CAS), a framework for determining how intervention intensity should be distributed among prescribed candidate input directions with respect to designated target variables, together with the target controllability score (TCS) as its nodewise specialization. We establish existence of the CASs and develop a general uniqueness theory based on restricted injectivity of the allocation-to-Gramian map, including generic uniqueness with respect to the time horizon. We show that restricting attention to target variables can fundamentally alter the optimal intervention allocation compared with the standard full-state setting. To enable scalability, we develop a general surrogate-Gramian framework and derive objective-performance guarantees from relative Gramian errors without requiring uniqueness or closeness of the CASs. For the TCS specialization, we further construct a target-only reduced virtual system and derive explicit bounds showing how the approximation error depends on the coupling between target and non-target nodes and on the dynamical growth rate. Experiments on human brain networks show that the reduced formulation accurately approximates the TCS at short horizons, whereas the two controllability criteria exhibit markedly different approximation accuracy at long horizons.

math.OC↗
arXiv2026

Solving the Offline and Online Min-Max Problem of Non-smooth Submodular-Concave Functions: A Zeroth-Order Approach

We consider max-min and min-max problems with objective functions that are possibly non-smooth, submodular with respect to the minimiser and concave with respect to the maximiser. We investigate the performance of a zeroth-order method applied to this problem. The method is based on the subgradient of the Lovász extension of the objective function with respect to the minimiser and based on Gaussian smoothing to estimate the smoothed function gradient with respect to the maximiser. In expectation sense, we prove the convergence of the algorithm to an $ε$-saddle point in the offline case. Moreover, we show that, in the expectation sense, in the online setting, the algorithm achieves $O(\sqrt{N(1+\bar{P}_N)})$ online duality gap, where $N$ is the number of iterations and $\bar{P}_N$ is the path length of the sequence of optimal decisions. The complexity analysis and hyperparameter selection are presented for all the cases. The theoretical results are illustrated via numerical examples.

math.OC↗