Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method

Steffen Dereich; Thang Do; Arnulf Jentzen

Home/Archive/arXiv

arXiv · 2603.18899

Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method

Steffen Dereich·Thang Do·Arnulf Jentzen

View original source ↗Read source document ↗Export metadata ↓

Abstract

The adaptive moment estimation (Adam) optimizer proposed by Kingma & Ba (2014) is presumably the most popular stochastic gradient descent (SGD) optimization method for the training of deep neural networks (DNNs) in artificial intelligence (AI) systems. Despite its groundbreaking success in the training of AI systems, it still remains an open research problem to provide a complete error analysis of Adam, not only for optimizing DNNs but even when applied to strongly convex stochastic optimization problems (SOPs). Previous error analysis results for strongly convex SOPs in the literature provide conditional convergence analyses that rely on the assumption that Adam does not diverge to infinity but remains uniformly bounded. It is the key contribution of this work to establish uniform a priori bounds for Adam and, thereby, to provide -- for the first time -- an unconditional error analysis for Adam for a large class of strongly convex SOPs.

Explore related subjects

cs.LG math.OC

Keep this discovery

BibTeX RIS

Steffen Dereich, Thang Do, Arnulf Jentzen. 2026-09-01. Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method. https://arxiv.org/abs/2603.18899

Cite the original work for its findings. Save a collection to share your selection of sources.

Discover connections

cs.LG subject math.OC subject

Connections use source metadata and explicit phrase matches, not verified experimental comparisons.

KEEP EXPLORING

Related papers

arXiv2026

Rates of Convergence in the Central Limit Theorem for Markov Chains, with an Application to TD Learning

We prove a non-asymptotic central limit theorem for vector-valued martingale differences using Stein's method, and use Poisson's equation to extend the result to functions of Markov Chains. We then show that these results can be applied to establish a non-asymptotic central limit theorem for Temporal Difference (TD) learning with averaging.

math.PR↗

arXiv2026

Sample Complexity of Linear Quadratic Regulator Without Initial Stability

Inspired by REINFORCE, we introduce a novel receding-horizon algorithm for the Linear Quadratic Regulator (LQR) problem with unknown dynamics. Unlike prior methods, our algorithm avoids reliance on two-point gradient estimates while maintaining the same order of sample complexity. Furthermore, it eliminates the restrictive requirement of starting with a stable initial policy, broadening its applicability. Beyond these improvements, we introduce a refined analysis of error propagation through the contraction of the Riccati operator under the Riemannian distance. This refinement leads to a better sample complexity and ensures improved convergence guarantees.

math.OC↗

arXiv2026

Learning to Optimize by Differentiable Programming

Solving massive-scale optimization problems requires scalable first-order methods with low per-iteration cost. This tutorial highlights a shift in optimization: using differentiable programming not only to execute algorithms but to learn how to design them. Modern frameworks such as PyTorch, TensorFlow, and JAX enable this paradigm through efficient automatic differentiation. Embedding first-order methods within these systems allows end-to-end training that improves convergence and solution quality. Guided by Fenchel-Rockafellar duality, the tutorial demonstrates how duality-informed iterative schemes such as the alternating direction method of multipliers, and the primal-dual hybrid gradient can be learned and adapted through representative case studies.

cs.MS↗