Optimal Algorithms for Nonsmooth Nonconvex Stochastic Optimization with Constraints

Gon Buzaglo; Guy Kornowski

Home/Archive/arXiv

arXiv · 2610.13787

Optimal Algorithms for Nonsmooth Nonconvex Stochastic Optimization with Constraints

Gon Buzaglo·Guy Kornowski

View original source ↗Read source document ↗Export metadata ↓

Abstract

We present new algorithms for stochastic optimization problems in which the objective is neither smooth nor convex, subject to a convex constraint set. Our algorithms return a $(δ,\varepsilon)$-Goldstein stationary point, as recently defined for constrained problems by Liu et al. (2024), with improved complexities, and are the first to achieve optimal convergence rates for this task in several settings. In particular, we design projection-based and projection-free algorithms that achieve a $\widetilde O(δ^{-1}\varepsilon^{-3})$ rate using stochastic gradients, which we show to be optimal, and a $\widetilde O(dδ^{-1}\varepsilon^{-3})$ rate using zeroth-order queries, which is also optimal in its dependence on the dimension $d$ and improves over prior work. Notably, our lower bound is given in terms of the Frank-Wolfe gap and holds even for smooth problems, which is of independent interest. These results show that in nonconvex stochastic optimization, the convergence rate is degraded neither by the presence of constraints nor by nonsmoothness. Our analysis is based on a reduction to online linear optimization, extending a related technique from unconstrained optimization.

Explore related subjects

math.OC cs.LG

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Gon Buzaglo, Guy Kornowski. 2026-10-09. Optimal Algorithms for Nonsmooth Nonconvex Stochastic Optimization with Constraints. https://arxiv.org/abs/2610.13787

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Unified Convergence Theory of Stochastic and Variance-Reduced Cubic Newton Methods

We study stochastic Cubic Newton methods for solving general possibly non-convex minimization problems. We propose a new framework, which we call the helper framework, that provides a unified view of the stochastic and variance-reduced second-order algorithms equipped with global complexity guarantees. It can also be applied to learning with auxiliary information. Our helper framework offers the algorithm designer high flexibility for constructing and analyzing the stochastic Cubic Newton methods, allowing arbitrary size batches, and the use of noisy and possibly biased estimates of the gradients and Hessians, incorporating both the variance reduction and the lazy Hessian updates. We recover the best-known complexities for the stochastic and variance-reduced Cubic Newton, under weak assumptions on the noise. A direct consequence of our theory is the new lazy stochastic second-order method, which significantly improves the arithmetic complexity for large dimension problems. We also establish complexity bounds for the classes of gradient-dominated objectives, that include convex and strongly convex problems. For Auxiliary Learning, we show that using a helper (auxiliary function) can outperform training alone if a given similarity measure is small.

math.OC↗

arXiv2026

The Riemannian Convex Bundle Method

We introduce the convex bundle method to solve convex, non-smooth optimization problems on Riemannian manifolds of bounded sectional curvature. Each step of our method is based on a model that involves the convex hull of previously collected subgradients, parallelly transported into the current serious iterate. This approach generalizes the dual form of classical bundle subproblems in Euclidean space. We prove that, under mild conditions, the convex bundle method converges to a minimizer. Several numerical examples implemented using Manopt$.$jl illustrate the performance of the proposed method and compare it to the subgradient method, the cyclic proximal point algorithm, as well as the proximal bundle method.

math.OC↗

arXiv2026

Leveraging Christoffel-Darboux Kernels to Strengthen Moment-SOS Relaxations

The classical Moment-SOS hierarchy allows one to approximate the global minimum of a polynomial optimization problem through semidefinite relaxations of increasing accuracy. For many optimization problems, however, moving to higher relaxation orders quickly becomes computationally prohibitive. While this difficulty is commonly addressed by exploiting structural properties such as symmetry or sparsity, we explore a complementary direction: strengthening a low-order relaxation using information contained in its own pseudo-moment solution. To this end, we leverage Christoffel-Darboux kernels to identify regions that are compatible with the current pseudo-moment information and use them to construct additional polynomial constraints. We propose two mechanisms: the first, denoted $\HH$, is based on successive global Christoffel restrictions, while the second, denoted $\hh$, combines marginal Christoffel information with an available local solution. For $\HH$, we establish a uniform separation property for successive pseudo-moment solutions and geometric convergence of the fixed-order bounds on a particular subclass of knapsack problems. Moreover, for specific Boolean quadratic problems, we derive an explicit $\hh$ selection rule and establish conditions under which the resulting marginal constraints preserve all global minimizers. Numerical experiments on dense and sparse nonconvex quadratically constrained quadratic programs show that the resulting strengthened low-order relaxations can significantly improve the initial bounds, sometimes attaining exactness, while remaining substantially cheaper than solving higher-order relaxations.

math.OC↗