arXiv Science⌕ Search

arXiv · 2610.26137

Structural conditions on trust-region radius updates. Part II: second-order convergence and asymptotic radius behaviour

Abstract

Continuing Part I (arXiv:2606.30202), we ask not which structural conditions on $\{Δ_k\}$ suffice for global convergence but what the radius does; the answers separate the mechanisms by whether the radius is tied to a criticality measure. We identify three asymptotic regimes: the criticality-anchored rules give $\sum_k Δ_k^2/M_k < +\infty$, hence $\liminf_k Δ_k = 0$, and $Δ_k \to 0$ once the model Hessians are bounded; the step-driven rules inherit the summability of the step series, hence $\sum_k Δ_k < +\infty$; the fixed-factor rules keep $Δ_k$ bounded away from zero. Near a critical point with positive definite Hessian, convergence of a subsequence forces convergence of the full sequence. We then extend the framework to second-order criticality through $τ_k = \max\{\|g_k\|, -λ_k^-\}$, obtaining convergence to second-order critical points with the worst-case complexity $\mathcal{O}(\max\{\varepsilon_g^{-2}\varepsilon_H^{-1}, \varepsilon_H^{-3}\})$, which reduces to $\mathcal{O}(\varepsilon^{-3})$ for a common tolerance and is sharp for quadratic models. Finally we establish eventual inactivity of the constraint and superlinear convergence, isolating the two properties of the radius rule through which the mechanism enters; the rate itself is mechanism-independent. The criticality-anchored and capped gradient-scaled rules reach inactivity only under a side condition, $ζ> \overlineκ$ or $\overlineμ > \overlineκ$, with $\overlineκ$ a constant of the solution that cannot be checked in advance. These conditions are necessary, not artefacts: on $f(x) = \sin(x)/x$, $\overlineκ$ is unbounded over the minimisers, so no parameter value meets the condition at all of them. Among the rules tied to a criticality measure, only those with an unbounded multiplier escape such a constant.

Explore related subjects

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Jérémy Rieussec, Fabian Bastin. 2026-10-19. Structural conditions on trust-region radius updates. Part II: second-order convergence and asymptotic radius behaviour. https://arxiv.org/abs/2610.26137

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

The Principle of Optimality in Dynamic Programming: A Pedagogical Note

The principle of optimality is a fundamental aspect of dynamic programming, which states that the optimal solution to a dynamic optimization problem can be found by combining the optimal solutions to its sub-problems. While this principle is generally applicable, it is often only taught for problems with finite or countable state spaces in order to sidestep measure-theoretic complexities. Therefore, it cannot be applied to classic models such as inventory management and dynamic pricing models that have continuous state spaces, and students may not be aware of the possible challenges involved in studying dynamic programming models with general state spaces. To address this, we provide conditions and a self-contained simple proof that establish when the principle of optimality for discounted dynamic programming is valid. These conditions shed light on the difficulties that may arise in the general state space case. We provide examples from the literature that include the relatively involved case of universally measurable dynamic programming and the simple case of finite dynamic programming where our main result can be applied to show that the principle of optimality holds.

math.OC↗

A Geometric Framework for Stochastic Iterations

This paper concerns models and convergence principles for dealing with stochasticity in a wide range of algorithms arising in nonlinear analysis and optimization in Hilbert spaces. It proposes a flexible geometric framework within which existing solution methods can be recast and improved, and new ones can be designed. Almost sure weak, strong, and linear convergence results are established in particular for stochastic fixed point iterations, the stochastic gradient descent method, and stochastic extrapolated parallel algorithms for feasibility problems. In these areas, the proposed algorithms exceed the features and convergence guarantees of the state of the art. Numerical applications to signal and image recovery are provided.

math.OC↗

Attention's forward pass and Frank-Wolfe

We study the hardmax limit of self-attention dynamics for token embeddings obtained in the zero-temperature ($β\to+\infty$) regime, and relate it to the finite-$β$ setting. In this limit, the update rule can be viewed as a Frank-Wolfe step for a quadratic objective over the convex hull of the current token embeddings. When the key-query matrix is negative semidefinite, the method linearly contracts all tokens to a single cluster at the origin. When it is positive semidefinite, extending the hardmax rule to the entire convex hull induces a Voronoi diagram: vertices are stationary, interior points remain in their initial cells, and each token moves along a straight line toward its cell's vertex, yielding (super-)exponential convergence. As a byproduct, we also establish well-posedness of the associated ODE limit in this regime. Returning to the finite-$β$ regime, we model self-attention dynamics as a Markov chain and prove dynamic metastability: with high probability, interior tokens reach near-vertex configurations in a constant number of steps and remain within a small neighborhood for times that grow exponentially in the inverse temperature $β$, before ultimately collapsing to the origin. Thus, the hardmax dynamics accurately approximate the finite-$β$ process over exponentially long time horizons.

math.OC↗