arXiv Science⌕ Search

arXiv subjects

Yanyi Li

Publications and source records attributed to Yanyi Li.

At least 19 recordsLinked to original sources
arXiv2026

Geodesically Convex Optimization in Hyperbolic Space: Matching Lower Bounds

We establish matching lower bounds for deterministic first-order optimization of globally Lipschitz geodesically convex functions on hyperbolic space. A global minimizer lies within distance $r$ of a known point, each query returns an exact function value and a selected Riemannian subgradient, and the target objective gap is $\varepsilon Lr$, where $L$ is the Lipschitz constant. For sectional curvature $-k^2<0$, let $ρ=kr$ and $ζ=ρ/\tanhρ$. The optimal worst-case query complexity $Q^\star$, taken uniformly over dimensions, satisfies $Q^\star(ρ,\varepsilon)=Θ(ζ\varepsilon^{-2})$ for every $ρ>0$ and $0<\varepsilon\le1/64$. The new lower bound covers arbitrary adaptive deterministic queries and arbitrary outputs, matching the projected-subgradient upper rate of Zhang and Sra (2016). It removes the geodesic-span and historical-halfspace restrictions of earlier matching lower bounds. The construction extends a radial source from a convex carrier with boundary. A small residual cut selects minimizer candidates while an exact hyperbolic persistence inequality keeps every historical contact valid under future carrier growth. One fixed globally convex objective and one fixed subgradient selection consequently realize all exact replies simultaneously. Each query consumes $O(\varepsilon^2)$ of a geometric potential whose available budget is $Θ(ζ)$, yielding the product lower bound in a dimension linear in the query horizon.

math.OC↗
arXiv2026

Near-Optimal Deterministic Exact-Value Complexity for Smooth Convex Optimization

We study the deterministic oracle complexity of smooth convex optimization when the algorithm receives only exact function values. The objective is a globally $β$-smooth convex function, all queries and the final output are restricted to the Euclidean ball of radius $R$, and the unique minimizer lies in the ball of radius $R/2$. We establish an upper bound of $O(d\sqrt{βR^2/ε})$ using coordinate finite differences together with an error-robust accelerated projected method. Our main contribution is a matching lower bound, up to the high-accuracy saturation of the construction: any deterministic adaptive value-oracle algorithm requires $Ω\!\left(d\min\{\sqrt{βR^2/ε},(d/\log(ed))^{1/3}\}\right)$ queries. Consequently, the minimax oracle complexity is $Θ(d\sqrt{βR^2/ε})$ throughout the moderate-accuracy regime $βR^2(\log(ed)/d)^{2/3}\leqε\leq cβR^2$ for a universal constant $c>0$. The lower bound must account for the fact that a single exact real value can encode arbitrarily much information. To overcome this difficulty, we construct a single fixed smooth convex hard instance using a Moreau-smoothed biased max chain, an exact prefix-shielding mechanism, and batched delayed rotations. These techniques preserve consistency with the full adaptive transcript and establish the optimality of the square-root complexity branch for deterministic bounded-query algorithms.

math.OC↗
arXiv2026

Near-Optimal Exact-Value Zeroth-Order Complexity for Smooth Strongly Convex Optimization

We study deterministic adaptive optimization of globally $β$-smooth, $μ$-strongly convex functions using exact scalar function values. Queries and outputs lie in $B_2^d(R)$, and the minimizer lies in $B_2^d(R/2)$. Set $κ=β/μ$, $Q=βR^2/ε$, and $D_d=(d/\log(ed))^{1/3}$. For sufficiently large $d$ and $0<ε\le c_εβR^2$, the minimax value complexity $N_ε$ satisfies \[ \begin{aligned} N_ε&\ge c d\min\{\sqrt Q,\sqrtκ,D_d\},\\ N_ε&\le C d\min\left\{ \sqrt Q,\sqrtκ[1+\log_+(Q/κ)] \right\}, \end{aligned} \] where $c,C,c_ε>0$ are universal constants and $\log_+(t)=\max\{0,\log t\}$. The lower bound uses an exactly shielded smooth chain and batched delayed rotations; the upper bound combines finite differences, acceleration, and restart. When $\min\{Q,κ\}\le D_d^2$, these bounds match up to constants in the accuracy-dominated regime $Q\leκ$ and at constant relative accuracy $ε=Θ(μR^2)$. For arbitrarily higher accuracy in the range $κ\le D_d^2$, the bounds differ by at most $1+\log(μR^2/ε)$; the optimal accuracy dependence remains unresolved in general.

math.OC↗
arXiv2026

Matching Upper and Lower Bounds for Higher-Order Nonconvex Finite-Sum Optimization

We establish tight randomized higher-order oracle complexity for finding first-order stationary points of nonconvex finite sums. Let $n$ be the number of components, $Δ>0$ the initial objective-gap bound, $L_p>0$ an individual $p$-th derivative Lipschitz bound, and $ε>0$ the target gradient norm. For every fixed integer $p\ge 2$, the minimax number of exact component queries returning the value and all derivatives through order $p$, with success probability at least $2/3$, is \[ Θ_p\!\left( n+ΔL_p^{1/p}n^{1-1/(2p)} ε^{-(p+1)/p} \right), \] where the constants depend only on $p$ and the worst case ranges over all finite dimensions. The lower bound holds for unrestricted randomized adaptive algorithms and closes the $\sqrt{n}$ gap between the previously known general-order upper and lower bounds in their dependence on $n$. We extend dense weak hiding to complete higher-order replies while keeping each component's regularity independent of the chain length. The matching upper bound retains the known finite-sum exponent, requires only mean-squared $p$-th derivative increments, and removes the fixed-confidence logarithmic loss by verifying entire recursive-estimation epochs with exact function values. The characterization includes the additive $n$ term for every positive parameter regime; it counts oracle calls with unrestricted internal computation.

math.OC↗
arXiv2026

Second-Order Stationarity with Common Random Losses: Matching Tolerance Bounds

We establish tight polynomial tolerance bounds for stochastic second-order stationarity when each fresh oracle response is a derivative of one common random scalar loss. For a population objective $F$ with Lipschitz gradient and Hessian, the target is $\|\nabla F(x)\|\le ε$ and $λ_{\min}(\nabla^2 F(x))\ge -γ$, with independent tolerances $ε,γ>0$. Under bounded gradient variance and almost-surely bounded Hessian error, the minimax number of fresh gradient or Hessian-vector-product calls is $\widetildeΘ\!\left(ε^{-3}+γ^{-5}\right)$. The characterization fixes positive gap, smoothness, and noise parameters, suppresses logarithmic factors, and allows dimension to grow within an explicit polynomial envelope. The upper bound removes the mixed term $ε^{-2}γ^{-2}$ from the earlier fresh-HVP guarantee. Direct random-line Hessian estimates and a dyadic gradient tracker separate gradient drift from randomly signed curvature motion. The lower bound realizes the endpoint costs through globally defined smooth random losses: a smooth partition localizes scalar noise without a chain-length penalty, while exact cancellation limits the information in the entire response. Consequently, the same tolerance exponents hold even for joint value, gradient, and full-Hessian responses with bounded value variance. For a population Hessian with Hölder exponent $ν\in(0,1]$, fresh gradient/HVP complexity becomes $\widetildeΘ\!\left(ε^{-3}+γ^{-(3+2/ν)}\right)$ under the corresponding dimension envelope.

math.OC↗
arXiv2026

Near-Optimal Higher-Order Oracle Complexity for Convex--Concave Minimax Optimization

For smooth convex--concave minimax optimization, the higher-order lower bound of Chen et al. (2026) applies to a restricted tensor-algorithm class with prescribed regularized Taylor-model updates. We establish the same bound for arbitrary adaptive deterministic and randomized algorithms, matching, up to logarithmic factors, the upper bound of Zhang et al. (2026). Fix an integer $p\ge 2$ and let $L_p>0$ bound the Lipschitz constant of the objective's $p$-th derivative on a compact convex product domain of diameter at most $D_Z>0$. Each feasible query returns the objective value and all derivatives through order $p$. For accuracy $ε>0$, set $Q_{\mathrm{tan}}=L_pD_Z^p/ε$ for tangent residual and $Q_{\mathrm{gap}}=L_pD_Z^{p+1}/ε$ for saddle gap. Let $T_E^{\mathrm{det}}(ε)$ and $T_E^{\mathrm{rand}}(ε)$ denote the high-dimensional minimax query complexities for criterion $E\in\{\mathrm{tan},\mathrm{gap}\}$, with randomized success probability at least $2/3$ on every instance. Our lower bounds and the existing upper bound give $c_pQ_E^{2/(3p-1)}\le T_E^{\mathrm{rand}}(ε)\le T_E^{\mathrm{det}}(ε)\le C_pQ_E^{2/(3p-1)}[1+\log(3+Q_E)]^{6(p-1)}$ for sufficiently large $Q_E$, where $c_p,C_p>0$ depend only on $p$. Thus the same accuracy exponent holds beyond tensor update rules, even for randomized queries and arbitrary feasible outputs. The proof constructs a scalar convex--concave chain with exactly flat gates that hide complete derivative information. Direct product-domain error witnesses and adaptive transcript arguments establish the lower bounds for both criteria.

math.OC↗
arXiv2026

Matrix-Vector Complexity of Low-Rank Approximation

We establish matching polynomial query bounds for low-rank approximation from exact matrix--vector products. Given an unknown matrix $A\in\mathbb{R}^{m\times n}$, at each step a randomized algorithm chooses either $v\in\mathbb{R}^n$ and receives $Av$, or $u\in\mathbb{R}^m$ and receives $A^\top u$. The choice may depend measurably on all previous queries and replies and on the algorithm's private randomness; each vector product costs one query. The output is a rank-$k$ right projector with Schatten-$p$ residual at most $1+\varepsilon$ times optimal. Write $N=\min\{m,n\}$ and let $Q_p^*$ denote the worst-case query budget for success probability $2/3$ on every input. For every $1\le k<N$ and sufficiently small $\varepsilon$, our lower bounds, combined with existing Krylov upper bounds, give $Q_p^*=\widetildeΘ\!\left(\min\{N,k\min\{p^{1/6}\varepsilon^{-1/3},\varepsilon^{-1/2}\}\}\right)$ $(2\le p<\infty)$, $Q_\infty^*=\widetildeΘ\!\left(\min\{N,k\varepsilon^{-1/2}\}\right)$. These bounds have universal constants and allow $p$ to vary with the problem parameters, identifying the transition at $p\varepsilon\asymp1$. A complementary result for each fixed $1\le p<2$ gives $\widetildeΘ_p(\min\{N,k\varepsilon^{-1/3}\})$, with constants and an accuracy threshold that may depend on $p$. Together, the results recover this fixed-norm rate for every fixed finite $p$, supplying the multiplicative rank dependence missing from previous lower bounds. Tildes suppress logarithmic factors. The proof extends adaptive Wishart deferred decisions to a rectangular factor with a $k$-dimensional nullspace. Posterior overlap gives a short fixed-norm argument, while persistence of small compression eigenvalues controls growing $p$ and the spectral endpoint. Exact range recovery handles target costs of order $k$; the Wishart family covers the remaining regimes.

math.OC↗
arXiv2026

Sharp Fresh-Gradient Complexity of Nonconvex-Strongly-Concave Minimax Optimization

We characterize the fresh-gradient oracle complexity of smooth nonconvex-strongly-concave minimax optimization, with matching upper and lower bounds up to logarithmic factors. Let $Φ(x)=\max_y f(x,y)$, where $f$ is jointly $L$-smooth and $μ$-strongly concave in $y$ on unconstrained Euclidean domains, and set $κ=L/μ$. Each query returns a fresh unbiased joint gradient with conditional variance at most $σ^2$. Given initial primal gap at most $Δ$ and dual residual $\|\nabla_y f(x_0,y_0)\|\le G$, the fixed-budget complexity of finding $\|\nablaΦ(\widehat x)\|\leε$ with probability at least $2/3$ is $\widetildeΘ(\sqrtκLΔ/ε^2+κLΔσ^2/ε^4+κ^2σ^2/ε^2+\sqrtκ\log_+(G/(ε\sqrtκ)))$, where $\log_+u=\log\max\{1,u\}$. This characterization holds when $κ$ and $LΔ/ε^2$ exceed universal constants, uniformly over finite dimensions, and the suppressed logarithms are independent of $G$. It establishes the necessity of linear condition-number dependence in the global stochastic cost and identifies a separate statistical refinement cost. A proximal method separates coarse primal progress from one final refinement, while the lower bounds apply to arbitrary adaptive randomized algorithms. Dual initialization contributes only an additive logarithmic cost, yet removing its control eliminates every finite dimension-free complexity bound, even with exact gradients.

math.OC↗
arXiv2026

Matching Lower Bounds for Randomized First-Order Methods in Hessian-Lipschitz Nonconvex Optimization

We establish a randomized first-order lower bound for finding an $ε$-stationary point, $\|\nabla f(x)\|\le ε$, of a nonconvex function with initial gap at most $Δ$, $L_1$-Lipschitz gradient, and $L_2$-Lipschitz Hessian. Each oracle call returns the exact function value and gradient. Let $Q_{\mathrm{rand},\mathrm{FO}}^{\infty}(ε;Δ,L_1,L_2)$ denote the minimax number of calls, maximized over finite dimensions, for arbitrary adaptive randomized algorithms with per-instance success probability at least $2/3$. In the regime $ε\lesssim L_1^2/L_2$ and $ΔL_2^{1/2}ε^{-3/2}\gtrsim 1$, we prove $ Q_{\mathrm{rand},\mathrm{FO}}^{\infty}(ε;Δ,L_1,L_2) \ge c\,ΔL_1^{1/2}L_2^{1/4}ε^{-7/4}, $ where $c>0$ is an absolute constant. This matches the deterministic restarted accelerated-gradient upper bound of Li and Lin (2023) and extends the sharp deterministic lower bound of Zhou (2026) to unrestricted randomized algorithms. Thus, randomization does not improve the high-dimensional worst-case query rate. The construction keeps quadratic curvature visible while revealing the forcing directions sequentially. Large eigenspaces limit preprocessing, and a scheduled-disclosure coupling makes each fresh direction incur a separate cost.

math.OC↗
arXiv2026

Sharp High-Probability Statistical Rates for Heavy-Tailed Nonsmooth Convex Optimization

We establish sharp high-probability statistical rates for nonsmooth stochastic convex optimization under heavy-tailed noise, closing the confidence-dependence gap in both general convex and strongly convex settings. We consider composite objectives with a Lipschitz convex loss and a known proximal regularizer, accessed through fresh unbiased stochastic subgradients. The centered noise has conditional norm and directional $p$th moments bounded by $L^p$ and $s^p$, respectively, where $1<p\le2$ and $0<s\le L$. For $T$ oracle calls and failure probability $0<δ<1/10$, set $q=1-1/p$, $\ell=\log(3/δ)$, and $A=s(L^2/s^2)^q$. With initial distance bound $D$, the minimax objective error is $Θ(D(A+s\ell^q)/T^q)$ in the nonsaturated statistical regime. For a $μ$-strongly convex regularizer, the corresponding quadratic-regime rate is $Θ((A^2+s^2\ell^{2q})/(μT^{2q}))$. These rates require domination of the deterministic and finite-variance terms; matching lower bounds require $d\ge\lceil L^2/s^2\rceil$, while the upper bounds hold in every finite dimension. The new upper bounds attain known statistical lower-bound orders, replacing the previous confidence coefficient $s^{1/p}L^{1-1/p}$ by $s$, and its square by $s^2$ under strong convexity. The key is an online inequality that separates directional fluctuations from the total second moment created by rejection sampling. Localization preserves this separation under strong convexity without additional horizon logarithms. The results characterize oracle complexity with unrestricted internal computation.

math.OC↗
arXiv2026

Joint Lower Bounds for Zeroth-Order Nonconvex Optimization on Euclidean Balls

We prove a joint stochastic zeroth-order lower bound for Goldstein stationarity on a Euclidean query ball, even when the ball is guaranteed to contain a stationary point. In dimension $d$, let $f=\mathbb{E}[F(\cdot;ξ)]$, assume $\mathbb{E}[\operatorname{Lip}(F(\cdot;ξ))^2]\le L_0^2$, and bound the initial objective gap over the ball by $Δ$. For neighborhood radius $δ>0$ and residual tolerance $\varepsilon>0$, our smooth hard family requires $Ω(dL_0^2Δ/(δ\varepsilon^3))$ scalar evaluations for success probability $1/2$, against randomized adaptive algorithms that may retain and repeatedly query each sampled function. The result holds for $\varepsilon\le cL_0$, $Δ\ge Cδ\varepsilon$, and $d\ge C[1+\log(2+ΔL_0^2/(δ\varepsilon^3))]$, on a constructed ball of radius $Θ(Δ/\varepsilon)$. A sequence of localized regions forces repeated direction estimation, and an adaptive Gaussian posterior argument controls sample reuse. The result establishes a joint dimension and accuracy obstruction on solvable bounded-domain instances. The gap is local to the query ball; a matching minimax characterization at a common radius and the corresponding unrestricted global-gap lower bound remain open here.

math.OC↗
arXiv2026

Sharp Minimax Rates for Highly Smooth Strongly Convex Zeroth-Order Optimization

We determine the minimax dimension dependence of noisy zeroth-order optimization for globally strongly convex functions with higher-order smoothness in multilinear operator norm. Each query returns one function value with fresh independent Gaussian noise. For dimension $d$, query budget $T$, and any fixed finite smoothness order $β>2$, the minimax expected objective error satisfies $\mathcal{E}_β(T,d)=Θ_β\left(\min\left\{1,(d^2/T)^{(β-1)/β}\right\}\right)$, with dimension-independent strong-convexity, gradient-smoothness, higher-order-smoothness, noise, and minimizer-radius bounds. Thus, for sufficiently small target error $\varepsilon$, the necessary and sufficient number of noisy values is $Θ_β(d^2\varepsilon^{-β/(β-1)})$, with no logarithmic loss. The new lower bound matches the dimension and budget dependence of the upper bound of Akhavan et al. (2024), and holds for arbitrary randomized adaptive algorithms with unrestricted query points. Its key step localizes a normalized hypercube with one radial cutoff: a query can emphasize one hidden coordinate, but the aggregate information about all coordinates remains small. A self-contained multiscale spherical estimator attains the matching upper bound by cancelling lower-order bias terms. Constants in the rates depend only on the fixed smoothness order under the stated normalization.

math.OC↗
arXiv2026

Closing the Accuracy Gap in Stochastic First-Order Bilevel Optimization

We characterize the optimal accuracy dependence for smooth nonconvex--strongly-convex bilevel optimization with stochastic first-order oracles. For globally unbiased fresh-gradient observations with bounded variance, we prove an $Ω(ε^{-6})$ lower bound matching the accuracy exponent of existing upper bounds. We consider globally Lipschitz gradients, a bounded upper gradient in the lower variable, and Lipschitz lower Hessian blocks, with target $\mathbb{E}|\nabla F(\widehat{x})|\leε$. Let $L$ be the common first-order scale, $μ$ the lower strong-convexity modulus, $ρ$ the lower Hessian-variation budget, $κ=L/μ$, and $χ=1+ρ/μ$. For $ρ\gtrsimμ$, $χ\lesssimκ$, sufficiently high accuracy, and sufficiently large dimension, we establish $Ω\left(Lκ^2Δε^{-2}+L^3χ^2κ^8σ_g^2Δε^{-6}\right)$, where $Δ$ is the initial gap budget and $σ_g^2$ is the lower-gradient variance budget. The bound holds over full Euclidean spaces against arbitrary randomized adaptive algorithms, even when every sample is the gradient of a scalar function. In the common-scale regime $ρ=Θ(L)$, the leading stochastic term has condition-number dependence $κ^{10}$, compared with the achievable $κ^{11}$ dependence. The proof embeds a sequential hard objective into an exact lower response while preserving global regularity and finite gap. A multiscale decomposition limits the gradient signal carrying each new direction, yielding the required sample complexity. We complement this lower bound with a frozen-linear-tilt estimator whose bias is proportional to lower Hessian variation. Its analysis makes this structural dependence explicit and yields matching leading rates in the specified curvature-controlled and small-gap regimes.

math.OC↗
arXiv2026

The Minimax Cost of Smoothness in Fully First-Order Stochastic Bilevel Optimization

We study the sample complexity of finding a point with expected hypergradient norm at most $ε>0$ in nonconvex--strongly-convex stochastic bilevel optimization using fresh, globally unbiased first-order samples with bounded variance. For lower-variable smoothness order $p\ge1$, F$^2$SA-$p$ achieves $\widetilde O(pε^{-4-2/p})$ (Chen et al., 2026), but necessity of the exponent $4+2/p$ was open. We prove matching upper and lower bounds $Θ_p(ε^{-4-2/p})$ under fixed nondegenerate regularity budgets, with constants allowed to depend on $p$. The lower bound holds for unrestricted randomized algorithms under a fixed sample budget, allows dimension to grow with accuracy, and preserves global unbiasedness and all prescribed lower-variable smoothness bounds, even with a scalar lower variable and exact upper gradients. We further characterize the minimax fixed-budget complexity under $L_j=MΛ^j(j!)^β$ for $j\ge3$: $Q_{p,β}^*(ε)=Θ\left(ε^{-4}\left[\inf_{1\le r\le p,,r\in\mathbb N} r^βε^{-1/r}\right]^2\right)$, for $p\in\mathbb N\cup{\infty}$, with constants independent of $p$. At infinite order, this gives $Θ(ε^{-4})$ for $β=0$ and $Θ(ε^{-4}\log^2(1/ε))$ for $β=1$. A gradient-difference method with weighted independent batches attains these bounds uniformly in order. Thus quantitative smoothness yields precise gains, whereas qualitative analyticity alone still permits $Θ(ε^{-6})$ complexity.

math.OC↗
arXiv2026

Dimension-Optimal Dynamic Regret in Exp-Concave Online Learning

Tracking changing comparators is a central problem in online learning. For exp-concave losses, previous work established the $T^{1/3}P_T^{2/3}$ dependence on the horizon $T$ and Euclidean path length $P_T$ for fixed dimension, while the optimal cost of dimension remained unresolved. We resolve this question by establishing a sharp $d^{1/3}$ curved dimension factor, improving the $d^{2/3}$ factor of recent switching and interval reductions [1, 2]. To achieve this improvement, we develop a proper, full-information learner on compact convex subsets of $\mathbb{R}^d$ that adapts to the unknown path length and, under fixed dimension-independent diameter, gradient, and curvature parameters, guarantees $$\operatorname{Reg}_T(u_{1:T}) \le \widetilde{O}\left(d+\min\left\{d^{1/3}T^{1/3}P_T^{2/3},\sqrt{TP_T},T\right\}\right).$$ A matching lower bound on a diameter-one Euclidean ball establishes the minimax frontier up to logarithmic factors when $T\gtrsim d^2$, including the transition from the curved rate to larger path budgets. The lower bound already holds for losses that depend on only one coordinate per round, isolating an unavoidable cost of learning across directions. Our upper bound follows from square-root Gaussian KL control of comparator movement and a cancellation argument that pays for history restarts. The curved learner maintains $O(\log^2 T)$ Gaussian states, uses one gradient per round, and runs in time near-linear in $T$ under a Mahalanobis-projection oracle.

cs.LG↗
arXiv2026

Gradient-Hessian Query Tradeoffs: Matching Lower Bounds in Logarithmic Dimension

We establish matching lower bounds for the tradeoff between gradient and Hessian queries in nonconvex optimization with Lipschitz Hessian. In normalized units, the initial objective gap is at most $T$, the Hessian Lipschitz constant is at most one, and the target gradient norm is one. For sufficiently large $T$, dimension $d \ge C\log(2+T)$, and a prescribed Hessian budget $h \ge 1$ with $h+1 \le aT$, every randomized algorithm requires $Ω(\min\{Td^{1/3}, T^{3/2}/\sqrt{h+1}\})$ ordinary queries. An ordinary query returns the value and gradient; a Hessian query returns the complete second jet and is charged only to its own budget. Both budgets are worst-case caps. Queries and outputs may be arbitrary real points, and no Hessian-norm bound is imposed. The hard input law consists of fixed, globally $C^3$, coercive functions. The construction combines sequential readouts of a continuous random matrix with localized completion blocks and reliability dimensions that vary across hierarchy levels. Premature disclosures are charged by their expected task cost, while every oracle reply remains the exact jet of the presampled objective. This yields a logarithmic dimension threshold and a depth-independent third-derivative bound. Together with a translation of the NALEN upper bound of Chen, Liu, Luo, and Zhang, the result determines the query tradeoff up to absolute constants throughout the stated range.

math.OC↗
arXiv2026

Sample-Adaptive Zeroth-Order Post-Training

Zeroth-order (ZO) optimization offers a memory-efficient alternative to backpropagation for fine-tuning large language models. However, existing ZO methods typically sample training examples uniformly, overlooking their evolving importance and leaving avoidable estimation variance. To address this limitation, we propose Sample-Adaptive Zeroth-Order Optimization (SAZO), which learns sample importance online and adaptively allocates queries to accelerate convergence. SAZO extracts gradient-norm information from existing directional differences to estimate sample importance without access to true gradients. Furthermore, to track evolving importance from sparse minibatch feedback, SAZO maintains per-example exponential moving averages that aggregate observations across updates. The method requires no additional model evaluations and only $O(N)$ additional scalar state compared with MeZO. We prove that SAZO asymptotically matches the leading convergence term of the minimum-variance sampling oracle, and achieves a strictly faster convergence rate than standard methods. Extensive experiments across four model families demonstrate improved average downstream performance and up to $6\times$ faster convergence to the same training loss in terms of query count.

cs.LG↗
arXiv2026

PRAC: Principal-Random Subspace for LLM Activation Compression and Memory-Efficient Training

Activations have become the primary memory bottleneck in large-batch LLM training. However, existing compression methods fail to exploit the spectral structure of activations, resulting in slow convergence or limited compression. To address this, we bridge the relationship between the algorithm's fast convergence and the requirements for subspace projection, and show that an effective compression should yield an unbiased estimate of the original activation with low variance. We propose Principal-Random Subspace for LLM Activation Compression (PRAC), which novelly decomposes activations into two components: a principal subspace captured via SVD to retain dominant information, and a random subspace sampled from the orthogonal complement to approximate the tail. By introducing a precise scaling factor, we prove that PRAC yields an unbiased gradient estimator with minimum variance under certain conditions. Extensive experiments on pre-training and fine-tuning tasks demonstrate that PRAC achieves up to 36% total memory reduction with negligible performance degradation and minimal computational cost.

cs.LG↗