arXiv Science⌕ Search

arXiv subjects

Kim-Chuan Toh

Publications and source records attributed to Kim-Chuan Toh.

At least 19 recordsLinked to original sources
arXiv2026

Exploring chordal sparsity in semidefinite programming with sparse plus low-rank data matrices

Semidefinite programming (SDP) problems are challenging to solve because of their high dimensionality. However, solving sparse SDP problems with small tree-width are known to be relatively easier because: (1) they can be decomposed into smaller multi-block SDP problems through chordal conversion; (2) they have low-rank optimal solutions. In this paper, we study more general SDP problems whose coefficient matrices have sparse plus low-rank (SPLR) structure. We develop a unified framework to convert such problems into sparse SDP problems with bounded tree-width. Based on this, we derive rank bounds for SDP problems with SPLR structure, which are tight in the worst case.

math.OC↗
arXiv2026

A Logarithmic-dual-update Bregman ADMM for Optimal Transport

This paper studies an entropy Bregman alternating direction method of multipliers (BADMM) with logarithmic dual update for solving discrete optimal transport (OT) problems. The algorithm uses simple row and column normalizations without inner iterations, allowing efficient CPU and GPU implementations. However, the convergence of this method for general OT problems has not been established. Starting from a strictly positive feasible primal point and a zero multiplier, here we establish the $R$-linear convergence of the primal--dual sequence for general OT costs with any fixed multiplier stepsize in $(0,2)$ and a sufficiently large fixed penalty, even when the primal limit lies on the boundary. Our analysis further characterizes the primal limit as the entropy Bregman projection of the initial point onto the optimal solution set and establishes strict complementarity of the limiting OT primal--dual solution. For separable costs on two-dimensional grids, we exploit the separable structure to accelerate matrix--vector products and reduce storage requirements. Numerical experiments show competitive performance of our BADMM for general cost matrices with substantial speedups over state-of-the-art solvers, including HOT and HALO, for squared Euclidean costs on two-dimensional-grid marginals. The largest tested problem, with $2048\times2048 \approx 4.19$M grid points in each marginal, is solved in approximately 100 seconds on a single GPU.

math.OC↗
arXiv2026

Echo: Error-Compensated 4-Bit Tensor-Parallel Communication for LLM Training and Inference

Tensor parallelism ends every layer with an all-reduce that sits between two dependent matrix multiplications, so its cost cannot be hidden and grows as compute outpaces interconnect bandwidth. Sending 4-bit instead of 16-bit values would cut this cost by nearly 4x, but the all-reduce must add the values it carries, and rounding the sum injects an error at every call. Model quality depends on how these errors accumulate over layers, yet existing methods treat each call on its own. We present Echo, a 4-bit all-reduce for training and inference that controls the accumulated error. Echo moves data in 4 bits and sums in high precision, so each value is quantized at most twice per call for any number of GPUs. The summation reveals each injected error, and Echo feeds it back into later calls, so old errors cancel instead of accumulating. We prove that the accumulated error stays bounded independently of depth. On A100 GPUs, Echo is 2.09x faster than the 16-bit NCCL all-reduce on 1 GB messages. It speeds up the time to first token by up to 1.16x and the training step by up to 1.09x on an idle interconnect, and by up to 1.44x and 1.38x as background traffic grows. It keeps the average accuracy over seven benchmarks within 0.4 points of 16-bit inference on Qwen3-8B. In 10B-token pretraining, its final loss is less than 0.9% above that of 16-bit training. Echo needs no FP8 or FP4 hardware.

cs.AI↗
arXiv2026

An Inexact Halpern-accelerated Preconditioned Generalized Proximal Point Algorithm for the Maximal Monotone Inclusion Problem

This paper studies an inexact Halpern-accelerated generalized proximal point algorithm with an admissible positive semidefinite preconditioner $\mathcal{M}$ for solving maximal monotone inclusion problems. The proposed framework combines Halpern anchoring, resolvent inexactness, and relaxation over the full weight range $ρ\in(0,2]$, thereby covering both under-relaxed and over-relaxed proximal iterations within a unified scheme. We first establish convergence of the inexact resolvent sequence under conditions that allow general anchoring parameters and certain nonsummable tolerances. For anchoring parameters $β_{k}=1/(k+r)$ with $r\geq2$, we then derive explicit bounds on the squared fixed-point residual in the $\mathcal{M}$-seminorm. In particular, if the tolerances satisfy $\varepsilon_{k}=\mathcal{O}((k+1)^{-α})$ with $α>3/2$ for $0<ρ<2$ and $α>2$ for $ρ=2$, these bounds yield an $\mathcal{O}(1/k^{2})$ convergence rate. The stronger decay condition on the inexactness tolerances at $ρ=2$ highlights a qualitative distinction between the endpoint and the interior regime in the inexact setting. Finally, we develop inexact accelerated versions of the preconditioned alternating direction method of multipliers (pADMM) and the preconditioned primal--dual hybrid gradient (PDHG) method based on this framework, derive inexactness criteria based on subproblem residuals, and establish a nonergodic $\mathcal{O}(1/k)$ KKT residual rate for their inexact iterates.

math.OC↗
arXiv2026

McADMM: A Multi-Clique Augmented Lagrangian-Based Algorithm for Large-Scale Sparse SDPs with Bound Constraints

sGS-PADMM [21, 14, 6] is a powerful and versatile class of convergent multi-block ADMM solvers for implementations on moderate-sized linear semidefinite programming (SDP) problems. In this paper, we further enhance this class of algorithms for solving SDP problems by proposing a new multi-clique decomposition approach, allowing substantial improvements in applications on large-scale sparse SDPs (e.g., where $n > 1000$) with conducive aggregate sparsity patterns. Our SDP decomposition strategy mainly aims to reduce the estimated PSD projection cost after decomposition, in contrast to common decomposition algorithms that are encumbered with minimizing the overlaps between cliques. This feature is made possible by our novel linear-space projection approach that is capable of efficiently processing a large number of overlap constraints via simple averaging steps. For the numerical experiments, we demonstrate the performance of our solver -- named McADMM for Multi-clique ADMM -- on a number of large-scale SDP instances that arise from relaxations of some important quadratically constrained quadratic programming (QCQP) problems. The performance of McADMM is contrasted against other state-of-the-art decomposition-based solvers as well as the non-decomposed sGS-PADMM to highlight our key contributions. We additionally develop a GPU implementation of McADMM and demonstrate that it can substantially accelerate the decomposed solver.

math.OC↗
arXiv2026

Extension on the convergence rates of moment-SOS hierarchies via approximation of truncated moment sequences

This paper continues our work on the convergence rates of moment-SOS hierarchies via approximation of truncated moment sequences. We extend the method developed for the Schmüdgen-type hierarchy to the Putinar-type, Krivine--Stengle-type, extended-Handelman-type, and Hol-Scherer-type hierarchies. The main idea is to lift a pseudo-moment sequence to a simple set, approximate it by a moment sequence, and project the atoms of a representing measure onto the original feasible set. The Łojasiewicz inequality then converts the error in the defining constraints into an error in the moments. With the Łojasiewicz exponent $0<L\leq 1$, we obtain the convergence rates $\mathrm{O}((\log_2 r)^{3L/2}/r^{L})$ for the Putinar-type hierarchy and $\mathrm{O}(1/r^{L/2})$ for the normalized Krivine--Stengle-type and extended-Handelman-type hierarchies. In the matrix setting, we utilize a Chebyshev-type kernel on $[-1,1]^n$ to derive the rate $\mathrm{O}((\log_2 r)^{3L/2}/r^{L})$ for the Hol-Scherer-type hierarchy. Together with our preceding work, the results provide a universal method for studying convergence rates of different types of moment-SOS hierarchies.

math.OC↗
arXiv2026

A Hybrid Subgradient Method for Nonsmooth Nonconvex Bilevel Optimization

In this paper, we focus on the nonconvex-nonconvex bilevel optimization problem (BLO), where both upper-level and lower-level objectives are nonconvex, with the upper-level problem potentially being nonsmooth. We develop a two-timescale momentum-accelerated subgradient method (TMG) that employs two-timescale stepsizes, and establish its local convergence when initialized within a sufficiently small neighborhood of the feasible region. To develop a globally convergent algorithm for (BLO), we introduce a feasibility restoration scheme (FRG) that drives iterates toward the feasible region. Both (TMG) and (FRG) only require the first-order derivatives of the upper-level and lower-level objective functions, ensuring efficient computations in practice. We then develop a novel hybrid method that alternates between (TMG) and (FRG) and adaptively estimates its hyperparameters. Under mild conditions, we establish the global convergence properties of our proposed algorithm. Preliminary numerical experiments demonstrate the high efficiency and promising potential of our proposed algorithm.

math.OC↗
arXiv2026

Establishing Boundary KKT Convergence of Mirror Descent through Reparameterization

Sequence convergence to a boundary Karush--Kuhn--Tucker (KKT) point has long remained unclear for nonconvex mirror descent with Legendre kernels. The difficulty arises from the blow-up of the gradient of the Legendre kernel at the boundary. Recent work~\cite{dingtoh2026nonkkt} shows that mirror descent can accumulate at non-KKT boundary points despite decreasing objective values, precluding a convergence guarantee to KKT points in general. Despite this negative result, mirror descent remains effective in many real applications. Motivated by this contrast, we address the boundary difficulty directly and establish KKT convergence of mirror descent for a broad class of structured nonconvex problems. We analyze mirror descent in reparameterized variables, where the Hessian metric is flattened and remains nondegenerate as the boundary is approached. Under extension and definability conditions jointly coupling the objective, the Legendre kernel, and the feasible region, the reparameterized sequence has finite length and converges, thereby recovering convergence to a KKT point of the original sequence. Our general framework applies to some concrete instances: Shannon entropy, Fermi--Dirac entropy, and power kernels on polyhedron.

math.OC↗
arXiv2026

Non-KKT Accumulation in Entropic Mirror Descent

For mirror descent generated by a Legendre kernel, perhaps one of the most basic question in optimization is this: must every accumulation point of a bounded mirror descent sequence be Karush--Kuhn--Tucker (KKT) stationary under proper stepsizes? We show that the answer is no. A longstanding obstacle to resolving this question is the boundary blow-up of the Legendre gradient: it keeps every mirror step in the interior, while at a boundary limit, the inverse entropy metric vanishes on active coordinates and can erase the dual-feasibility in the KKT system. We construct $C^\infty$ objectives and bounded sequences generated by the Shannon-entropic mirror descent on the nonnegative orthant $\R_+^n$, for every $n\geq 3$, and on the probability simplex $Δ_n$, for every $n\geq 4$, such that, in each case, the set of accumulation points is a smooth boundary circle containing a nonempty relatively open arc of non-KKT points. The steps satisfy $α_k\asymp k^{-β}$ with $β\in(1/2,1)$, the objective values are nonincreasing, and the objectives are entropy-relatively smooth. Hence the pathology stems from the degeneracy of the Bregman geometry at the boundary, rather than from failure of descent, or improper stepsizes. To the best of our knowledge, these provide the first counterexamples to KKT accumulation for bounded mirror descent sequences with nonincreasing objective values.

math.OC↗
arXiv2026

Stochastic Subgradient Methods with Guaranteed Global Stability in Nonsmooth Nonconvex Optimization

In this paper, we focus on providing convergence guarantees for stochastic subgradient methods in minimizing nonsmooth nonconvex functions. We first investigate the global stability of a general framework for stochastic subgradient methods, where the corresponding differential inclusion admits a coercive Lyapunov function. We prove that, for any sequence of sufficiently small stepsizes and approximation parameters, coupled with sufficiently controlled noises, the iterates are uniformly bounded and asymptotically stabilize around the stable set of its corresponding differential inclusion. Moreover, we develop an improved analysis to apply our proposed framework to establish the global stability of a wide range of stochastic subgradient methods, where the corresponding Lyapunov functions are possibly non-coercive. These theoretical results illustrate the promising potential of our proposed framework for establishing the global stability of various stochastic subgradient methods.

math.OC↗
arXiv2026

Sparsity-Cone SDP Relaxations and Applications to Variable Fixing for Sparse Quadratic Programs

Quadratic programs (QPs) with sparsity constraint are generally NP-hard, and their efficient global solution depends crucially on tractable tight convex relaxations. In this paper, we propose a sparsity-cone semidefinite programming (SC-SDP) relaxation for sparse (indefinite) QPs. Unlike standard SDP liftings, such as the SDP--RLT relaxation, which involve a $(2n+1)$-dimensional semidefinite matrix, the proposed SC-SDP formulation uses only a $(n+1)$-dimensional matrix together with a single sparsity-cone constraint $\mathcal{K}$ to handle the relaxation of the $\ell_0$-norm constraint. We prove that SC-SDP is equivalent in strength to the SDP--RLT relaxation. We further study the sparsity cone $\mathcal{K}$, deriving structural characterizations and showing that projection onto $\mathcal{K}$ can be computed efficiently via a one-dimensional subproblem. Building on the dual of SC-SDP, we derive explicit presolving mechanisms, including a dual-fixing rule for individual variables, a screening-cut rule for excluding larger support patterns, and a dual-refinement step for improving presolving certificates. To solve the resulting relaxation SC-SDP efficiently, we develop a two-phase Riemannian-based augmented Lagrangian method and exploits the structured projection subproblems. Numerical experiments on several classes of sparse QPs show that SC-SDP preserves the bound quality of SDP--RLT while offering substantial computational advantages and practically effective presolving capabilities.

math.OC↗
arXiv2026

Optimization Hyper-parameter Laws for Large Language Models

Large Language Models have driven significant AI advancements, yet their training is resource-intensive and highly sensitive to hyper-parameter selection. While scaling laws provide valuable guidance on model size and data requirements, they fall short in choosing dynamic hyper-parameters, such as learning-rate (LR) schedules, that evolve during training. To bridge this gap, we present Optimization Hyper-parameter Laws (Opt-Laws), a framework that predicts final training loss as a function of LR schedule, model size, and data size. Grounded in SDE-based convergence and escape analyses, Opt-Laws yield interpretable convergence and escape features that predict final training loss across model scales, enabling schedule pre-selection from small-scale experiments. Empirically, Opt-Laws achieve a 94% Top-2 hit rate for identifying near-optimal schedule candidates on held-out configurations, correctly identify the best-performing schedule family in all five evaluated out-of-family settings, and detect training divergence with F1 = 0.92.

cs.LG↗
arXiv2026

A preconditioned augmented Lagrangian method for solving semidefinite programming problems

In this work, we propose a preconditioned augmented Lagrangian method (ALM) for solving semidefinite programming (SDP) problems. The preconditioner is implemented via a weighted penalty function in the ALM subproblem, with the weight matrix derived from the projection operator onto the tangent space of the feasible region. This simple yet effective modification significantly accelerates ALM, particularly for ill-conditioned SDPs. By combining the preconditioned ALM with our previously developed feasible method SDPF, we develop SDPF+, an SDP solver capable of handling convex problems with possibly nonlinear objective functions. Extensive numerical experiments demonstrate the efficiency and robustness of SDPF+, showing that it can generally outperform other solvers on large-scale SDPs whose optimal solutions exhibit low-rank structure.

math.OC↗
arXiv2026

ripALM: A Relative-Type Inexact Proximal Augmented Lagrangian Method for Linearly Constrained Convex Optimization

Inexact proximal augmented Lagrangian methods (ipALMs) have been widely used for solving linearly constrained convex optimization problems, owing to their strong theoretical guarantees and excellent numerical performance. In practice, however, existing ipALMs typically employ Rockafellar-type absolute error criteria for solving the subproblems, which require delicate problem-dependent tuning of error-tolerance sequences. In this paper, we propose ripALM, a relative-type ipALM whose subproblem error criterion has only a \textit{single} tolerance parameter in $[0,1)$. This makes the method simpler to implement and less sensitive to parameter tuning in practice. On the other hand, the use of such a relative-type error criterion renders the convergence of our ripALM beyond the scope of the convergence theory of existing ipALMs. To address this gap, we develop a new analysis framework under which ripALM is shown to admit desirable global convergence properties and it achieves an asymptotic (super)linear convergence rate under a standard error bound condition. While there exist other relative-type inexact pALMs, to ensure convergence, they require additional correct steps that generally impede the convergence speed. To the best of our knowledge, ripALM is the first relative-type inexact version of the vanilla pALM that avoids both summable tolerance parameter sequences and correction steps, while retaining rigorous convergence guarantees. Numerical experiments on quadratically regularized optimal transport and basis pursuit denoising problems demonstrate the effectiveness and robustness of our proposed method.

math.OC↗
arXiv2026

Gradient flow for finding E-optimal designs

The $E$-optimality criterion for a regression model maximizes the smallest eigenvalue of the information matrix and becomes non-differentiable when this eigenvalue has multiplicity greater than one. Working in the $2$-Wasserstein space, we show that the Wasserstein gradient at an empirical measure coincides, up to a constant factor, with the Euclidean particle gradient for smooth criteria such as $D$- and $L$-optimality, and that the approximation gap for equal-weight $N$-particle designs vanishes at an explicit rate. The main challenge is the nonsmooth $E$-criterion, for which the Wasserstein gradient does not exist. We replace it with a constrained Wasserstein steepest-ascent field obtained by maximizing feasible directional derivatives over the tangent cone of the design space, and prove that the resulting flow satisfies an exact energy identity and that every limit point is first-order stationary. The particle ascent computation reduces to a convex semidefinite programme whose dimension equals the multiplicity of the smallest eigenvalue. In numerical comparisons on second-order response surface models and a seven-dimensional logistic regression model, the constrained Wasserstein steepest-ascent method attains near-optimal $E$-criterion values and is markedly more reliable than particle swarm optimization in higher-dimensional settings. The framework applies more broadly to other nonsmooth minimax criteria in optimal design, and a numerical experiment on the minimax-single-parameter criterion confirms that the method attains the theoretical optimum.

math.OC↗
arXiv2026

Convergence Analysis of a Relative-type Inexact Preconditioned Proximal ALM for Convex Nonlinear Programming

This article investigates the convergence properties of a relative-type inexact preconditioned proximal augmented Lagrangian method (rip$^2$ALM) for convex nonlinear programming, a fundamental class of optimization problems with broad applications in science and engineering. Inexact proximal augmented Lagrangian methods have proven to be highly effective for solving such problems, owing to their attractive theoretical properties and strong practical performance. However, the convergence behavior of the relative-type inexact preconditioned variant remains insufficiently understood. This work aims to reduce this gap by rigorously establishing the global convergence of the sequence generated by rip$^2$ALM and proving its asymptotic (super)linear convergence rate under standard assumptions. In addition, we derive the global ergodic convergence rate with respect to both the primal feasibility violation and the primal objective residual, thereby offering a more comprehensive understanding of the overall performance of rip$^2$ALM. These results deepen our theoretical understanding of the family of proximal augmented Lagrangian methods and motivate their development for practical, large-scale structured application problems.

math.OC↗
arXiv2026

Slow-Fast Inference: Training-Free Inference Acceleration via Within-Sentence Support Stability

Long-context autoregressive decoding remains expensive because each decoding step must repeatedly process a growing history. We observe a consistent pattern during decoding: within a sentence, and more generally within a short semantically coherent span, the dominant attention support often remains largely stable. Motivated by this observation, we propose Slow-Fast Inference (SFI), a training-free decoding framework that decouples generation into frequent low-cost fast steps and occasional dense-attention slow steps. Fast steps reuse a compact sparse memory for efficient decoding. Slow steps are triggered near semantic boundaries. At slow steps, the model revisits the broader context and uses the Selector to refresh the selected memory for subsequent fast steps. Across the evaluated context lengths, SFI delivers approximately $1.6\times$--$14.4\times$ higher decoding throughput while generally maintaining quality on par with the full-KV baseline across long-context and long-CoT settings. Because SFI is training-free and applies directly to existing checkpoints, it offers a practical path to reducing inference cost for contemporary autoregressive reasoning models in long-context, long-horizon, and agentic workloads.

cs.LG↗
arXiv2026

Robust principal component analysis with rank and cardinality regularization under matrix factorization

Robust principal component analysis is an important representative method in data analysis. It is usually viewed as an optimization problem involving the rank and $\ell_0$-norm of matrices. In this paper, we study the rank and $\ell_0$ regularized optimization problem and its matrix factorization problem. We establish their equivalences on global minimizers and stationary points, respectively. Furthermore, we construct a broadly applicable equivalent nonconvex relaxation framework for the constrained factorization model in the sense of global minimizers and stationary points with strong optimality conditions (called strong stationary points). For the general factorization problem with lower semicontinuous regularizers and a loss function whose gradient is locally Lipschitz, we propose a novel proximal gradient-based algorithm based on joint and alternating calculation with convergence to its limiting-critical points. The algorithm can attain the stationary points of the original problem and its adaptive counterpart can attain the strong stationary points of the factorization problem.

math.OC↗