arXiv subjects

Defeng Sun

Publications and source records attributed to Defeng Sun.

At least 37 records · Page 2Linked to original sources

arXiv2026

Robust mean field control: an application to optimal execution under composite uncertainty

We provide a framework for robust mean field control problems that describe multi-dimensional optimal liquidation problems under uncertainty from both the underlying stochastic process and the deterministic model parameters. The verification results are established with Hamilton-Jacobi-Bellman-Isaacs (HJBI) equations where the variables are probability measures and the Hamiltonian nonlinearly involves the joint distribution of position and momentum. Using novel a priori estimates, we establish the well-posedness of the HJBI equations featuring general or quadratic Hamiltonians that are neither displacement convex nor concave in their momentum. The a priori estimates and well-posedness results are extended during their application to optimal liquidation problems, where we allow the Hamiltonian to have derivatives of linear growth and solve the constrained multi-dimensional linear quadratic optimal liquidation problem under composite uncertainty.

math.OC↗

arXiv2026

Standard Transformers Achieve the Minimax Rate in Nonparametric Regression with $C^{s,λ}$ Targets

The tremendous success of Transformer models in fields such as large language models and computer vision necessitates a rigorous theoretical investigation. To the best of our knowledge, this paper is the first work proving that standard Transformers can approximate Hölder functions $ C^{s,λ}\left([0,1]^{d\times n}\right) $$ (s\in\mathbb{N}_{\geq0},0<λ\leq1) $ under the $L^t$ distance ($t \in [1, \infty]$) with arbitrary precision. Building upon this approximation result, we demonstrate that standard Transformers achieve the minimax optimal rate in nonparametric regression for Hölder target functions. It is worth mentioning that, by introducing two metrics: the size tuple and the dimension vector, we provide a fine-grained characterization of Transformer structures, which facilitates future research on the generalization and optimization errors of Transformers with different structures. As intermediate results, we also derive the upper bounds for the Lipschitz constant of standard Transformers and their memorization capacity, which may be of independent interest. These findings provide theoretical justification for the powerful capabilities of Transformer models.

stat.ML↗

arXiv2026

Approximation of Analytic Functions by ReLU Neural Networks with Adjustable Depth and Width

In contrast to most studies on neural network approximation theory that characterize results through a single parameter, such as the total number of network parameters, \cite{shen2020deep} pioneered the characterization of approximation rates as a joint function of the width parameter $N$ and the depth parameter $L$, thereby granting greater architectural flexibility. Existing works using the $(N,L)$-characterization focus on function classes with finite smoothness $s$, establishing a typical approximation rate of $\mathcal{O}\left(N^{-2s/d}L^{-2s/d}\right)$ with $d$ denoting the input dimension, which indicates that network depth and width play symmetric roles for these classes. In contrast, this paper establishes upper bounds for the approximation of analytic functions, which possess infinite smoothness, via ReLU networks under the $(N,L)$-characterization. Specifically, we derive approximation rates of $\mathcal{O}\left(N^{-C L^τ}\right)$, where $C>0$ is some constant and $τ>0$ is a parameter influenced by the relation between $L$ and $N$. In particular, $τ=1$ if $N$ scales roughly as $L^d$. Our findings reveal that depth plays a more critical role than width in the context of analytic function approximation. The main technical difficulty of obtaining such upper bounds lies in the trade-off between the smoothness parameters and the approximation accuracy. To overcome this difficulty, we employ refined constructions of several ReLU networks to approximate power functions, multivariate multiplication, and polynomials, which may be of independent interest.

stat.ML↗

arXiv2026

Convergence Analysis of the Restarted Moving-Anchored Extra-Gradient Method in the Absence of Local Lipschitz Continuity

In this paper, we introduce the moving-anchored extra-gradient (MAEG) method for solving monotone inclusion problems involving the sum of a continuous monotone operator and a maximal monotone operator. Notably, the distance from the anchor point to the solution set is designed to be monotonically non-increasing. Under Lipschitz continuity of the forward operator, MAEG attains an $\mathcal{O}(1/k)$ non-asymptotic iteration complexity, and when a positive anchor-update parameter is used, it further achieves an $o(1/k)$ asymptotic rate. Furthermore, leveraging the specific behavior of the anchor point, we propose a tailored restart strategy. We demonstrate that this strategy ensures convergence even in the absence of local Lipschitz continuity, while preserving the original iteration complexity guarantees whenever the Lipschitz condition holds.

math.OC↗

arXiv2026

A Semismooth Newton Augmented Lagrangian Method for Sparse Spectral Risk Optimization

Empirical risk minimization is a standard and effective paradigm for learning predictive models by minimizing average loss. In high-stakes decision-making, however, an average-loss criterion may underrepresent rare but severe losses. Spectral risk measures (SRMs) provide a principled framework by incorporating weighted order statistics of losses, but the induced nonsmoothness and nonseparability from sorting make the resulting optimization problems challenging. We propose a relative inexact proximal augmented Lagrangian method with a semismooth Newton subproblem solver for solving SRM-based optimization problems. Exploiting a dual reformulation and properties of the Moreau envelope, we reduce the subproblems to structured dual-variable formulations, significantly simplifying computation. We provide explicit generalized Jacobian characterizations and tailor the pool adjacent violators algorithm for their efficient evaluation. Numerical results on synthetic and real-data instances show that the proposed method attains lower running times than the tested ADMM baseline while producing comparable stationarity residuals and sparse solutions.

math.OC↗

arXiv2026

Factorized low-rank matrix recovery problem, Schatten-$q$ quasi-norm, Error bound for critical point, Kurdyka-Łojasiewicz property, Inexact proximal alternating linearized minimization

The Schatten-$q$ quasi-norm is a widely used nonconvex rank surrogate and matrix factorization is an effective approach to reduce computational cost. In this paper, we consider the equivalent group-sparse factorized reformulation of Schatten-$q$ norm regularized low-rank matrix recovery problem. Though this factorized model exhibits favorable performance, two issues remain: (i) the error bound of critical points is unexplored; (ii) the proximal operator of $\|\cdot\|_2^q$ lacks a closed-form solution for general $q$, limiting algorithms to adopt fixed $q$ like $1/2$ or $2/3$. This paper addresses both issues. We investigate the properties of critical points for the factorized problem and show that, compared to nuclear norm, the Schatten-$q$ norm implicitly endows critical points with column orthogonality. From this insight, we introduce the notion of S-critical points under mild conditions that ensure column orthogonality with easily operable criterion for identifying. We show that global optimal points must be S-critical points and we derive an error bound between S-critical points and the true matrix. We further present an inexact proximal alternating linearized minimization method for the factorized problem, along with practically computable inexact proximal operator for $\|\cdot\|_2^q$ and criteria to find solutions satisfying inexactness conditions, and we establish the whole sequence convergence and a convergence rate guarantee under Kurdyka--Łojasiewicz condition. Moreover, we prove that the factorized model with least-squares loss has KL exponent $1/2$ at S-critical points, then the iteration converges linearly under suitable condition. Extensive numerical experiments validate the effectiveness of our algorithm and confirm the theoretical properties of the factorized model.

math.OC↗

arXiv2026

Low-Rank Tensor Completion using Tensor Train Decomposition via Riemannian Optimization on the Quotient Geometry

Owing to the effectiveness of Tensor Train (TT) decomposition in managing high-order tensors, low-rank tensor completion within the TT-format has emerged as a prominent research focus. In this paper, we leverage the left-orthogonal property of the TT-decomposition to construct a novel quotient manifold and introduce a family of admissible Riemannian metrics. Within this geometric framework, we propose a new approach to constructing retractions compatible with the quotient structure, realized via two novel retractions based on recursive polar and QR decompositions that respect the recursive orthogonalization structure of the TT format. We then derive Riemannian gradient descent and conjugate gradient methods to solve the tensor completion problem. Theoretically, our approach streamlines the horizontal projection by reducing the number of unknowns per block from a quadratic dependence on the TT-ranks to a near-half scaling, thereby enhancing computational efficiency over conventional quotient-based methods. Numerical experiments demonstrate that the proposed algorithms achieve reconstruction accuracy comparable to state-of-the-art TT-based geometric methods.

math.NA↗

arXiv2026

A symmetric Gauss-Seidel based alternating proximal ALM for generalized Nash Equilibrium problems in Banach spaces

In this paper, we study a class of monotone generalized Nash equilibrium problems (GNEPs) with jointly linear constraints. The players' strategy spaces are real Hilbert spaces, while the joint constraint is formulated in a Banach space. To solve such problems, we propose a novel symmetric Gauss-Seidel (sGS) based alternating proximal augmented Lagrangian method (sGS-APALM) which incorporates newly designed quadratic surrogates. In contrast to existing regularization and ALM-type methods, the proposed method avoids solving coupled Nash equilibrium subproblems at each iteration and instead updates the players' strategies alternately by solving a sequence of unconstrained quadratic programs. Moreover, unlike many existing splitting-based methods, our global convergence analysis and convergence rate estimation require only monotonicity and Lipschitz continuity of the pseudo-gradient mapping, without imposing stronger assumptions such as strong monotonicity or co-coercivity. Finally, we apply the method to a class of risk-neutral PDE-constrained GNEPs with joint state constraints, and preliminary numerical results demonstrate its efficiency and effectiveness.

math.OC↗

arXiv2026

Beyond the Prompt in Large Language Models: Comprehension, In-Context Learning, and Chain-of-Thought

Large Language Models (LLMs) have demonstrated remarkable proficiency across diverse tasks, exhibiting emergent properties such as semantic prompt comprehension, In-Context Learning (ICL), and Chain-of-Thought (CoT) reasoning. Despite their empirical success, the theoretical mechanisms driving these phenomena remain poorly understood. This study dives into the foundations of these observations by addressing three critical questions: (1) How do LLMs accurately decode prompt semantics despite being trained solely on a next-token prediction objective? (2) Through what mechanism does ICL facilitate performance gains without explicit parameter updates? and (3) Why do intermediate reasoning steps in CoT prompting effectively unlock capabilities for complex, multi-step problems? Our results demonstrate that, through the autoregressive process, LLMs are capable of exactly inferring the transition probabilities between tokens across distinct tasks using provided prompts. We show that ICL enhances performance by reducing prompt ambiguity and facilitating posterior concentration on the intended task. Furthermore, we find that CoT prompting activates the model's capacity for task decomposition, breaking complex problems into a sequence of simpler sub-tasks that the model has mastered during the pretraining phase. By comparing their individual error bounds, we provide novel theoretical insights into the statistical superiority of advanced prompt engineering techniques.

cs.CL↗

arXiv2026

Complexity of normalized stochastic first-order methods with momentum under heavy-tailed noise

In this paper, we propose practical normalized stochastic first-order methods with Polyak momentum, multi-extrapolated momentum, and recursive momentum for solving unconstrained optimization problems. These methods employ dynamically updated algorithmic parameters and do not require explicit knowledge of problem-dependent quantities such as the Lipschitz constant or noise bound. We establish first-order oracle complexity results for finding approximate stochastic stationary points under heavy-tailed noise and weakly average smoothness conditions -- both of which are weaker than the commonly used bounded variance and mean-squared smoothness assumptions. Our complexity bounds either improve upon or match the best-known results in the literature. Numerical experiments are presented to demonstrate the practical effectiveness of the proposed methods.

math.OC↗

arXiv2026

The global well-posedness for master equations of mean field games of controls

In this manuscript, we establish the global well-posedness for master equations of mean field games of controls, where the interaction is through the joint law of the state and control. Our results are proved under two different conditions: the Lasry-Lions monotonicity and the displacement $λ$-monotonicity, both considered in their integral forms. We provide a detailed analysis of both the differential and integral versions of these monotonicity conditions for the corresponding nonseparable Hamiltonian and examine their relation. The proof of global well-posedness relies on the propagation of these monotonicity conditions in their integral forms and a priori uniform Lipschitz continuity of the solution with respect to the measure variable.

math.PR↗

arXiv2025

dHPR: A Distributed Halpern Peaceman--Rachford Method for Non-smooth Distributed Optimization Problems

This paper introduces the distributed Halpern Peaceman--Rachford (dHPR) method, an efficient algorithm for solving distributed convex composite optimization problems with non-smooth objectives, which achieves a non-ergodic $O(1/k)$ iteration complexity regarding Karush--Kuhn--Tucker residual. By leveraging the symmetric Gauss--Seidel decomposition, the dHPR effectively decouples the linear operators in the objective functions and consensus constraints while maintaining parallelizability and avoiding additional large proximal terms, leading to a decentralized implementation with provably fast convergence. The superior performance of dHPR is demonstrated through comprehensive numerical experiments on distributed LASSO, group LASSO, and $L_1$-regularized logistic regression problems.

math.OC↗

arXiv2025

Progressive Bound Strengthening via Doubly Nonnegative Cutting Planes for Nonconvex Quadratic Programs

We introduce a cutting-plane framework for nonconvex quadratic programs (QPs) that progressively tightens convex relaxations. Our approach leverages the doubly nonnegative (DNN) relaxation to compute strong lower bounds and generate separating cuts, which are iteratively added to improve the relaxation. We establish that, at any Karush-Kuhn-Tucker (KKT) point satisfying a second-order sufficient condition, a valid cut can be obtained by solving a linear semidefinite program (SDP), and we devise a finite-termination local search procedure to identify such points. Extensive computational experiments on both benchmark and synthetic instances demonstrate that our approach yields tighter bounds and consistently outperforms leading commercial and academic solvers in terms of efficiency, robustness, and scalability. Notably, on a standard desktop, our algorithm reduces the relative optimality gap to 0.01% on 138 out of 140 instances of dimension 100 within one hour, without resorting to branch-and-bound.

math.OC↗

arXiv2025

On Error Bounds for Rank-Constrained Affine Matrix Sets

Rank-constrained matrix problems appear frequently across science and engineering. The convergence analysis of iterative algorithms developed for these problems often hinges on local error bounds, which correlate the distance to the feasible set with a measure of how much the constraints are violated. Foundational results in semi-algebraic geometry guarantee that such bounds exist, yet the associated exponents are generally not explicitly determined. This paper establishes a local Hölderian error bound with an explicit exponent for the canonical rank-constrained affine feasibility set. This paper proves that, on any compact set, the distance to the feasible set is bounded by a power of a natural residual function capturing violations in both the rank and affine constraints. The exponent in this bound is given explicitly in terms of the problem's dimensions. This provides a fundamental quantitative result on the geometry of the solution set, paving the way for the convergence analysis of a broad class of numerical methods.

math.OC↗

arXiv2025

On the Relationships among GPU-Accelerated First-Order Methods for Solving Linear Programming

This paper aims to understand the relationships among recently developed GPU-accelerated first-order methods (FOMs) for linear programming (LP), with particular emphasis on HPR-LP -- a Halpern Peaceman--Rachford (HPR) method for LP. Our findings can be summarized as follows: (i) the base algorithm of cuPDLPx, a recently released GPU solver, is a special case of the base algorithm of HPR-LP, thereby showing that cuPDLPx is another concrete implementation instance of HPR-LP; (ii) once the active sets have been identified, HPR-LP and EPR-LP -- an ergodic PR method for LP -- become equivalent under the same initialization; and (iii) extensive numerical experiments on benchmark datasets demonstrate that HPR-LP achieves the best overall performance among current GPU-accelerated LP solvers. These findings provide a strong motivation for using the HPR method as a baseline to further develop GPU-accelerated LP solvers and beyond.

math.OC↗

arXiv2025

Robust Gradient Descent Estimation for Tensor Models under Heavy-Tailed Distributions

Low-rank tensor models are widely used in statistics. However, most existing methods rely heavily on the assumption that data follows a sub-Gaussian distribution. To address the challenges associated with heavy-tailed distributions encountered in real-world applications, we propose a novel robust estimation procedure based on truncated gradient descent for general low-rank tensor models. We establish the computational convergence of the proposed method and derive optimal statistical rates under heavy-tailed distributional settings of both covariates and noise for various low-rank models. Notably, the statistical error rates are governed by a local moment condition, which captures the distributional properties of tensor variables projected onto certain low-dimensional local regions. Furthermore, we present numerical results to demonstrate the effectiveness of our method.

stat.ME↗

arXiv2025

A Survey on Large Language Model-based Agents for Statistics and Data Science

In recent years, data science agents powered by Large Language Models (LLMs), known as "data agents," have shown significant potential to transform the traditional data analysis paradigm. This survey provides an overview of the evolution, capabilities, and applications of LLM-based data agents, highlighting their role in simplifying complex data tasks and lowering the entry barrier for users without related expertise. We explore current trends in the design of LLM-based frameworks, detailing essential features such as planning, reasoning, reflection, multi-agent collaboration, user interface, knowledge integration, and system design, which enable agents to address data-centric problems with minimal human intervention. Furthermore, we analyze several case studies to demonstrate the practical applications of various data agents in real-world scenarios. Finally, we identify key challenges and propose future research directions to advance the development of data agents into intelligent statistical analysis software.

cs.AI↗

arXiv2025

On the $p$-order Semismoothness of the Metric Projection onto Slices of the Positive Semidefinite Cone

The metric projection onto the positive semidefinite (PSD) cone is strongly semismooth, a property that guarantees local quadratic convergence for many powerful algorithms in semidefinite programming. In this paper, we investigate whether this essential property holds for the metric projection onto an affine slice of the PSD cone, which is the operator implicitly used by many algorithms that handle linear constraints directly. Although this property is known to be preserved for the second-order cone, we conclusively demonstrate that this is not the case for the PSD cone. Specifically, we provide a constructive example that for any $p > 0$, there exists an affine slice of a PSD cone for which the metric projection operator fails to be $p$-order semismooth. This finding establishes a fundamental difference between the geometry of the second-order cone and the PSD cone and necessitates new approaches for both analysis and algorithm design for linear semidefinite programming problems.

math.OC↗