arXiv subjects

Defeng Sun

Publications and source records attributed to Defeng Sun.

At least 37 records · Page 2Linked to original sources

arXiv2025

On Error Bounds for Rank-Constrained Affine Matrix Sets

Rank-constrained matrix problems appear frequently across science and engineering. The convergence analysis of iterative algorithms developed for these problems often hinges on local error bounds, which correlate the distance to the feasible set with a measure of how much the constraints are violated. Foundational results in semi-algebraic geometry guarantee that such bounds exist, yet the associated exponents are generally not explicitly determined. This paper establishes a local H\"olderian error bound with an explicit exponent for the canonical rank-constrained affine feasibility set. This paper proves that, on any compact set, the distance to the feasible set is bounded by a power of a natural residual function capturing violations in both the rank and affine constraints. The exponent in this bound is given explicitly in terms of the problem's dimensions. This provides a fundamental quantitative result on the geometry of the solution set, paving the way for the convergence analysis of a broad class of numerical methods.

math.OC↗

arXiv2025

On the Relationships among GPU-Accelerated First-Order Methods for Solving Linear Programming

This paper aims to understand the relationships among recently developed GPU-accelerated first-order methods (FOMs) for linear programming (LP), with particular emphasis on HPR-LP -- a Halpern Peaceman--Rachford (HPR) method for LP. Our findings can be summarized as follows: (i) the base algorithm of cuPDLPx, a recently released GPU solver, is a special case of the base algorithm of HPR-LP, thereby showing that cuPDLPx is another concrete implementation instance of HPR-LP; (ii) once the active sets have been identified, HPR-LP and EPR-LP -- an ergodic PR method for LP -- become equivalent under the same initialization; and (iii) extensive numerical experiments on benchmark datasets demonstrate that HPR-LP achieves the best overall performance among current GPU-accelerated LP solvers. These findings provide a strong motivation for using the HPR method as a baseline to further develop GPU-accelerated LP solvers and beyond.

math.OC↗

arXiv2025

The Aubin Property for Generalized Equations over $C^2$-cone Reducible Sets

This paper establishes the equivalence between the Aubin property and strong regularity for generalized equations over $C^2$-cone reducible sets. This result resolves a long-standing question in variational analysis and extends the classical equivalence theorem for polyhedral sets to a broad class of non-polyhedral sets. Our proof strategy departs from traditional variational techniques, integrating insights from convex geometry with powerful tools from algebraic topology. At the heart of our analysis is a novel index theorem for a class of functions involving metric projections onto arbitrary closed convex sets. Its proof exploits the geometry of normal cones and topological degree theory. We then use a lift of the diffeomorphism provided by the $C^2$-cone reduction to reduce the original generalized equations to the setting of the index theorem. The homological inverse mapping theorem then yields the desired strong regularity. We show that the $C^2$-cone reducibility assumption cannot be removed in general by constructing a three-dimensional semialgebraic counterexample. This result unifies and extends existing stability results for conventional nonlinear programming, nonlinear second-order cone programming, and nonlinear semidefinite programming under a single general framework, and removes the local optimality assumption imposed in the existing non-polyhedral results.

math.OC↗

arXiv2025

On the $p$-order Semismoothness of the Metric Projection onto Slices of the Positive Semidefinite Cone

The metric projection onto the positive semidefinite (PSD) cone is strongly semismooth, a property that guarantees local quadratic convergence for many powerful algorithms in semidefinite programming. In this paper, we investigate whether this essential property holds for the metric projection onto an affine slice of the PSD cone, which is the operator implicitly used by many algorithms that handle linear constraints directly. Although this property is known to be preserved for the second-order cone, we conclusively demonstrate that this is not the case for the PSD cone. Specifically, we provide a constructive example that for any $p > 0$, there exists an affine slice of a PSD cone for which the metric projection operator fails to be $p$-order semismooth. This finding establishes a fundamental difference between the geometry of the second-order cone and the PSD cone and necessitates new approaches for both analysis and algorithm design for linear semidefinite programming problems.

math.OC↗

arXiv2025

HPR-QP: A dual Halpern Peaceman-Rachford method for solving large-scale convex composite quadratic programming

In this paper, we introduce HPR-QP, a dual Halpern Peaceman-Rachford (HPR) method designed for solving large-scale convex composite quadratic programming. One distinctive feature of HPR-QP is that, instead of working with the primal formulations, it builds on the novel restricted Wolfe dual introduced in recent years. It also leverages the symmetric Gauss-Seidel technique to simplify subproblem updates without introducing auxiliary slack variables that typically lead to slow convergence. By restricting updates to the range space of the Hessian of the quadratic objective function, HPR-QP employs proximal operators of smaller spectral norms to speed up the convergence. Shadow sequences are elaborately constructed to deal with the range space constraints. Additionally, HPR-QP incorporates adaptive restart and penalty parameter update strategies, derived from the HPR method's $O(1/k)$ convergence in terms of the Karush-Kuhn-Tucker residual, to further enhance its performance and robustness. Extensive numerical experiments on benchmark data sets using a GPU demonstrate that our Julia implementation of HPR-QP significantly outperforms state-of-the-art solvers in both speed and scalability.

math.OC↗

arXiv2025

Complexity of normalized stochastic first-order methods with momentum under heavy-tailed noise

In this paper, we propose practical normalized stochastic first-order methods with Polyak momentum, multi-extrapolated momentum, and recursive momentum for solving unconstrained optimization problems. These methods employ dynamically updated algorithmic parameters and do not require explicit knowledge of problem-dependent quantities such as the Lipschitz constant or noise bound. We establish first-order oracle complexity results for finding approximate stochastic stationary points under heavy-tailed noise and weakly average smoothness conditions -- both of which are weaker than the commonly used bounded variance and mean-squared smoothness assumptions. Our complexity bounds either improve upon or match the best-known results in the literature. Numerical experiments are presented to demonstrate the practical effectiveness of the proposed methods.

math.OC↗

arXiv2025

Approximation Bounds for Transformer Networks with Application to Regression

We explore the approximation capabilities of Transformer networks for H\"older and Sobolev functions, and apply these results to address nonparametric regression estimation with dependent observations. First, we establish novel upper bounds for standard Transformer networks approximating sequence-to-sequence mappings whose component functions are H\"older continuous with smoothness index $\gamma \in (0,1]$. To achieve an approximation error $\varepsilon$ under the $L^p$-norm for $p \in [1, \infty]$, it suffices to use a fixed-depth Transformer network whose total number of parameters scales as $\varepsilon^{-d_x n / \gamma}$. This result not only extends existing findings to include the case $p = \infty$, but also matches the best known upper bounds on number of parameters previously obtained for fixed-depth FNNs and RNNs. Similar bounds are also derived for Sobolev functions. Second, we derive explicit convergence rates for the nonparametric regression problem under various $\beta$-mixing data assumptions, which allow the dependence between observations to weaken over time. Our bounds on the sample complexity impose no constraints on weight magnitudes. Lastly, we propose a novel proof strategy to establish approximation bounds, inspired by the Kolmogorov-Arnold representation theorem. We show that if the self-attention layer in a Transformer can perform column averaging, the network can approximate sequence-to-sequence H\"older functions, offering new insights into the interpretability of self-attention mechanisms.

stat.ML↗

arXiv2025

Bringing Generative Learning to Representation Learning: Self-Supervised Transfer Learning as Distribution Matching

Most self-supervised learning objectives defend against collapse but leave the target representation law unspecified. We formulate representation learning as Distribution Matching (DM), learning an augmentation-invariant encoder whose induced law matches an explicit geometric reference. The reference law specifies what the learned representation distribution should look like, whereas a separately chosen discrepancy determines how deviations from this target are measured; here we use Mallows distance. The DM framework reveals a directional inverse: generative learning maps a tractable reference to data, whereas representation learning maps data to a designed reference law. We connect the population objective to class-centre separation and classification error and prove a non-asymptotic neural-sieve guarantee. Simulations and image benchmarks show manifold rectification, fine-grained structure and transfer across label spaces.

stat.ML↗

arXiv2025

Peaceman-Rachford Splitting Method Converges Ergodically for Solving Convex Optimization Problems

In this paper, we prove that the ergodic sequence generated by the Peaceman-Rachford (PR) splitting method with semi-proximal terms converges for convex optimization problems (COPs). Numerical experiments on the linear programming benchmark dataset further demonstrate that, with a restart strategy, the ergodic sequence of the PR splitting method with semi-proximal terms consistently outperforms both the point-wise and ergodic sequences of the Douglas-Rachford (DR) splitting method. These findings indicate that the restarted ergodic PR splitting method is a more effective choice for tackling large-scale COPs compared to its DR counterparts.

math.OC↗

arXiv2024

A Survey on Large Language Model-based Agents for Statistics and Data Science

In recent years, data science agents powered by Large Language Models (LLMs), known as "data agents," have shown significant potential to transform the traditional data analysis paradigm. This survey provides an overview of the evolution, capabilities, and applications of LLM-based data agents, highlighting their role in simplifying complex data tasks and lowering the entry barrier for users without related expertise. We explore current trends in the design of LLM-based frameworks, detailing essential features such as planning, reasoning, reflection, multi-agent collaboration, user interface, knowledge integration, and system design, which enable agents to address data-centric problems with minimal human intervention. Furthermore, we analyze several case studies to demonstrate the practical applications of various data agents in real-world scenarios. Finally, we identify key challenges and propose future research directions to advance the development of data agents into intelligent statistical analysis software.

cs.AI↗

arXiv2024

Support matrix machine: exploring sample sparsity, low rank, and adaptive sieving in high-performance computing

Support matrix machine (SMM) is a successful supervised classification model for matrix-type samples. Unlike support vector machines, it employs low-rank regularization on the regression matrix to effectively capture the intrinsic structure embedded in each input matrix. When solving a large-scale SMM, a major challenge arises from the potential increase in sample size, leading to substantial computational and storage burdens. To address these issues, we design a semismooth Newton-CG (SNCG) based augmented Lagrangian method (ALM) for solving the SMM. The ALM exhibits an asymptotic R-superlinear convergence if a strict complementarity condition is satisfied. The SNCG method is employed to solve the ALM subproblems, achieving at least a superlinear convergence rate under the nonemptiness of an index set. Furthermore, the sparsity of samples and the low-rank nature of solutions enable us to reduce the computational cost and storage demands for the Newton linear systems. Additionally, we develop an adaptive sieving strategy that generates a solution path for the SMM by exploiting sample sparsity. The finite convergence of this strategy is also demonstrated. Numerical experiments on both large-scale real and synthetic datasets validate the effectiveness of the proposed methods.

math.OC↗

arXiv2024

Robust Gradient Descent Estimation for Tensor Models under Heavy-Tailed Distributions

Low-rank tensor models are widely used in statistics. However, most existing methods rely heavily on the assumption that data follows a sub-Gaussian distribution. To address the challenges associated with heavy-tailed distributions encountered in real-world applications, we propose a novel robust estimation procedure based on truncated gradient descent for general low-rank tensor models. We establish the computational convergence of the proposed method and derive optimal statistical rates under heavy-tailed distributional settings of both covariates and noise for various low-rank models. Notably, the statistical error rates are governed by a local moment condition, which captures the distributional properties of tensor variables projected onto certain low-dimensional local regions. Furthermore, we present numerical results to demonstrate the effectiveness of our method.

stat.ME↗

arXiv2024

Lipschitz stability of least-squares problems regularized by functions with $\mathcal{C}^2$-cone reducible conjugates

In this paper, we study Lipschitz continuity of the solution mappings of regularized least-squares problems for which the convex regularizers have (Fenchel) conjugates that are $\mathcal{C}^2$-cone reducible. Our approach, by using Robinson's strong regularity on the dual problem, allows us to obtain new characterizations of Lipschitz stability that rely solely on first-order information, thus bypassing the need to explore second-order information (curvature) of the regularizer. We show that these solution mappings are automatically Lipschitz continuous around the points in question whenever they are locally single-valued. We leverage our findings to obtain new characterizations of full stability and tilt stability for a broader class of convex additive-composite problems.

math.OC↗

arXiv2024

HPR-LP: An implementation of an HPR method for solving linear programming

In this paper, we introduce an HPR-LP solver, an implementation of a Halpern Peaceman-Rachford (HPR) method with semi-proximal terms for solving linear programming (LP). The HPR method enjoys the iteration complexity of $O(1/k)$ in terms of the Karush-Kuhn-Tucker residual and the objective error. Based on the complexity results, we design an adaptive strategy of restart and penalty parameter update to improve the efficiency and robustness of the HPR method. We conduct extensive numerical experiments on different LP benchmark datasets using NVIDIA A100-SXM4-80GB GPU in different stopping tolerances. Our solver's Julia version achieves a $\textbf{2.39x}$ to $\textbf{5.70x}$ speedup measured by SGM10 on benchmark datasets with presolve ($\textbf{2.03x}$ to $\textbf{4.06x}$ without presolve) over the award-winning solver PDLP with the tolerance of $10^{-8}$.

math.OC↗

arXiv2024

Characterizations of the Aubin Property of the Solution Mapping for Nonlinear Semidefinite Programming

In this paper, we study the Aubin property of the Karush-Kuhn-Tucker solution mapping for the nonlinear semidefinite programming (NLSDP) problem at a locally optimal solution. In the literature, it is known that the Aubin property implies the constraint nondegeneracy by Fusek [SIAM J. Optim. 23 (2013), pp. 1041-1061] and the second-order sufficient condition by Ding et al. [SIAM J. Optim. 27 (2017), pp. 67-90]. Based on the Mordukhovich criterion, here we further prove that the strong second-order sufficient condition is also necessary for the Aubin property to hold. Consequently, several equivalent conditions including the strong regularity are established for NLSDP's Aubin property. Together with the recent progress made by Chen et al. on the equivalence between the Aubin property and the strong regularity for nonlinear second-order cone programming [SIAM J. Optim., in press; arXiv:2406.13798v3 (2024)], this paper constitutes a significant step forward in characterizing the Aubin property for general non-polyhedral $C^2$-cone reducible constrained optimization problems.

math.OC↗

arXiv2024

HOT: An Efficient Halpern Accelerating Algorithm for Optimal Transport Problems

This paper proposes an efficient HOT algorithm for solving the optimal transport (OT) problems with finite supports. We particularly focus on an efficient implementation of the HOT algorithm for the case where the supports are in $\mathbb{R}^2$ with ground distances calculated by $L_2^2$-norm. Specifically, we design a Halpern accelerating algorithm to solve the equivalent reduced model of the discrete OT problem. Moreover, we derive a novel procedure to solve the involved linear systems in the HOT algorithm in linear time complexity. Consequently, we can obtain an $\varepsilon$-approximate solution to the optimal transport problem with $M$ supports in $O(M^{1.5}/\varepsilon)$ flops, which significantly improves the best-known computational complexity. We further propose an efficient procedure to recover an optimal transport plan for the original OT problem based on a solution to the reduced model, thereby overcoming the limitations of the reduced OT model in applications that require the transport plan. We implement the HOT algorithm in PyTorch and extensive numerical results show the superior performance of the HOT algorithm compared to existing state-of-the-art algorithms for solving the OT problems.

math.OC↗

arXiv2024

LAMBDA: A Large Model Based Data Agent

We introduce LArge Model Based Data Agent (LAMBDA), a novel open-source, code-free multi-agent data analysis system that leverages the power of large language models. LAMBDA is designed to address data analysis challenges in data-driven applications through innovatively designed data agents using natural language. At the core of LAMBDA are two key agent roles: the programmer and the inspector, which are engineered to work together seamlessly. Specifically, the programmer generates code based on the user's instructions and domain-specific knowledge, while the inspector debugs the code when necessary. To ensure robustness and handle adverse scenarios, LAMBDA features a user interface that allows direct user intervention. Moreover, LAMBDA can flexibly integrate external models and algorithms through our proposed Knowledge Integration Mechanism, catering to the needs of customized data analysis. LAMBDA has demonstrated strong performance on various data analysis tasks. It has the potential to enhance data analysis paradigms by seamlessly integrating human and artificial intelligence, making it more accessible, effective, and efficient for users from diverse backgrounds. The strong performance of LAMBDA in solving data analysis problems is demonstrated using real-world data examples. The code for LAMBDA is available at https://github.com/AMA-CMFAI/LAMBDA and videos of three case studies can be viewed at https://www.polyu.edu.hk/ama/cmfai/lambda.html.

cs.AI↗

arXiv2024

Two Typical Implementable Semismooth* Newton Methods for Generalized Equations are G-Semismooth Newton Methods

Semismooth* Newton methods have been proposed in recent years targeting multi-valued inclusion problems and have been successfully implemented to deal with several concrete generalized equations. In this paper, we show that two typical implementations of them that are available are exactly the applications of G-semismooth Newton methods for solving nonsmooth equations localized from these generalized equations. This new understanding expands the breadth of G-semismooth Newton methods in theory, results in a few interesting problems regarding the two categories of nonsmooth Newton methods, and more importantly, provides informative observations in facilitating the design and implementation of practical Newton-type algorithms for solving generalized equations.

math.OC↗