arXiv subjects

Lixin Shen

Publications and source records attributed to Lixin Shen.

At least 19 recordsLinked to original sources

arXiv2026

Exact Series and Error-Controlled Evaluation of the Nonconvex $\ell_p$ Proximity Operator

We study the scalar proximity operator of the nonconvex power penalty with exponent between zero and one. Lagrange--Bürmann inversion gives an exact series for the larger stationary root, and the series converges uniformly throughout the active domain of the proximity operator. For rational exponents, an explicit residue-class decomposition reorganizes the coefficients into finite sums of generalized hypergeometric functions, recovering the familiar special cases and producing an additional formula for $p=\tfrac13$. A Mellin--Barnes formula yields an equivalent residue-corrected representation and analytic truncation bounds. We distinguish the proximal activation threshold from the lower saddle-node threshold and prove that the active proximal branch is uniformly well-conditioned; the practical difficulty near activation is slow series convergence when the exponent approaches one, rather than singular root sensitivity. We also formulate an inexact proximal-gradient step using a computable optimality residual and a descent test. With summable descent errors and vanishing residuals, the objective values converge, the steps vanish, and every cluster point is critical. Tests over a broad parameter grid and in sparse-recovery proximal-gradient runs illustrate both evaluator accuracy and the implementability of the acceptance certificates.

math.OC↗

arXiv2026

Proximity Operator of the $\ell_1$ over $\ell_2$ Function

We study the complete, possibly set-valued proximity operator of the nonconvex, scale-invariant ratio $h(\vx)=\|\vx\|_{1}/\|\vx\|_{2}$. A polar decomposition of the nonzero branch, followed by sign and permutation reductions, transforms the proximal problem into a smooth rank-one quadratic problem over the nonnegative orthant of the unit sphere. After the magnitudes of the data are sorted, every canonical proximal direction belongs to a finite candidate set indexed by its support size $k\in\{1,\ldots,n\}$. For each $k\geq 2$, a candidate exists if and only if an explicit quartic equation has a simple root satisfying both a positive-orthant condition and a second-order derivative condition. This characterization eliminates the need to guess the unknown sparsity. We further derive a necessary-and-sufficient $O(1)$ prefix test that enforces positivity and prunes support sizes for which no candidate can exist. Using prefix sums, the optimized method performs an $O(n)$ scan after an $O(n\log n)$ sorting step; the construction of multiple tied outputs is necessarily output-sensitive. Numerical experiments illustrate the finite-candidate selection, identify inputs on which a sparsity-guessing baseline is suboptimal, quantify the benefit of pruning, and demonstrate the numerical accuracy of the secular-equation solver. The observed runtimes exhibit near-linear growth over the tested range, consistent with the predicted $O(n)$ post-sorting scan and $O(n\log n)$ total complexity.

math.OC↗

arXiv2026

MG-SpaIR: Multi-grade Sparse-guided Implicit Representation for Training-Data-Free Image Restoration

MG-SpaIR is a training-data-free framework for restoring a clean image from a single observation corrupted by a mixture of blur, downsampling, noise, and missing pixels. Building on implicit neural representations (INRs), we introduce a multi-grade coarse-to-fine residual hierarchy that progressively refines the reconstruction across resolution grades, improving representational fidelity and mitigating spectral limitations. To stabilize reconstruction optimization and suppress INR-induced artifacts, we further propose an explicit sparse proximal regularization (e.g., $\ell_0$-type) applied directly in the high-resolution image domain, which discourages spurious high-frequency patterns while preserving sharp structures. The resulting optimization is solved efficiently via a multi-grade proximal alternating scheme, and we establish convergence guarantees for the associated updates under standard regularity conditions. Experiments on mixed-degradation benchmarks demonstrate that MG-SpaIR consistently outperforms strong training-data-free baselines such as Deep Image Prior, providing a stable, interpretable, and data-efficient alternative to conventional learning-based restoration methods.

cs.CV↗

arXiv2026

Oracle Supervision Transfers for Hyperparameter Prediction in Model-Based Image Denoising

Hyperparameter prediction is a critical practical bottleneck for model-based image denoisers, ranging from classical TV/TGV variational solvers to modern diffusion-based models such as DiffPIR. While existing learned predictors can achieve near-oracle performance, this approach scales poorly: each new configuration conventionally requires its own oracle-labeled training set, and each label requires a hierarchical grid search evaluated against clean ground truth. We therefore ask whether oracle supervision collected on source configurations can transfer to target configurations with few or no target oracle labels. We propose HyperDn, a single configuration-conditioned predictor that pools oracle supervision across source configurations and predicts heterogeneous hyperparameters for new denoiser--noise configurations. In a cross-paradigm experiment, HyperDn transfers from relatively cheap TV/TGV variational sources to more expensive diffusion-based DiffPIR. With only $2$ target oracle labels, it reaches $30.23$\,dB, within $0.90$\,dB of the oracle, and outperforms the $64$-label per-configuration predictor trained from scratch, using $1/32$ as many target labels as that baseline point. Without any target oracle labels, HyperDn also reaches near-oracle PSNR on two unseen mixtures of seen noise types and on transfer from relatively cheap $96\times 96$ source images to $512\times 768$ targets. Together, these results show that expensive oracle supervision for hyperparameter prediction can be transferred from source to new target configurations, reducing the need to rebuild oracle labels for each new denoising configuration.

cs.CV↗

arXiv2025

Signal and Image Recovery with Scale and Signed Permutation Invariant Sparsity-Promoting Functions

Sparse signal recovery has been a cornerstone of advancements in data processing and imaging. Recently, the squared ratio of $\ell_1$ to $\ell_2$ norms, $(\ell_1/\ell_2)^2$, has been introduced as a sparsity-prompting function, showing superior performance compared to traditional $\ell_1$ minimization, particularly in challenging scenarios with high coherence and dynamic range. This paper explores the integration of the proximity operator of $(\ell_1/\ell_2)^2$ and $\ell_1/\ell_2$ into efficient optimization frameworks, including the Accelerated Proximal Gradient (APG) and Alternating Direction Method of Multipliers (ADMM). We rigorously analyze the convergence properties of these algorithms and demonstrate their effectiveness in compressed sensing and image restoration applications. Numerical experiments highlight the advantages of our proposed methods in terms of recovery accuracy and computational efficiency, particularly under noise and high-coherence conditions.

math.OC↗

arXiv2024

Sparse Recovery: The Square of $\ell_1/\ell_2$ Norms

This paper introduces a nonconvex approach for sparse signal recovery, proposing a novel model termed the $τ_2$-model, which utilizes the squared $\ell_1/\ell_2$ norms for this purpose. Our model offers an advancement over the $\ell_0$ norm, which is often computationally intractable and less effective in practical scenarios. Grounded in the concept of effective sparsity, our approach robustly measures the number of significant coordinates in a signal, making it a powerful alternative for sparse signal estimation. The $τ_2$-model is particularly advantageous due to its computational efficiency and practical applicability. We detail two accompanying algorithms based on Dinkelbach's procedure and a difference of convex functions strategy. The first algorithm views the model as a linear-constrained quadratic programming problem in noiseless scenarios and as a quadratic-constrained quadratic programming problem in noisy scenarios. The second algorithm, capable of handling both noiseless and noisy cases, is based on the alternating direction linearized proximal method of multipliers. We also explore the model's properties, including the existence of solutions under certain conditions, and discuss the convergence properties of the algorithms. Numerical experiments with various sensing matrices validate the effectiveness of our proposed model.

math.OC↗

arXiv2024

Sparse Deep Learning Models with the $\ell_1$ Regularization

Sparse neural networks are highly desirable in deep learning in reducing its complexity. The goal of this paper is to study how choices of regularization parameters influence the sparsity level of learned neural networks. We first derive the $\ell_1$-norm sparsity-promoting deep learning models including single and multiple regularization parameters models, from a statistical viewpoint. We then characterize the sparsity level of a regularized neural network in terms of the choice of the regularization parameters. Based on the characterizations, we develop iterative algorithms for selecting regularization parameters so that the weight parameters of the resulting deep neural network enjoy prescribed sparsity levels. Numerical experiments are presented to demonstrate the effectiveness of the proposed algorithms in choosing desirable regularization parameters and obtaining corresponding neural networks having both of predetermined sparsity levels and satisfactory approximation accuracy.

cs.LG↗

arXiv2024

Large-Scale Non-convex Stochastic Constrained Distributionally Robust Optimization

Distributionally robust optimization (DRO) is a powerful framework for training robust models against data distribution shifts. This paper focuses on constrained DRO, which has an explicit characterization of the robustness level. Existing studies on constrained DRO mostly focus on convex loss function, and exclude the practical and challenging case with non-convex loss function, e.g., neural network. This paper develops a stochastic algorithm and its performance analysis for non-convex constrained DRO. The computational complexity of our stochastic algorithm at each iteration is independent of the overall dataset size, and thus is suitable for large-scale applications. We focus on the general Cressie-Read family divergence defined uncertainty set which includes $χ^2$-divergences as a special case. We prove that our algorithm finds an $ε$-stationary point with a computational complexity of $\mathcal O(ε^{-3k_*-5})$, where $k_*$ is the parameter of the Cressie-Read divergence. The numerical results indicate that our method outperforms existing methods.} Our method also applies to the smoothed conditional value at risk (CVaR) DRO.

stat.ML↗

arXiv2024

Computing Proximity Operators of Scale and Signed Permutation Invariant Functions

This paper investigates the computation of proximity operators for scale and signed permutation invariant functions. A scale-invariant function remains unchanged under uniform scaling, while a signed permutation invariant function retains its structure despite permutations and sign changes applied to its input variables. Noteworthy examples include the $\ell_0$ function and the ratios of $\ell_1/\ell_2$ and its square, with their proximity operators being particularly crucial in sparse signal recovery. We delve into the properties of scale and signed permutation invariant functions, delineating the computation of their proximity operators into three sequential steps: the $\mathbf{w}$-step, $r$-step, and $d$-step. These steps collectively form a procedure termed as WRD, with the $\mathbf{w}$-step being of utmost importance and requiring careful treatment. Leveraging this procedure, we present a method for explicitly computing the proximity operator of $(\ell_1/\ell_2)^2$ and introduce an efficient algorithm for the proximity operator of $\ell_1/\ell_2$.

math.OC↗

arXiv2024

Hyperparameter Estimation for Sparse Bayesian Learning Models

Sparse Bayesian Learning (SBL) models are extensively used in signal processing and machine learning for promoting sparsity through hierarchical priors. The hyperparameters in SBL models are crucial for the model's performance, but they are often difficult to estimate due to the non-convexity and the high-dimensionality of the associated objective function. This paper presents a comprehensive framework for hyperparameter estimation in SBL models, encompassing well-known algorithms such as the expectation-maximization (EM), MacKay, and convex bounding (CB) algorithms. These algorithms are cohesively interpreted within an alternating minimization and linearization (AML) paradigm, distinguished by their unique linearized surrogate functions. Additionally, a novel algorithm within the AML framework is introduced, showing enhanced efficiency, especially under low signal noise ratios. This is further improved by a new alternating minimization and quadratic approximation (AMQ) paradigm, which includes a proximal regularization term. The paper substantiates these advancements with thorough convergence analysis and numerical experiments, demonstrating the algorithm's effectiveness in various noise conditions and signal-to-noise ratios.

cs.LG↗

arXiv2023

A Successive Two-stage Method for Sparse Generalized Eigenvalue Problems

The Sparse Generalized Eigenvalue Problem (sGEP), a pervasive challenge in statistical learning methods including sparse principal component analysis, sparse Fisher's discriminant analysis, and sparse canonical correlation analysis, presents significant computational complexity due to its NP-hardness. The primary aim of sGEP is to derive a sparse vector approximation of the largest generalized eigenvector, effectively posing this as a sparse optimization problem. Conventional algorithms for sGEP, however, often succumb to local optima and exhibit significant dependency on initial points. This predicament necessitates a more refined approach to avoid local optima and achieve an improved solution in terms of sGEP's objective value, which we address in this paper through a novel successive two-stage method. The first stage of this method incorporates an algorithm for sGEP capable of yielding a stationary point from any initial point. The subsequent stage refines this stationary point by adjusting its support, resulting in a point with an enhanced objective value relative to the original stationary point. This support adjustment is achieved through a novel procedure we have named support alteration. The final point derived from the second stage then serves as the initial point for the algorithm in the first stage, creating a cyclical process that continues until a predetermined stopping criterion is satisfied. We also provide a comprehensive convergence analysis of this process. Through extensive experimentation under various settings, our method has demonstrated significant improvements in the objective value of sGEP compared to existing methodologies, underscoring its potential as a valuable tool in statistical learning and optimization.

math.OC↗

arXiv2022

A Super-Resolution Framework for Tensor Decomposition

This work considers a super-resolution framework for overcomplete tensor decomposition. Specifically, we view tensor decomposition as a super-resolution problem of recovering a sum of Dirac measures on the sphere and solve it by minimizing a continuous analog of the $\ell_1$ norm on the space of measures. The optimal value of this optimization defines the tensor nuclear norm. Similar to the separation condition in the super-resolution problem, by explicitly constructing a dual certificate, we develop incoherence conditions of the tensor factors so that they form the unique optimal solution of the continuous analog of $\ell_1$ norm minimization. Remarkably, the derived incoherence conditions are satisfied with high probability by random tensor factors uniformly distributed on the sphere, implying global identifiability of random tensor factors.

cs.IT↗

arXiv2021

On Distributed Online Convex Optimization with Sublinear Dynamic Regret and Fit

In this work, we consider a distributed online convex optimization problem, with time-varying (potentially adversarial) constraints. A set of nodes, jointly aim to minimize a global objective function, which is the sum of local convex functions. The objective and constraint functions are revealed locally to the nodes, at each time, after taking an action. Naturally, the constraints cannot be instantaneously satisfied. Therefore, we reformulate the problem to satisfy these constraints in the long term. To this end, we propose a distributed primal-dual mirror descent based approach, in which the primal and dual updates are carried out locally at all the nodes. This is followed by sharing and mixing of the primal variables by the local nodes via communication with the immediate neighbors. To quantify the performance of the proposed algorithm, we utilize the challenging, but more realistic metrics of dynamic regret and fit. Dynamic regret measures the cumulative loss incurred by the algorithm, compared to the best dynamic strategy. On the other hand, fit measures the long term cumulative constraint violations. Without assuming the restrictive Slater's conditions, we show that the proposed algorithm achieves sublinear regret and fit under mild, commonly used assumptions.

math.OC↗

arXiv2021

The Proximity Operator of the Log-Sum Penalty

The log-sum penalty is often adopted as a replacement for the $\ell_0$ pseudo-norm in compressive sensing and low-rank optimization. The hard-thresholding operator, i.e., the proximity operator of the $\ell_0$ penalty, plays an essential role in applications; similarly, we require an efficient method for evaluating the proximity operator of the log-sum penalty. Due to the nonconvexity of this function, its proximity operator is commonly computed through the iteratively reweighted $\ell_1$ method, which replaces the log-sum term with its first-order approximation. This paper reports that the proximity operator of the log-sum penalty actually has an explicit expression. With it, we show that the iteratively reweighted $\ell_1$ solution disagrees with the true proximity operator of the log-sum penalty in certain regions. As a by-product, the iteratively reweighted $\ell_1$ solution is precisely characterized in terms of the chosen initialization. We also give the explicit form of the proximity operator for the composition of the log-sum penalty with the singular value function, as seen in low-rank applications. These results should be useful in the development of efficient and accurate algorithms for optimization problems involving the log-sum penalty.

math.OC↗

arXiv2021

Regularization with Multilevel Non-stationary Tight Framelets for Image Restoration

Variational regularization models are one of the popular and efficient approaches for image restoration. The regularization functional in the model carries prior knowledge about the image to be restored. The prior knowledge, in particular for natural images, are the first-order (i.e. variance in luminance) and second-order (i.e. contrast and texture) information. In this paper, we propose a model for image restoration, using a multilevel non-stationary tight framelet system that can capture the image's first-order and second-order information. We develop an algorithm to solve the proposed model and the numerical experiments show that the model is effective and efficient as compared to other higher-order models.

math.OC↗

arXiv2020

Multiplicative Noise Removal: Nonlocal Low-Rank Model and Its Proximal Alternating Reweighted Minimization Algorithm

The goal of this paper is to develop a novel numerical method for efficient multiplicative noise removal. The nonlocal self-similarity of natural images implies that the matrices formed by their nonlocal similar patches are low-rank. By exploiting this low-rank prior with application to multiplicative noise removal, we propose a nonlocal low-rank model for this task and develop a proximal alternating reweighted minimization (PARM) algorithm to solve the optimization problem resulting from the model. Specifically, we utilize a generalized nonconvex surrogate of the rank function to regularize the patch matrices and develop a new nonlocal low-rank model, which is a nonconvex nonsmooth optimization problem having a patchwise data fidelity and a generalized nonlocal low-rank regularization term. To solve this optimization problem, we propose the PARM algorithm, which has a proximal alternating scheme with a reweighted approximation of its subproblem. A theoretical analysis of the proposed PARM algorithm is conducted to guarantee its global convergence to a critical point. Numerical experiments demonstrate that the proposed method for multiplicative noise removal significantly outperforms existing methods such as the benchmark SAR-BM3D method in terms of the visual quality of the denoised images, and the PSNR (the peak-signal-to-noise ratio) and SSIM (the structural similarity index measure) values.

math.OC↗

arXiv2019

Algorithmic Versatility of SPF-regularization Methods

Sparsity promoting functions (SPFs) are commonly used in optimization problems to find solutions which are assumed or desired to be sparse in some basis. For example, the l1-regularized variation model and the Rudin-Osher-Fatemi total variation (ROF-TV) model are some of the most well-known variational models for signal and image denoising, respectively. However, recent work demonstrates that convexity is not always desirable in sparsity promoting functions. In this paper, we replace convex SPFs with their induced nonconvex SPFs and develop algorithms for the resulting model by exploring the intrinsic structures of the nonconvex SPFs. We also present simulations illustrating the performance of the SPF and the developed algorithms in image denoising.

math.OC↗

arXiv2019

Principal Component Projection with Low-Degree Polynomials

In this paper, we consider approximations of principal component projection (PCP) without explicitly computing principal components. This problem has been studied in several recent works. The main feature of existing approaches is viewing the PCP matrix as a matrix function. This underlying function is the composition of a step function with a rational function. To find an approximate PCP, the step function is approximated by a polynomial while the rational function is evaluated by a fast ridge regression solver. In this work, we further improve this process by replacing the rational function with carefully constructed polynomials of low degree. We characterize the properties of polynomials that are suitable for approximating PCP, and establish an optimization problem to select the optimal one from those polynomials. We show theoretically and confirm numerically that the resulting approximate PCP approach with optimal polynomials is indeed effective for approximations of principal component projection.

math.NA↗