arXiv subjects

Zhaojun Bai

Publications and source records attributed to Zhaojun Bai.

At least 19 recordsLinked to original sources

arXiv2026

NEPv Approach for Optimization on Stiefel Manifold with the $(2,1)$-norm Regularization

Row-sparse projection provides a useful tool in machine learning (ML) when it comes to, for example, feature selection, aiming to choose most relevant features for various ML objectives. One way to seek a high quality row-sparse projection is to combine an ML objective, such as the ones for PCA, LDA, and OCCA, with the matrix $(2,1)$-norm regularization which is nonsmooth. Such combinations result in challenging optimization problems on the Stiefel manifold that need to be solved efficiently. In this paper, a unifying NEPv framework is established to efficiently deal with optimization on the Stiefel manifold with the $(2,1)$-norm regularization. The effect of the $(2,1)$-norm regularization is also investigated. The wide applicability of the framework is demonstrated through the combinations of common learning objectives in today's data science applications with the $(2,1)$-norm regularization. Numerical experiments are presented to illustrate the use of the NEPv approach and to gain insights as to what a proper regularizing parameter should have in real-world applications.

math.NA↗

arXiv2025

Mixed Precision Photonic Computing with 3D Electronic-Photonic Integrated Circuits

We propose advancing photonic in-memory computing through three-dimensional photonic-electronic integrated circuits using phase-change materials (PCM) and AlGaAs-CMOS technology. These circuits offer high precision (greater than 12 bits), scalability (greater than 1024 by 1024), and massive parallelism (greater than 1 million operations) across the wavelength, spatial, and temporal domains at ultra-low power (less than 1 watt per PetaOPS). Monolithically integrated hybrid PCM-AlGaAs memory resonators handle coarse-precision iterations (greater than 5-bit most significant bit precision) through reversible PCM phase transitions. Electro-optic memristive tuning enables fine-precision updates (greater than 8-bit least significant bit precision), resulting in over 12-bit precision for in-memory computing. The use of low-loss PCM (less than 0.01 dB per cm) and electro-optical tuning yields memristive optical resonators with high Q-factors (greater than 1 million), low insertion loss, and low tuning power. A W by W photonic tensor core composed of PCM-AlGaAs memresonators performs general matrix multiplication (GEMM) across W wavelengths from optical frequency combs, with minimal crosstalk and loss. Hierarchical scaling in the wavelength domain (K) and spatial domain (L) enables this system to address high-dimensional (N) scientific partial differential equation (PDE) problems in a single constant-time operation, compared to the conventional quadratic-time (N squared) computational complexity.

physics.optics↗

arXiv2025

Jacobian-Free Newton-Krylov with a globalization method for solving groundwater flow models of multi-layer aquifer systems

A Jacobian free Newton Krylov (JFNK) method with a globalization scheme is introduced to solve large and complex nonlinear systems of equations that arise in groundwater flow models of multi-layer aquifer systems. We explore the advantages of the JFNK method relative to the Newton-Krylov (NK) method and identify the circumstances in which the JFNK method demonstrates computing efficiency. We perform the validation and efficiency of the JFNK method on various test cases involving an unconfined single-layer aquifer and a two-layer aquifer with both confined and unconfined conditions. The results are validated by the NK method. The JFNK method is incorporated in Integrated Water Flow Model (IWFM), an integrated hydrologic model developed and maintained by California Department of Water Resources. We examine the determinacy of the JFNK's adaptability on practical models such as the California Central Valley Groundwater-Surface Water Simulation Model (C2VSim).

math.NA↗

arXiv2025

Hidden Convexity of Fair PCA and Fast Solver via Eigenvalue Optimization

Principal Component Analysis (PCA) is a foundational technique in machine learning for dimensionality reduction of high-dimensional datasets. However, PCA could lead to biased outcomes that disadvantage certain subgroups of the underlying datasets. To address the bias issue, a Fair PCA (FPCA) model was introduced by Samadi et al. (2018) for equalizing the reconstruction loss between subgroups. The semidefinite relaxation (SDR) based approach proposed by Samadi et al. (2018) is computationally expensive even for suboptimal solutions. To improve efficiency, several alternative variants of the FPCA model have been developed. These variants often shift the focus away from equalizing the reconstruction loss. In this paper, we identify a hidden convexity in the FPCA model and introduce an algorithm for convex optimization via eigenvalue optimization. Our approach achieves the desired fairness in reconstruction loss without sacrificing performance. As demonstrated in real-world datasets, the proposed FPCA algorithm runs $8\times$ faster than the SDR-based algorithm, and only at most 85% slower than the standard PCA.

cs.LG↗

arXiv2024

EPIC: a provable accelerated Eigensolver based on Preconditioning and Implicit Convexity

This paper is concerned with the extraction of the smallest eigenvalue and the corresponding eigenvector of a symmetric positive definite matrix pencil. We reveal implicit convexity of the eigenvalue problem in Euclidean space. A provable accelerated eigensolver based on preconditioning and implicit convexity (EPIC) is proposed. Theoretical analysis shows the acceleration of EPIC with the rate of convergence resembling the expected rate of convergence of the well-known locally optimal preconditioned conjugate gradient (LOPCG). A complete proof of the expected rate of convergence of LOPCG is elusive so far. Numerical results confirm our theoretical findings of EPIC.

math.NA↗

arXiv2023

A Self-Consistent Field Solution for Robust Common Spatial Pattern Analysis

The common spatial pattern analysis (CSP) is a widely used signal processing technique in brain-computer interface (BCI) systems to increase the signal-to-noise ratio in electroencephalogram (EEG) recordings. Despite its popularity, the CSP's performance is often hindered by the nonstationarity and artifacts in EEG signals. The minmax CSP improves the robustness of the CSP by using data-driven covariance matrices to accommodate the uncertainties. We show that by utilizing the optimality conditions, the minmax CSP can be recast as an eigenvector-dependent nonlinear eigenvalue problem (NEPv). We introduce a self-consistent field (SCF) iteration with line search that solves the NEPv of the minmax CSP. Local quadratic convergence of the SCF for solving the NEPv is illustrated using synthetic datasets. More importantly, experiments with real-world EEG datasets show the improved motor imagery classification rates and shorter running time of the proposed SCF-based solver compared to the existing algorithm for the minmax CSP.

math.NA↗

arXiv2023

A Bi-level Nonlinear Eigenvector Algorithm for Wasserstein Discriminant Analysis

Much like the classical Fisher linear discriminant analysis (LDA), the recently proposed Wasserstein discriminant analysis (WDA) is a linear dimensionality reduction method that seeks a projection matrix to maximize the dispersion of different data classes and minimize the dispersion of same data classes via a bi-level optimization. In contrast to LDA, WDA can account for both global and local interconnections between data classes by using the underlying principles of optimal transport. In this paper, a bi-level nonlinear eigenvector algorithm (WDA-nepv) is presented to fully exploit the structures of the bi-level optimization of WDA. The inner level of WDA-nepv for computing the optimal transport matrices is formulated as an eigenvector-dependent nonlinear eigenvalue problem (NEPv), and meanwhile, the outer level for trace ratio optimizations is formulated as another NEPv. Both NEPvs can be computed efficiently under the self-consistent field (SCF) framework. WDA-nepv is derivative-free and surrogate-model-free when compared with existing algorithms. Convergence analysis of the proposed WDA-nepv justifies the utilization of the SCF for solving the bi-level optimization of WDA. Numerical experiments with synthetic and real-life datasets demonstrate the classification accuracy and scalability of WDA-nepv.

stat.ML↗

arXiv2023

Scalable Spectral Clustering with Group Fairness Constraints

There are synergies of research interests and industrial efforts in modeling fairness and correcting algorithmic bias in machine learning. In this paper, we present a scalable algorithm for spectral clustering (SC) with group fairness constraints. Group fairness is also known as statistical parity where in each cluster, each protected group is represented with the same proportion as in the entirety. While FairSC algorithm (Kleindessner et al., 2019) is able to find the fairer clustering, it is compromised by high costs due to the kernels of computing nullspaces and the square roots of dense matrices explicitly. We present a new formulation of underlying spectral computation by incorporating nullspace projection and Hotelling's deflation such that the resulting algorithm, called s-FairSC, only involves the sparse matrix-vector products and is able to fully exploit the sparsity of the fair SC model. The experimental results on the modified stochastic block model demonstrate that s-FairSC is comparable with FairSC in recovering fair clustering. Meanwhile, it is sped up by a factor of 12 for moderate model sizes. s-FairSC is further demonstrated to be scalable in the sense that the computational costs of s-FairSC only increase marginally compared to the SC without fairness constraints.

cs.LG↗

arXiv2023

2D Eigenvalue Problem III: Convergence Analysis of the 2D Rayleigh Quotient Iteration

In Part I of this paper, we introduced a two dimensional eigenvalue problem (2DEVP) of a matrix pair and investigated its fundamental theory such as existence, variational characterization and number of 2D-eigenvalues. In Part II, we proposed a Rayleigh quotient iteration (RQI)-like algorithm (2DRQI) for computing a 2D-eigentriplet of the 2DEVP near a prescribed point, and discussed applications of 2DEVP and 2DRQI for solving the minimax problem of Rayleigh quotients, and computing the distance to instability. In this third part, we present convergence analysis of the 2DRQI. We show that under some mild conditions, the 2DRQI is locally quadratically convergent for computing a nonsingular 2D-eigentriplet.

math.NA↗

arXiv2022

Variational Characterization of Monotone Nonlinear Eigenvector Problems and Geometry of Self-Consistent-Field Iteration

This paper concerns a class of monotone eigenvalue problems with eigenvector nonlinearities (mNEPv). The mNEPv is encountered in applications such as the computation of joint numerical radius of matrices, best rank-one approximation of third-order partial symmetric tensors, and distance to singularity for dissipative Hamiltonian differential-algebraic equations. We first present a variational characterization of the mNEPv. Based on the variational characterization, we provide a geometric interpretation of the self-consistent-field (SCF) iterations for solving the mNEPv, prove the global convergence of the SCF, and devise an accelerated SCF. Numerical examples from a variety of applications demonstrate the theoretical properties and computational efficiency of the SCF and its acceleration.

math.NA↗

arXiv2022

2D Eigenvalue Problem II: Rayleigh Quotient Iteration and Applications

In Part I of this paper, we introduced a 2D eigenvalue problem (2DEVP) and presented theoretical results of the 2DEVP and its intrinsic connetion with the eigenvalue optimizations. In this part, we devise a Rayleigh quotient iteration (RQI)-like algorithm, 2DRQI in short, for computing a 2D-eigentriplet of the 2DEVP. The 2DRQI performs $2\times$ to $8\times$ faster than the existing algorithms for large scale eigenvalue optimizations arising from the minmax of Rayleigh quotients and the distance to instability of a stable matrix.

math.NA↗

arXiv2022

2D Eigenvalue Problem I: Existence and Number of Solutions

A two dimensional eigenvalue problem (2DEVP) of a Hermitian matrix pair $(A, C)$ is introduced in this paper. The 2DEVP can be viewed as a linear algebraic formulation of the well-known eigenvalue optimization problem of the parameter matrix $H(μ) = A - μC$. We present fundamental properties of the 2DEVP such as the existence, the necessary and sufficient condition for the finite number of 2D-eigenvalues and variational characterizations. We use eigenvalue optimization problems from the minmax of two Rayleigh quotients and the computation of distance to instability to show their connections with the 2DEVP and new insights of these problems derived from the properties of the 2DEVP.

math.NA↗

arXiv2022

Convergence analysis of a block preconditioned steepest descent eigensolver with implicit deflation

Gradient-type iterative methods for solving Hermitian eigenvalue problems can be accelerated by using preconditioning and deflation techniques. A preconditioned steepest descent iteration with implicit deflation (PSD-id) is one of such methods. The convergence behavior of the PSD-id is recently investigated based on the pioneering work of Samokish on the preconditioned steepest descent method (PSD). The resulting non-asymptotic estimates indicate a superlinear convergence of the PSD-id under strong assumptions on the initial guess. The present paper utilizes an alternative convergence analysis of the PSD by Neymeyr under much weaker assumptions. We embed Neymeyr's approach into the analysis of the PSD-id using a restricted formulation of the PSD-id. More importantly, we extend the new convergence analysis of the PSD-id to a practically preferred block version of the PSD-id, or BPSD-id, and show the cluster robustness of the BPSD-id. Numerical examples are provided to validate the theoretical estimates.

math.NA↗

arXiv2021

Backward Stability of Explicit External Deflation for the Symmetric Eigenvalue Problem

A thorough backward stability analysis of Hotelling's deflation, an explicit external deflation procedure through low-rank updates for computing many eigenpairs of a symmetric matrix, is presented. Computable upper bounds of the loss of the orthogonality of the computed eigenvectors and the symmetric backward error norm of the computed eigenpairs are derived. Sufficient conditions for the backward stability of the explicit external deflation procedure are revealed. Based on these theoretical results, the strategy for achieving numerical backward stability by dynamically selecting the shifts is proposed. Numerical results are presented to corroborate the theoretical analysis and to demonstrate the stability of the procedure for computing many eigenpairs of large symmetric matrices arising from applications.

math.NA↗

arXiv2020

Optimal Convergence Rate of Self-Consistent Field Iteration for Solving Eigenvector-dependent Nonlinear Eigenvalue Problems

We present a comprehensive convergence analysis for Self-Consistent Field (SCF) iteration to solve a class of nonlinear eigenvalue problems with eigenvector-dependency (NEPv). Using a tangent-angle matrix as an intermediate measure for approximation error, we establish new formulas for two fundamental quantities that optimally characterize the local convergence of the plain SCF: the local contraction factor and the local average contraction factor. In comparison with previously established results, new convergence rate estimates provide much sharper bounds on the convergence speed. As an application, we extend the convergence analysis to a popular SCF variant -- the level-shifted SCF. The effectiveness of the convergence rate estimates is demonstrated numerically for NEPv arising from solving the Kohn-Sham equation in electronic structure calculation and the Gross-Pitaevskii equation in the modeling of Bose-Einstein condensation.

math.NA↗

arXiv2019

Linear Constrained Rayleigh Quotient Optimization: Theory and Algorithms

We consider the following constrained Rayleigh quotient optimization problem (CRQopt) $$ \min_{x\in \mathbb{R}^n} x^{T}Ax\,\,\mbox{subject to}\,\, x^{T}x=1\,\mbox{and}\,C^{T}x=b, $$ where $A$ is an $n\times n$ real symmetric matrix and $C$ is an $n\times m$ real matrix. Usually, $m\ll n$. The problem is also known as the constrained eigenvalue problem in the literature because it becomes an eigenvalue problem if the linear constraint $C^{T}x=b$ is removed. We start by equivalently transforming CRQopt into an optimization problem, called LGopt, of minimizing the Lagrangian multiplier of CRQopt, and then an problem, called QEPmin, of finding the smallest eigenvalue of a quadratic eigenvalue problem. Although such equivalences has been discussed in the literature, it appears to be the first time that these equivalences are rigorously justified. Then we propose to numerically solve LGopt and QEPmin by the Krylov subspace projection method via the Lanczos process. The basic idea, as the Lanczos method for the symmetric eigenvalue problem, is to first reduce LGopt and QEPmin by projecting them onto Krylov subspaces to yield problems of the same types but of much smaller sizes, and then solve the reduced problems by some direct methods, which is either a secular equation solver (in the case of LGopt) or an eigensolver (in the case of QEPmin). The resulting algorithm is called the Lanczos algorithm. We perform convergence analysis for the proposed method and obtain error bounds. The sharpness of the error bound is demonstrated by artificial examples, although in applications the method often converges much faster than the bounds suggest. Finally, we apply the Lanczos algorithm to semi-supervised learning in the context of constrained clustering.

math.NA↗

arXiv2019

On the shift-invert Lanczos method for the buckling eigenvalue problem

We consider the problem of extracting a few desired eigenpairs of the buckling eigenvalue problem $Kx = λK_Gx$, where $K$ is symmetric positive semi-definite, $K_G$ is symmetric indefinite, and the pencil $K - λK_G$ is singular, namely, $K$ and $K_G$ share a non-trivial common nullspace. Moreover, in practical buckling analysis of structures, bases for the nullspace of $K$ and the common nullspace of $K$ and $K_G$ are available. There are two open issues for developing an industrial strength shift-invert Lanczos method: (1) the shift-invert operator $(K - σK_G)^{-1}$ does not exist or is extremely ill-conditioned, and (2) the use of the semi-inner product induced by $K$ drives the Lanczos vectors rapidly towards the nullspace of $K$, which leads to a rapid growth of the Lanczos vectors in norms and cause permanent loss of information and the failure of the method. In this paper, we address these two issues by proposing a generalized buckling spectral transformation of the singular pencil $K - λK_G$ and a regularization of the inner product via a low-rank updating of the semi-positive definiteness of $K$. The efficacy of our approach is demonstrated by numerical examples, including one from industrial buckling analysis.

math.NA↗

arXiv2019

A Self-consistent-field Iteration for Orthogonal Canonical Correlation Analysis

We propose an efficient algorithm for solving orthogonal canonical correlation analysis (OCCA) in the form of trace-fractional structure and orthogonal linear projections. Even though orthogonality has been widely used and proved to be a useful criterion for pattern recognition and feature extraction, existing methods for solving OCCA problem are either numerical unstable by relying on a deflation scheme, or less efficient by directly using generic optimization methods. In this paper, we propose an alternating numerical scheme whose core is the sub-maximization problem in the trace-fractional form with an orthogonal constraint. A customized self-consistent-field (SCF) iteration for this sub-maximization problem is devised. It is proved that the SCF iteration is globally convergent to a KKT point and that the alternating numerical scheme always converges. We further formulate a new trace-fractional maximization problem for orthogonal multiset CCA (OMCCA) and then propose an efficient algorithm with an either Jacobi-style or Gauss-Seidel-style updating scheme based on the same SCF iteration. Extensive experiments are conducted to evaluate the proposed algorithms against existing methods including two real world applications: multi-label classification and multi-view feature extraction. Experimental results show that our methods not only perform competitively to or better than baselines but also are more efficient.

cs.LG↗