arXiv subjects

Arvind K. Saibaba

Publications and source records attributed to Arvind K. Saibaba.

At least 19 recordsLinked to original sources

arXiv2026

Revisiting column subset selection through the lens of submodularity

The problem is to select k columns with maximal volume from a real mxn matrix X. We show that the logarithm of the volume is a set submodular function on columns of X, and for full column-rank matrices X with sufficiently large singular values, it is a non-negative non-decreasing function. As a consequence, traditional Businger-Golub QR with column pivoting is a greedy algorithm, with a relative error of at most 37 percent. In contrast, Gu-Eisenstat strong rank-revealing QR is a 1-interchange algorithm, with a relative error of at most 50 percent. The higher accuracy, under this metric, of the simple QR with column pivoting confirms its well known effectiveness in practice. For general, possibly rank-deficient matrices, we derive probabilistic bounds for the absolute error based on a smoothed analysis. The above analyses are extended to finding kxk submatrices of maximal volume in symmetric positive-definite matrices.

math.NA↗

arXiv2026

A Majorization-Minimization with Monte Carlo Approach for Hyperparameter Estimation

We consider inverse problems with linear forward models and Gaussian priors, but with unknown hyperparameters that may arise from the model, the noise, or the specification of the prior. We model this using a hierarchical Bayes framework resulting in a posterior distribution that is non-Gaussian, in general, and challenging to sample from. Consequently, we use an empirical Bayes framework for estimating the maximum a posteriori estimate of the hyperpameters by considering the marginalized posterior distribution. However, the optimization problem is also computationally challenging due to the need for repeated evaluation of log determinants. To address this issue, we propose a Majorization-Minimization with Monte Carlo approach, which we call M$^{3}$C, for hyperparameter estimation. Specifically, we replace the challenging optimization problem with a sequence of simpler ones by utilizing a majorization function (or majorant) for the log-determinant term, combined with a Monte Carlo estimator to approximate the majorant. We provide theoretical results, showing that under certain assumptions, the M$^{3}$C iterates converge with high probability to a critical point of the original cost function. A variety of numerical examples are provided from seismic tomography, super-resolution imaging, and contaminant source identification.

math.NA↗

arXiv2026

FlexTrace: Exchangeable Randomized Trace Estimation for Matrix Functions

We consider the task of estimating the trace of a matrix function, ${\rm tr}(f({\bf A}))$, of a large symmetric positive semi-definite matrix ${\bf A}$. This problem arises in multiple applications, including kernel methods and inverse problems. A key challenge across existing trace estimation methods is the need for matrix-vector products (matvecs) with $f({\bf A})$, which can be very expensive. In this article, we introduce a novel trace estimator, FlexTrace, an exchangeable, single-pass method that estimates ${\rm tr}(f({\bf A}))$ solely using matvecs with ${\bf A}$. We consider the case where $f$ is an operator monotone matrix function with $f(0)=0$, which includes functions such as $\log(1+x)$ and $x^{1/2}$, and derive probabilistic bounds showcasing the theoretical advantages of FlexTrace. Numerical experiments across synthetic examples and application domains demonstrate that FlexTrace provides substantially more accurate estimates of the trace of $f({\bf A})$ compared to existing methods.

math.NA↗

arXiv2026

Multifidelity sensor placement in Bayesian state estimation problems

We study optimal sensor placement for Bayesian state estimation problems in which sensors vary in cost and fidelity, resulting in a budget-constrained multifidelity optimal experimental design problem. Sensor placement optimality is quantified using the D-optimality criterion, and the problem is approached by leveraging connections with the column subset selection problem in numerical linear algebra. We implement a greedy approach for this problem, whose computational efficiency we improve using rank-one updates via the Sherman-Morrison formula. We additionally present an iterative algorithm that, for each feasible allocation of sensors, greedily optimizes over each sensor fidelity subject to previous sensor choices, repeating this process until a termination criterion is satisfied. To our knowledge, these algorithms are novel in the context of cost-constrained multifidelity sensor placement. We evaluate our methods on several benchmark state estimation problems, including reconstructions of sea surface temperature and flow around a cylinder, and empirically demonstrate improved performance over random designs.

math.NA↗

arXiv2026

Optimal Sensor Placement in Gaussian Processes via Column Subset Selection

Gaussian process regression uses data measured at sensor locations to reconstruct a spatially dependent function with quantified uncertainty. However, if only a limited number of sensors can be deployed, it is important to determine how to optimally place the sensors to minimize a measure of the uncertainty in the reconstruction. We consider the Bayesian D-optimal criterion to determine the optimal sensor locations by choosing sensors from a candidate set of sensors. Since this is an NP-hard problem, our approach models sensor placement as a column subset selection problem (CSSP) on the covariance matrix, computed using the kernel function on the candidate sensor points. We propose an algorithm that uses the Golub-Klema-Stewart framework (GKS) to select sensors and provide an analysis of lower bounds on the D-optimality of these sensor placements. To reduce the computational cost in the GKS step, we propose and analyze algorithms for the D-optimal sensor placements using Nystr\"om approximations on the covariance matrix. Moreover, we propose several algorithms that select sensors via Nystr\"om approximation of the covariance matrix, utilizing the randomized Nystr\"om approximation, random pivoted Cholesky and greedy pivoted Cholesky. We demonstrate the performance of our method on two applications: thin liquid film dynamics and sea surface temperature.

math.NA↗

arXiv2025

Parametric Hierarchical Matrix Approximations to Kernel Matrices

Kernel matrices are ubiquitous in computational mathematics, often arising from applications in machine learning and scientific computing. In two or three spatial or feature dimensions, such problems can be approximated efficiently by a class of matrices known as hierarchical matrices. A hierarchical matrix consists of a hierarchy of small near-field blocks (or sub-matrices) stored in a dense format and large far-field blocks approximated by low-rank matrices. Standard methods for forming hierarchical matrices do not account for the fact that kernel matrices depend on specific hyperparameters; for example, in the context of Gaussian processes, hyperparameters must be optimized over a fixed parameter space. We introduce a new class of hierarchical matrices, namely, parametric (parameter-dependent) hierarchical matrices. Members of this new class are parametric $\mathcal{H}$-matrices and parametric $\mathcal{H}^{2}$-matrices. The construction of a parametric hierarchical matrix follows an offline-online paradigm. In the offline stage, the near-field and far-field blocks are approximated by using polynomial approximation and tensor compression. In the online stage, for a particular hyperparameter, the parametric hierarchical matrix is instantiated efficiently as a standard hierarchical matrix. The asymptotic costs for storage and computation in the offline stage are comparable to the corresponding standard approaches of forming a hierarchical matrix. However, the online stage of our approach requires no new kernel evaluations, and the far-field blocks can be computed more efficiently than standard approaches. {Numerical experiments show over $100\times$ speedups compared with existing techniques.}

math.NA↗

arXiv2025

Adaptive Randomized Tensor Train Rounding using Khatri-Rao Products

Approximating a tensor in the tensor train (TT) format has many important applications in scientific computing. Rounding a TT tensor involves further compressing a tensor that is already in the TT format. This paper proposes new randomized algorithms for TT-rounding that uses sketches based on Khatri-Rao products (KRP). When the TT-ranks are known in advance, the proposed methods are comparable in cost to the sketches that used a sketching matrix in the TT-format~\cite{al2023randomized}. However, the use of KRP sketches enables adaptive algorithms to round the tensor in the TT-format within a fixed user-specified tolerance. An important component of the adaptivity is the estimation of error using KRP sketching, for which we develop theoretical guarantees. We report numerical experiments on synthetic tensors, parametric low-rank kernel approximations, and the solution of parametric partial differential equations. The numerical experiments show that we obtain speed-ups of up to $50\times$ compared to deterministic TT-rounding. Both the computational cost analysis and numerical experiments verify that the adaptive algorithms are competitive with the fixed rank algorithms, suggesting the adaptivity introduces only a low overhead.

math.NA↗

arXiv2025

Many (most?) column subset selection criteria are NP hard for a few columns

We consider a variety of criteria for selecting k representative columns from a real mxn matrix A, when sufficiently few columns are required, i.e., 1<= k<= min{rank(A), m/3}. The criteria include the following optimization problems: absolute volume and S-optimality maximization; norm, pseudo-inverse norm, and condition minimization number in the two-norm, Frobenius norm and Schatten p-norms for p>2; stable rank maximization; and the new criterion of relative volume maximization, which is inversely proportional to a power of the condition number. We show that these criteria are NP hard and many do not admit polynomial time approximation schemes (PTAS). To formulate the optimization problems as decision problems, we derive optimal values for the subset selection criteria, as well as expressions for partitioned pseudo-inverses. The results for minimization of the pseudo-inverse in the Frobenius norm are applicable to trace optimization in A-optimal design.

math.NA↗

arXiv2025

A tensor-based dynamic mode decomposition based on the $\star_{\boldsymbol{M}}$-product

Dynamic mode decomposition (DMD) is a data-driven method for estimating the dynamics of a discrete dynamical system. This paper proposes a tensor-based approach to DMD for applications in which the states can be viewed as tensors. Specifically, we use the $\star_{\boldsymbol{M}}$-product framework for tensor decompositions which we demonstrate offers excellent compression compared to matrix-based methods and can be implemented in a computationally efficient manner. We show how the proposed approach is connected to the traditional DMD and physics-informed DMD frameworks. We give a computational framework for computing the tensor-based DMD and detail the computational costs. We also give a randomized algorithm that enables efficient $\star_{\boldsymbol{M}}$-DMD computations in the streaming setting. The numerical results show that the proposed method achieves equal or better accuracy for the same storage compared to the standard DMD on these examples and is more efficient to compute.

math.NA↗

arXiv2025

Stochastic Trace Optimization of Parameter Dependent Matrices Based on Statistical Learning Theory

We consider matrices $\boldsymbol{A}(\boldsymbol\theta)\in\mathbb{R}^{m\times m}$ that depend, possibly nonlinearly, on a parameter $\boldsymbol\theta$ from a compact parameter space $\Theta$. We present a Monte Carlo estimator for minimizing $\text{trace}(\boldsymbol{A}(\boldsymbol\theta))$ over all $\boldsymbol\theta\in\Theta$, and determine the sampling amount so that the backward error of the estimator is bounded with high probability. We derive two types of bounds, based on epsilon nets and on generic chaining. Both types predict a small sampling amount for matrices $\boldsymbol{A}(\boldsymbol\theta)$ with small offdiagonal mass, and parameter spaces $\Theta$ of small ``size.'' Dependence on the matrix dimension~$m$ is only weak or not explicit. The bounds based on epsilon nets are easier to evaluate and come with fully specified constants. In contrast, the bounds based on chaining depend on the Talagrand functionals which are difficult to evaluate, except in very special cases. Comparisons between the two types of bounds are difficult, although the literature suggests that chaining bounds can be superior.

stat.ML↗

arXiv2025

Improved Analysis of Khatri-Rao Random Projections and Applications

Randomization has emerged as a powerful set of tools for large-scale matrix and tensor decompositions. Randomized algorithms involve computing sketches with random matrices. A prevalent approach is to take the random matrix as a standard Gaussian random matrix, for which the theory is well developed. However, this approach has the drawback that the cost of generating and multiplying by the random matrix can be prohibitively expensive. Khatri-Rao random projections (KRPs), obtained by sketching with Khatri-Rao products of random matrices, offer a viable alternative and are much cheaper to generate. However, the theoretical guarantees of using KRPs are much more pessimistic compared to their accuracy observed in practice. We attempt to close this gap by obtaining improved analysis of the use of KRPs in matrix and tensor low-rank decompositions. We propose and analyze a new algorithm for low-rank approximations of block-structured matrices (e.g., block Hankel) using KRPs. We also show how to accelerate tensor computations in the Tucker format using KRPs and give theoretical guarantees of the resulting low-rank approximations. Numerical experiments on synthetic and real-world tensors show the computational benefits of the proposed methods.

math.NA↗

arXiv2025

Structured Column Subset Selection for Bayesian Optimal Experimental Design

We consider optimal experimental design (OED) for Bayesian inverse problems, where the experimental design variables have a certain multiway structure. Given $d$ different experimental variables with $m_i$ choices per design variable $1 \le i\le d$, the goal is to select $k_i \le m_i$ experiments per design variable. Previous work has related OED to the column subset selection problem by mapping the design variables to the columns of a matrix $\mathbf{A}$. However, this approach is applicable only to the case $d=1$ in which the columns can be selected independently. We develop an extension to the case where the design variables have a multi-way structure. Our approach is to map the matrix $\mathbf{A}$ to a tensor and perform column subset selection on mode unfoldings of the tensor. We develop an algorithmic framework with three different algorithmic templates, and randomized variants of these algorithms. We analyze the computational cost of all the proposed algorithms and also develop greedy versions to facilitate comparisons. Numerical experiments on four different applications -- time-dependent inverse problems, seismic tomography, X-ray tomography, and flow reconstruction -- demonstrate the effectiveness and scalability of our methods for structured experimental design in Bayesian inverse problems.

math.NA↗

arXiv2025

Bridging the Gap Between Deterministic and Probabilistic Approaches to State Estimation

We consider the problem of state estimation from limited discrete and noisy measurements. In particular, we focus on modal state estimation, which approximates the unknown state of the system within a prescribed basis. We estimate the coefficients of the modal expansion using available observational data. This is usually accomplished through two distinct frameworks. One is deterministic and estimates the expansion coefficients by solving a least-squares (LS) problem. The second is probabilistic and uses a Bayesian approach to derive a distribution for the coefficients, resulting in the maximum-a-posteriori (MAP) estimate. Here, we seek to quantify and compare the accuracy of these two approaches. To this end, we derive a computable expression for the difference in Bayes risk between the deterministic LS and the Bayesian MAP estimates. We prove that this difference is always nonnegative, indicating that the MAP estimate is always more reliable than the LS estimate. We further show that this difference comprises two nonnegative components representing measurement noise and prior uncertainty, and identify regimes where one component dominates the other in magnitude. We also derive a novel prior distribution from the sample covariance matrix of the training data, and examine the greedy Bayesian and column-pivoted QR (CPQR) sensor placement algorithms with this prior as an input. Using numerical examples, we show that the greedy Bayesian algorithm returns nearly optimal sensor locations. We show that, under certain conditions, the greedy Bayesian sensor locations are identical or nearly identical to those of CPQR when applied to a regularized modal basis.

math.NA↗

arXiv2025

A control-oriented approach to optimal sensor placement

We propose a control-oriented optimal experimental design (cOED) approach for linear PDE-constrained Bayesian inverse problems. In particular, we consider optimal control problems with uncertain parameters that need to be estimated by solving an inverse problem, which in turn requires measurement data. We consider the case where data is collected at a set of sensors. While classical Bayesian OED techniques provide experimental designs (sensor placements) that minimize the posterior uncertainty in the inversion parameter, these designs are not tailored to the demands of the optimal control problem. In the present control-oriented setting, we prioritize the designs that minimize the uncertainty in the state variable being controlled or the control objective. We propose a mathematical framework for uncertainty quantification and cOED for parameterized PDE-constrained optimal control problems with linear dependence to the control variable and the inversion parameter. We also present scalable computational methods for computing control-oriented sensor placements and for quantifying the uncertainty in the control objective. Additionally, we present illustrative numerical results in the context of a model problem motivated by heat transfer applications.

math.OC↗

arXiv2025

Optimal sensor placement under model uncertainty in the weak-constraint 4D-Var framework

In data assimilation, the model may be subject to uncertainties and errors. The weak-constraint data assimilation framework enables incorporating model uncertainty in the dynamics of the governing equations. We propose a new framework for near-optimal sensor placement in the weak-constrained setting. This is achieved by first deriving a design criterion based on the expected information gain, which involves the Kullback-Leibler divergence from the forecast prior to the posterior distribution. An explicit formula for this criterion is provided, assuming that the model error and background are independent and Gaussian and the dynamics are linear. We discuss algorithmic approaches to efficiently evaluate this criterion through randomized approximations. To provide further insight and flexibility in computations, we also provide alternative expressions for the criteria. We provide an algorithm to find near-optimal experimental designs using column subset selection, including a randomized algorithm that avoids computing the adjoint of the forward operator. Through numerical experiments in one and two spatial dimensions, we show the effectiveness of our proposed methods.

math.NA↗

arXiv2024

Efficient hyperparameter estimation in Bayesian inverse problems using sample average approximation

In Bayesian inverse problems, it is common to consider several hyperparameters that define the prior and the noise model that must be estimated from the data. In particular, we are interested in linear inverse problems with additive Gaussian noise and Gaussian priors defined using Mat\'{e}rn covariance models. In this case, we estimate the hyperparameters using the maximum a posteriori (MAP) estimate of the marginalized posterior distribution. However, this is a computationally intensive task since it involves computing log determinants. To address this challenge, we consider a stochastic average approximation (SAA) of the objective function and use the preconditioned Lanczos method to compute efficient approximations of the function and gradient evaluations. We propose a new preconditioner that can be updated cheaply for new values of the hyperparameters and an approach to compute approximations of the gradient evaluations, by reutilizing information from the function evaluations. We demonstrate the performance of our approach on static and dynamic seismic tomography problems.

math.NA↗

arXiv2024

Stable Rank and Intrinsic Dimension of Real and Complex Matrices

The notion of `stable rank' of a matrix is central to the analysis of randomized matrix algorithms, covariance estimation, deep neural networks, and recommender systems. We compare the properties of the stable rank and intrinsic dimension of real and complex matrices to those of the classical rank. Basic proofs and examples illustrate that the stable rank does not satisfy any of the fundamental rank properties, while the intrinsic dimension satisfies a few. In particular, the stable rank and intrinsic dimension of a submatrix can exceed those of the original matrix; adding a Hermitian positive semi-definite matrix can lower the intrinsic dimension of the sum; and multiplication by a nonsingular matrix can drastically change the stable rank and the intrinsic dimension. We generalize the concept of stable rank to the p-stable in any Schatten p-norm, thereby unifying the concepts of stable rank and intrinsic dimension: The stable rank is the 2-stable rank, while the intrinsic dimension is the 1-stable rank of a Hermitian positive semi-definite matrix. We derive sum and product inequalities for the pth root of the p-stable rank, and show that it is well-conditioned in the norm-wise absolute sense. The conditioning improves if the matrix and the perturbation are Hermitian positive semi-definite.

math.NA↗

arXiv2024

Parametric kernel low-rank approximations using tensor train decomposition

Computing low-rank approximations of kernel matrices is an important problem with many applications in scientific computing and data science. We propose methods to efficiently approximate and store low-rank approximations to kernel matrices that depend on certain hyperparameters. The main idea behind our method is to use multivariate Chebyshev function approximation along with the tensor train decomposition of the coefficient tensor. The computations are in two stages: an offline stage, which dominates the computational cost and is parameter-independent, and an online stage, which is inexpensive and instantiated for specific hyperparameters. A variation of this method addresses the case that the kernel matrix is symmetric and positive semi-definite. The resulting algorithms have linear complexity in terms of the sizes of the kernel matrices. We investigate the efficiency and accuracy of our method on parametric kernel matrices induced by various kernels, such as the Mat\'ern kernel, through various numerical experiments. Our methods have speedups up to $200\times$ in the online time compared to other methods with similar complexity and comparable accuracy.

math.NA↗