arXiv subjects

Babak Hassibi

Publications and source records attributed to Babak Hassibi.

At least 19 recordsLinked to original sources

arXiv2026

Rate-cost Trade-offs in H-infinity Control with Initial State Uncertainty

We consider the problem of H-infinity control in the presence of a digital communication channel between the observer and the controller and investigate the fundamental trade-off between the required communication data rate and H-infinity performance. The system follows linear dynamics and has no additive internal noise, so that the uncertainty is confined to the initial state. For the scalar system, using a deterministic time-zero covering argument, we establish a lower bound on the required data rate for a given H-infinity cost and demonstrate why logarithmic quantization (Elia, 2000) is a natural choice. We further develop an achievability scheme and show that, for the scalar system, its rate matches the converse asymptotically in the high-data-rate limit.

eess.SY↗

arXiv2026

Dual Space Preconditioning for Gradient Descent in the Overparameterized Regime

In this work, we study the convergence properties of the Dual Space Preconditioned Gradient Descent, encompassing optimizers such as Normalized Gradient Descent and Gradient Clipping. We consider preconditioners of the form $\nabla K$, where $K: \mathbb{R}^{d \times k} \to \mathbb{R}$ is convex and apply $\nabla K(\cdot)$ to train an over-parameterized linear model with a convex loss of the form $\ell(X W - Y)$, for weights $W \in \mathbb{R}^{d \times k}$, labels $Y \in \mathbb{R}^{n \times k}$ and data $X \in \mathbb{R}^{n \times d}$. Under the aforementioned assumptions, we prove that the iterates of the full-batch preconditioned gradient descent converge at an exponential rate to a point $W_{\infty} \in \mathbb{R}^{d \times k}$ satisfying $XW_{\infty} = Y$. We also study the implicit bias of Dual Space Preconditioned Gradient Descent. First, we demonstrate analytically and empirically that, for general $K(\cdot)$, $W_\infty$ depends on the chosen constant step size, hindering a precise characterization of the implicit bias. We also provide an approximate implicit bias property for general preconditioners, namely, $\|W_0 - W_{\infty}\|_F \le c \|W_0 - W_{\text{GD}, \infty}\|_F$ for a constant $c>0$ and $W_{\text{GD}, \infty}$ denoting the convergence point of GD initialized at $W_0$. Furthermore, for preconditioners of the form $K(G) = h(\|G\|_F)$, known as {\it isotropic preconditioners}, and for the stochastic variation of the algorithm with arbitrary batch-size, we prove linear convergence to $W_{\text{GD}, \infty}$. Finally, in the experiments, we demonstrate faster convergence on a nonlinear model obtained using the smoothed matrix elastic-net as a preconditioner.

stat.ML↗

arXiv2026

Deriving Approximate Message Passing from the Convex Gaussian Min-Max Theorem

Approximate message passing (AMP) provides fast iterative algorithms for high-dimensional signal recovery with Gaussian design matrices, while the Convex Gaussian Min-max Theorem (CGMT) gives a static optimization framework for obtaining sharp asymptotic characterizations of convex estimators. Although these two frameworks often lead to the same scalar state-evolution equations, their connection is usually indirect. In this paper, we establish a direct connection between the two for regularized linear regression in the proportional high-dimensional regime. When the CGMT Auxiliary Optimization (AO) and Primary Optimization (PO) give the same primal-dual solution, we show that the CGMT framework recovers the AMP fixed-point equations, including the Onsager correction. We further identify the AO Gaussian vectors with the Gaussian perturbations in the primal and residual AMP channels. For regularized M-estimation, the same viewpoint recovers the fixed point of scalar-variance max-sum Generalized AMP (GAMP). These results show that the AMP (and GAMP) iterations are suggested, and can be derived, from the CGMT framework, and may further suggest a way to derive AMP-like algorithms in settings where CGMT applies but standard AMP derivations are unavailable.

cs.IT↗

arXiv2026

On Codes with Support-Constrained Parity Checks

We study linear codes that maximize minimum distance subject to arbitrary support constraints on the parity-check matrix. Such constraints arise naturally in the design of LDPC codes, locally repairable codes, and hardware-constrained systems where each parity check must involve only a limited number of code symbols. They are also essential in quantum error correction, where sparse stabilizers reduce measurement noise and respect the connectivity constraints of physical qubit architectures. We derive the optimal minimum distance possible given support constraints on the parity-check matrix and show it is achievable over sufficiently large fields. When this maximum distance coincides with the Singleton bound for unconstrained parity check matrices, the dual GM-MDS construction yields generalized Reed--Solomon codes obeying the mask. In the generator-matrix setting, the GM-MDS theorem guarantees that the optimal distance can always be achieved by a subcode of a generalized Reed--Solomon code while satisfying arbitrary support constraints. We show that this is not true for the parity-check setting. We exhibit a set of support constraints, derived from the vertex-edge incidence of $K_{6,6}$, for which the optimal minimum distance cannot be realized by any subcode of a generalized Reed--Solomon code over any field. We also analyze structured constraint families -- regular, balanced, and cyclic masks -- through numerical optimization, providing design guidance for practical code constructions.

cs.IT↗

arXiv2026

Distributionally Robust K-Means Clustering

K-means clustering is a workhorse of unsupervised learning, but it is notoriously brittle to outliers, distribution shifts, and limited sample sizes. Viewing k-means as Lloyd--Max quantization of the empirical distribution, we develop a distributionally robust variant that protects against such pathologies. We posit that the unknown population distribution lies within a Wasserstein-2 ball around the empirical distribution. In this setting, one seeks cluster centers that minimize the worst-case expected squared distance over this ambiguity set, leading to a minimax formulation. A tractable dual yields a soft-clustering scheme that replaces hard assignments with smoothly weighted ones. We propose an efficient block coordinate descent algorithm with provable monotonic decrease and local linear convergence. Experiments on standard benchmarks and large-scale synthetic data demonstrate substantial gains in outlier detection and robustness to noise.

cs.LG↗

arXiv2026

Quantized Online LQR

We study online linear-quadratic regulation (LQR) with unknown dynamics under communication rate constraints. Classical networked control quantizes the plant state at every time step, requiring $O(T)$ total bits while injecting persistent quantization noise that limits control performance. We consider a setting where the plant observes its state locally and can estimate system dynamics via ordinary least squares, while a remote controller possesses knowledge of the control cost. Rather than quantizing the raw state, the plant transmits learned dynamics estimates over a rate-limited uplink, and the controller returns the optimal control policy so that the plant can compute actions locally using its superior state knowledge. We first prove a fundamental information-theoretic lower bound: any scheme achieving $O(T^α)$ regret for $α\in [1/2,1)$ compared to the optimal infinite horizon LQR controller that knows the true system dynamics must transmit at least $Ω(\log T)$ bits. We then design the \textbf{Quantized Certainty Equivalent (QCE-LQR)} algorithm, which matches this bound. The resulting regret bound contains inflation factors $Q_{\mathrm{slow}}(\varrho)$ and $Q_{\mathrm{fast}}(\varrho)$ that vanish as the codebook resolution increases, smoothly recovering the unquantized baseline regret. Numerical experiments on four benchmark systems -- from a scalar unstable plant to a 24-parameter Boeing 747 lateral model -- confirm that a variant of QCE-LQR achieves regret comparable to an unquantized certainty equivalent controller over a horizon of $T=10{,}000$ steps.

eess.SY↗

arXiv2026

Precise Performance of Linear Denoisers in the Proportional Regime

In the present paper we study the performance of linear denoisers for noisy data of the form $\mathbf{x} + \mathbf{z}$, where $\mathbf{x} \in \mathbb{R}^d$ is the desired data with zero mean and unknown covariance $\mathbfΣ$, and $\mathbf{z} \sim \mathcal{N}(0, \mathbfΣ_{\mathbf{z}})$ is additive noise. Since the covariance $\mathbfΣ$ is not known, the standard Wiener filter cannot be employed for denoising. Instead we assume we are given samples $\mathbf{x}_1,\dots,\mathbf{x}_n \in \mathbb{R}^d$ from the true distribution. A standard approach would then be to estimate $\mathbfΣ$ from the samples and use it to construct an ``empirical" Wiener filter. However, in this paper, motivated by the denoising step in diffusion models, we take a different approach whereby we train a linear denoiser $\mathbf{W}$ from the data itself. In particular, we synthetically construct noisy samples $\hat{\mathbf{x}}_i$ of the data by injecting the samples with Gaussian noise with covariance $\mathbfΣ_1 \neq \mathbfΣ_{\mathbf{z}}$ and find the best $\mathbf{W}$ that approximates $\mathbf{W}\hat{\mathbf{x}}_i \approx \mathbf{x}_i$ in a least-squares sense. In the proportional regime $\frac{n}{d} \rightarrow κ> 1$ we use the {\it Convex Gaussian Min-Max Theorem (CGMT)} to analytically find the closed form expression for the generalization error of the denoiser obtained from this process. Using this expression one can optimize over $\mathbfΣ_1$ to find the best possible denoiser. Our numerical simulations show that our denoiser outperforms the ``empirical" Wiener filter in many scenarios and approaches the optimal Wiener filter as $κ\rightarrow\infty$.

stat.ML↗

arXiv2026

Implicit Bias and Convergence of Matrix Stochastic Mirror Descent

We investigate Stochastic Mirror Descent (SMD) with matrix parameters and vector-valued predictions, a framework relevant to multi-class classification and matrix completion problems. Focusing on the overparameterized regime, where the total number of parameters exceeds the number of training samples, we prove that SMD with matrix mirror functions $ψ(\cdot)$ converges exponentially to a global interpolator. Furthermore, we generalize classical implicit bias results of vector SMD by demonstrating that the matrix SMD algorithm converges to the unique solution minimizing the Bregman divergence induced by $ψ(\cdot)$ from initialization subject to interpolating the data. These findings reveal how matrix mirror maps dictate inductive bias in high-dimensional, multi-output problems.

stat.ML↗

arXiv2025

One-Bit Quantization for Random Features Models

Recent advances in neural networks have led to significant computational and memory demands, spurring interest in one-bit weight compression to enable efficient inference on resource-constrained devices. However, the theoretical underpinnings of such compression remain poorly understood. We address this gap by analyzing one-bit quantization in the Random Features model, a simplified framework that corresponds to neural networks with random representations. We prove that, asymptotically, quantizing weights of all layers except the last incurs no loss in generalization error, compared to the full precision random features model. Our findings offer theoretical insights into neural network compression. We also demonstrate empirically that one-bit quantization leads to significant inference speed ups for the Random Features models even on a laptop GPU, confirming the practical benefits of our work. Additionally, we provide an asymptotically precise characterization of the generalization error for Random Features with an arbitrary number of layers. To the best of our knowledge, our analysis yields more general results than all previous works in the related literature.

cs.LG↗

arXiv2025

Learn to Change the World: Multi-level Reinforcement Learning with Model-Changing Actions

Reinforcement learning usually assumes a given or sometimes even fixed environment in which an agent seeks an optimal policy to maximize its long-term discounted reward. In contrast, we consider agents that are not limited to passive adaptations: they instead have model-changing actions that actively modify the RL model of world dynamics itself. Reconfiguring the underlying transition processes can potentially increase the agents' rewards. Motivated by this setting, we introduce the multi-layer configurable time-varying Markov decision process (MCTVMDP). In an MCTVMDP, the lower-level MDP has a non-stationary transition function that is configurable through upper-level model-changing actions. The agent's objective consists of two parts: Optimize the configuration policies in the upper-level MDP and optimize the primitive action policies in the lower-level MDP to jointly improve its expected long-term reward.

cs.LG↗

arXiv2025

A Precise Performance Analysis of the Randomized Singular Value Decomposition

The Randomized Singular Value Decomposition (RSVD) is a widely used algorithm for efficiently computing low-rank approximations of large matrices, without the need to construct a full-blown SVD. Of interest, of course, is the approximation error of RSVD compared to the optimal low-rank approximation error obtained from the SVD. While the literature provides various upper and lower error bounds for RSVD, in this paper we derive precise asymptotic expressions that characterize its approximation error as the matrix dimensions grow to infinity. Our expressions depend only on the singular values of the matrix, and we evaluate them for two important matrix ensembles: those with power law and bilevel singular value distributions. Our results aim to quantify the gap between the existing theoretical bounds and the actual performance of RSVD. Furthermore, we extend our analysis to polynomial-filtered RSVD, deriving performance characterizations that provide insights into optimal filter selection.

math.NA↗

arXiv2025

Gaussian Universality for Diffusion Models

We investigate Gaussian Universality for data distributions generated via diffusion models. By Gaussian Universality we mean that the test error of a generalized linear model $f(\mathbf{W})$ trained for a classification task on the diffusion data matches the test error of $f(\mathbf{W})$ trained on the Gaussian Mixture with matching means and covariances per class.In other words, the test error depends only on the first and second order statistics of the diffusion-generated data in the linear setting. As a corollary, the analysis of the test error for linear classifiers can be reduced to Gaussian data from diffusion-generated data. Analysing the performance of models trained on synthetic data is a pertinent problem due to the surge of methods such as \cite{sehwag2024stretchingdollardiffusiontraining}. Moreover, we show that, for any $1$- Lipschitz scalar function $ϕ$, $ϕ(\mathbf{x})$ is close to $\mathbb{E} ϕ(\mathbf{x})$ with high probability for $\mathbf{x}$ sampled from the conditional diffusion model corresponding to each class. Finally, we note that current approaches for proving universality do not apply to diffusion-generated data as the covariance matrices of the data tend to have vanishing minimum singular values, contrary to the assumption made in the literature. This leaves extending previous mathematical universality results as an intriguing open question.

stat.ML↗

arXiv2025

Beyond Quadratic Costs: A Bregman Divergence Approach to H$_\infty$ Control

In the past couple of decades, non-quadratic convex penalties have reshaped signal processing and machine learning; in robust control, however, general convex costs break the Riccati and storage function structure that make the design tractable. Practitioners thus default to approximations, heuristics or robust model predictive control that are solved online for short horizons. We close this gap by extending $H_\infty$ control of discrete-time linear systems to strictly convex penalties on state, input, and disturbance, recasting the objective with Bregman divergences that admit a completion-of-squares decomposition. The result is a closed-form, time-invariant, full-information stabilizing controller that minimizes a worst-case performance ratio over the infinite horizon. Necessary and sufficient existence/optimality conditions are given by a Riccati-like identity together with a concavity requirement; with quadratic costs, these collapse to the classical $H_\infty$ algebraic Riccati equation and the associated negative-semidefinite condition, recovering the linear central controller. Otherwise, the optimal controller is nonlinear and can enable safety envelopes, sparse actuation, and bang-bang policies with rigorous $H_\infty$ guarantees.

eess.SY↗

arXiv2025

Coded Kalman Filtering over MIMO Gaussian Channels with Feedback

We consider the problem of remotely stabilizing a dynamical system. A sensor (encoder) co-located with the system communicates with a controller (decoder), whose goal is to stabilize the system, over a noisy communication channel with feedback. To accomplish this, the controller must estimate the system state with finite mean squared error (MSE). The vector-valued dynamical system state follows a Gauss-Markov law with additive control. The channel is a multiple-input multiple-output (MIMO) additive white Gaussian noise (AWGN) channel with feedback. For such a source, a linear encoder, and a MIMO AWGN channel, the minimal MSE decoder is a Kalman filter. The parameters of the Kalman filter and the linear encoder can be jointly optimized, under a power constraint at the channel input. We term the resulting encoder-decoder pair a coded Kalman filter. We establish sufficient and necessary conditions for the coded Kalman filter to achieve a finite MSE in the real-time estimation of the state. For sufficiency, we introduce a coding scheme where each unstable mode of the state is estimated using the channel outputs of a single sub-channel. We prove a coinciding necessity condition when either the source or channel is scalar and present a matrix-algebraic condition which implies the condition is necessary in general. Finally, we provide a new counter-example demonstrating that linear codes are generally sub-optimal for coding over MIMO channels.

cs.IT↗

arXiv2025

Optimal Implicit Bias in Linear Regression

Most modern learning problems are over-parameterized, where the number of learnable parameters is much greater than the number of training data points. In this over-parameterized regime, the training loss typically has infinitely many global optima that completely interpolate the data with varying generalization performance. The particular global optimum we converge to depends on the implicit bias of the optimization algorithm. The question we address in this paper is, ``What is the implicit bias that leads to the best generalization performance?". To find the optimal implicit bias, we provide a precise asymptotic analysis of the generalization performance of interpolators obtained from the minimization of convex functions/potentials for over-parameterized linear regression with non-isotropic Gaussian data. In particular, we obtain a tight lower bound on the best generalization error possible among this class of interpolators in terms of the over-parameterization ratio, the variance of the noise in the labels, the eigenspectrum of the data covariance, and the underlying distribution of the parameter to be estimated. Finally, we find the optimal convex implicit bias that achieves this lower bound under certain sufficient conditions involving the log-concavity of the distribution of a Gaussian convolved with the prior of the true underlying parameter.

cs.LG↗

arXiv2025

Beyond Quadratic Costs in LQR: Bregman Divergence Control

In the past couple of decades, the use of ``non-quadratic" convex cost functions has revolutionized signal processing, machine learning, and statistics, allowing one to customize solutions to have desired structures and properties. However, the situation is not the same in control where the use of quadratic costs still dominates, ostensibly because determining the ``value function", i.e., the optimal expected cost-to-go, which is critical to the construction of the optimal controller, becomes computationally intractable as soon as one considers general convex costs. As a result, practitioners often resort to heuristics and approximations, such as model predictive control that only looks a few steps into the future. In the quadratic case, the value function is easily determined by solving Riccati equations. In this work, we consider a special class of convex cost functions constructed from Bregman divergence and show how, with appropriate choices, they can be used to fully extend the framework developed for the quadratic case. The resulting optimal controllers are infinite horizon, come with stability guarantees, and have state-feedback, or estimated state-feedback, laws. They exhibit a much wider range of behavior than their quadratic counterparts since the feedback laws are nonlinear. The approach can be applied to several cases of interest, including safety control, sparse control, and bang-bang control.

eess.SY↗

arXiv2025

Universality in Transfer Learning for Linear Models

We study the problem of transfer learning and fine-tuning in linear models for both regression and binary classification. In particular, we consider the use of stochastic gradient descent (SGD) on a linear model initialized with pretrained weights and using a small training data set from the target distribution. In the asymptotic regime of large models, we provide an exact and rigorous analysis and relate the generalization errors (in regression) and classification errors (in binary classification) for the pretrained and fine-tuned models. In particular, we give conditions under which the fine-tuned model outperforms the pretrained one. An important aspect of our work is that all the results are "universal", in the sense that they depend only on the first and second order statistics of the target distribution. They thus extend well beyond the standard Gaussian assumptions commonly made in the literature. Furthermore, our universality results extend beyond standard SGD training to the test error of a classification task trained using a ridge regression.

cs.LG↗

arXiv2025

Robust Mean Estimation With Auxiliary Samples

In data-driven learning and inference tasks, the high cost of acquiring samples from the target distribution often limits performance. A common strategy to mitigate this challenge is to augment the limited target samples with data from a more accessible "auxiliary" distribution. This paper establishes fundamental limits of this approach by analyzing the improvement in the mean square error (MSE) when estimating the mean of the target distribution. Using the Wasserstein-2 metric to quantify the distance between distributions, we derive expressions for the worst-case MSE when samples are drawn (with labels) from both a target distribution and an auxiliary distribution within a specified Wasserstein-2 distance from the target distribution. We explicitly characterize the achievable MSE and the optimal estimator in terms of the problem dimension, the number of samples from the target and auxiliary distributions, the Wasserstein-2 distance, and the covariance of the target distribution. We note that utilizing samples from the auxiliary distribution effectively improves the MSE when the squared radius of the Wasserstein-2 uncertainty ball is small compared to the variance of the true distribution and the number of samples from the true distribution is limited. Numerical simulations in the Gaussian location model illustrate the theoretical findings.

math.ST↗