arXiv subjects

Sumanta Basu

Publications and source records attributed to Sumanta Basu.

At least 19 recordsLinked to original sources

arXiv2026

Inference for High-Dimensional Sparse Spectral Precision Matrices

Statistical inference for the spectral precision matrix at a given frequency allows us to assess frequency-specific conditional relationships among the components of a stationary multivariate time series. Compared with classical analogues, inference in the spectral domain is more challenging due to the absence of closed-form asymptotic variance expressions for complex-valued estimators, the limited asymptotic theory in high-dimensional settings, and the presence of truncation and smoothing biases in finite samples. We construct a debiased complex graphical lasso estimator at any fixed frequency by leveraging the full likelihood structure of neighboring discrete Fourier transforms. Using asymptotic distributions for bilinear forms of stationary multivariate time series, we establish the joint asymptotic normality of the real and imaginary parts of the debiased estimator. Our main theoretical contributions include deriving a closed-form asymptotic covariance matrix for the real and imaginary parts, establishing a central limit theorem for bilinear forms of the underlying time series, and controlling smoothing and truncation biases in covariance estimation to ensure valid inference. Simulation studies demonstrate reliable coverage and improved statistical power relative to the benchmark, while maintaining false discovery rates near the nominal level. An application to real fMRI data further reveals distinct patterns of functional connectivity across selected frequencies.

stat.ME↗

arXiv2026

LARGE: A Locally Adaptive Regularization Approach for Estimating Gaussian Graphical Models

The graphical Lasso (GLASSO) is a widely used algorithm for learning high-dimensional undirected Gaussian graphical models (GGM). Given i.i.d. observations from a multivariate normal distribution, GLASSO estimates the precision matrix by maximizing the log-likelihood with an \ell_1-penalty on the off-diagonal entries. However, selecting an optimal regularization parameter \lambda in this unsupervised setting remains a significant challenge. A well-known issue is that existing methods, such as out-of-sample likelihood maximization, select a single global \lambda and do not account for heterogeneity in variable scaling or partial variances. Standardizing the data to unit variances, although a common workaround, has been shown to negatively affect graph recovery. Addressing the problem of nodewise adaptive tuning in graph estimation is crucial for applications like computational neuroscience, where brain networks are constructed from highly heterogeneous, region-specific fMRI data. In this work, we develop Locally Adaptive Regularization for Graph Estimation (LARGE), an approach to adaptively learn nodewise tuning parameters to improve graph estimation and selection. In each block coordinate descent step of GLASSO, we augment the nodewise Lasso regression to jointly estimate the regression coefficients and error variance, which in turn guides the adaptive learning of nodewise penalties. In simulations, LARGE consistently outperforms benchmark methods in graph recovery, demonstrates greater stability across replications, and achieves the best estimation accuracy in the most difficult simulation settings. We demonstrate the practical utility of our method by estimating brain functional connectivity from a real fMRI data set.

stat.ME↗

arXiv2025

Autotune: fast, accurate, and automatic tuning parameter selection for Lasso

Least absolute shrinkage and selection operator (Lasso), the popular variable selection engine for high-dimensional regression, is commonly tuned using cross-validation (CV). This is known to be slow and loses accuracy in high-dimension, low signal-to-noise ratio (SNR) settings. These issues are exacerbated for high-dimensional time series models such as the vector autoregression (VAR), where time series cross-validaiton (TSCV) is used for tuning. We propose $\texttt{autotune}$, a strategy for the Lasso to tune itself automatically by exploiting information contained in partial residuals computed during a single Lasso fit. The strategy can also be viewed as alternately estimating noise standard deviation and column space of relevant predictors. Numerical experiments on regression and VAR models show that $\texttt{autotune}$ is faster than existing alternatives, and more accurate when the SNR is low. It also provides an accurate estimator of noise scale and diagnostic plots akin to screeplots for checking model sparsity. We demonstrate the benefit of $\texttt{autotune}$ on several real data sets and develop an R package available on CRAN.

stat.ME↗

arXiv2025

Counterfactual Forecasting for Panel Data

We address the challenge of forecasting counterfactual outcomes in a panel data with missing entries and temporally dependent latent factors -- a common scenario in causal inference, where estimating unobserved potential outcomes ahead of time is essential. We propose Forecasting Counterfactuals under Stochastic Dynamics (FOCUS), a method that extends traditional matrix completion methods by leveraging time series dynamics of the factors, thereby enhancing the prediction accuracy of future counterfactuals. Building upon a consistent estimator of the factors, our method accommodates both stochastic and deterministic components within the factors, and provides a flexible framework for various applications. In case of stationary autoregressive factors and under standard conditions, we derive error bounds and establish asymptotic normality of our estimator. Empirical evaluations demonstrate that our method outperforms existing benchmarks when the latent factors have an autoregressive component. We illustrate FOCUS results on HeartSteps, a mobile health study, illustrating its effectiveness in forecasting step counts for users receiving activity prompts, thereby leveraging temporal patterns in user behavior.

stat.ME↗

arXiv2025

Identification of Emotionally Stressful Periods Through Tracking Changes in Statistical Features of mHealth Data

Identifying the onset of emotional stress in older patients with mood disorders and chronic pain is crucial in mental health studies. To this end, studying the associations between passively sensed variables that measure human behaviors and self-reported stress levels collected from mobile devices is emerging. Existing algorithms rely on conventional change point detection (CPD) methods due to the nonstationary nature of the data. They also require explicit modeling of the associations between variables and output only discrete time points, which can lead to misinterpretation of stress onset timings. This is problematic when distributional shifts are complex, dependencies between variables are difficult to capture, and changes occur asynchronously across series with weak signals. In this study, we propose an algorithm that detects hotspots, defined as collections of time intervals during which statistical features of passive sensing variables and stress indicators shift, highlighting periods that require investigation. We first extend the moving sum (MOSUM) scheme to detect simultaneous changes both within and across series, and then define hotspots in two ways: using distance-based test statistics and confidence intervals. The proposed method tracks local changes in combined distributional features, enabling it to capture all types of simultaneous and asynchronous change. It does not require a specific functional relationship between series, and the results are expressed as intervals rather than as individual time points. We conduct simulations under varying signal strengths with mixed and asynchronous distributional shifts, where the proposed method outperforms benchmarks. Results on hotspot identification indicate that the two definitions are complementary. We further apply our method to ALACRITY Phase I data, analyzing hotspots from patients' stress levels and activity measures.

stat.ME↗

arXiv2025

Exact Coordinate Descent for High-Dimensional Regularized Huber Regression

This study develops a coordinate descent algorithm for high-dimensional Huber regression with an elastic-net penalty. Unlike existing gradient descent algorithms and coordinate descent methods based on second-order optimization, the proposed algorithm performs each coordinate update exactly, without approximation or gradient and Hessian computations. Consequently, it remains stable and converges rapidly even in the presence of highly correlated covariates or heavy-tailed noise. Building on these exact updates, adaptive variable screening rules and optimality-condition validation procedures are introduced to determine which coordinates to update at each iteration, substantially accelerating convergence in high-dimensional settings. Theoretical guarantees are established for both the convergence of the proposed algorithm and the validity of validation procedures. Extensive simulation studies under heavy-tailed noise and highly correlated designs, together with a real-data application, demonstrate that the proposed method is both accurate and computationally efficient in these challenging settings.

stat.ME↗

arXiv2025

BLOG: Bayesian Longitudinal Omics with Group Constraints

Clinical investigators are increasingly interested in discovering computational biomarkers from short-term longitudinal omics data sets. This work focuses on Bayesian regression and variable selection for longitudinal omics datasets, which can quantify uncertainty and control false discovery. In our univariate approach, Zellner's $g$ prior is used with two different options of the tuning parameter $g$: $g=\sqrt{n}$ and a $g$ that minimizes Stein's unbiased risk estimate (SURE). Bayes Factors were used to quantify uncertainty and control for false discovery. In the multivariate approach, we use Bayesian Group LASSO with a spike and slab prior for group variable selection. In both approaches, we use the first difference ($\Delta$) scale of longitudinal predictor and the response. These methods work together to enhance our understanding of biomarker identification, improving inference and prediction. We compare our method against commonly used linear mixed effect models on simulated data and real data from a Tuberculosis (TB) study on metabolite biomarker selection. With an automated selection of hyperparameters, the Zellner's $g$ prior approach correctly identifies target metabolites with high specificity and sensitivity across various simulation and real data scenarios. The Multivariate Bayesian Group Lasso spike and slab approach also correctly selects target metabolites across various simulation scenarios.

stat.ME↗

arXiv2025

A Pathwise Coordinate Descent Algorithm for LASSO Penalized Quantile Regression

$\ell_1$ penalized quantile regression is used in many fields as an alternative to penalized least squares regressions for high-dimensional data analysis. Existing algorithms for penalized quantile regression either use linear programming, which does not scale well in high dimension, or an approximate coordinate descent (CD) which does not solve for exact coordinatewise minimum of the nonsmooth loss function. Further, neither approaches build fast, pathwise algorithms commonly used in high-dimensional statistics to leverage sparsity structure of the problem in large-scale data sets. To avoid the computational challenges associated with the nonsmooth quantile loss, some recent works have even advocated using smooth approximations to the exact problem. In this work, we develop a fast, pathwise coordinate descent algorithm to compute exact $\ell_1$ penalized quantile regression estimates for high-dimensional data. We derive an easy-to-compute exact solution for the coordinatewise nonsmooth loss minimization, which, to the best of our knowledge, has not been reported in the literature. We also employ a random perturbation strategy to help the algorithm avoid getting stuck along the regularization path. In simulated data sets, we show that our algorithm runs substantially faster than existing alternatives based on approximate CD and linear program, while retaining the same level of estimation accuracy.

stat.CO↗

arXiv2025

A co-segmentation algorithm to predict emotional stress from passively sensed mHealth data

We develop a data-driven co-segmentation algorithm of passively sensed and self-reported active variables collected through smartphones to identify emotionally stressful states in middle-aged and older patients with mood disorders undergoing therapy, some of whom also have chronic pain. Our method leverages the association between the different types of time series. These data are typically non-stationary, with meaningful associations often occurring only over short time windows. Traditional machine learning (ML) methods, when applied globally on the entire time series, often fail to capture these time-varying local patterns. Our approach first segments the passive sensing variables by detecting their change points, then examines segment-specific associations with the active variable to identify co-segmented periods that exhibit distinct relationships between stress and passively sensed measures. We then use these periods to predict future emotional stress states using standard ML methods. By shifting the unit of analysis from individual time points to data-driven segments of time and allowing for different associations in different segments, our algorithm helps detect patterns that only exist within short-time windows. We apply our method to detect periods of stress in patient data collected during ALACRITY Phase I study. Our findings indicate that the data-driven segmentation algorithm identifies stress periods more accurately than traditional ML methods that do not incorporate segmentation.

stat.AP↗

arXiv2024

Regularized Estimation of Sparse Spectral Precision Matrices

Estimation of a sparse spectral precision matrix, the inverse of a spectral density matrix, is a canonical problem in frequency-domain analysis of high-dimensional time series (HDTS), with applications in neurosciences and environmental sciences. Existing estimators use off-the-shelf optimizers for complex variables that limit scalability, uniform (non-adaptive) penalization that is not tailored to handle heterogeneity across time series components, and lack a formal non-asymptotic theory that systematically analyzes approximation and estimation errors in high-dimension. In this work, develop fast pathwise coordinate descent (CD) algorithms and non-asymptotic theory for a complex graphical lasso (CGLASSO) and an adaptive version CAGLASSO, that adapts penalization to the underlying scale of variability. For fast algorithms, we devise a realification procedure based on ring isomorphism, a notion from abstract algebra, that can be used for other high-dimensional optimization problems over complex variables. Our non-asymptotic analysis shows that consistency is possible in high-dimension under suitable sparsity assumptions. A key step is to separately bound the approximation and estimation error arising from treating the finite-sample discrete Fourier Transforms (DFTs) as i.i.d. complex-valued data, an issue well-addressed in classical time series but relatively less explored in HDTS literature. We demonstrate the performance of our proposed estimators in several simulated data sets and a real data application from neuroscience.

stat.ME↗

arXiv2023

Sparse reconstruction of ordinary differential equations with inference

Sparse regression has emerged as a popular technique for learning dynamical systems from temporal data, beginning with the SINDy (Sparse Identification of Nonlinear Dynamics) framework proposed by arXiv:1509.03580. Quantifying the uncertainty inherent in differential equations learned from data remains an open problem, thus we propose leveraging recent advances in statistical inference for sparse regression to address this issue. Focusing on systems of ordinary differential equations (ODEs), SINDy assumes that each equation is a parsimonious linear combination of a few candidate functions, such as polynomials, and uses methods such as sequentially-thresholded least squares or the Lasso to identify a small subset of these functions that govern the system's dynamics. We instead employ bias-corrected versions of the Lasso and ridge regression estimators, as well as an empirical Bayes variable selection technique known as SEMMS, to estimate each ODE as a linear combination of terms that are statistically significant. We demonstrate through simulations that this approach allows us to recover the functional terms that correctly describe the dynamics more often than existing methods that do not account for uncertainty.

stat.ME↗

arXiv2023

Modeling Multiple Irregularly Spaced Financial Time Series

In this paper we propose univariate volatility models for irregularly spaced financial time series by modifying the regularly spaced stochastic volatility models. We also extend this approach to propose multivariate stochastic volatility (MSV) models for multiple irregularly spaced time series by modifying the MSV model that was used with daily data. We use these proposed models for modeling intraday logarithmic returns from health sector stocks data obtained from Trade and Quotes (TAQ) database at Wharton Research Data Services (WRDS).

stat.AP↗

arXiv2022

Learning Financial Networks with High-frequency Trade Data

Financial networks are typically estimated by applying standard time series analyses to price-based economic variables collected at low-frequency (e.g., daily or monthly stock returns or realized volatility). These networks are used for risk monitoring and for studying information flows in financial markets. High-frequency intraday trade data sets may provide additional insights into network linkages by leveraging high-resolution information. However, such data sets pose significant modeling challenges due to their asynchronous nature, nonlinear dynamics, and nonstationarity. To tackle these challenges, we estimate financial networks using random forests. The edges in our network are determined by using microstructure measures of one firm to forecast the sign of the change in a market measure (either realized volatility or returns kurtosis) of another firm. We first investigate the evolution of network connectivity in the period leading up to the U.S. financial crisis of 2007-09. We find that the networks have the highest density in 2007, with high degree connectivity associated with Lehman Brothers in 2006. A second analysis into the nature of linkages among firms suggests that larger firms tend to offer better predictive power than smaller firms, a finding qualitatively consistent with prior works in the market microstructure literature.

q-fin.ST↗

arXiv2022

Exploring Financial Networks Using Quantile Regression and Granger Causality

In the post-crisis era, financial regulators and policymakers are increasingly interested in data-driven tools to measure systemic risk and to identify systemically important firms. Granger Causality (GC) based techniques to build networks among financial firms using time series of their stock returns have received significant attention in recent years. Existing GC network methods model conditional means, and do not distinguish between connectivity in lower and upper tails of the return distribution - an aspect crucial for systemic risk analysis. We propose statistical methods that measure connectivity in the financial sector using system-wide tail-based analysis and is able to distinguish between connectivity in lower and upper tails of the return distribution. This is achieved using bivariate and multivariate GC analysis based on regular and Lasso penalized quantile regressions, an approach we call quantile Granger causality (QGC). By considering centrality measures of these financial networks, we can assess the build-up of systemic risk and identify risk propagation channels. We provide an asymptotic theory of QGC estimators under a quantile vector autoregressive model, and show its benefit over regular GC analysis on simulated data. We apply our method to the monthly stock returns of large U.S. firms and demonstrate that lower tail based networks can detect systemically risky periods in historical data with higher accuracy than mean-based networks. In a similar analysis of large Indian banks, we find that upper and lower tail networks convey different information and have the potential to distinguish between periods of high connectivity that are governed by positive vs negative news in the market.

q-fin.ST↗

arXiv2022

Modeling Multivariate Positive-Valued Time Series Using R-INLA

In this paper we describe fast Bayesian statistical analysis of vector positive-valued time series, with application to interesting financial data streams. We discuss a flexible level correlated model (LCM) framework for building hierarchical models for vector positive-valued time series. The LCM allows us to combine marginal gamma distributions for the positive-valued component responses, while accounting for association among the components at a latent level. We use integrated nested Laplace approximation (INLA) for fast approximate Bayesian modeling via the R-INLA package, building custom functions to handle this setup. We use the proposed method to model interdependencies between realized volatility measures from several stock indexes.

stat.ME↗

arXiv2021

An empirical Bayes approach to estimating dynamic models of co-regulated gene expression

Time-course gene expression datasets provide insight into the dynamics of complex biological processes, such as immune response and organ development. It is of interest to identify genes with similar temporal expression patterns because such genes are often biologically related. However, this task is challenging due to the high dimensionality of these datasets and the nonlinearity of gene expression time dynamics. We propose an empirical Bayes approach to estimating ordinary differential equation (ODE) models of gene expression, from which we derive a similarity metric between genes called the Bayesian lead-lag $R^2$ (LLR2). Importantly, the calculation of the LLR2 leverages biological databases that document known interactions amongst genes; this information is automatically used to define informative prior distributions on the ODE model's parameters. As a result, the LLR2 is a biologically-informed metric that can be used to identify clusters or networks of functionally-related genes with co-moving or time-delayed expression patterns. We then derive data-driven shrinkage parameters from Stein's unbiased risk estimate that optimally balance the ODE model's fit to both data and external biological information. Using real gene expression data, we demonstrate that our methodology allows us to recover interpretable gene clusters and sparse networks. These results reveal new insights about the dynamics of biological systems.

stat.ME↗

arXiv2021

Graphical models for nonstationary time series

We propose NonStGM, a general nonparametric graphical modeling framework for studying dynamic associations among the components of a nonstationary multivariate time series. It builds on the framework of Gaussian Graphical Models (GGM) and stationary time series Gaussian Graphical model (StGM), and complements existing works on parametric graphical models based on change point vector autoregressions (VAR). Analogous to StGM, the proposed framework captures conditional noncorrelations (both intertemporal and contemporaneous) in the form of an undirected graph. In addition, to describe the more nuanced nonstationary relationships among the components of the time series, we introduce the new notion of conditional nonstationarity/stationarity and incorporate it within the graph architecture. This allows one to distinguish between direct and indirect nonstationary relationships among system components, and can be used to search for small subnetworks that serve as the "source" of nonstationarity in a large system. Together, the two concepts of conditional noncorrelation and nonstationarity/stationarity provide a parsimonious description of the dependence structure of the time series.

math.ST↗

arXiv2021

A Survey of Estimation Methods for Sparse High-dimensional Time Series Models

High-dimensional time series datasets are becoming increasingly common in many areas of biological and social sciences. Some important applications include gene regulatory network reconstruction using time course gene expression data, brain connectivity analysis from neuroimaging data, structural analysis of a large panel of macroeconomic indicators, and studying linkages among financial firms for more robust financial regulation. These applications have led to renewed interest in developing principled statistical methods and theory for estimating large time series models given only a relatively small number of temporally dependent samples. Sparse modeling approaches have gained popularity over the last two decades in statistics and machine learning for their interpretability and predictive accuracy. Although there is a rich literature on several sparsity inducing methods when samples are independent, research on the statistical properties of these methods for estimating time series models is still in progress. We survey some recent advances in this area, focusing on empirically successful lasso based estimation methods for two canonical multivariate time series models - stochastic regression and vector autoregression. We discuss key technical challenges arising in high-dimensional time series analysis and outline several interesting research directions.

stat.ME↗