arXiv Science⌕ Search

arXiv subjects

Rajarshi Mukherjee

Publications and source records attributed to Rajarshi Mukherjee.

At least 19 recordsLinked to original sources

Sharp minimax risks and phase transitions in sparse submatrix detection

In this paper, we study the minimax risk of detecting a sparse, elevated-mean Gaussian submatrix planted in a larger noisy matrix. When the planted submatrix has size $n\times n$ and the ambient matrix has size $N\times N$ with $N=n^{1+α}$, the seminal work of Butucea and Ingster (2013) identifies the sharp detection boundary around which the risk converges to $0$ or $1$. We extend that zero-one theory by determining sharp risk asymptotics throughout a two-parameter phase diagram. Above the boundary, the risk decays stretched- or super-exponentially fast, and we determine its precise exponent. Below the boundary, where the risk tends to $1$, we obtain matching absolute-constant bounds for its polynomial rate of convergence. In both regimes, the asymptotics change across the line $α+δ=1/2$, where $δ$ parametrizes the signed distance from the detection boundary. Finally, on the boundary, we show that the minimax risk converges to the nondegenerate constant $\frac12$ in a very sparse regime where $n$ remains fixed and $N\to\infty$. Each of these rates corresponds to the risk of a suitably calibrated scan or sum test, whence follow the upper bounds. The main problem is to derive the matching regime-specific lower bounds. The mixture likelihood ratio is an average rather than a product and thus its logarithm lacks the additive structure required by standard large deviation formulations. On the other hand, classical second-moment estimates fail to be sharp due to overlap-induced correlations. To address these issues, we develop two general lower-bound principles based on truncated mixture likelihoods, tailored to the target risks on both sides of the detection boundary. The same framework yields sharp risk asymptotics for sparse subtensors with equal side lengths and also applies to more general testing problems including the detection of rectangular subtensors.

math.ST↗

A Statistical Framework for Understanding Causal Effects that Vary by Treatment Initiation Time in EHR-based Studies

Standard practice in electronic health record (EHR)-based studies evaluating the comparative effectiveness of bariatric surgery relative to no surgery is to estimate and report a constant treatment effect across calendar time. However, real-world treatment strategies can evolve, particularly when comparators include standard of care or surgical procedures where techniques may improve, making it clinically important to ascertain whether efficacy of bariatric surgery has changed over time. Efforts to determine whether treatment efficacy itself is evolving are complicated by changing patient populations, with potential covariate shift in key effect modifiers. Through a comprehensive analysis of EHR data from Kaiser Permanente comparing two bariatric surgical procedures to standard of care, we develop a statistical framework to estimate calendar time-specific average treatment effects and describe both how and why effects vary across treatment initiation time in EHR-based studies. Our approach projects doubly robust, time-specific treatment effect estimates onto candidate marginal structural models and uses model selection to best describe how effects vary by calendar time. We further introduce a novel summary metric, based on standardization, to quantify the degree to which observed effect changes may be attributed to covariate shift in the population receiving treatment, as opposed to changes in underlying treatment efficacy.

stat.ME↗

A plug-in estimator of the non-iterative conditional expectation g-formula using long short-term memory models

The non-iterative conditional expectation (NICE) plug-in estimator of the g-formula can be used to estimate the causal effects of sustained treatment strategies, even in the presence of treatment-confounder feedback, under the assumptions of exchangeability, positivity, and consistency. In practical applications, parametric models are commonly used to estimate the joint distribution of confounders, treatments, and outcomes over time. As an alternative to parametric models, long short-term memory (LSTM) models, designed to capture both short-term patterns and long-range dependencies in sequential data, may better accommodate complex temporal dependencies among time-varying confounders, treatments, and outcomes. However, the validity of statistical inferences of a plug-in estimator based on LSTM models has not yet been investigated. Here, we provide proof of principle that, under a complex data generation mechanism that resembles data from a cohort of persons with HIV and sufficiently large sample sizes, an LSTM-based NICE g-formula plug-in estimator results in less bias than parametric estimators while achieving near-nominal coverage for 95\% confidence intervals. This work is intended to spur the development of a theoretical framework that encompasses deep learning plug-in estimators for causal inference based on high-dimensional longitudinal data.

stat.ME↗

Robust Causal Inference for EHR-based Studies of Point Exposures with Missingness in Eligibility Criteria

Missingness in variables that define study eligibility criteria is a seldom addressed challenge in electronic health record (EHR)-based settings. Patients with incomplete eligibility information are typically excluded from analysis without consideration of the (implicit) assumptions being made, leaving study conclusions subject to potential selection bias. In an effort to ascertain eligibility for more patients, researchers may look back further in time prior to study baseline, inappropriately including individuals who, unbeknownst to the researcher, would fail to meet eligibility at baseline. To the best of our knowledge, however, little work has been done to mitigate these concerns. We propose a robust and efficient estimator of the causal average treatment effect on the treated, defined in the study-eligible population, where eligibility-defining covariates are missing at random. The approach facilitates flexible machine-learning strategies for component nuisance functions while maintaining valid asymptotic inference. This method is applied to EHR data from Kaiser Permanente on 14,809 patients undergoing bariatric surgery, comparing two surgical interventions for long-term weight and glycemic outcomes among severely obese patients with type II diabetes mellitus. Our results confirm a weight-loss advantage for Roux-en-Y gastric bypass over sleeve gastrectomy, but are less suggestive of a diabetes remission advantage than complete-case analyses indicate.

stat.ME↗

Statistical Inference for Continuous Action Bandits

Statistical inference under adaptively collected data is becoming increasingly popular across e-commerce and mobile health. Many methods exist for discrete action settings, ranging from weighting to debiasing approaches. Despite the ubiquity of continuous actions in experimentation from optimal pricing to precision dosing, to the best of our knowledge, methods that support statistical inference under continuous action, adaptively collected data remain underdeveloped. In this work, we extend kernel-smoothed doubly robust estimation from i.i.d. data to adaptively collected data with continuous actions. We study a family of adaptively weighted estimators, establish their mean squared error rates, asymptotic normality, and characterize an estimation lower bound as a function of regret. We conclude with recommendations for experimental design informing efficiency and regret, and support our theory with an extensive simulation study.

stat.ME↗

Improved Variance Estimation in Homoskedastic Nonparametric Random-Design Regression via a Two-Scale Approach

We study estimation of a constant conditional variance $σ^2$ in nonparametric regression with a $d$-dimensional random design. This is an important problem, and similar questions arise in causal inference. The regression function is $β_b$-Hölder smooth, the design density is $β_g$-Hölder smooth and bounded above and away from zero, and we consider the nonparametric regime $β_b>1$ and $d>4β_b$. Set $β_g^\star=β_b(1-4β_b/d)/\{1+2β_b/d+8(β_b/d)^2\}$. We give an estimator whose mean squared error is upper bounded by $Cn^{-4(β_b+1)/(d+4)}$ in the low-regularity regime when $0<β_g\leqβ_g^\star$. The low-regularity branch is based on a new two-scale construction: the covariate space is partitioned into cells, the local polynomial trend is projected out within each suitable cell, and the squared normalized contrast from one eligible close pair per cell is averaged across cells. In the high regularity regime when $β_g>β_g^\star$, a higher-order influence function estimator of Robins, Li, Tchetgen Tchetgen, and van der Vaart (2008) provides the rate $Cn^{-8β_b/(d+4β_b)}$. We also give an all-pairs ridge extension, which achieves the same two-scale rate, and evaluate the methods alongside a range of existing estimators in simulations.

math.ST↗

Nuisance Function Tuning and Sample Splitting for Optimally Estimating a Doubly Robust Functional

Estimators of doubly robust functionals typically rely on estimating two complex nuisance functions, such as the propensity score and conditional outcome mean for the average treatment effect functional. We consider the problem of how to estimate nuisance functions to obtain optimal rates of convergence for a doubly robust nonparametric functional that has witnessed applications across the causal inference and conditional independence testing literature. For several plug-in estimators and a first-order bias-corrected estimator, we illustrate the interplay between different tuning parameter choices for the nuisance function estimators and sample splitting strategies on the optimal rate of estimating the functional of interest. For each of these estimators and each sample splitting strategy, we show the necessity to either undersmooth or oversmooth the nuisance function estimators under low regularity conditions to obtain optimal rates of convergence for the functional of interest. Unlike the existing literature, we show that plug-in and first-order bias-corrected estimators can achieve minimax rates of convergence across all Hölder smoothness classes of the nuisance functions by careful combinations of sample splitting and nuisance function tuning strategies. We complement these results with numerical simulations illustrating the impact of different nuisance function tuning and sample splitting strategies.

math.ST↗

On the Asymptotic Inadmissibility of Double Machine Learning Estimators Under Structure-Agnostic Models

Structure-agnostic (SA) models introduced by Balakrishnan et al. (2026) aim to reflect the general lack of knowledge of structural assumptions on data-generating laws such as smoothness or sparsity in practice. Roughly speaking, SA models restrict the observed-data generating law to be in some rn-neighborhood of (black-box machine learning) estimates, treated as given and fixed, where rn encodes the convergence rates of the estimates to the truth. Under SA models, Balakrishnan et al. (2026) show that the popular Double Machine Learning (DML) estimators for three functionals, the quadratic functional in the Gaussian sequence model, the quadratic density integral functional and the expected conditional covariance, are minimax. However, minimax estimators may be inadmissible. In this paper, we show that, for the first two of the three functionals, the DML estimator is asymptotically inadmissible under the SA model. In particular, we show that these two functionals fall into a class of functionals, which we refer to as the monotone bias class. For this class, we exhibit second-order (U-statistic) estimators, which asymptotically dominate DML estimators, under the SA model. These second-order estimators are empirical higher-order influence function (HOIF) estimators introduced in Liu et al. (2017). Furthermore, the empirical HOIF estimator, like the DML estimator, is minimax for the third functional (the expected conditional covariance), although neither asymptotically dominates the other.

math.ST↗

Pooling Versus Ensembling for Ridge Regression Under Covariate Shift

Datasets in many settings naturally partition into clusters arising from sub-populations, batch effects, or aggregation across multiple sources. A common response to such heterogeneity is to ensemble learners trained on each cluster rather than fit a single model to the pooled data. Prior work motivating such approaches has typically considered settings in which both the covariate distribution and the conditional outcome model differ across clusters; the role of cluster-aware partitioning and ensembling based solely on the covariate distribution remains to be explored. We address this case for ridge-regularized least-squares regression under a linear outcome model and consider all ridge penalty values $λ\geq 0$, including the special case of the ridgeless predictor at $λ= 0$. By considering both fixed-effects and random-effects models, we argue that under random effects, an optimally tuned pooled ridge predictor always outperforms ensembles of individually optimally tuned predictors. For fixed effects, we derive a general formula for the pooled and ensembled predictors to characterize the role of both regression coefficients as well as the predictor distribution shifts. Together, these results generalize prior risk analyses of bagging and random-partition estimation using ridge and ridgeless regression predictors from the i.i.d. setting to encompass covariate shift and heterogeneity-aware partition structure.

math.ST↗

Toward Joint Prediction of a Longitudinal Marker and a Terminal Event: A bivariate discrete-time framework

Sudden cardiac death (SCD) is a leading cause of death in the U.S. Patients at elevated risk of SCD are primarily treated with an implantable cardioverter-defibrillator (ICD), which may prevent death from cardiovascular causes but may cause severe side effects, such as reduced quality of life from shock-induced pain. Decisions about ICD treatment therefore involve complex personal trade-offs across multiple health events, including mortality and quality of life. While prediction tools could help weigh these trade-offs, they commonly focus on univariate outcomes; at best, they treat other clinical endpoints as inputs, so trade-offs cannot be directly informed. To address this, we propose a novel general Bayesian framework that jointly models a terminal event and a longitudinal marker as a bivariate process over discrete time, for settings where prediction is the primary goal. Discretization of study time lets the framework capture the dynamic interplay between outcomes while avoiding implicit extrapolation beyond truncation by a terminal event. The framework flexibly accommodates phenomena arising in applied contexts, including global time-invariant and local time-dependent dependence structures between the terminal event and the longitudinal marker, and latent association via a shared frailty term. Estimation proceeds via the Bayesian paradigm, yielding patient-specific joint posterior predictions for the time to terminal event and the future marker trajectory. We introduce the framework with a focus on its modeling flexibility, provide guidance on discretization and on Bayesian model construction and selection, and discuss insights from the joint posterior predictions. Finally, we demonstrate the framework's clinical relevance and practicality for SCD and ICD therapy using data from the Sudden Cardiac Death in Heart Failure Trial (SCD-HeFT), an important ICD-related benchmark trial.

stat.ME↗

Cross-Cluster Weighted Forests

Building trustworthy machine learning algorithms for biological applications requires adapting to data heterogeneity from different sources, batches, distributions, or studies. We propose the 'Cross-Cluster Weighted Forest' (CCWF), an ensembling approach that explicitly leverages heterogeneity in the feature distribution to produce more accurate and more generalizable predictors than the standard Random Forest in cases when data can be naturally clustered. CCWF generalizes the RF architecture to an outer unsupervised layer, supervised subtasks, and ensembling. Specifically it involves unsupervised clustering of the training data, fitting a Random Forest on each cluster, and combining the forests via stacked regression weights that reward cross-cluster generalizability. We provide a theoretical analysis of an analytically tractable forest model showing that cluster-based ensembling is asymptotically more accurate than training a single forest on the full data, with the gain driven by bias reduction. In simulations, we find that CCWF is robust across data-generating regimes and outcome models; furthermore, we explore the influence of data partitioning and ensemble weighting strategies on the benefits of our method. Finally, we apply our approach to cancer molecular profiling and gene expression datasets that are naturally divisible into clusters; in both simulations and real data examples, we illustrate that our approach outperforms classic Random Forest by margins of 30-40%, aligning with our theoretical results. Overall, we show that CCWF provides a statistically grounded prediction algorithm for data spanning multiple domains or sub-populations, a structure common in biological applications.

stat.ML↗

Semiparametric Efficient Empirical Higher Order Influence Function Estimators

Robins et al. (2008, 2017) applied the theory of higher order influence functions (HOIFs) to derive an estimator of the mean $ψ$ of an outcome Y in a missing data model with Y missing at random conditional on a vector X of continuous covariates; their estimator, in contrast to previous estimators, is semiparametric efficient under the minimal conditions of Robins et al. (2009b), together with an additional (non-minimal) smoothness condition on the density g of X, because the Robins et al. (2008, 2017) estimator depends on a nonparametric estimate of g. In this paper, we introduce a new HOIF estimator that has the same asymptotic properties as the original one, but does not impose any smoothness requirement on g. This is important for two reasons. First, one rarely has the knowledge about the properties of g. Second, even when g is smooth, if the dimension of X is even moderate, accurate nonparametric estimation of its density is not feasible at the sample sizes often encountered in applications. In fact, to the best of our knowledge, this new HOIF estimator remains the only semiparametric efficient estimator of $ψ$ under minimal conditions, despite the rapidly growing literature on causal effect estimation. We also show that our estimator can be generalized to the entire class of functionals considered by Robins et al. (2008) which include the average effect of a treatment on a response Y when a vector X suffices to control confounding and the expected conditional variance of a response Y given a vector X. Simulation experiments are also conducted, which demonstrate that our new estimator outperforms those of Robins et al. (2008, 2017) in finite samples, when g is not very smooth.

math.ST↗

Asymptotic Inference for Constrained Regression

We consider statistical inference in high-dimensional regression problems under affine constraints on the parameter space. The theoretical study of this is motivated by the study of genetic determinants of diseases, such as diabetes, using external information from mediating protein expression levels. Specifically, we develop rigorous methods for estimating genetic effects on diabetes-related continuous outcomes when these associations are constrained based on external information about genetic determinants of proteins, and genetic relationships between proteins and the outcome of interest. In this regard, we discuss multiple candidate estimators and study their theoretical properties, sharp large sample optimality, and numerical qualities under a high-dimensional proportional asymptotic framework.

stat.ME↗

Optimal Nuisance Function Tuning for Estimating a Doubly Robust Functional under Proportional Asymptotics

In this paper, we explore the asymptotically optimal tuning parameter choice in ridge regression for estimating nuisance functions of a statistical functional that has recently gained prominence in conditional independence testing and causal inference. Given a sample of size $n$, we study estimators of the Expected Conditional Covariance (ECC) between variables $Y$ and $A$ given a high-dimensional covariate $X \in \mathbb{R}^p$. Under linear regression models for $Y$ and $A$ on $X$ and the proportional asymptotic regime $p/n \to c \in (0, \infty)$, we evaluate three existing ECC estimators and two sample splitting strategies for estimating the required nuisance functions. Since no consistent estimator of the nuisance functions exists in the proportional asymptotic regime without imposing further structure on the problem, we first derive debiased versions of the ECC estimators that utilize the ridge regression nuisance function estimators. We show that our bias correction strategy yields $\sqrt{n}$-consistent estimators of the ECC across different sample splitting strategies and estimator choices. We then derive the asymptotic variances of these debiased estimators to illustrate the nuanced interplay between the sample splitting strategy, estimator choice, and tuning parameters of the nuisance function estimators for optimally estimating the ECC. Our analysis reveals that prediction-optimal tuning parameters (i.e., those that optimally estimate the nuisance functions) may not lead to the lowest asymptotic variance of the ECC estimator -- thereby demonstrating the need to be careful in selecting tuning parameters based on the final goal of inference. Finally, we verify our theoretical results through extensive numerical experiments.

math.ST↗

Inference on Gaussian mixture models with dependent labels

Gaussian mixture models are widely used to model data generated from multiple latent sources. Despite its popularity, most theoretical research assumes that the labels are either independent and identically distributed, or follows a Markov chain. It remains unclear how the fundamental limits of estimation change under more complex dependence. In this paper, we address this question for the spherical two-component Gaussian mixture model. We first show that for labels with an arbitrary dependence, a naive estimator based on the misspecified likelihood is $\sqrt{n}$-consistent. Additionally, under labels that follow an Ising model, we establish the information theoretic limitations for estimation, and discover an interesting phase transition as dependence becomes stronger. When the dependence is smaller than a threshold, the optimal estimator and its limiting variance exactly matches the independent case, for a wide class of Ising models. On the other hand, under stronger dependence, estimation becomes easier and the naive estimator is no longer optimal. Hence, we propose an alternative estimator based on the variational approximation of the likelihood, and argue its optimality under a specific Ising model.

math.ST↗

PC Adjusted Testing for Low Dimensional Parameters

In this paper, we investigate the impact of high-dimensional Principal Component (PC) adjustments on inferring the effects of variables on outcomes, with a focus on applications in genetic association studies where PC adjustment is commonly used to account for population stratification. We consider high-dimensional linear regression in the regime where the number of covariates grows proportionally to the number of samples. In this setting, we provide an asymptotically precise understanding of when PC adjustments yield valid tests with controlled Type I error rates. Our results demonstrate that, under both fixed and diverging signal strengths, PC regression often fails to control the Type I error at the desired nominal level. Furthermore, we establish necessary and sufficient conditions for Type I error inflation based on covariate distributions. These theoretical findings are further supported by a series of numerical experiments.

math.ST↗

Method-of-Moments Inference for GLMs and Doubly Robust Functionals under Proportional Asymptotics

In this paper, we consider the estimation of regression coefficients and signal-to-noise (SNR) ratio in high-dimensional Generalized Linear Models (GLMs), and explore their implications in inferring popular estimands such as average treatment effects in high-dimensional observational studies. Under the ``proportional asymptotic'' regime and Gaussian covariates with known (population) covariance $Σ$, we derive Consistent and Asymptotically Normal (CAN) estimators of our targets of inference through a Method-of-Moments type of estimators that bypasses estimation of high dimensional nuisance functions and hyperparameter tuning altogether. Additionally, under non-Gaussian covariates, we demonstrate universality of our results under certain additional assumptions on the regression coefficients and $Σ$. We also demonstrate that knowing $Σ$ is not essential to our proposed methodology when the sample covariance matrix estimator is invertible. Finally, we complement our theoretical results with numerical experiments and comparisons with existing literature.

math.ST↗

Sensitivity analysis for nonignorable missing values in blended analysis framework: a study on the effect of bariatric surgery via electronic health records

This paper establishes a series of sensitivity analyses to investigate the impact of missing values in the electronic health records (EHR) that are possibly missing not at random (MNAR). EHRs have gained tremendous interest due to their cost-effectiveness, but their employment for research involves numerous challenges, such as selection bias due to missing data. The blended analysis has been suggested to overcome such challenges, which decomposes the data provenance into a sequence of sub-mechanisms and uses a combination of inverse-probability weighting (IPW) and multiple imputation (MI) under missing at random assumption (MAR). In this paper, we expand the blended analysis under the MNAR assumption and present a sensitivity analysis framework to investigate the effect of MNAR missing values on the analysis results. We illustrate the performance of my proposed framework via numerical studies and conclude with strategies for interpreting the results of sensitivity analyses. In addition, we present an application of our framework to the DURABLE data set, an EHR from a study examining long-term outcomes of patients who underwent bariatric surgery.

stat.ME↗