arXiv Science⌕ Search

arXiv subjects

Debsurya De

Publications and source records attributed to Debsurya De.

6 recordsLinked to original sources

Statistical-to-computational gaps in central limit theorems for high-dimensional tensors

We study high-dimensional central limit theorems (CLTs) as hypothesis testing problems, comparing the number of summands needed for statistical and computational indistinguishability of the normalized sum from the limiting Gaussian distribution as the dimension and number of summands grow together. For the off-diagonal entries of sums of i.i.d. rank-one tensors formed from standard Gaussian vectors (equivalently, the problem of distinguishing Wigner from Wishart tensors), we prove total variation bounds sharpening previous estimates of Mikulincer (2020) and give matching polynomial-time tests. This shows that the statistical and computational thresholds coincide for this CLT, contrary to a proposal by Kunisky, Moore, and Wein (2024). On the other hand, we also show that, for tensors of order at least three, sufficient Gaussian dilution of the summands by adding independent Gaussian noise and rescaling gives a CLT model with a statistical-to-computational gap (in the above setting and ones with more general rank-one summand distributions). To the best of our knowledge, such a phenomenon has not been observed before in any high-dimensional CLT. In particular, in the relevant parameter regimes, we show that a brute-force search achieves strong detection, while polynomials of degree up to $n^{Ω(1)}$ cannot achieve strong separation. The latter lower bounds control low-degree $χ^2$-divergences by a novel method based on cumulant generating functions, and we further illustrate the use of these tools by applying them to masked tensor CLT models where only a prescribed subset of tensor entries is revealed.

math.ST↗

Global synchronization for multi-source data integration under blockwise missing patterns

Multi-source data integration problems over datasets from different sources covering different but possibly overlapping sets of entities have become increasingly important in many applications, including genomics, single-cell analysis, and healthcare research. We study a model of multi-source data integration where we observe overlapping submatrices of a low-rank matrix, each corrupted by additive noise. In such problems, one often first learns a low-dimensional representation of the entities within each source and then integrates these representations across sources. As the representations are only identifiable up to some transformation, aligning them using the sources' overlapping entities becomes a key challenge. Existing methods align these estimates in a sequential or tree-structured manner, and are therefore sensitive to the chosen order and exploit only part of the overlapping information. We propose Global Synchronized Multiple Matrix Integration (GSMMI), which formulates this alignment problem as a global synchronization problem and jointly aligns all sources using all pairwise overlaps at once. We develop an efficient, scalable iterative algorithm for GSMMI, and show both theoretically and empirically that it improves alignment accuracy, with clear gains over prior approaches even under moderate overlap. We additionally establish information-theoretic and computational lower bounds for detection in data integration, and find GSMMI to be empirically near-optimal. GSMMI applies broadly across symmetric positive semidefinite, symmetric indefinite, and asymmetric or rectangular matrices, as well as settings where sources overlap only in rows or only in columns.

stat.ME↗

Computational and statistical lower bounds for low-rank estimation under general inhomogeneous noise

Recent work has generalized several results concerning the well-understood spiked Wigner matrix model of a low-rank signal matrix corrupted by additive i.i.d. Gaussian noise to the inhomogeneous case, where the noise has a variance profile. In particular, for the special case where the variance profile has a block structure, a series of results identified an effective spectral algorithm for detecting and estimating the signal, identified the threshold signal strength required for that algorithm to succeed, and proved information-theoretic lower bounds that, for some special signal distributions, match the above threshold. We complement these results by studying the computational optimality of this spectral algorithm. Namely, we show that, for a much broader range of signal distributions, whenever the spectral algorithm cannot detect a low-rank signal, then neither can any low-degree polynomial algorithm. This gives the first evidence for a computational hardness conjecture of Guionnet, Ko, Krzakala, and Zdeborová (2023). With similar techniques, we also prove sharp information-theoretic lower bounds for a class of signal distributions not treated by prior work. Unlike all of the above results on inhomogeneous models, our results do not assume that the variance profile has a block structure, and suggest that the same spectral algorithm might remain optimal for quite general profiles. We include a numerical study of this claim for an example of a smoothly-varying rather than piecewise-constant profile. Our proofs involve analyzing the graph sums of a matrix, which also appear in free and traffic probability, but we require new bounds on these quantities that are tighter than existing ones for non-negative matrices, which may be of independent interest.

math.ST↗

Clustering sequence data with mixture Markov chains with covariates using multiple simplex constrained optimization routine (MSiCOR)

Mixture Markov Model (MMM) is a widely used tool to cluster sequences of events coming from a finite state-space. However the MMM likelihood being multi-modal, the challenge remains in its maximization. Although Expectation-Maximization (EM) algorithm remains one of the most popular ways to estimate the MMM parameters, however convergence of EM algorithm is not always guaranteed. Given the computational challenges in maximizing the mixture likelihood on the constrained parameter space, we develop a pattern search-based global optimization technique which can optimize any objective function on a collection of simplexes, which is eventually used to maximize MMM likelihood. This is shown to outperform other related global optimization techniques. In simulation experiments, the proposed method is shown to outperform the expectation-maximization (EM) algorithm in the context of MMM estimation performance. The proposed method is applied to cluster Multiple sclerosis (MS) patients based on their treatment sequences of disease-modifying therapies (DMTs). We also propose a novel method to cluster people with MS based on DMT prescriptions and associated clinical features (covariates) using MMM with covariates. Based on the analysis, we divided MS patients into 3 clusters. Further cluster-specific summaries of relevant covariates indicate patient differences among the clusters.

math.OC↗

Estimating the Optimal Linear Combination of Biomarkers using Spherically Constrained Optimization

In the context of a binary classification problem, the optimal linear combination of continuous predictors can be estimated by maximizing an empirical estimate of the area under the receiver operating characteristic (ROC) curve (AUC). For multi-category responses, the optimal predictor combination can similarly be obtained by maximization of the empirical hypervolume under the manifold (HUM). This problem is particularly relevant to medical research, where it may be of interest to diagnose a disease with various subtypes or predict a multi-category outcome. Since the empirical HUM is discontinuous, non-differentiable, and possibly multi-modal, solving this maximization problem requires a global optimization technique. Estimation of the optimal coefficient vector using existing global optimization techniques is computationally expensive, becoming prohibitive as the number of predictors and the number of outcome categories increases. We propose an efficient derivative-free black-box optimization technique based on pattern search to solve this problem. Through extensive simulation studies, we demonstrate that the proposed method achieves better performance compared to existing methods including the step-down algorithm. Finally, we illustrate the proposed method to predict swallowing difficulty after radiation therapy for oropharyngeal cancer based on radiation dose to various structures in the head and neck.

stat.ME↗

Some More Properties of the Unit-Gompertz Distribution

In a recent paper, Mazucheli et al. (2019) introduced the unit-Gompertz (UG) distribution and studied some of its properties. It is a continuous distribution with bounded support, and hence may be useful for modelling life-time phenomena. We present counter-examples to point out some subtle errors in their work, and subsequently correct them. We also look at some other interesting properties of this new distribution. Further, we also study some important reliability measures and consider some stochastic orderings associated with this new distribution.

math.ST↗