arXiv subjects

Shu Ge

Publications and source records attributed to Shu Ge.

8 recordsLinked to original sources

arXiv2026

$(\alpha,\beta)$-Stability for Boosting Vector-Valued Prediction

Despite the widespread use of boosting in structured prediction, a general theoretical understanding of aggregation beyond scalar prediction remains incomplete. We study vector-valued prediction under a target divergence and identify a geometric stability property under which aggregation amplifies weak guarantees into strong ones. We formalize this property as $(\alpha,\beta)$-stability by geometric median and show how it supports a boosting framework based on exponential reweighting and geometric-median aggregation. For vector-valued prediction, we characterize this stability property under several natural divergences: $\ell_1$ and $\ell_2$ distances for unconstrained vector-valued prediction, and TV, Hellinger, and KL for density estimation over finite probability vectors. Building on these results, we propose a generic boosting framework \geomedboost. Under a weak learner condition and $(\alpha,\beta)$-stability, we obtain exponential decay of the empirical divergence error, which then yields population guarantees through a generalization bound.

cs.LG↗

arXiv2025

Sigmoid-FTRL: Design-Based Adaptive Neyman Allocation for AIPW Estimators

We consider the problem of Adaptive Neyman Allocation for the class of AIPW estimators in a design-based setting, where potential outcomes and covariates are deterministic. As each subject arrives, an adaptive procedure must select both a treatment assignment probability and a pair of linear predictors to be used in the AIPW estimator. Our goal is to construct an adaptive procedure that minimizes the Neyman Regret, which is the difference between the variance of the adaptive procedure and an oracle variance which uses the optimal non-adaptive choice of assignment probabilities and linear predictors. While previous work has drawn insightful connections between Neyman Regret and online convex optimization for the Horvitz--Thompson estimator, one of the central challenges for the AIPW estimator is that the underlying optimization is non-convex. In this paper, we propose Sigmoid-FTRL, an adaptive experimental design which addresses the non-convexity via simultaneous minimization of two convex regrets. We prove that under standard regularity conditions, the Neyman Regret of Sigmoid-FTRL converges at a $T^{-1/2} R$ rate, where $T$ is the number of subjects in the experiment and $R$ is the maximum norm of covariate vectors. Moreover, we show that no adaptive design can improve upon the $T^{-1/2} R$ rate under our regularity conditions, establishing the minimax rate of Neyman Regret. Finally, we establish a central limit theorem and a consistently conservative variance estimator which facilitate the construction of asymptotically valid Wald-type confidence intervals.

math.ST↗

arXiv2025

Normalization in Attention Dynamics

We study the effect of normalization schemes on token representations in deep transformers. Modeling their evolution as interacting particles on the sphere, we show that normalization acts as a form of speed regulation. This perspective enables a unified analysis of several schemes -- including Post-LN, Pre-LN, Mix-LN, Peri-LN, nGPT -- revealing how they influence clustering dynamics and representation collapse. Our framework clarifies how different schemes shape token representations across layers and provides a principled basis for comparing them, identifying Peri-LN as a particularly effective choice.

cs.LG↗

arXiv2024

Bridging multiple worlds: multi-marginal optimal transport for causal partial-identification problem

Under the prevalent potential outcome model in causal inference, each unit is associated with multiple potential outcomes but at most one of which is observed, leading to many causal quantities being only partially identified. The inherent missing data issue echoes the multi-marginal optimal transport (MOT) problem, where marginal distributions are known, but how the marginals couple to form the joint distribution is unavailable. In this paper, we cast the causal partial identification problem in the framework of MOT with $K$ margins and $d$-dimensional outcomes and obtain the exact partial identified set. In order to estimate the partial identified set via MOT, statistically, we establish a convergence rate of the plug-in MOT estimator for the $\ell_2$ cost function stemming from the variance minimization problem and prove it is minimax optimal for arbitrary $K$ and $d \le 4$. We also extend the convergence result to general quadratic objective functions. Numerically, we demonstrate the efficacy of our method over synthetic datasets and several real-world datasets where our proposal consistently outperforms the baseline by a significant margin (over 70%). In addition, we provide efficient off-the-shelf implementations of MOT with general objective functions.

stat.ME↗

arXiv2023

A Collapsible Polynomial Hierarchy for Promise Problems

The polynomial hierarchy has been widely studied in classical complexity theory. In this paper, we will generalize some commonly known results about the polynomial hierarchy to a version of the hierarchy extended to promise problems. This paper proposes new definitions of existential and universal operators for classes of promise problems. Applying these to BQP, we recover the hierarchy proposed by Gharibian et al. (MFCS 2018). Moreover, using our definition, we give an easy proof of the collapse of this hierarchy under a Karp-Lipton-like scenario, which was an open question for the original definition of Gharibian et al.

cs.CC↗

arXiv2023

A Knowledge Distillation Approach for Sepsis Outcome Prediction from Multivariate Clinical Time Series

Sepsis is a life-threatening condition triggered by an extreme infection response. Our objective is to forecast sepsis patient outcomes using their medical history and treatments, while learning interpretable state representations to assess patients' risks in developing various adverse outcomes. While neural networks excel in outcome prediction, their limited interpretability remains a key issue. In this work, we use knowledge distillation via constrained variational inference to distill the knowledge of a powerful "teacher" neural network model with high predictive power to train a "student" latent variable model to learn interpretable hidden state representations to achieve high predictive performance for sepsis outcome prediction. Using real-world data from the MIMIC-IV database, we trained an LSTM as the "teacher" model to predict mortality for sepsis patients, given information about their recent history of vital signs, lab values and treatments. For our student model, we use an autoregressive hidden Markov model (AR-HMM) to learn interpretable hidden states from patients' clinical time series, and use the posterior distribution of the learned state representations to predict various downstream outcomes, including hospital mortality, pulmonary edema, need for diuretics, dialysis, and mechanical ventilation. Our results show that our approach successfully incorporates the constraint to achieve high predictive power similar to the teacher model, while maintaining the generative performance.

cs.LG↗

arXiv2023

Faster-than-Clifford Simulations of Entanglement Purification Circuits and Their Full-stack Optimization

Quantum Entanglement is a fundamentally important resource in Quantum Information Science; however, generating it in practice is plagued by noise and decoherence, limiting its utility. Entanglement distillation and forward error correction are the tools we employ to combat this noise, but designing the best distillation and error correction circuits that function well, especially on today's imperfect hardware, is still challenging. Here, we develop a simulation algorithm for distillation circuits with gate-simulation complexity of $\mathcal{O}(1)$ steps, providing for drastically faster modeling compared to $\mathcal{O}(n)$ Clifford simulators or $\mathcal{O}(2^n)$ wavefunction simulators over $n$ qubits. This new simulator made it possible to not only model but also optimize practically interesting purification circuits. It enabled us to use a simple discrete optimization algorithm to design purification circuits from $n$ raw Bell pairs to $k$ purified pairs and study the use of these circuits in the teleportation of logical qubits in second-generation quantum repeaters. The resulting purification circuits are the best-known purification circuits for finite-size noisy hardware and can be fine-tuned for specific hardware error models. Furthermore, we design purification circuits that shape the correlations of errors in the purified pairs such that the performance of the error-correcting code used in teleportation or other higher-level protocols is greatly improved. Our approach of optimizing multiple layers of the networking stack, both the low-level entanglement purification, and the forward error correction on top of it, are shown to be indispensable for the design of high-performance second-generation quantum repeaters.

quant-ph↗

arXiv2023

Model-agnostic Measure of Generalization Difficulty

The measure of a machine learning algorithm is the difficulty of the tasks it can perform, and sufficiently difficult tasks are critical drivers of strong machine learning models. However, quantifying the generalization difficulty of machine learning benchmarks has remained challenging. We propose what is to our knowledge the first model-agnostic measure of the inherent generalization difficulty of tasks. Our inductive bias complexity measure quantifies the total information required to generalize well on a task minus the information provided by the data. It does so by measuring the fractional volume occupied by hypotheses that generalize on a task given that they fit the training data. It scales exponentially with the intrinsic dimensionality of the space over which the model must generalize but only polynomially in resolution per dimension, showing that tasks which require generalizing over many dimensions are drastically more difficult than tasks involving more detail in fewer dimensions. Our measure can be applied to compute and compare supervised learning, reinforcement learning and meta-learning generalization difficulties against each other. We show that applied empirically, it formally quantifies intuitively expected trends, e.g. that in terms of required inductive bias, MNIST < CIFAR10 < Imagenet and fully observable Markov decision processes (MDPs) < partially observable MDPs. Further, we show that classification of complex images < few-shot meta-learning with simple images. Our measure provides a quantitative metric to guide the construction of more complex tasks requiring greater inductive bias, and thereby encourages the development of more sophisticated architectures and learning algorithms with more powerful generalization capabilities.

cs.LG↗