arXiv Science⌕ Search

arXiv · 2610.41255

Discrete total variation under the multinomial, Part 3: expectation under a general distribution

Abstract

For multinomial counts $X\sim\operatorname{Mult}(N;\mathbf p)$ on $n$ ordered bins and a reference law $\mathbf r$ we study the expectation of the reference-centered discrete total variation $d_{\mathbf r}(X)=\sum_{j=1}^{n-1}|(X_{j+1}-Nr_{j+1})-(X_j-Nr_j)|$, the natural extension of the comb-test statistic to a non-uniform null. For a non-uniform null the raw total variation of the counts carries a deterministic staircase of order $N$ that no centering of the statistic removes; the counts are therefore centered at the reference law, and under the null ($\mathbf p=\mathbf r$) the statistic is the matched $d_{\mathbf p}$. Exact finite-sample expectations are obtained by conditioning on the total mass in each adjacent pair, which reduces every edge to a translated binomial absolute-deviation problem. For fixed $n\ge3$ and a sampling law bounded away from the boundary of the simplex, the matched statistic admits a three-term asymptotic expansion in powers of $1/N$ whose coefficients are explicit continuous periodic functions of $N(p_{j+1}-p_j)$ (a generalized, non-Poincaré asymptotic expansion), obtained by a weighted local Edgeworth estimate and a corner Euler--Maclaurin summation; all half-integer orders vanish by parity. Each correction separates into a smooth Edgeworth part and an oscillatory lattice part, the latter an Edgeworth perturbation of one formal Euler--Maclaurin series of a Gaussian lattice sum. Under a sampling law $\mathbf p\neq\mathbf r$ the leading expectation is the folded-normal profile implied by the central limit theorem; under local alternatives we identify the $O(1)$ correction (a skewness term, plus a variance-gradient term for Pitman alternatives), and fixed alternatives give a law-of-large-numbers limit, the $\ell_1$ distance between the adjacent-gradient profiles of $\mathbf p$ and $\mathbf r$, on which threshold tests are consistent.

Explore related subjects

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Nikola Banić, Neven Elezović. 2026-10-30. Discrete total variation under the multinomial, Part 3: expectation under a general distribution. https://arxiv.org/abs/2610.41255

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

Gaussian Approximation for High-Dimensional Weighted and Two-Sample $U$-statistics with Size-Dependent Kernels

We develop Gaussian approximations for high-dimensional vectors formed by second-order $U$-and $V$-statistics whose kernels depend on sample size. To accommodate weighted and two-sample $U$-and $V$-statistics within a unified framework, we establish our results under independent but not identically distributed (i.n.i.d.) sampling. The resulting theorems apply to a broad range of problems, including estimation in linear regression models with many covariates or many-weak instruments, variance component analysis, smoothed two-sample testing, and inference for separately exchangeable arrays, among others. Our results do not require either component of the Hoeffding decomposition to dominate, thereby covering both non-degenerate and degenerate regimes. In addition, we develop maximal inequalities for high-dimensional $U$-statistics with size-dependent kernels under the i.n.i.d.~setting, in a form that remains sharp across a broad range of applications, which may be of independent interest. Finally, applications to estimation and testing with many weak instrumental variables demonstrate that, our results are sharp enough to recover the corresponding fixed-dimensional rate conditions up to logarithmic factors.

math.ST↗

Asymmetric conformal prediction with penalized kernel sum-of-squares

Conformal prediction (CP) is a distribution-free method to construct reliable prediction intervals that has gained significant attention in recent years. Despite its success and various proposed extensions, a significant practical feature which has been overlooked in previous research is the potential skewed nature of the noise, or of the residuals when the predictive model exhibits bias. In this work, we leverage recent developments in CP to propose a new asymmetric procedure that bridges the gap between skewed and non-skewed noise distributions, while still maintaining adaptivity of the prediction intervals. We introduce a new statistical learning problem to construct adaptive and asymmetric prediction bands, with a unique feature based on a penalty which promotes symmetry: when its intensity varies, the intervals smoothly change from symmetric to asymmetric ones. This learning problem is based on reproducing kernel Hilbert spaces and the recently introduced kernel sum-of-squares framework. First, we establish representer theorems to make our problem tractable in practice, and derive dual formulations which are essential for scalability to larger datasets. Second, the intensity of the penalty is chosen using a novel data-driven method which automatically identifies the symmetric nature of the noise. We show that consenting to some asymmetry can let the learned prediction bands better adapt to small sample regimes or biased predictive models.

math.ST↗

The Pivotal Information Criterion

The Bayesian and Akaike information criteria aim at finding a good balance between under- and over-fitting. They are extensively used every day by practitioners. Yet we contend they suffer from at least two afflictions: their penalty parameters $λ=\log n$ and $λ=2$ are too small, leading to many false discoveries, and their inherent (best subset) discrete optimization is infeasible in high dimension. We alleviate these issues with the pivotal information criterion: PIC is defined as a continuous optimization problem, and the PIC penalty parameter $λ$ is selected at the detection boundary (under pure noise). PIC's choice of $λ$ is an upper quantile of a statistic that we prove to be (asymptotically) pivotal, provided the loss function is appropriately transformed. As a result, simulations show a phase transition in the probability of exact support recovery with PIC, a phenomenon studied with no noise in compressed sensing. Applied to real data, for similar predictive performances, PIC selects the least complex model among state-of-the-art learners.

math.ST↗