Priority-preserving augmentation of goodness-of-fit tests by conditional calibration

Abstract

An omnibus goodness-of-fit statistic can fail to exploit simple diagnostic evidence efficiently. For example, under a standard normal null, an exceptionally large observation is direct evidence of a scale or tail departure even when the primary omnibus statistic does not cross its critical value. This motivates a simple augmentation principle: retain an established primary test, but reserve a small part of its null rejection budget for secondary statistics that encode natural features such as variation, asymmetry, or tail behavior. We implement this principle by calibrating each secondary acceptance region under the null conditional on acceptance at all preceding stages. Once calibrated, the resulting test is a fixed rectangular acceptance rule; the ordering is a mechanism for choosing its boundaries and assigning ordered first-rejection contributions, not a sequential-sampling scheme. An unconditional stage-budget parameterization makes the central trade-off explicit: additional sensitivity is purchased by removing a prespecified, usually small, amount of rejection probability from the primary stage. We establish strong consistency of the quantile-based Monte Carlo calibration and give an observation-specific pooled-rank version with exact randomized finite-m null size. In an experiment under a standard normal null with n=10, we augment the Kolmogorov--Smirnov statistic with sample variance and sample skewness. Assigning only 0.75% of the total Type I error budget to the two secondary statistics changes power against N(0.5,1) from 0.2742 to 0.2733, while increasing power against N(0,0.2^2) from 0.4693 to 0.9827. This focused experiment is a proof of principle rather than an exhaustive comparison of normality tests: a small unconditional allocation to prespecified diagnostics can greatly broaden power while preserving almost all of the primary test's power.