arXiv subjects

Reid Dale

Publications and source records attributed to Reid Dale.

7 recordsLinked to original sources

arXiv2026

Data Reuse and the Long Shadow of Error: Splitting, Subsampling, and Prospectively Managing Inferential Errors

When multiple investigators analyze a common dataset, the data reuse induces dependence across testing procedures, affecting the distribution of errors. Existing techniques of managing dependent tests require either cross-study coordination or post-hoc correction. These methods do not apply to the current practice of uncoordinated groups of researchers independently evaluating hypotheses on a shared dataset. We investigate the use of subsampling techniques implemented at the level of individual investigators to remedy dependence with minimal coordination. To this end, we establish the asymptotic joint normality of test statistics for the class of asymptotically linear test statistics, decomposing the covariance matrix as the product of a data overlap term and a test statistic association term. This decomposition shows that controlling data overlap is sufficient to control dependence, which we formalize through the notion of Expected Variance Ratio. This enables the closed form derivation of the variance of the joint rejection region under the global null as a function of pairwise correlations of test statistics. We adopt mean-variance portfolio theory to measure risk, defining the Expected Variance Ratio (EVR) as the ratio of the expected variance of the Type I error count to the independent baseline. We show that data splitting is asymptotically optimal among rules that ensure exact independence. We then use concentration inequalities to establish that subsampling techniques implementable by individual investigators can ensure an EVR close to $1$. Finally, we show that such subsampling techniques are able to simultaneously perform a number of tests while ensuring sufficient power and that the bounded EVR is $O\left(\frac{1}{r^2}\right)$ compared to data splitting's $O\left(\frac{1}{r}\right)$, where $r$ is the per-statistic fraction of data required.

math.ST↗

arXiv2026

Synthetic Data, Information, and Prior Knowledge: Why Synthetic Data Augmentation to Boost Sample Doesn't Work for Statistical Inference

The use of synthetic data to deidentify data and to improve predictive models is well-attested to. The augmentation of datasets using synthetically generated data is an alluring proposition: in the best case, it generates realistic data \textit{in silico} at a fraction of the cost of authentic data which may be found \textit{in vivo} or \textit{in vitro}. This poses novel epistemic challenges. We contend that synthetic data augmentation is best understood as a novel way of accounting for prior knowledge. In this manuscript, we propose a definition of synthetic distributions and analyze how synthetic data augmentation interplays with standard accounts of maximum likelihood and Bayesian estimation. We observe that the marginal Fisher information contributed by synthetic data processes is subject to fundamental bounds, and enumerate obstacles to the use of synthetic data augmentation to aid in inferential tasks. We then articulate a Bayesian formulation of the way that synthetic data augmentation can be coherently understood, but argue that naive approaches to the specification of the prior are epistemically unjustifiable. This suggests that enhanced scrutiny must be placed on identifying justifiable priors to warrant the use and inclusion of data drawn from specific synthetic distributions. While our analysis shows the challenges and limitations of using synthetic data augmentation to improve upon traditional statistical model reasoning, it does suggest that augmentation is the principal approach analysts using outcome reasoning (i.e. using train/test splits to justify the analysis) can constrain an otherwise high-dimensional model space, providing an alternative to trying to encode the constraints into the potentially complex architecture of the algorithm.

stat.ME↗

arXiv2025

Data Gluttony: Epistemic Risks, Dependent Testing and Data Reuse in Large Datasets

Large-scale registries have collected vast amounts of data which has enabled investigators to efficiently conduct studies of observational data. Common practice is for investigators to use all data meeting the inclusion criteria of their study to perform their analysis. We term this common practice data gluttony. It has apparent formal justification insofar as this approach maximizes per-study power. But this comes at a cost: data reuse affects the shape of the tail distribution of inferential errors. Using the theory of risk orderings we demonstrate how positively dependent testing procedures result in strictly riskier distributions of inferential error. We identify two remedies to this state of affairs: research portfolio optimization and what we term data temperance. Research portfolio optimization requires that we formulate the enterprise of inference in a utility theoretic framework: associated to each hypothesis to be evaluated is some utility dependent on its truth as well as the impact of the statistical decision rendered on the basis of the data. Under certain models of data governance, this approach can be used to optimally allocate data usage across multiple inferential tasks. On the other hand, data temperance is a more flexible strategy for managing the distribution of inferential errors. Data temperance is the principle that an investigator use only as much data as is necessary to perform the task at hand. This is possible due to the diminishing marginal returns in power and precision in sample size. We analyze the effectiveness of data temperance at reducing the dependence across testing and develop a theory of the capacity of a static database to sustain large numbers of inferential tasks with low probability of inducing pairwise dependent testing procedures.

math.ST↗

arXiv2024

Confirming the Null: Remarks on Equivalence Testing and the Topology of Confirmation

Null Hypothesis Statistical Testing is a dominant framework for conducting statistical analysis across the sciences. There remains considerable debate as to whether, and under what circumstances, evidence can be said to be confirmatory of a null hypothesis. This paper presents a modal logic of short-run frequentist confirmation developed by leveraging the duality between hypothesis testing and statistical estimation. It is shown that a hypothesis is confirmable if and only if it satisfies the topological condition of having nonempty interior. Consequently, two-sided hypotheses are not statistically confirmable owing to defects in their topological structure. Equivalence hypotheses are, by contrast, confirmable.

math.ST↗

arXiv2022

The Combinatorics of Falsification and Hypothesis Testing

The present paper is concerned with the question of how falsifiable a single proposition is in the short and long run. Formal Learning theorists such as Schulte and Juhl have argued that long-run falsifiability is characterized by the topological notion of nowhere density in a suitable topological space. I argue that the short-run falsifiability of a hypothesis is in turn characterized by the VC finiteness of the hypothesis. Crucially, VC finite hypotheses correspond precisely to definable sets in NIP structures. I end the chapter by giving rigorous foundations for Mayo's conception of severe testing by way of a combinatorial, non-probabilistic notion of surprise. VC finite hypotheses again appear as the hypotheses with guaranteed short-run surprise bounds. Therefore, NIP theories and VC finite hypotheses capture the notion of short-run falsifiability.

math.LO↗

arXiv2022

The Model Theory of Falsification

This paper is concerned with the question of when a theory is refutable with certainty on the basis of sequence of primitive observations. Beginning with the simple definition of falsifiability as the ability to be refuted by some finite collection of observations, I assess the literature on falsification and its descendants within the context of the dividing lines of contemporary model theory. The static case is broadly concerned with the question of how much of a theory can be subjected to falsifying experiments. In much of the literature, this question is tied up with whether the theory in question is axiomatizable by a collection of universal first-order sentences. I argue that this is too narrow a conception of falsification by demonstrating that a natural class of theories of distinct model-theoretic interest -- so-called NIP theories -- are themselves highly falsifiable.

math.LO↗

arXiv2022

On Rational Jurisprudence: A Problem in Bayesian Confirmation Theory

This paper is concerned with the epistemic question of confirming a hypothesis -- the guilt of a defendant -- by way of testimony heard by a juror over the course of an American-style criminal trial. In it, I attempt to settle a dispute between two strands of the legal community over the issue of whether the methods of Bayesian rationality are incompatible with jurisprudential principles such as the Presumption of Innocence. To this end, I prove a representation theorem that shows that so long as a juror would not convict the defendant having heard no testimony (the Presumption of Innocence) but would convict upon hearing some collection of testimony (Willingness to Convict), then this juror's disposition to convict the defendant is representable as the disposition of a Bayesian threshold juror in Posner's sense. This result indicates that relevant notion of a Bayesian threshold juror is insufficiently specified to render this debate a substantive one.

math.HO↗