Not Just Why, but When and for Whom: Revisiting Common Estimands in Causal Inference

Abstract

Causal inference commonly defines an individual treatment effect as a quantity that varies across units. But we know such effects, even for a fixed unit, can also vary over time, across repeated studies, or over different methods of measurement, and they can even exhibit inherent stochasticity. In this work, we consider how such additional sources of variation create a need for more general notions of average treatment effects, including some estimands that have yet to be more formally explored by the causal inference community. This perspective also motivates a reconceptualization of our most common existing estimands, such as conditional average, sample average, and individual treatment effects (CATEs, SATEs, and ITEs), as subject to more axes of variation than are typically considered. We explore two main consequences of this reframing in the case where we have data from a single randomized controlled trial: (1) decision making in fixed samples and (2) uncertainty quantification during estimation. We find that different estimands, even for the same unit or the same sample, can qualitatively disagree on whether an effect is positive or negative. We demonstrate these issues empirically through a case study with real data and characterize through a simulation study how our assumptions about these sources of variation, even if we cannot observe the variation, can be consequential for estimation and proper coverage performance. Importantly, these phenomena arise even in the simplest of cases, where all structural and parametric assumptions are satisfied. More broadly, this paper aims to provide a preliminary investigation into an underexplored class of assumptions --- assumptions about unobserved non-confounders --- that should be considered when estimating treatment effects.