A Plasmode Benchmark Can Refute but Cannot Certify: Projection and Regularity in Causal-Inference Simulation

Abstract

Plasmode simulation - resampling covariates, and often treatment, from a real cohort and regenerating the outcome from a model with an injected effect - is a standard tool for comparing causal-inference estimators. We argue that its validity turns on two separable design choices, and that separating them clarifies a recent debate over resampling versus regenerating the treatment. Our central message limits what such a benchmark can show: a plasmode can refute a universal claim about an estimator but cannot certify one. Refuting needs only that the simulated data genuinely belong to the class over which a guarantee is quoted; certifying also needs the benchmark's difficulty to carry over to the target population, which the analyst cannot verify. Two choices set that difficulty. Regularity is whether the simulated law preserves the overlap that inverse-weighting and targeted-learning theory require. Resampling the real treatment with fine-grained covariates destroys it, and the resulting coverage failure falls on flexible, machine-learning-based estimators, while parametric-nuisance estimators - including inverse-probability weighting - stay near nominal; it disappears once the treatment is regenerated from a bounded propensity or the covariates are coarsened, and cross-fitting removes only part of it. Projection is that the injected outcome model hands a matching estimator the right answer by construction; this survives the regularity fix and can reverse estimator rankings even when overlap is intact. We give a regular-plasmode algorithm, a confirming real-data simulation, and practical recommendations. A plasmode is a filter against estimators, not a warrant for them.