More Treatments More Problems: Designing the 2026 American Causal Inference Conference Data Challenge

Abstract

This paper documents the design of the 2026 American Causal Inference Conference (ACIC) Data Challenge, which uses a large set of simulations to evaluate the performance of a wide variety of causal inference estimators chosen and implemented by challenge participants. Unlike previous ACIC Data Challenges, which tended to mimic observational studies with a single binary treatment, the 2026 Data Challenge focused on randomized control trials with five arms (one control and four active treatments) to isolate key estimation challenges that arise when simultaneously estimating multiple treatment effects. We created 9,000 distinct synthetic populations corresponding to 18 distinct settings to explore variation in performance corresponding to patterns of treatment effect heterogeneity and response surface specifications. From each population, a sample of n = 1,000 was drawn for participants to estimate individual, subgroup, sample, and population-level treatment effects and identify the optimal treatment arm at each level. We describe several strategies used to generate the data to prevent Challenge participants from exploiting structural sim across datasets and ``gaming the system.'' By detailing these simulation design considerations and data generation choices, this paper provides a framework for conducting large-scale, unconfounded synthetic benchmark studies to rigorously evaluate multi-treatment causal inference procedures.