arXiv Science⌕ Search

arXiv · 2611.11003

Do Modern Difference-in-Differences Estimators Change Published Findings in Business Research?

Abstract

Staggered-adoption difference-in-differences (DiD) designs are a workhorse of empirical research in business-school journals. The two-way fixed effects (TWFE) estimator that typically implements them can place negative weights on heterogeneous treatment effects. We ask whether negative weights change published findings by reanalyzing staggered-adoption specifications in business-school journals with journal-wide public replication archives. Following Hull (2026), we use large language model agents to identify eligible specifications, replicate the published TWFE estimate, and re-estimate each specification with the DiD-averaging estimator of Callaway and Sant'Anna (2021) and the imputation estimator of Borusyak et al. (2024). We re-estimate 355 specifications from 40 papers. Across the 283 DiD-averaging and 306 imputation comparisons that pass Hull's leverage screen, from 28 and 29 papers, TWFE and modern estimates have correlations of 0.61 and 0.70, change sign in 25.1 and 18.0 percent of comparisons, and differ significantly with opposite signs in 10.2 and 7.2 percent. Modern standard errors are 1.61 and 1.30 times those of TWFE. Including the comparisons that fail the screen leaves these figures essentially unchanged. Our results suggest that TWFE and heterogeneity-robust estimators disagree in a non-negligible share of the published staggered designs we re-estimate, and more often than in economics.

Explore related subjects

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Davide Proserpio. 2026-11-10. Do Modern Difference-in-Differences Estimators Change Published Findings in Business Research?. https://arxiv.science/abs/2611.11003

Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite arXiv Science by Indiciae: BibTeX · RIS.

KEEP EXPLORING

Related papers

Difference-in-Differences Estimators for Treatments Continuously Distributed at Every Period

When studying the effects of taxes, tariffs, or prices using panel data, the treatment is often continuously distributed in every period. We develop difference-in-differences (DID) estimators for such settings. We partition units into switchers, whose treatment changes between consecutive periods, and stayers, whose treatment remains constant. Under a parallel-trends assumption, we show that the slopes of switchers' potential outcomes with respect to the treatment are nonparametrically identified by DID comparisons between switchers and stayers sharing the same baseline treatment level. Conditioning on the baseline treatment is key, as it ensures that the underlying parallel-trends assumption accommodates time-varying treatment effects. We then study two weighted averages of these slopes, discuss their respective advantages, and propose for each a doubly robust, semiparametrically efficient, and $\sqrt{n}$-consistent estimator. Finally, we extend our framework to instrumental variables and illustrate it by estimating the effects of gasoline taxes on prices and fuel consumption.

econ.EM↗

Difference-in-differences for mediation analysis using double machine learning

We propose a difference-in-differences (DiD) framework with mediation for possibly multivalued discrete or continuous treatments and mediators, aimed at identifying the direct effect of the treatment on the outcome (net of effects operating through the mediator), the indirect effect via the mediator, and the joint effects of treatment and mediator, consistent with the framework of dynamic treatment effects. Identification relies on a conditional parallel trends assumption imposed on the mean potential outcome across treatment and mediator states, or (depending on the causal parameter) additionally on the mean potential outcomes and potential mediator distributions across treatment states. We propose ATET estimators for repeated cross sections and panel data within the double/debiased machine learning framework, which allows for data-driven control of covariates, and we establish their asymptotic normality under standard regularity conditions. We investigate the finite-sample performance of the proposed methods in a simulation study and illustrate our approach in an empirical application to the US National Longitudinal Survey of Youth, estimating the direct effect of health care coverage on general health as well as the indirect effect operating through routine checkups.

econ.EM↗

The Online LSTM Prediction Models and The Term Structure of Predictions

We examine and illustrate the properties of the Long Short Term Memory (LSTM) prediction models applied online to stationary inputs. We discuss the effects of: a) the network architecture that defines the admissible prediction functions, b) the objective function used to measure the prediction accuracy, and c) the Stochastic Gradient Descent (SGD) algorithm. We also study the prediction optimality, the term structure of predictions, the filtered memory dynamics. We also introduce a set of diagnostic tools for memory-based prediction models that focus on the estimated dynamics of the hidden states.

econ.EM↗