When Predictive Fit Misleads: Selecting Marketing Mix Models Using Experimental RoAS Benchmarks

Abstract

Marketing Mix Models (MMMs) are used to estimate channel-level Return on Ad Spend (RoAS) and guide marketing investment, yet competing specifications are usually compared on predictive-fit metrics such as out-of-sample R-squared, MAPE, or RMSE. We show that predictive fit can diverge from recovery of the causal quantities that drive marketing decisions. In simulations with known RoAS, the best-fitting specification is often not the one that recovers RoAS most accurately; replicated studies show that this disagreement is systematic and persists under media-spend collinearity, noise, and a misspecified carryover shape. Selecting specifications against experimental RoAS benchmarks, even imprecise ones, improves RoAS recovery relative to fit-based selection, and a real-world application with experimental benchmarks shows the same pattern. We propose an applied workflow: calibrate MMM specifications against estimand-aligned experimental benchmarks when they exist, and otherwise report RoAS sensitivity across plausible specifications rather than relying on a single best-fitting model.