arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 1808.04904

False Discovery Rate Controlled Heterogeneous Treatment Effect Detection for Online Controlled Experiments

Yuxiang Xie·Nanyu Chen·Xiaolin Shi
View original source ↗Read source document ↗Export metadata ↓

Abstract

Online controlled experiments (a.k.a. A/B testing) have been used as the mantra for data-driven decision making on feature changing and product shipping in many Internet companies. However, it is still a great challenge to systematically measure how every code or feature change impacts millions of users with great heterogeneity (e.g. countries, ages, devices). The most commonly used A/B testing framework in many companies is based on Average Treatment Effect (ATE), which cannot detect the heterogeneity of treatment effect on users with different characteristics. In this paper, we propose statistical methods that can systematically and accurately identify Heterogeneous Treatment Effect (HTE) of any user cohort of interest (e.g. mobile device type, country), and determine which factors (e.g. age, gender) of users contribute to the heterogeneity of the treatment effect in an A/B test. By applying these methods on both simulation data and real-world experimentation data, we show how they work robustly with controlled low False Discover Rate (FDR), and at the same time, provides us with useful insights about the heterogeneity of identified user groups. We have deployed a toolkit based on these methods, and have used it to measure the Heterogeneous Treatment Effect of many A/B tests at Snap.

Explore related subjects

stat.AP

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Yuxiang Xie, Nanyu Chen, Xiaolin Shi. 2018-08-14. False Discovery Rate Controlled Heterogeneous Treatment Effect Detection for Online Controlled Experiments. https://doi.org/10.1145/3219819.3219860

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Probabilistic Estimation of Hidden Migrant Fatalities Along the Central Mediterranean Route

Estimating the number of migrants who die or go missing along dangerous routes such as the Central Mediterranean remains challenging as available records are incomplete. Some incidents are never documented, and fatalities associated with such unobserved incidents are absent from observed totals. We propose a Bayesian approach for probabilistic estimation of total migrant fatalities in such settings. Building on recent developments in multiple-systems estimation, we develop a time-stratified latent-class framework that accommodates missing fatality counts for unobserved incidents. We apply the method to recoded incident-level data from the Missing Migrants Project for the Central Mediterranean route from 2014 to 2025, encompassing 25,712 fatalities across 1,562 incidents. Our model yields 95% credible intervals of 31,922-41,413 fatalities and 2,170-2,952 deadly incidents, indicating that approximately 62%-81% of fatalities and 53%-72% of incidents are reflected in the available data. We estimate that unreported fatalities were concentrated between 2014 and 2016. Furthermore, we document that reporting likelihood increases with incident severity, implying that smaller incidents are most likely to remain undetected. While contingent on modeling assumptions and incomplete data, our method provides a broadly applicable and principled alternative to naive data adjustment methods.

stat.AP↗
arXiv2026

Privacy-Preserving Causal Meta-Mediation Analysis with Survival Outcomes

Privacy and data-governance constraints often prevent pooling individual-level data across studies, limiting the use of conventional approaches for causal media- tion analysis in multicenter settings. We propose a federated causal meta-mediation framework for right-censored time-to-event outcomes that enables collaborative es- timation without sharing individual-level data. Our framework targets natural indirect effects in a prespecified population by combining information on mediator and outcome mechanisms across distributed data sources. A site-by-site identifi- cation strategy further allows heterogeneity across data sources to be character- ized, with a variance decomposition separating outcome-related, mediator-related, and interaction components. We develop federated one-step and targeted maxi- mum likelihood estimators that accommodate data-adaptive and machine-learning methods for nuisance-function estimation. The finite-sample performance of the proposed estimators is evaluated through numerical simulations. To illustrate the practical utility of the framework, we apply it on data from the French National Health Data System to evaluate the role of methotrexate coprescription in explain- ing the effect of TNFi versus IL-12/23 inhibitor therapy on treatment persistence among psoriatic patients.

stat.AP↗
arXiv2026

Geospatial Foundation Models Capture Health-Relevant Dimensions of Place Beyond Conventional Social Risk Indices

Area-based social risk indices summarize residents' socioeconomic conditions but incompletely capture physical features of place that may affect health. We evaluated whether numerical representations of physical place produced by four geospatial foundation model families from 2022 satellite data explained residual variance in tract-level associations between the Area Deprivation Index, Social Deprivation Index, and Social Vulnerability Index with health outcomes. We used LightGBM to predict variables from the American Community Survey and 40 chronic disease and health-behavior outcomes from CDC PLACES across 82,646 census tracts in the contiguous United States, evaluating performance across 10 held-out states. Among survey variables, models were moderately predictive of some variables including housing type (R-squared up to 0.54) but weak for disability, unemployment, and income disparity. For health outcomes, models explained up to 54% of variance left unexplained by social risk indices, with the largest gains for annual checkups, arthritis, and high blood pressure. Mean total variance explained by geospatial foundation models across the 40 health-related outcomes increased from 0.31 in the smallest tract-size decile to 0.39 in the largest. Geospatial foundation models capture health-relevant features of place not represented by conventional social risk indices and may usefully augment them in epidemiological analyses.

stat.AP↗