arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2610.22930

Conformal Policy Selection: When to Deploy AI Decisions for Targeted Intervention

Junyu Cao·Ruijiang Gao·Maytal Saar-Tsechansky·Mingzhang Yin
View original source ↗Read source document ↗Export metadata ↓

Abstract

Artificial-intelligence (AI) policies are increasingly used to make consequential decisions in medicine, finance, education, and public services, yet deploying them responsibly requires knowing not only whether a policy helps on average, but for whom. Existing tools for off-policy evaluation and safe policy learning provide population-level guarantees, but some individuals may experience adverse treatment effects. We recast targeted deployment as a multiple-testing problem: for each candidate unit, test whether the outcome under the AI policy would exceed a user-specified threshold, and control the false discovery rate (FDR) among the selected set. Our method, Conformal Policy Selection, combines weighted conformal $p$-values that account for the distributional shift from behavior to target policy with a two-step selection-and-pruning procedure. We prove finite-sample FDR control and establish a doubly robust asymptotic guarantee, where FDR remains controlled if either the outcome model or the propensity model is correctly specified. Synthetic experiments verify these guarantees across four misspecification regimes, and a semi-synthetic study built on real electronic-health-record covariates illustrates how the same FDR control can act as a deployment safety gate, bounding the harmful treatments that a globally trained policy would otherwise administer to a high-risk subpopulation where it misgeneralizes. The framework offers a principled method for safe AI deployment in high-stakes applications.

Explore related subjects

stat.MEcs.LGstat.ML

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Junyu Cao, Ruijiang Gao, Maytal Saar-Tsechansky, Mingzhang Yin. 2026-10-16. Conformal Policy Selection: When to Deploy AI Decisions for Targeted Intervention. https://arxiv.org/abs/2610.22930

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Logistic Regression Equivalent Weights for Survey Inference: Construction and Asymptotic Properties

Equivalent weights from regression models provide a bridge between design-based and model- based survey inference. We develop a frequentist survey-weighting framework for logistic regres- sion equivalent weights under categorical poststratification. Because the logistic model-based population estimator is nonlinear in the observed outcomes, we define equivalent weights through a local first-order representation, with each weight given by the scaled derivative of the popula- tion estimator with respect to the corresponding observed outcome. This construction yields an explicit closed-form expression in terms of the fitted logistic model and population cell structure. We establish convergence of the effective sample size ratio, asymptotic linearity of the resulting weighted estimator, and a consistent plug-in variance estimator under a finite-cell superpop- ulation framework. Simulation results examine population recovery, regression performance, effective sample size, and the finite-sample behavior of the asymptotic variance approximation. The results show that logistic equivalent weighting provides competitive finite-sample point esti- mation and approximately valid plug-in variance estimation, effective sample size is competitive with alternative model-based weighting in the settings considered. An application to the Fu- ture of Families and Child Wellbeing Study illustrates the construction and use of the proposed weights and variance estimator in an empirical survey setting. We emphasize the relationship between logistic equivalent weighting and recent work on locally equivalent weights for non- linear multilevel regression and poststratification, while focusing specifically on the frequentist interpretation of the weights, their survey-weight properties, and their use for inference.

stat.ME↗
arXiv2026

Small-study effects on the hierarchical summary ROC curve: latent accuracy and threshold trends in meta-analysis of diagnostic test accuracy

Diagnostic meta-analyses commonly assess small-study effects using the Deeks test, which tests for a study-size trend in the log diagnostic odds ratio. Under the hierarchical summary receiver operating characteristic (HSROC) model, this quantity reflects both latent accuracy and threshold unless the curve is symmetric. A threshold trend can therefore generate a log diagnostic odds ratio trend without a latent accuracy trend, or conceal an existing one. We develop a likelihood-ratio test of the latent accuracy trend under a common-shape binomial HSROC model. Its null hypothesis states that the summary curve is unchanged across study sizes. A log diagnostic odds ratio test using the same binomial fit provides a comparison with a different null hypothesis. In simulations, the proposed test remained near its nominal level under its null and had higher power than Deeks with a latent accuracy trend alone. In a colorectal cancer review of faecal immunochemical tests, it detected a negative latent accuracy trend obscured on the log diagnostic odds ratio scale by an opposing threshold contribution. All three tests detected size trends for impedance plethysmography in deep-vein thrombosis, where the fitted curve was nearly symmetric. The proposed test distinguishes changes in the summary curve from changes in operating position.

stat.ME↗
arXiv2026

Hierarchical Gaussian Processes for Survey Weighting: Local Equivalent Weights in Structured Populations

Regression-based survey weighting connects model-based population estimation with design-based inference through equivalent weights. For nonlinear models, equivalent weights can be defined locally through the sensitivity of a poststratified population estimator. However, conventional regression models may inadequately capture nonlinear relationships and complex interactions across sparse poststratification cells. We develop a survey-weighting framework based on hierarchical Gaussian process (HGP) regression, combining structured group-level effects with kernel-based smoothing to share information across related cells. We derive Gaussian posterior prediction formulas that exploit the structured covariance representation to reduce computational costs. We also establish a uniform convergence result for the posterior predictive mean under stated regularity conditions and construct local equivalent weights by matching the first-order sensitivity of a plug-in poststratified estimand to that of a weighted average of fitted response means, with both derivatives taken with respect to the latent mean-scale vector. Simulations show that HGP achieves lower population-weighted prediction error under nonparametric structure, whereas logistic regression performs well under correct parametric specification. Additional analyses examine sample size, frame coverage, subgroup prediction, and population-mean agreement. An application to the Future of Families and Child Wellbeing Study illustrates the method. Overall, this work extends local equivalent-weight construction to nonparametric modeling and provides a computational framework and conditional theoretical results for survey weighting in structured, sparse populations.

stat.ME↗