arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 1507.03063

Incentive-Compatible Experimental Design

Panos Toulis·David C. Parkes·Elery Pfeffer·James Zou
View original source ↗Read source document ↗Export metadata ↓

Abstract

We consider the design of experiments to evaluate treatments that are administered by self-interested agents, each seeking to achieve the highest evaluation and win the experiment. For example, in an advertising experiment, a company wishes to evaluate two marketing agents in terms of their efficacy in viral marketing, and assign a contract to the winner agent. Contrary to traditional experimental design, this problem has two new implications. First, the experiment induces a game among agents, where each agent can select from multiple versions of the treatment it administers. Second, the action of one agent -- selection of treatment version -- may affect the actions of another agent, with the resulting strategic interference complicating the evaluation of agents. An incentive-compatible experiment design is one with an equilibrium where each agent selects its natural action, which is the action that would maximize the performance of the agent if there was no competition (e.g., expected number of conversions if agent was assigned the contract). Under a general formulation of experimental design, we identify sufficient conditions that guarantee incentive-compatible experiments. These conditions rely on the existence of statistics that can estimate how agents would perform without competition, and their use in constructing score functions to evaluate the agents. In the setting with no strategic interference, we also study the power of the design, i.e., the probability that the best agent wins, and show how to improve the power of incentive-compatible designs. From the technical side, our theory uses a range of statistical methods such as hypothesis testing, variance-stabilizing transformations and the Delta method, all of which rely on asymptotics.

Explore related subjects

stat.ME

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Panos Toulis, David C. Parkes, Elery Pfeffer, James Zou. 2015-07-11. Incentive-Compatible Experimental Design. https://doi.org/10.1145/2764468.2764525

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Logistic Regression Equivalent Weights for Survey Inference: Construction and Asymptotic Properties

Equivalent weights from regression models provide a bridge between design-based and model- based survey inference. We develop a frequentist survey-weighting framework for logistic regres- sion equivalent weights under categorical poststratification. Because the logistic model-based population estimator is nonlinear in the observed outcomes, we define equivalent weights through a local first-order representation, with each weight given by the scaled derivative of the popula- tion estimator with respect to the corresponding observed outcome. This construction yields an explicit closed-form expression in terms of the fitted logistic model and population cell structure. We establish convergence of the effective sample size ratio, asymptotic linearity of the resulting weighted estimator, and a consistent plug-in variance estimator under a finite-cell superpop- ulation framework. Simulation results examine population recovery, regression performance, effective sample size, and the finite-sample behavior of the asymptotic variance approximation. The results show that logistic equivalent weighting provides competitive finite-sample point esti- mation and approximately valid plug-in variance estimation, effective sample size is competitive with alternative model-based weighting in the settings considered. An application to the Fu- ture of Families and Child Wellbeing Study illustrates the construction and use of the proposed weights and variance estimator in an empirical survey setting. We emphasize the relationship between logistic equivalent weighting and recent work on locally equivalent weights for non- linear multilevel regression and poststratification, while focusing specifically on the frequentist interpretation of the weights, their survey-weight properties, and their use for inference.

stat.ME↗
arXiv2026

Small-study effects on the hierarchical summary ROC curve: latent accuracy and threshold trends in meta-analysis of diagnostic test accuracy

Diagnostic meta-analyses commonly assess small-study effects using the Deeks test, which tests for a study-size trend in the log diagnostic odds ratio. Under the hierarchical summary receiver operating characteristic (HSROC) model, this quantity reflects both latent accuracy and threshold unless the curve is symmetric. A threshold trend can therefore generate a log diagnostic odds ratio trend without a latent accuracy trend, or conceal an existing one. We develop a likelihood-ratio test of the latent accuracy trend under a common-shape binomial HSROC model. Its null hypothesis states that the summary curve is unchanged across study sizes. A log diagnostic odds ratio test using the same binomial fit provides a comparison with a different null hypothesis. In simulations, the proposed test remained near its nominal level under its null and had higher power than Deeks with a latent accuracy trend alone. In a colorectal cancer review of faecal immunochemical tests, it detected a negative latent accuracy trend obscured on the log diagnostic odds ratio scale by an opposing threshold contribution. All three tests detected size trends for impedance plethysmography in deep-vein thrombosis, where the fitted curve was nearly symmetric. The proposed test distinguishes changes in the summary curve from changes in operating position.

stat.ME↗
arXiv2026

Hierarchical Gaussian Processes for Survey Weighting: Local Equivalent Weights in Structured Populations

Regression-based survey weighting connects model-based population estimation with design-based inference through equivalent weights. For nonlinear models, equivalent weights can be defined locally through the sensitivity of a poststratified population estimator. However, conventional regression models may inadequately capture nonlinear relationships and complex interactions across sparse poststratification cells. We develop a survey-weighting framework based on hierarchical Gaussian process (HGP) regression, combining structured group-level effects with kernel-based smoothing to share information across related cells. We derive Gaussian posterior prediction formulas that exploit the structured covariance representation to reduce computational costs. We also establish a uniform convergence result for the posterior predictive mean under stated regularity conditions and construct local equivalent weights by matching the first-order sensitivity of a plug-in poststratified estimand to that of a weighted average of fitted response means, with both derivatives taken with respect to the latent mean-scale vector. Simulations show that HGP achieves lower population-weighted prediction error under nonparametric structure, whereas logistic regression performs well under correct parametric specification. Additional analyses examine sample size, frame coverage, subgroup prediction, and population-mean agreement. An application to the Future of Families and Child Wellbeing Study illustrates the method. Overall, this work extends local equivalent-weight construction to nonparametric modeling and provides a computational framework and conditional theoretical results for survey weighting in structured, sparse populations.

stat.ME↗