arXiv · 2611.09670
Goodhart in the Planner: Sampling-Based Control Exploits Rewards Learned from Human Preferences
Abstract
Robots increasingly plan by searching against a reward model learned from human preferences. We measure what this search does to the true objective. On HalfCheetah and Hopper with crowd-sourced labels, and on Meta-World drawer-open and plate-slide with synthetic and blind human labels, harder MPPI search cuts true return by up to 76% while held-out label accuracy stays at 0.75-0.97; with human labels the drawer opens in 0 of 20 runs, against 10 of 10 for the cloned policy the planner starts from. The effect persists with a learned dynamics model and an image scorer. We prove that $χ^2$-regularized MPPI at a temperature set by the scorer's error spread on the proposals is worst-case optimal, and that no gain can be certified once that spread exceeds the scorer's own. Calibrated from 100 ranked proposal pairs, this rule is the safest fix we test, keeping 86-94% of the cloned policy's return on drawer-open and plate-slide.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Soovadeep Bakshi, Apurba Bose. 2026-11-17. Goodhart in the Planner: Sampling-Based Control Exploits Rewards Learned from Human Preferences. https://arxiv.science/abs/2611.09670
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite arXiv Science by Indiciae: BibTeX · RIS.