arXiv · 2611.10457
Represented but Not Used: Value-Policy Dissociation in LLM Agents
Abstract
Large language model (LLM) agents need to choose actions whose consequences unfold over multiple future steps, yet poor decisions do not reveal whether the model failed to represent those consequences or failed to use them. Mechanistic interpretability has studied internal representations of environment state and plans, but sequential decision making also requires evaluating how candidate actions change future outcomes. Reinforcement learning formalizes this as action value. We study this computation in Qwen2.5-7B-Instruct, Qwen2.5-32B-Instruct, and GPT-OSS-20B across a partially observed Grid environment and WebShop, a simulated web shopping environment. We find that the relative values of candidate actions are linearly decodable from intermediate hidden states, including at states where the policy prefers a worse action. We call this gap between value representation and policy choice value-policy dissociation. To investigate how represented value relates to action selection, we compare direct interventions along the value readout direction with Natural Value Transport (NVT), a transformation learned from hidden state changes that naturally result from differences in action value. While readout-direction interventions have limited and inconsistent effects, NVT more consistently influences action preferences, particularly at later layers. We also test a closed-loop controller that uses the value readout as a sensor and NVT as an actuator, observing improvements in task outcomes. These results suggest that LLM agents can represent useful prospective value without reliably translating it into policy choice.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Yuzhang Luo, Mohamed Aghzal, Liangming Pan, Ziyu Yao. 2026-11-07. Represented but Not Used: Value-Policy Dissociation in LLM Agents. https://arxiv.science/abs/2611.10457
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite arXiv Science by Indiciae: BibTeX · RIS.