arXiv · 2611.10297
Uncertainty, Not Concealment: A Mechanistic Reappraisal of Sycophancy in Vision-Language Models
Abstract
Vision-language models (VLMs) in medical retrieval-augmented generation must be able to say when a retrieved report contradicts the X-ray, so understanding whether they can detect such conflicts is important for safe deployment. The sycophancy mask hypothesis holds that these models internally detect conflicts but refuse to report them, concealing a detected contradiction. We introduce a three-part diagnostic that separates concealment from uncertainty: representation probing, activation steering, and two-way forced choice. Across four instruction-tuned VLMs on chest X-rays, we confirm behavioral suppression of contradiction and find that a linear probe separates matched from mismatched inputs, beyond what a simple CLIP model achieves. Steering the learned conflict direction never elicits contradiction, and forcing a binary choice does not recover one; instead, cross-modal attention to the image drops on mismatched pairs. We conclude that these models fail to contradict because the mismatch registers as hedged, gated uncertainty rather than concealed knowledge, calling for re-grounding and uncertainty surfacing. Code is available at https://github.com/mohamed-eldagla/Uncertainty-Not-Concealment/.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Mohamed I. Eldagla, Zeyad H. ElShafei, Ahmed M. Anter. 2026-11-06. Uncertainty, Not Concealment: A Mechanistic Reappraisal of Sycophancy in Vision-Language Models. https://arxiv.org/abs/2611.10297
Cite the original work for its findings. Save a collection to share your selection of sources.