arXiv · 2611.22815
Visual Harness Scales General VLMs to Medical Images
Abstract
We show that an appropriate visual harness can substantially improve the performance of general vision-language models (VLMs) across diverse medical imaging tasks. We introduce \ours, a medical visual harness that equips general VLMs with interactive visual tools and systematic diagnostic evidence management. With \ours, general VLMs interactively explore medical images and progressively acquire diagnostic evidence, which is preserved in a lossless memory for subsequent retrieval and use. After submitting an initial answer, the VLM receives an assessment of potential omissions and the reliability of identified findings. This assessment guides further image inspection and measurement, followed by answer revision in an iterative agentic loop until sufficient evidence is obtained. Across 11 tasks and four imaging modalities, \ours improves performance on all tasks with Qwen3.8-27B, with notable gains in spleen measurement (63.3\% to 98.3\%), renal search (57.5\% to 71.2\%), and pulmonary nodule detection (0.0\% to 28.2\%). High-capability API-based VLMs achieve even higher performance with \ours. For example, GPT-6.1 Sol improves pulmonary nodule detection from 10.8\% to 69.2\% with \ours, demonstrating the complementary benefits of stronger model capabilities and an effective medical visual harness. Our results highlight the potential of visual harnesses as a new paradigm for scaling general VLMs to medical imaging tasks.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Shaowen Wan, Hanqi Jiang, Zihao Wu, Junhao Chen, Weihang You, Ruiyu Yan, Bowen Chen, Tianming Liu, Lin Zhao. 2026-11-17. Visual Harness Scales General VLMs to Medical Images. https://arxiv.science/abs/2611.22815
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite arXiv Science by Indiciae: BibTeX · RIS.