VisRAG2.0: Mitigating Visual Hallucinations via Evidence-Guided Multi-Image Reasoning in Visual Retrieval-Augmented Generation

Abstract

Visual Retrieval-Augmented Generation (VRAG) has emerged as a promising paradigm for equipping Vision-Language Models (VLMs) with external visual evidence, enabling them to go beyond parametric knowledge when answering visually grounded questions. However, in such multi-image settings, VLMs still often suffer from visual hallucinations and struggle to accurately identify the question-relevant evidence needed for reliable reasoning. Existing methods usually lack an explicit cross-image evidence collection process, and also provide limited credit assignment when jointly optimizing perception and reasoning. To address this issue, we propose EVisRAG, an evidence-guided visual retrieval-augmented framework for multi-image reasoning. EVisRAG first observes the retrieved images, records question-relevant visual evidence from each image, and then performs reasoning and answer generation based on the aggregated evidence. We further introduce RS-GRPO, which aligns reward signals with token spans from different stages, improving training stability and strengthening the joint optimization of evidence localization and reasoning. Experiments on multiple visual question answering benchmarks show that EVisRAG consistently outperforms the backbone VLM by an average of about 19\%, while substantially reducing visual hallucinations. These results demonstrate that explicit evidence collection and scoped reward design are effective for improving visual grounding and reasoning reliability in multi-image settings. Codes and data are available at https://github.com/OpenBMB/VisRAG