Bounding Hallucinations: Merlin-Arthur Protocols for Mutual-Information Bounds in Language Models

Abstract

Retrieval-augmented generation (RAG) relies on retrieved context to guide large language models (LLM), yet treats the retrieval as a heuristic rather than verifiable evidence -- leading to unsupported answers, hallucinations, and reliance on spurious context. We introduce a novel evaluation, data augmentation, and training framework that treats the RAG pipeline as an interactive proof system by adapting the Merlin-Arthur (M/A) protocol: Arthur (the generator LLM) receives context of unknown provenance and Merlin gives helpful evidence, while Morgana injects adversarial, misleading context. We implement them both with an XAI method to self-assess and modify evidence most influential to Arthur. Based on this framework we propose the Explained Information Fraction (EIF) score, that disentangles explanation fidelity from model predictive errors and imperfect benchmarks, and normalizes M/A mutual-information lower bounds to realistic empirical settings. When trained with those contexts, Arthur learns to answer when evidence supports the answer and abstain when evidence is insufficient. Across five QA benchmarks and four LLMs (1B to 32B parameters), M/A reduces incorrect answers under insufficient context by up to 35pp during training and by 18-20pp over vanilla finetuning. Abstention emerges even \emph{without any manually annotated unanswerable example or preference pair}. We improve EIF-cond by 0.1-0.4 during M/A training and by 0.33-0.38 over vanilla finetuning. Reusing Merlin/Morgana contexts as automatic hard positives and negatives also raises retriever Recall@1 by 2pp. While high accuracy does not guarantee entropy flow from context to answer, our EIF scores -- to our knowledge, the first context-to-answer information bound for RAG generators -- show that autonomous interactive-proof-style supervision enables RAG systems that treat retrieved documents as verifiable evidence.