arXiv · 2611.15597
When Local Definitions Are Missing: Stable Errors in Language Models
Abstract
Contracts, technical manuals, and online communities often give familiar words special meanings and define them in one place. A language model that reads a passage without that definition may use the usual meaning and give an answer that sounds right but is wrong. We study this with multiple-choice questions whose correct answers depend on a local definition. The questions come from a synthetic vocabulary, commercial contracts, Reddit community glossaries, and the documentation of a decentralized exchange. We ask three language models each question with and without the definition. Without using the answer key, we then test whether their repeated answers and stated confidence reveal their own errors. Separately, we give a small open model the answer key and test whether it can predict which questions larger models get wrong. We report three findings. (1) Familiar words that a source redefines cause the most errors. Models often repeat the same wrong answer, and neither repeated answers nor stated confidence reliably flag it. (2) The source definition fixes most errors on synthetic questions, but a definition drawn from the model's own prior knowledge does not. In real contracts, the source definition fixes only about half of the errors. (3) Given an answer key, a small model can rank questions by how likely larger models are to get them wrong. These results call for tests that check answers against the full source, including its local definitions.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Zengqing Wu, Chuan Xiao. 2026-11-11. When Local Definitions Are Missing: Stable Errors in Language Models. https://arxiv.science/abs/2611.15597
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite arXiv Science by Indiciae: BibTeX · RIS.