arXiv · 2611.10502
Distinguishing Extraction Failures and Clinical Discordance with LLM Uncertainty on Radiology Reports
Abstract
Large language models (LLMs) are increasingly used to structure data from unstructured text, but extraction errors remain inevitable. We evaluated whether LLM uncertainty could distinguish extraction failure from true clinical discordance. In 5{,}225 vessels from 1{,}045 patients undergoing coronary CT angiography (CCTA) for suspected coronary artery disease (CAD), we evaluated token-level and self-reported LLM uncertainty during extraction of vessel-level stenosis severity. Extraction failure was defined as disagreement between LLM-extracted stenosis and expert annotations. Clinical discordance was defined as disagreement in obstructive CAD status between CCTA and subsequent invasive coronary angiography (ICA). Incremental discrimination was assessed after adjustment for stenosis severity, calcium burden, and coronary vessel. Extraction failure was evaluated in 1{,}077 vessels from 253 patients with expert annotations, and clinical discordance in 1{,}641 vessels from 729 patients undergoing ICA within 90 days. Token-level Stenosis Margin showed the strongest association with extraction failure (adjusted OR, 4.15 [95\% CI, 2.57--6.69]; $P<0.001$) and clinical discordance (adjusted OR, 1.22 [95\% CI, 1.04--1.43]; $P=0.118$). Stenosis Margin significantly improved discrimination of extraction failure ($Δ$AUC, +0.06 [95\% CI, +0.03 to +0.09]; $P=0.001$), but not clinical discordance ($Δ$AUC, +0.01 [95\% CI, $-0.00$ to +0.02]; $P=0.245$). In a nested cohort of 138 vessels from 57 patients, 38 of 42 clinically discordant vessels (90\%) were extracted correctly. LLM uncertainty improved identification of extraction failure but not clinical discordance, suggesting that uncertainty may identify errors introduced during data extraction but may not generalize to discordance in underlying clinical data.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Jerjes Aguirre-Chavez, Shant Malkasian, Roshun Sankaran, Brendan Crabb, Albert Hsiao. 2026-11-17. Distinguishing Extraction Failures and Clinical Discordance with LLM Uncertainty on Radiology Reports. https://arxiv.science/abs/2611.10502
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite arXiv Science by Indiciae: BibTeX · RIS.