arXiv Science⌕ Search

arXiv subjects

Brendan Crabb

Publications and source records attributed to Brendan Crabb.

2 recordsLinked to original sources

Distinguishing Extraction Failures and Clinical Discordance with LLM Uncertainty on Radiology Reports

Large language models (LLMs) are increasingly used to structure data from unstructured text, but extraction errors remain inevitable. We evaluated whether LLM uncertainty could distinguish extraction failure from true clinical discordance. In 5{,}225 vessels from 1{,}045 patients undergoing coronary CT angiography (CCTA) for suspected coronary artery disease (CAD), we evaluated token-level and self-reported LLM uncertainty during extraction of vessel-level stenosis severity. Extraction failure was defined as disagreement between LLM-extracted stenosis and expert annotations. Clinical discordance was defined as disagreement in obstructive CAD status between CCTA and subsequent invasive coronary angiography (ICA). Incremental discrimination was assessed after adjustment for stenosis severity, calcium burden, and coronary vessel. Extraction failure was evaluated in 1{,}077 vessels from 253 patients with expert annotations, and clinical discordance in 1{,}641 vessels from 729 patients undergoing ICA within 90 days. Token-level Stenosis Margin showed the strongest association with extraction failure (adjusted OR, 4.15 [95\% CI, 2.57--6.69]; $P<0.001$) and clinical discordance (adjusted OR, 1.22 [95\% CI, 1.04--1.43]; $P=0.118$). Stenosis Margin significantly improved discrimination of extraction failure ($Δ$AUC, +0.06 [95\% CI, +0.03 to +0.09]; $P=0.001$), but not clinical discordance ($Δ$AUC, +0.01 [95\% CI, $-0.00$ to +0.02]; $P=0.245$). In a nested cohort of 138 vessels from 57 patients, 38 of 42 clinically discordant vessels (90\%) were extracted correctly. LLM uncertainty improved identification of extraction failure but not clinical discordance, suggesting that uncertainty may identify errors introduced during data extraction but may not generalize to discordance in underlying clinical data.

cs.CL↗

Image quality assessment for determining efficacy and limitations of Super-Resolution Convolutional Neural Network (SRCNN)

Traditional metrics for evaluating the efficacy of image processing techniques do not lend themselves to understanding the capabilities and limitations of modern image processing methods - particularly those enabled by deep learning. When applying image processing in engineering solutions, a scientist or engineer has a need to justify their design decisions with clear metrics. By applying blind/referenceless image spatial quality (BRISQUE), Structural SIMilarity (SSIM) index scores, and Peak signal-to-noise ratio (PSNR) to images before and after image processing, we can quantify quality improvements in a meaningful way and determine the lowest recoverable image quality for a given method.

cs.CV↗