arXiv Science⌕ Search

arXiv subjects

Bhaskar Gurram

Publications and source records attributed to Bhaskar Gurram.

3 recordsLinked to original sources

Measuring and Attributing Evidence Sensitivity in Medical Vision-Language Models via Symmetric Counterfactual Controls

Medical vision-language models (VLMs) asked closed clinical questions ("Is there a polyp?") can score well by relying on the question rather than the image, and current hallucination benchmarks cannot separate true visual discrimination from editing shortcuts or shifts in the answer threshold. We introduce a Counterfactual Evidence Framework that pairs each lesion erasure with a shape- and tool-matched control edit elsewhere in the image, adds same-site ring controls, restored and partial erasures, degraded views and natural negatives, tests on held-out inpainting tools, separates discrimination (the Evidence Sensitivity Gap, ESG, and AUROC) from the answer threshold, and checks the labels with blinded physician readers and hidden-state probes. Across 507 main-test cases (1,115 views), nine frontier VLMs claim 41.1% to 61.6% of erased lesions (ESG 0.269 to 0.470), whereas MedScope-4B, a Qwen3.5-4B agent trained on automatically built counterfactual trajectories, claims 9.6% (ESG 0.758; p <= 0.001 in all 27 primary comparisons). Physician readers support automated erasure in colonoscopy, where frontier models still claim 45% of erased lesions against 1% for MedScope-4B, but call a finding present on 54.7% of pooled erased reads across all sources, which bounds what unverified erasure can certify. Across 12 pre-registered training or system arms and 270 decision policies screened under a rule fixed in advance, every apparent utility gain comes from shifting the threshold rather than improving held-out discrimination, and probes point to perception as the remaining limit. We also report a 0/48 pneumothorax failure on an external cohort; retraining with pneumothorax patients in every stage finds 53% of pneumothoraxes on new patients without changing the main-test results.

cs.AI↗

Measuring What Tool-Using LLM Agents Actually Do: Structured Validation, Causal Error Propagation, and Fabricated Tool Use

Tool-using agents built on large language models (LLMs) are usually scored by comparing the final answer with a reference text, using a string heuristic or an LLM judge. This is unreliable when the references describe the task rather than its answer. We replace it with a deterministic validator that checks the tools called, the executed parameters, the resulting state, and whether the values in the answer are supported by the correct tool observations. We re-scored 33,583 traces from 13 agents on simulated calendar, weather, medical and retail tools. The simulator reproduced all 10,396 logged observations, and the gold observations supported the validator in 16 of 18 disagreements with a three-annotator majority (the annotators in none). The instrument changes conclusions: on the same 306 traces, Gemini-2.0-Flash scores 63.4% under the substring heuristic and 5.6% under the validator. In a paired design (same task and model, with and without one schema-valid wrong argument injected into the first tool call), the error rate rises from 52.4% to 74.5% (+22.0 percentage points, 95% CI 18.4-25.5; 1,085 pairs, 7 models), and by 44.7 points (38.4-51.0) where the corrupted call executed without errors. An LLM-judge ensemble estimates +2.5 points on the same pairs. Agents also report tool results without executing any call, in 0-42.5% of traces depending on the model (rule-based lower bounds), largely because the harness accepts an action and a final answer in one completion. A rule-based runtime monitor costs about 7 microseconds per trace but reduces wrong answers mainly by abstaining.

cs.AI↗

Valid Per-Field Selective Risk Control for Document Extraction: Three Failure Modes, a Validity Ladder, and When Conditioning Pays

Per-field accept/review with selective risk at most alpha -- accept a field only if the error rate among accepted fields is controlled -- is the trust contract document-extraction systems need, and the natural procedure silently violates it on real documents. On 13,859 genuine claude-sonnet-5 fields from 800 CORD receipts (49.0% correct) we diagnose three failure modes: document clustering (design effect 1.84-2.45), score-refit leakage (coverage 0.416 at risk 0.127, violating alpha=0.10 in 95% of splits), and a tie-mass pathology (a degenerate score collapses the threshold grid, 0.030 to 0.001). We organize the fixes as a validity ladder, guarantee form stated per tier. A fit/val split protocol restores expected-selective-risk control for a learned fusion: coverage 0.318 at risk 0.096 at nominal alpha=0.10, no tolerance band (production variant 0.326) -- an on-average point whose realized risk exceeds alpha in 47.5% of resplits, not a certificate. Mondrian Learn-then-Test with exact binomial tails yields per-group PAC certificates: field-iid 0.171 at risk 0.068, cluster-corrected 0.140, doc-iid 0.060 -- the only tier matching documents, honestly near-vacuous today. Support-bin, the pre-specified provenance taxonomy, wins every rigor tier on the sonnet CORD capture (p<1e-4, Bonferroni-corrected) -- a win that does not replicate on the same documents under haiku or qwen -- while on higher-accuracy corpora pooled thresholds win: conditioning helps exactly where pooled cannot certify, subsumed by a learned score elsewhere. A frozen-configuration confirmation on selection-untouched claude-haiku-4-5 held at both risk levels, and a blind three-annotator human-gold audit verifies the practical tier's accepted-set risk at 1.3% against its 10% budget (Fleiss' kappa=0.83; labels err one-sidedly pessimistic). Released Apache-2.0 with seed-pinned, regression-gated procedures.

cs.LG↗