arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2610.24902

RXSynth: A Seed-Grounded Corpus for Studying Java Runtime-Exception Bugs

Parnian Eslamloo·Banani Roy
View original source ↗Read source document ↗Export metadata ↓

Abstract

Existing Java bug benchmarks offer limited method-level data for studying runtime-exception detection. Filtering Defects4J, Bugs.jar, Bears, and BugSwarm yields only 170 real buggy/fixed method pairs across seven unchecked exception types. We introduce RXSynth, a corpus of 4,275 synthetic Java method pairs generated from real bug/fix exemplars. We select the generator using structural alignment with real seeds and validate pairs using a two-stage patch-anchored procedure that identifies the labelled exception risk and checks whether the repair removes it. Independent JVM execution confirms patch-local validity for 84.2% of a stratified sample of 500 pairs. Provenance records enable evaluation without overlap between generation exemplars and test bugs. Structural analyses show good alignment within the represented size range, but compression of larger methods. We use RXSynth to compare isolated-method detection with pair discrimination when the repair is visible. Under provenance-controlled evaluation, supervised isolated-method ROC-AUC is about 0.55-0.64. Stronger zero-shot models reach about 69% accuracy but identify only 41.2% of repaired methods correctly. In contrast, fine-tuned encoders achieve 70.4-74.8% accuracy with both versions available. However, a simple token-growth rule achieves 84.7% expected accuracy, highlighting repair-direction cues. On 36 pairs without token growth, encoder accuracy falls to approximately chance, while GPT-5.5 and Gemini 3.1 Pro exceed 94%. RXSynth thus supports contrastive, patch-oriented, and context-aware studies; pairwise evaluations should control for provenance and directional surface cues before interpreting high accuracy as defect understanding.

Explore related subjects

cs.SE

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Parnian Eslamloo, Banani Roy. 2026-10-17. RXSynth: A Seed-Grounded Corpus for Studying Java Runtime-Exception Bugs. https://arxiv.org/abs/2610.24902

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Real Faults in Deep Learning Fault Benchmarks: How Real Are They?

As the adoption of Deep Learning (DL) systems continues to rise, an increasing number of approaches are being proposed to test these systems, localise faults within them, and repair those faults. The best attestation of effectiveness for such techniques is an evaluation that showcases their capability to detect, localise and fix real faults. To facilitate these evaluations, the research community has collected multiple benchmarks of real faults in DL systems. In this work, we perform a manual analysis of 490 faults from five different benchmarks and identify that 305 of them are eligible for our study. Our investigation focuses specifically on how well the bugs correspond to the sources they were extracted from, which fault types are represented, and whether the bugs are reproducible. Our findings indicate that only 15.4% of the faults satisfy our realism conditions. Our attempts to reproduce these faults were successful only in 54% of cases.

cs.SE↗
arXiv2026

LLM-as-a-Judge for Contest-Style Human-AI Co-Creation: A Reliability-Aware Evaluation Framework for Coding

Large language models now judge open-ended outputs and co-create code with programmers, yet judgments of human-AI co-creation must be reliable, comparable across models, and interpretable over multi-turn interaction. This work presents a rubric-driven LLM-as-a-Judge framework for contest-style human-AI co-creation in coding, evaluated on 15 participants, 13 problems, 184 trajectories, and 517 attempts from an AI-permitted track run in parallel with a programming contest. Four judges are evaluated under grouped cross-validation with cluster-bootstrap 95% intervals and probed by injecting single defects into accepted programs against behavior-preserving controls. Three findings concern the judges as instruments and likely extend beyond this setting. First, the judge input is a first-order design variable: supplying the problem statement in place of an identifier rebuilds the judges' ranking (rank correlation 0.2617 [0.123, 0.391] for the three judges whose model was unchanged) and raises their ROC-AUC by 0.130 [0.061, 0.202] to 0.295 [0.214, 0.381]. Second, schema compliance does not imply scale validity: the judges emit as few as 8 distinct probability values, leaving as few as 6 of 99 thresholds effective. Third, accuracy and defect sensitivity are separable: Claude and Gemini are indistinguishable on accuracy (ROC-AUC 0.6985 [0.633, 0.763] against 0.7691 [0.719, 0.817]) yet separate on defect detection (0.8380 [0.806, 0.871] against 0.7512 [0.719, 0.783]). For co-creation, Success@Turn rises from 0.6033 [0.533, 0.674] at the first attempt to 0.8370 [0.783, 0.886] by the tenth, 44 of the 73 trajectories unsolved at first (60.3%) succeed later, and revision size is unrelated to judged improvement (Spearman 0.0007 [-0.085, +0.090]).

cs.SE↗
arXiv2026

Step-Aligned Script Generation with Verification and Recovery for Web Automation Testing

Automated generation of end-to-end web tests remains costly: developers must identify stable selectors, encode multi-step workflows, attach assertions, and maintain scripts as interfaces evolve. Recent LLM-based web agents can complete natural-language tasks in a live browser session, but task completion is not the same as producing a test-ware artifact - code that is executable, replay-able from a reset state, inspectable by developers, and tied to an oracle. We present TESSARA, an adaptive test-script generation framework that converts natural-language test intent into Playwright scripts consisting of executable interaction code, goal-state assertions, and step-level provenance linking each step to its generated code fragment, selector evidence, execution outcome, and repair history. TESSARA grounds each planned step via Step-Aligned Scripting (SAS), enabling fine-grained failure localization, selective regeneration of failing suffixes, and reversibility-aware backtracking that avoids undoing likely commit points (e.g., form submissions). A three-level verification pipeline (syntactic, execution, and semantic/oracle), detects and repairs malformed code, interaction failures, and incorrect landing states. We evaluate TESSARA on three public web benchmarks, measuring script executability, oracle satisfaction, replay stability, and repair efficiency, and show substantial improvements over baselines and underlying VLMs used directly as script generators.

cs.SE↗