arXiv · 2611.13395
A Taxonomy of Failing Bug Reproduction Tests on SWT-Bench
Abstract
Automatic Program Repair (APR) tools require high-quality test generation models to validate candidate patches. Despite the advancements of automated bug reproduction testing tools, they continue to struggle with issues of higher complexity. These struggles may be caused by a framework's lack of problem-specific context. We address the unclassified nature of these framework failures by providing a taxonomy of bug reproduction tests. This study uses the AssertFlip and OpenHands frameworks, the two highest-scoring open-source tools on SWT-Bench Verified. The significance of model selection on the causes of failure is also explored by utilizing GPT-5-mini and GPT-4o-mini on both tools. The error traces of these frameworks are categorized to determine granular modes of failure. Through this process, mechanical and implementation failures were the most common failure classes. Overall, a tool's generation approach has a significant impact on precision and coverage, while the selection of the model alters the quality and quantity of generated tests. Specifically, more powerful models with higher reasoning capability tended to present fewer failures due to a lack of technical or problem-specific context. Our experiment also shows evidence that our taxonomy can be used as concrete guidance to augment the issue description to improve the generation quality.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Luca Filippelli, Hamed Taherkhani, Hung Viet Pham, Hadi Hemmati. 2026-11-10. A Taxonomy of Failing Bug Reproduction Tests on SWT-Bench. https://arxiv.science/abs/2611.13395
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite arXiv Science by Indiciae: BibTeX · RIS.