arXiv · 2611.10098
Autonomous Code Review: Hidden-Test Results on GitHub Issues
Abstract
Can an autonomous cross-provider review step change whether code written for a real issue passes hidden tests? We report a retrospective benchmark of 12 historical GitHub issues (five C#, five JavaScript, and two Python) from five repositories. Two directed author--reviewer pairs, OpenAI GPT-6.1-Sol and Anthropic Claude Sonnet 5.5, each ran twice per issue, yielding 48 planned workflows. A bounded terminal agent produced an initial patch; a cross-provider model reviewed it; the author model adjudicated findings and could revise the patch. The same hidden evaluator assessed initial and final patches. Forty workflows completed, while eight terminated before a complete evaluator outcome. Four completed workflows passed, and all four already passed before review. No initial-to-final pass/fail transitions occurred among completed workflows.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Adam Shakhabov. 2026-11-06. Autonomous Code Review: Hidden-Test Results on GitHub Issues. https://arxiv.org/abs/2611.10098
Cite the original work for its findings. Save a collection to share your selection of sources.