arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2610.26394

When Coding Agents Outpace Review: Evaluating PRD-Driven Development in Mature Enterprise Repositories

Tianqi Liu·Yandong Guan·Songze Li·Wenkai Wang·Gang Yu·Chengfei Lv
View original source ↗Read source document ↗Export metadata ↓

Abstract

Coding agents increasingly undertake development tasks in mature enterprise repositories, yet existing benchmarks offer limited insight into whether they fulfill complete Product Requirement Documents (PRDs) and where delivery breaks down. Issue resolution and development from scratch leave incremental PRD fulfillment underexplored, while isolated tests provide incomplete evidence of business and integration requirements. We introduce PACE, a benchmark of 43 enterprise tasks with 750 expert-audited Code Points. It combines requirement reconstruction, criterion construction, and evidence-based evaluation to measure PRD fulfillment and connect delivery gaps to agent behavior. Across 11 model--scaffold configurations, most scores fall between 52.1 and 78.1; GPT-5.6-Sol achieves the highest complete-delivery rate of 14.0\%. Incomplete implementation and missing integration are the principal bottlenecks: producing individual components does not ensure a complete delivery. Evaluation achieves 95.2\% agreement with expert consensus, with lower case-level variation across judging records than repeated executions. Beyond model comparison, the benchmark provides actionable feedback for harness refinement: diagnosis guides targeted improvements, and successive harness releases raise mean performance from 59.4 to 63.0 using the same model and task suite. Code is available at https://github.com/TianqiLiu1/PACE-Bench.

Explore related subjects

cs.SEcs.AI

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Tianqi Liu, Yandong Guan, Songze Li, Wenkai Wang, Gang Yu, Chengfei Lv. 2026-10-19. When Coding Agents Outpace Review: Evaluating PRD-Driven Development in Mature Enterprise Repositories. https://arxiv.org/abs/2610.26394

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Real Faults in Deep Learning Fault Benchmarks: How Real Are They?

As the adoption of Deep Learning (DL) systems continues to rise, an increasing number of approaches are being proposed to test these systems, localise faults within them, and repair those faults. The best attestation of effectiveness for such techniques is an evaluation that showcases their capability to detect, localise and fix real faults. To facilitate these evaluations, the research community has collected multiple benchmarks of real faults in DL systems. In this work, we perform a manual analysis of 490 faults from five different benchmarks and identify that 305 of them are eligible for our study. Our investigation focuses specifically on how well the bugs correspond to the sources they were extracted from, which fault types are represented, and whether the bugs are reproducible. Our findings indicate that only 15.4% of the faults satisfy our realism conditions. Our attempts to reproduce these faults were successful only in 54% of cases.

cs.SE↗
arXiv2026

LLM-as-a-Judge for Contest-Style Human-AI Co-Creation: A Reliability-Aware Evaluation Framework for Coding

Large language models now judge open-ended outputs and co-create code with programmers, yet judgments of human-AI co-creation must be reliable, comparable across models, and interpretable over multi-turn interaction. This work presents a rubric-driven LLM-as-a-Judge framework for contest-style human-AI co-creation in coding, evaluated on 15 participants, 13 problems, 184 trajectories, and 517 attempts from an AI-permitted track run in parallel with a programming contest. Four judges are evaluated under grouped cross-validation with cluster-bootstrap 95% intervals and probed by injecting single defects into accepted programs against behavior-preserving controls. Three findings concern the judges as instruments and likely extend beyond this setting. First, the judge input is a first-order design variable: supplying the problem statement in place of an identifier rebuilds the judges' ranking (rank correlation 0.2617 [0.123, 0.391] for the three judges whose model was unchanged) and raises their ROC-AUC by 0.130 [0.061, 0.202] to 0.295 [0.214, 0.381]. Second, schema compliance does not imply scale validity: the judges emit as few as 8 distinct probability values, leaving as few as 6 of 99 thresholds effective. Third, accuracy and defect sensitivity are separable: Claude and Gemini are indistinguishable on accuracy (ROC-AUC 0.6985 [0.633, 0.763] against 0.7691 [0.719, 0.817]) yet separate on defect detection (0.8380 [0.806, 0.871] against 0.7512 [0.719, 0.783]). For co-creation, Success@Turn rises from 0.6033 [0.533, 0.674] at the first attempt to 0.8370 [0.783, 0.886] by the tenth, 44 of the 73 trajectories unsolved at first (60.3%) succeed later, and revision size is unrelated to judged improvement (Spearman 0.0007 [-0.085, +0.090]).

cs.SE↗
arXiv2026

Step-Aligned Script Generation with Verification and Recovery for Web Automation Testing

Automated generation of end-to-end web tests remains costly: developers must identify stable selectors, encode multi-step workflows, attach assertions, and maintain scripts as interfaces evolve. Recent LLM-based web agents can complete natural-language tasks in a live browser session, but task completion is not the same as producing a test-ware artifact - code that is executable, replay-able from a reset state, inspectable by developers, and tied to an oracle. We present TESSARA, an adaptive test-script generation framework that converts natural-language test intent into Playwright scripts consisting of executable interaction code, goal-state assertions, and step-level provenance linking each step to its generated code fragment, selector evidence, execution outcome, and repair history. TESSARA grounds each planned step via Step-Aligned Scripting (SAS), enabling fine-grained failure localization, selective regeneration of failing suffixes, and reversibility-aware backtracking that avoids undoing likely commit points (e.g., form submissions). A three-level verification pipeline (syntactic, execution, and semantic/oracle), detects and repairs malformed code, interaction failures, and incorrect landing states. We evaluate TESSARA on three public web benchmarks, measuring script executability, oracle satisfaction, replay stability, and repair efficiency, and show substantial improvements over baselines and underlying VLMs used directly as script generators.

cs.SE↗