arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2610.30916

When Code Gets Cheap, Evaluation Gets Scarce: An Experience Report on AI-Assisted Agile Software Engineering

Barbara Weber·Olaf Zimmermann·Marco Franceschetti
View original source ↗Read source document ↗Export metadata ↓

Abstract

Coding assistants are widely reported to speed up code production; most accounts stop there. We report the end-to-end construction and deployment of a compliance-critical, human-in-the-loop exam-grading system, built by a single developer who wrote no code by hand, protected by a gated process from stakeholder persona to audited behavior. The initial system was built in three working days and could grade a real cohort. We draw one claim from this experience: AI changes software engineering less by making programming faster than by changing what is scarce. Writing code becomes abundant. Reliable evaluation-knowing whether a result is correct and meets the actual need-remains necessary at every iteration and becomes the binding constraint. Engineering judgment remains a second scarcity. Several defects passed the test suite: they sat at the boundaries the tests had mocked, the browser and the language-model API, or appeared only in real use. Deployment exposed assumptions that local tests had missed. Moreover, an AI assistant acting across many steps bypassed process checkpoints until we made them enforceable gates. All of this was observed in a triple union: the developer was also the end user and domain expert, making expert judgment immediately available and evaluation as cheap as it can be. This favors the iteration gains, which may shrink where evaluation requires additional people or expertise, but makes the case a hard test for evaluation scarcity: evaluation still constrained progress. We draw five lessons: run a gated, traceable construction process; specify what is knowable and discover the rest against a running prototype; validate in the real environment; gate the assistant, not only the work; record decisions with their assumptions and the process state the assistant forgets. A second project showed similar patterns; an independent single-expert case corroborates the shift.

Explore related subjects

cs.SE

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Barbara Weber, Olaf Zimmermann, Marco Franceschetti. 2026-10-22. When Code Gets Cheap, Evaluation Gets Scarce: An Experience Report on AI-Assisted Agile Software Engineering. https://arxiv.org/abs/2610.30916

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

DynamiQ: Unlocking the Potential of Dynamic Task Allocation in Parallel Fuzzing

We present DynamiQ, a full-fledged and optimized successor to AFLTeam that supports dynamic and adaptive parallel fuzzing. Unlike most existing approaches that treat individual seeds as tasks, DynamiQ leverages structural information from the program's call graph to define tasks and continuously refines task allocation using runtime feedback. This design significantly reduces redundant exploration and enhances fuzzing efficiency at scale. Built on top of the state-of-the-art LibAFL framework, DynamiQ incorporates several practical optimizations in both task allocation and task-aware fuzzing. We evaluate DynamiQ on eighteen targets from Magma and FuzzBench with a total budget of over 40 CPU years. It achieves the highest mean branch coverage on fourteen targets, with average relative gains ranging from 8.37% to 19.53% over five baseline fuzzers. Across our experiments, DynamiQ identifies nine previously unknown bugs.

cs.SE↗
arXiv2026

Measuring the Permission Gate: A Stress-Test Evaluation of Claude Code's Auto Mode

Claude Code's auto mode is the first deployed permission system for AI coding agents, using a two-stage transcript classifier to gate dangerous tool calls. Anthropic reports a 0.4% false positive rate and 17% false negative rate on production traffic. We present the first independent evaluation of this system on deliberately ambiguous authorization scenarios, i.e., tasks where the user's intent is clear but the target scope, blast radius, or risk level is underspecified. Using AmPermBench, a 128-prompt benchmark spanning four DevOps task families and three controlled ambiguity axes, we evaluate 253 state-changing actions at the individual action level against oracle ground truth. Our findings characterize auto mode's scope-escalation coverage under this stress-test workload. The end-to-end false negative rate is 81.0% (95% CI: 73.8%-87.4%), substantially higher than the 17% reported on production traffic, reflecting a fundamentally different workload rather than a contradiction. Notably, 36.8% of all state-changing actions fall outside the classifier's scope via Tier 2 (in-project file edits), contributing to the elevated end-to-end FNR. Even restricting to the 160 actions the classifier actually evaluates (Tier 3), the FNR remains 70.3%, while the FPR rises to 31.9%. The Tier 2 coverage gap is most pronounced on artifact cleanup (92.9% FNR), where agents naturally fall back to editing state files when the expected CLI is unavailable. These results highlight a coverage boundary worth examining: auto mode assumes dangerous actions transit the shell, but agents routinely achieve equivalent effects through file edits that the classifier does not evaluate.

cs.SE↗
arXiv2026

A Multi-Dimensional, Per-Pass Empirical Study of the LLVM Optimization Pipeline

Quantifying the marginal impact of individual optimization passes underpins phase ordering, pass selection, optimization design, and analysis of pass/hardware interactions. In LLVM, the standard backend for C/C++, Rust, and ML stacks via MLIR, interactions among passes, measurement noise, and pipeline scale make this difficult. We present a systematic empirical study of the LLVM -O3 pipeline. We decompose it into 113 cumulative per-pass prefixes and measure execution time, compile time, binary size, hardware counters, and RAPL energy on 30 PolyBench/C kernels (84,750 measurements). Transitions whose binary is byte-identical to their predecessor give us a built-in null distribution, which we use to calibrate every significance claim. On these compute-bound affine kernels, the pipeline is non-monotone: 36% of the transitions that significantly change runtime are regressions, found in 28 of 30 kernels. Gains are back-loaded (the median kernel needs 84.8% of the pipeline for 80% of its speedup) and concentrated in a small core of passes, a result that holds under both absolute and scale-free impact measures. IR instruction count is an unreliable runtime predictor, and runtime-targeted passes are de facto energy-targeted (30-60% savings). Finally, we bound the speedup lost to non-monotone behavior from above by 46.35% and decompose it: 9.7 points are measurement noise, 15.3 points are transient dips repaired within three passes, and only 21.3 points persist to the end of the pipeline. These findings inform pass pruning, cost-model calibration, and autotuning.

cs.SE↗