arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2610.14768

Do Modifications Made by Coding Agents Make Subsequent Software Maintenance Harder?

Ruwei Pan·John Zhang·Linhao Wu·Yakun Zhang·Hongyu Zhang·Lu Zhang
View original source ↗Read source document ↗Export metadata ↓

Abstract

Coding agents increasingly maintain repositories over time, where each change becomes the starting point for subsequent tasks. Yet existing benchmarks either start tasks from developer-written code or cannot separate later-task difficulty from the effects of an agent's earlier changes. We ask whether agent-written code makes subsequent maintenance harder and whether the cost stems from inherited test failures or the code itself. We introduce SWE-Maintain, comprising 100 three-event maintenance episodes from 50 real-world Python repositories, with hidden tests certified across each sequence. Each later task is evaluated twice with the same agent and tests: from the developers' preceding patches (Developer-History) and from the agent's own changes (Agent-History). Across five coding systems, two backbone models, and 1,996 paired later events, Agent-History reduces later-task resolution by 5.6 percentage points on average. When success additionally requires all earlier tests to pass, the gap reaches 14.0 points. Two-thirds of Agent-History starting repositories already fail earlier tests, and only 1.9% of events starting from those states end with all accumulated tests passing. When the inherited code passes earlier tests and the next task is not already solved, the disadvantage disappears (+2.7 points; 95% CI [-0.5, +6.1]). These findings suggest that the main downstream cost of agent-written changes lies in inherited test failures rather than a measurable maintenance disadvantage in code that preserves earlier tested behavior. Evaluations should therefore include agent-produced history, and coding agents should preserve earlier behavior across changes.

Explore related subjects

cs.SE

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Ruwei Pan, John Zhang, Linhao Wu, Yakun Zhang, Hongyu Zhang, Lu Zhang. 2026-10-10. Do Modifications Made by Coding Agents Make Subsequent Software Maintenance Harder?. https://arxiv.org/abs/2610.14768

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Chain-of-Thought in Neural Code Generation: From and For Lightweight Language Models

Large Language Models (LLMs) have demonstrated remarkable potential in code generation. The integration of Chain of Thought (CoT) reasoning can further boost their performance. However, current CoT methods often require manual writing or LLMs with over 100 billion parameters to generate, impeding their applicability in resource-constrained scenarios. In this study, we investigate lightweight Language Models (lLMs), which are defined to have fewer than 10 billion parameters. Empirically, we find that most lLMs cannot generate high-quality CoTs when prompted by the few-shot method, but can take advantage of high-quality CoTs generated elsewhere to improve their performance in code generation. Based on these findings, we design a novel approach COTTON which can leverage lLMs to automatically generate CoTs for code generation. We synthesize new datasets and conduct extensive experiments on various benchmarks. The results show that the CoTs generated by COTTON outperform the baselines in terms of automated and human evaluation metrics. In particular, the CoTs generated by COTTON boost various lLMs to achieve higher performance gains than those generated by LLMs such as ChatGLM (130B), and are competitive with those generated by gpt-3.5-turbo (175B). Our study also showcases the potential of lLMs in software engineering applications.

cs.SE↗
arXiv2026

From Legacy Fortran to Portable Kokkos: An Autonomous Agentic AI Workflow

Scientific applications still rely heavily on legacy Fortran codebases built for homogeneous, CPU-based systems. As High-Performance Computing (HPC) shifts toward GPU-accelerated architectures, many accelerators lack native Fortran bindings, creating a need to translate and optimize legacy code for portability. Kokkos offers performance portability through a single-source C++ abstraction, but manual porting from Fortran demands significant expertise and time. While large language models (LLMs) show promise in source-to-source code generation, their use in autonomous workflows for translating and optimizing parallel code is largely unexplored. This paper presents an agentic AI workflow in which specialized LLM agents collaborate to translate, validate, compile, run, test, debug, and optimize Fortran kernels into portable Kokkos C++ programs without human intervention, bounded by configurable retry and optimization thresholds. We evaluate the pipeline over five runs per configuration on an NVIDIA A100, characterizing its variability in success rates, token cost, and optimization behavior, on benchmark kernels including a real-world multi-file kernel from a production climate model. Proprietary models such as GPT-5 and o4-mini-high completed the full workflow most reliably, producing optimized code that exceeded the Fortran baselines for only a few U.S. dollars, whereas open-weights models such as gpt-oss-120b and Qwen3.5-397B-A17B handled simpler kernels but struggled with complex Kokkos abstractions such as parallel reductions and hierarchical parallelism. This work demonstrates, at the kernel level, the feasibility of agentic AI for Fortran-to-Kokkos transformation and, more broadly, for structured software-engineering tasks such as automated code modernization and refactoring.

cs.SE↗
arXiv2026

ScratchWorld: Evaluating If World Models Compute Executable Consequences

Evaluations of world models commonly score a predicted future state by its overlap with the ground truth. Yet in interactive worlds, this rewards copying: a model that merely preserves persistent state appears accurate without predicting any consequence of an action. We introduce ScratchWorld, an execution-grounded diagnostic benchmark that treats Scratch projects as executable worlds and derives every label from replay on a pinned Scratch VM, including next-state transitions, hidden variables, causal traces, and counterfactual outcomes. Its primary metric, value-aware changed-field F1 ($F_1^{\mathrm{VA}}$), credits a prediction only when it names a changed field and computes its executed value. Model-free anchors make the confound explicit: a no-change prior attains 91.4% full-state field accuracy on synthetic long-horizon items and 0% changed-field credit on non-empty one-step items, whereas VM replay attains 100% wherever the corresponding metric is defined. Across seven frontier LLMs prompted as world models, state-only $F_1^{\mathrm{VA}}$ reaches at most 30.0% on controlled synthetic mechanisms and declines steeply as projects approach real-world complexity. Revealing the full program raises every model's value-aware score by 22.3-51.7 points and substantially reorders the models, yet the best reaches only 55.4%, an execution gap that persists even with the rules visible. Auxiliary diagnostics complete the error profile: causal-event accuracy remains within 2.5 points of the 50% no-event baseline, with no correct exact chain; counterfactual prediction trades changed-outcome accuracy against invariance; hidden-state accuracy degrades roughly twice as fast as full-state overlap over long horizons. In a word, ScratchWorld separates what a model knows about the rules from what it can actually execute.

cs.SE↗