arXiv subjects

Sebastian Baltes

Publications and source records attributed to Sebastian Baltes.

At least 19 recordsLinked to original sources

arXiv2026

Loop Engineering: Building Blocks, Adoption, and Impact

Over the past months, the way developers direct agentic AI coding tools has moved up several levels of abstraction, from phrasing prompts to engineering context to configuring the harness around the model. In June 2026, practitioners began to describe a further level called loop engineering: Instead of prompting an agent interactively, developers design systems that prompt agents for them. These systems start agent runs on a schedule or on repository events and stop them when a machine-checkable condition holds. The term spread rapidly, accompanied by bold claims and vocal skepticism, but its adoption in software projects has not been measured. We present an exploratory review of the emerging gray literature, which largely agrees on what a well-engineered loop contains: triggered agent runs bounded by machine-checkable stop conditions, persistent state files, verifier sub-agents, token budgets, and defined points of escalation to humans. From this review, we derive a research agenda for the empirical study of loop engineering in open-source projects, analyze which of its aspects are traceable from repository data, and report an exploratory mining study of 36,710 software repositories. We confirmed the operation of autonomous agent loops in 217 of the 256 repositories our heuristics matched. The repositories commit the configuration around these loops, but almost none commits the state files the discourse prescribes, and the loops' runtime state remains outside version control. We conclude by outlining a planned controlled study of agent autonomy levels and their effect on effort and outcomes.

cs.SE↗

arXiv2026

An Exploratory Study of Agent Plans for Agentic AI Coding Tools in Open-Source Software

Repository-level configuration artifacts allow developers to provide guidance for agentic AI coding tools, such as Claude Code, Gemini, etc. Although prior research has examined repository-shared context files that capture project-level instructions and conventions (e.g., AGENTS.md files), little is known about more task-oriented artifacts such as Agent Plans. We present an exploratory study of Agent Plans in open-source software repositories, examining how plan files are preserved, which development activities they support, and what information they provide to guide agent execution. We screened 36,710 GitHub repositories belonging to engineered software projects and identified 85 Markdown plan files from 10 repositories. Within this highly concentrated corpus, Agent Plans supported several kinds of software engineering work, including maintenance, design, construction, quality-related work, and process support. They also provided task-oriented execution guidance, most commonly through implementation steps, concrete files and locations, and testing and validation information. Overall, repository-preserved Agent Plans under these tool-specific directories appear to be a narrow but informative artifact for studying task intent and execution guidance in human-agent workflows.

cs.SE↗

arXiv2026

Motivations and Barriers to Communicating Software Engineering Research: Insights from Early Career Researchers

Science communication is increasingly becoming a part of modern research careers, involving researchers to disseminate knowledge, engage broader communities, and increase the societal impact of their work. Despite its growing importance, little is known about how early-career software engineering researchers perceive and navigate science communication in practice. In this paper, we investigate how PhD students in software engineering experience science communication. We conducted semi-structured interviews with 18 doctoral candidates from diverse international backgrounds. Using thematic analysis, we examine three interconnected dimensions: motivations, communication channels, and barriers. Our findings reveal a strong tension between aspiration and practice. Participants were highly motivated to engage in science communication due to opportunities for collaboration, professional recognition, broader impact, and advocacy for themselves and their research. However, translating these motivations into action was frequently constrained by social anxiety, uncertainty regarding appropriate audiences and communication venues, limited feedback mechanisms, insufficient institutional guidance, and challenges associated with navigating an increasingly fragmented communication landscape. Our findings highlight the need for tailored, balanced support systems that empower software engineering PhD students to engage in science communication effectively and confidently across diverse cultural and institutional environments. We outline practical implications that offer initial guidance for addressing these challenges in future work.

cs.SE↗

arXiv2026

How Far Are We from Detecting Flaky Tests? On the Limits of Code-Based Detection

Flaky tests pass and fail on the same code version, weakening the signal of test results and disrupting continuous integration (CI) pipelines. Code-based flakiness detectors report strong benchmark results, yet their use in practice remains limited. We argue that the field is studying the wrong problem: Flakiness is not a static property of test code, which often lacks the information needed to decide whether a test is flaky. Analyzing three code-based detectors operating on test code, we found that widely used benchmarks contain shortcuts that inflate reported F1 scores and that evaluation protocols overstate generalizability. To control for these shortcuts, we curated two datasets. The first, C-IDoFT (54,468 unit tests from 57 GitHub projects), keeps a developer-confirmed subset of IDoFT's flaky tests and rebuilds only the non-flaky class from repeated executions instead of fixed versions of flaky tests. C-IDoFT is a controlled counterfactual, not a benchmark for reuse. Our CodeBERT reimplementations of two published detectors scored far above its constant baselines under the published cross-validation protocol but no better than them once projects were separated. The high scores rested on the labeling shortcut and the evaluation protocol, not on the test code. On FlakeBench, a benchmark restricted to flakiness types typically recognizable from test code, and the same project-disjoint protocol, the models identified nearly all flaky tests. The second dataset, mined from CI logs, contains 86 flaky end-to-end tests that passed and failed on the same commit. The test code and CI log yielded a cause for 42% of them; the other 58% required further execution evidence. Rather than abandoning flakiness prediction, we reframe it around whether an observed failure is flaky and how likely a test is to fail given its execution environment. Our datasets and CI-mining method support this direction.

cs.SE↗

arXiv2026

From Registry to Repository: How AI Agent Skills Are Written, Adapted, and Maintained

AI coding agents increasingly rely on skills: structured context bundles, typically a SKILL$.$md file with a YAML header and Markdown body, loaded on demand for domain knowledge, workflows, and scripts. Public registries such as skills$.$sh now host tens of thousands of skills, making them an emerging unit of reuse in agent-based software engineering. Yet skills have largely been viewed as agent capabilities rather than software artefacts whose content and evolution shape agent behaviour. We present the first empirical study of AI agent skills as engineered artefacts that are authored, reused, customised and maintained, across public registries and personal-use repositories. We mined 18,463 skills from skills$.$sh and 23,199 personal-use skills from 5,876 GitHub repositories, identifying 3,709 reuse links. LLM-based classification into SWEBOK knowledge areas (KAs) shows Software Construction dominates alongside a long tail of specialised areas. A thematic analysis of 180 skills identifies six content categories. Qualitative coding of 444 modifications reveals six themes, of which reworking operational specifications and adapting knowledge and resources are the primary target of change. Our findings show that reuse is largely a one-time copy operation: most reused skills remain near-verbatim, 53% are never modified after adoption, and subsequent local maintenance is overwhelmingly additive. Customisation primarily adapts skills to local environments, whereas evolution accretes new inline domain knowledge. Across both, a stable behavioural contract - how a skill interacts with users, monitors runtime state, and recovers from failures - remains almost untouched. These results suggest maintenance effort should focus on project-specific bindings, and that registries and tool support should enable consolidating the domain knowledge skills re-author in isolation.

cs.SE↗

arXiv2026

Context Rot in AI-Assisted Software Development: Repurposing Documentation Consistency for AI Configuration Artifacts

Developers increasingly provide AI coding assistants with persistent context through configuration files such as CLAUDE.md, AGENTS.md, and .cursorrules. These files describe code elements, architecture, and development conventions, forming the context that guides AI tool behavior across sessions. As software evolves, this context can become stale, a phenomenon we call context rot. While AI configuration artifacts are new, the underlying consistency problem connects to decades of software documentation research. Researchers have built tools to check consistency between documentation and code, spanning README files, code comments, API documentation, architecture descriptions, and installation instructions. We argue that this existing toolbox is an immediate starting point for detecting context rot, and we present a research roadmap mapping documentation consistency approaches to corresponding problems in this new setting. As preliminary evidence, applying an existing README/wiki consistency checker to a statistically representative sample of 356 repositories identifies stale code element references in 23.0% of repositories, showing that traditional documentation consistency tools can already surface context rot.

cs.SE↗

arXiv2026

The Impact of Configuring Agentic AI Coding Tools on Build-vs-Buy Decisions: A Study Protocol

Agentic AI coding tools write code with increasing autonomy and in doing so decide when to import a library and when to implement functionality from scratch. These decisions, whether to build functionality from scratch or buy into an external library, hereafter build-versus-buy, carry direct consequences for software security, licensing compliance, performance, and long-term maintainability. Yet no controlled experimental study has examined what governs build-versus-buy decisions in agentic AI coding tools. Configuration mechanisms, i.e., the means by which developers tailor agentic AI coding tool behavior to a project or workflow, are one of the primary means by which practitioners can influence these decisions. However, it is unclear which configuration mechanisms influence build-versus-buy decisions most effectively. We present a pre-registered protocol to study how configuration mechanisms alter build-versus-buy behavior in two popular agentic AI coding tools: Claude Code and OpenAI Codex. We will execute controlled programming tasks drawn from a benchmark of staged projects, each constructed around identifiable build-versus-buy points, and will manipulate the configuration supplied to each tool, ranging from no configuration, through context files with soft preferences and explicit prohibitions, to Skills (instructions that can be autonomously discovered), MCP-enabled library discovery tools, and permission controls, measuring which libraries the tool selects, whether it discloses newly introduced libraries, and whether those disclosures are complete and accurate. Nine pre-registered hypotheses structure the protocol. The resulting benchmark dataset and analysis pipeline will be released as a reusable artifact for evaluating build-versus-buy behavior in agentic AI coding tools.

cs.SE↗

arXiv2026

A Dataset of Agentic AI Coding Tool Configurations

Agentic AI coding tools such as Claude Code and OpenAI Codex execute multi-step coding tasks with limited human oversight. To steer these tools, developers create repository-level configuration artifacts (e.g., Markdown files) for configuration mechanisms such as Context Files, Skills, Rules, and Hooks. There is no curated dataset yet that captures these configurations at scale. This dataset, collected from open-source GitHub repositories, fills that gap. We selected 40,585 actively maintained repositories through metadata filtering, classified them using GPT-5.2 to identify 36,710 as belonging to engineered software projects, and systematically detected configuration artifacts in these repositories. The dataset covers 4,738 repositories across five tools (Claude Code, GitHub Copilot, OpenAI Codex, Cursor, Gemini) and eight configuration mechanisms. We collected 15,591 configuration artifacts, the full content of 18,167 configuration files associated with these configuration artifacts, and 148,519 AI-co-authored commits. The dataset and the construction pipeline are publicly available on Zenodo under CC BY 4.0. An interactive website allows researchers to browse and explore the data. This data supports research on context engineering, AI tool adoption patterns, and human-AI collaboration.

cs.SE↗

arXiv2026

Operationalizing Ethics for AI Agents: How Developers Encode Values into Repository Context Files

As AI coding agents become embedded in software development workflows, developers are beginning to operationalize ethical principles by encoding behavioral rules into repository-level context files for AI agents, such as AGENTS.md files. Rather than examining the ethics of AI agents in the abstract, this vision paper investigates how ethics and values are already being translated for AI agents into actionable instructions that shape agent behavior. Through a preliminary investigation, we find that developers are already embedding guidance related to fairness, accessibility, sustainability, tone, and privacy. These artifacts function as a developer-authored governance layer, translating abstract principles into situated, natural-language directives within development workflows. We outline a research agenda for studying this emerging practice, including how encoded values vary across communities, what governance dynamics emerge when multiple contributors negotiate these files, and whether agents reliably adhere to the constraints specified. Understanding how ethics and values are operationalized for AI agents is essential to ground AI governance in modern software engineering practice.

cs.SE↗

arXiv2026

AI Slop and the Software Commons

In this article, we argue that AI slop in software is creating a tragedy of the commons. Individual productivity gains from AI-generated content externalize costs onto reviewer capacity, codebase integrity, public knowledge resources, collaborative trust, and the talent pipeline. AI slop is cheap to generate and expensive to review, and the review layer is already thin. Commons problems are not solved by individual restraint. We outline concrete next steps for tool developers, team leads, and educators, grounded in Ostrom's design principles for enduring commons institutions.

cs.SE↗

arXiv2026

"An Endless Stream of AI Slop": How Developers Discuss the Burden of AI-Assisted Software Development

"AI slop", that is, low-quality AI-generated content, is increasingly affecting software development, from generated code and pull requests to documentation and bug reports. However, there is limited empirical research on how developers perceive and respond to this phenomenon. We qualitatively analyzed how developers discuss AI slop in 1,154 Reddit and Hacker News posts, developing a codebook of 15 codes organized into three thematic clusters: Review Friction (how AI slop burdens reviewers, erodes trust, and prompts countermeasures), Quality Degradation (damage to codebases, knowledge resources, and developer competence), and Forces and Consequences (systemic incentives, mandated adoption, craft erosion, and workforce disruption). Our findings frame AI slop as a tragedy of the commons, where individual productivity gains externalize costs onto reviewers, maintainers, and the broader community. We report the concerns developers raise and the mitigation strategies they propose, with implications for tool developers, team leads, and educators.

cs.SE↗

arXiv2026

Harness Engineering for Agentic AI Coding Tools: An Exploratory Study

Agentic AI coding tools increasingly automate software development tasks. Developers can configure these tools through versioned repository-level artifacts such as Markdown and JSON files. We present a systematic analysis of configuration mechanisms for agentic AI coding tools, covering Claude Code, GitHub Copilot, Cursor, Gemini, and Codex. We identify eight configuration mechanisms spanning from static context to executable and external integrations and, in an empirical study of 2,853 GitHub repositories, examine whether and how they are adopted, with a detailed analysis of Context Files, Skills, and Subagents. First, Context Files dominate the configuration landscape and are often the sole mechanism in a repository, with AGENTS$.$md emerging as an interoperable standard across tools. Second, few repositories adopt advanced mechanisms such as Skills and Subagents. Skills predominantly rely on static instructions rather than executable scripts. Third, distinct configuration practices are forming around different tools, with Claude Code users employing the broadest range of mechanisms. These findings establish an empirical baseline for understanding how developers configure agentic tools, suggest that AGENTS$.$md serves as a natural starting point, and motivate longitudinal and experimental research on how configuration strategies evolve and affect agent performance.

cs.SE↗

arXiv2026

Forecasting Developer Environments with GenAI: A Research Perspective

Generative Artificial Intelligence (GenAI) models are achieving remarkable performance in various tasks, including code generation, testing, code review, and program repair. The ability to increase the level of abstraction away from writing code has the potential to change the Human-AI interaction within the integrated development environment (IDE). To explore the impact of GenAI on IDEs, 33 experts from the Software Engineering, Artificial Intelligence, and Human-Computer Interaction domains gathered to discuss challenges and opportunities at Shonan Meeting 222, a four-day intensive research meeting. Four themes emerged as areas of interest for researchers and practitioners.

cs.SE↗

arXiv2026

Can We Classify Flaky Tests Using Only Test Code? An LLM-Based Empirical Study

Flaky tests yield inconsistent results when they are repeatedly executed on the same code revision. They interfere with automated quality assurance of code changes and hinder efficient software testing. Previous work evaluated approaches to train machine learning models to classify flaky tests based on identifiers in the test code. However, the resulting classifiers have been shown to lack generalizability, hindering their applicability in practical environments. Recently, pre-trained Large Language Models (LLMs) have shown the capability to generalize across various tasks. Thus, they represent a promising approach to address the generalizability problem of previous approaches. In this study, we evaluated three LLMs (two general-purpose models, one code-specific model) using three prompting techniques on two benchmark datasets from prior studies on flaky test classification. Furthermore, we manually investigated 50 samples from the given datasets to determine whether classifying flaky tests based only on test code is feasible for humans. Our findings indicate that LLMs struggle to classify flaky tests given only the test code. The results of our best prompt-model combination were only marginally better than random guessing. In our manual analysis, we found that the test code does not necessarily contain sufficient information for a flakiness classification. Our findings motivate future work to evaluate LLMs for flakiness classification with additional context, for example, using retrieval-augmented generation or agentic AI.

cs.SE↗

arXiv2026

Flaky Tests in a Large Industrial Database Management System: An Empirical Study of Fixed Issue Reports for SAP HANA

Flaky tests yield different results when executed multiple times for the same version of the source code. Thus, they provide an ambiguous signal about the quality of the code and interfere with the automated assessment of code changes. While a variety of factors can cause test flakiness, approaches to fix flaky tests are typically tailored to address specific causes. However, the prevalent root causes of flaky tests can vary depending on the programming language, application domain, or size of the software project. Since manually labeling flaky tests is time-consuming and tedious, this work proposes an LLMs-as-annotators approach that leverages intra- and inter-model consistency to label issue reports related to fixed flakiness issues with the relevant root cause category. This allows us to gain an overview of prevalent flakiness categories in the issue reports. We evaluated our labeling approach in the context of SAP HANA, a large industrial database management system. Our results suggest that SAP HANA's tests most commonly suffer from issues related to concurrency (23%, 130 of 559 analyzed issue reports). Moreover, our results suggest that different test types face different flakiness challenges. Therefore, we encourage future research on flakiness mitigation to consider evaluating the generalizability of proposed approaches across different test types.

cs.SE↗

arXiv2026

On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents

AI coding agents such as Codex and Claude Code are increasingly used to autonomously contribute to software repositories. However, little is known about how repository-level configuration artifacts affect operational efficiency of the agents. In this paper, we study the impact of AGENTS$.$md files on the runtime and token consumption of AI coding agents operating on GitHub pull requests. We analyze 10 repositories and 124 pull requests, executing agents under two conditions: with and without an AGENTS$.$md file. We measure wall-clock execution time and token usage during agent execution. Our results show that the presence of AGENTS$.$md is associated with a lower median runtime ($\Delta 28.64$%) and reduced output token consumption ($\Delta 16.58$%), while maintaining a comparable task completion behavior. Based on these results, we discuss immediate implications for the configuration and deployment of AI coding agents in practice, and outline a broader research agenda on the role of repository-level instructions in shaping the behavior, efficiency, and integration of AI coding agents in software development workflows.

cs.SE↗

arXiv2026

Political and Ideological Pressure in Software Engineering Research: The Case of DEI Backlash

Political and ideological pressures shape global research. Recently, these pressures have become particularly visible in research related to diversity, equity, and inclusion (DEI). Drastic changes in national funding and governmental guidance, especially in the US, have affected the global software engineering research ecosystem. The impacts of these pressures on research are not always direct, as they operate at multiple levels. However, what is clear is that these pressures affect every field, including software engineering (SE), despite the belief that our field is politically and ideologically neutral. In this position paper, we examine cases of political and ideological pressures on the SE research ecosystem. We investigate the community's perceptions of political and ideological pressures by analyzing community survey responses and outlining case examples of DEI backlash in SE research across three levels: macro, meso, and micro. Our research shows how recent political and ideological pressures have affected SE research across these levels, and, as a result, we propose actionable steps for the community to address these issues at different levels.

cs.SE↗

arXiv2026

Ethics of Care for Software Engineering

Software engineering researchers repeatedly argue that the impact of their research on industrial practice, while desired and intended, is rarely achieved. We believe that a possible explanation of this phenomenon is the opposition of "caring about" and "caring for", based on the ethics of care. Indeed, while software engineering is collaborative and hence builds on interpersonal relations, researchers tend to care about "industrial impact" and "practitioners" in abstract terms, but rarely care for specific individuals working in specific contexts facing specific challenges. In this position paper, we advocate for the adoption of ethics of care in software engineering and discuss the implications of this adoption for researchers and conference organizers.

cs.SE↗