arXiv subjects

Matthias Galster

Publications and source records attributed to Matthias Galster.

16 recordsLinked to original sources

arXiv2026

An Exploratory Study of Agent Plans for Agentic AI Coding Tools in Open-Source Software

Repository-level configuration artifacts allow developers to provide guidance for agentic AI coding tools, such as Claude Code, Gemini, etc. Although prior research has examined repository-shared context files that capture project-level instructions and conventions (e.g., AGENTS.md files), little is known about more task-oriented artifacts such as Agent Plans. We present an exploratory study of Agent Plans in open-source software repositories, examining how plan files are preserved, which development activities they support, and what information they provide to guide agent execution. We screened 36,710 GitHub repositories belonging to engineered software projects and identified 85 Markdown plan files from 10 repositories. Within this highly concentrated corpus, Agent Plans supported several kinds of software engineering work, including maintenance, design, construction, quality-related work, and process support. They also provided task-oriented execution guidance, most commonly through implementation steps, concrete files and locations, and testing and validation information. Overall, repository-preserved Agent Plans under these tool-specific directories appear to be a narrow but informative artifact for studying task intent and execution guidance in human-agent workflows.

cs.SE↗

arXiv2026

Identifying Quality Indicators in Student Self-Reflections in Software Engineering

Context: Reflection is a fundamental skill in software engineering education, particularly in project-based courses where students learn through extended group work and need to develop their ability to reflect iteratively throughout their work. For students to benefit from reflection, their written reflections need to be assessed so that feedback can guide and improve their reflective practice. However, manually assessing written reflections to guide reflections is time-consuming, and often results in broad, non-specific feedback for a student to improve. Objective: This study builds on reflective writing frameworks to produce an eight-indicator scheme for assessing student reflections in software engineering. Furthermore, this study validates an automated classifier for assessing reflections against the framework, enabling scalable and structured feedback whilst reducing instructor workload. Method: We adapted existing reflection frameworks through iterative refinement to create our eight-indicator framework. Three annotators labelled student reflection texts, establishing moderate to reliable inter-rater agreement. We then trained and evaluated multiple encoder-only transformer models and compared them with decoder-only large language models using zero-shot prompting. Results: The fine-tuned RoBERTa model achieved the strongest performance, substantially outperforming decoder-only models in both accuracy and speed. The classifier demonstrated human-level agreement on most indicators whilst enabling near-instantaneous classification. We provide two model variants optimised for different assessment priorities. Conclusions: Our fine-tuned encoder-only models enable efficient automated assessment of reflective writing. The framework and automated classifier offer a means to provide timely, structured feedback on student reflections in software engineering.

cs.SE↗

arXiv2026

The Impact of Configuring Agentic AI Coding Tools on Build-vs-Buy Decisions: A Study Protocol

Agentic AI coding tools write code with increasing autonomy and in doing so decide when to import a library and when to implement functionality from scratch. These decisions, whether to build functionality from scratch or buy into an external library, hereafter build-versus-buy, carry direct consequences for software security, licensing compliance, performance, and long-term maintainability. Yet no controlled experimental study has examined what governs build-versus-buy decisions in agentic AI coding tools. Configuration mechanisms, i.e., the means by which developers tailor agentic AI coding tool behavior to a project or workflow, are one of the primary means by which practitioners can influence these decisions. However, it is unclear which configuration mechanisms influence build-versus-buy decisions most effectively. We present a pre-registered protocol to study how configuration mechanisms alter build-versus-buy behavior in two popular agentic AI coding tools: Claude Code and OpenAI Codex. We will execute controlled programming tasks drawn from a benchmark of staged projects, each constructed around identifiable build-versus-buy points, and will manipulate the configuration supplied to each tool, ranging from no configuration, through context files with soft preferences and explicit prohibitions, to Skills (instructions that can be autonomously discovered), MCP-enabled library discovery tools, and permission controls, measuring which libraries the tool selects, whether it discloses newly introduced libraries, and whether those disclosures are complete and accurate. Nine pre-registered hypotheses structure the protocol. The resulting benchmark dataset and analysis pipeline will be released as a reusable artifact for evaluating build-versus-buy behavior in agentic AI coding tools.

cs.SE↗

arXiv2026

A Dataset of Agentic AI Coding Tool Configurations

Agentic AI coding tools such as Claude Code and OpenAI Codex execute multi-step coding tasks with limited human oversight. To steer these tools, developers create repository-level configuration artifacts (e.g., Markdown files) for configuration mechanisms such as Context Files, Skills, Rules, and Hooks. There is no curated dataset yet that captures these configurations at scale. This dataset, collected from open-source GitHub repositories, fills that gap. We selected 40,585 actively maintained repositories through metadata filtering, classified them using GPT-5.2 to identify 36,710 as belonging to engineered software projects, and systematically detected configuration artifacts in these repositories. The dataset covers 4,738 repositories across five tools (Claude Code, GitHub Copilot, OpenAI Codex, Cursor, Gemini) and eight configuration mechanisms. We collected 15,591 configuration artifacts, the full content of 18,167 configuration files associated with these configuration artifacts, and 148,519 AI-co-authored commits. The dataset and the construction pipeline are publicly available on Zenodo under CC BY 4.0. An interactive website allows researchers to browse and explore the data. This data supports research on context engineering, AI tool adoption patterns, and human-AI collaboration.

cs.SE↗

arXiv2026

Towards Understanding Views on Combining Videos and Gamification in Software Engineering Training

Watching training videos passively leads to superficial learning. Adding gamification can increase engagement. We study how software engineering students and industry practitioners view gamifying video-based training. We conducted a survey with students and professionals. Students and professionals share similar perceptions toward video-based training in general and support combining gamification and video-based training. Our findings can inform the design of gamified training solutions for software engineers.

cs.SE↗

arXiv2026

Harness Engineering for Agentic AI Coding Tools: An Exploratory Study

Agentic AI coding tools increasingly automate software development tasks. Developers can configure these tools through versioned repository-level artifacts such as Markdown and JSON files. We present a systematic analysis of configuration mechanisms for agentic AI coding tools, covering Claude Code, GitHub Copilot, Cursor, Gemini, and Codex. We identify eight configuration mechanisms spanning from static context to executable and external integrations and, in an empirical study of 2,853 GitHub repositories, examine whether and how they are adopted, with a detailed analysis of Context Files, Skills, and Subagents. First, Context Files dominate the configuration landscape and are often the sole mechanism in a repository, with AGENTS$.$md emerging as an interoperable standard across tools. Second, few repositories adopt advanced mechanisms such as Skills and Subagents. Skills predominantly rely on static instructions rather than executable scripts. Third, distinct configuration practices are forming around different tools, with Claude Code users employing the broadest range of mechanisms. These findings establish an empirical baseline for understanding how developers configure agentic tools, suggest that AGENTS$.$md serves as a natural starting point, and motivate longitudinal and experimental research on how configuration strategies evolve and affect agent performance.

cs.SE↗

arXiv2026

On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents

AI coding agents such as Codex and Claude Code are increasingly used to autonomously contribute to software repositories. However, little is known about how repository-level configuration artifacts affect operational efficiency of the agents. In this paper, we study the impact of AGENTS$.$md files on the runtime and token consumption of AI coding agents operating on GitHub pull requests. We analyze 10 repositories and 124 pull requests, executing agents under two conditions: with and without an AGENTS$.$md file. We measure wall-clock execution time and token usage during agent execution. Our results show that the presence of AGENTS$.$md is associated with a lower median runtime ($\Delta 28.64$%) and reduced output token consumption ($\Delta 16.58$%), while maintaining a comparable task completion behavior. Based on these results, we discuss immediate implications for the configuration and deployment of AI coding agents in practice, and outline a broader research agenda on the role of repository-level instructions in shaping the behavior, efficiency, and integration of AI coding agents in software development workflows.

cs.SE↗

arXiv2025

Context Engineering for AI Agents in Open-Source Software

GenAI-based coding assistants have disrupted software development. The next generation of these tools is agent-based, operating with more autonomy and potentially without human oversight. Like human developers, AI agents require contextual information to develop solutions that are in line with the standards, policies, and workflows of the software projects they operate in. Vendors of popular agentic tools (e.g., Claude Code) recommend maintaining version-controlled Markdown files that describe aspects such as the project structure, code style, or building and testing. The content of these files is then automatically added to each prompt. Recently, AGENTS$.$md has emerged as a potential standard that consolidates existing tool-specific formats. However, little is known about whether and how developers adopt this format. Therefore, in this paper, we present the results of a preliminary study investigating the adoption of AI context files in 466 open-source software projects. We analyze the information that developers provide in AGENTS$.$md files, how they present that information, and how the files evolve over time. Our findings indicate that there is no established content structure yet and that there is a lot of variation in terms of how context is provided (descriptive, prescriptive, prohibitive, explanatory, conditional). Our commit-level analysis provides first insights into the evolution of the provided context. AI context files provide a unique opportunity to study real-world context engineering. In particular, we see great potential in studying which structural or presentational modifications can positively affect the quality of the generated content.

cs.SE↗

arXiv2024

Towards Quantifying Requirements Technical Debt for Software Requirements concerning Veracity: A Perspective and Research Roadmap

Software practitioners can make sub-optimal decisions concerning requirements during gathering, documenting, prioritizing, and implementing requirements as software features or architectural design decisions -- this is captured by the metaphor `Requirements Technical Debt (RTD).' In our prior work, we developed a conceptual model to understand the quantification of RTD and support its management. In this paper, we present our perspective and the vision to apply the lens of RTD to software requirements concerning veracity, i.e., requirements related to truth, trust, authenticity, and demonstrability in software-intensive systems. Our goal is to cultivate awareness of veracity as an important concern and eventually support the management of RTD for software requirements concerning veracity, what we term as `Veracity Debt,' through its quantification.

cs.SE↗

arXiv2024

Exploring Data Management Challenges and Solutions in Agile Software Development: A Literature Review and Practitioner Survey

Context: Managing data related to a software product and its development poses significant challenges for software projects and agile development teams. These include integrating data from diverse sources and ensuring data quality amidst continuous change and adaptation. Objective: The paper systematically explores data management challenges and potential solutions in agile projects, aiming to provide insights into data management challenges and solutions for both researchers and practitioners. Method: We employed a mixed-methods approach, including a systematic literature review (SLR) to understand the state-of-research followed by a survey with practitioners to reflect on the state-of-practice. The SLR reviewed 45 studies, identifying and categorizing data management aspects along with their associated challenges and solutions. The practitioner survey captured practical experiences and solutions from 32 industry practitioners who were significantly involved in data management to complement the findings from the SLR. Results: Our findings identified major data management challenges in practice, such as managing data integration processes, capturing diverse data, automating data collection, and meeting real-time analysis requirements. To address the challenges, solutions such as automation tools, decentralized data management practices, and ontology-based approaches have been identified. The solutions enhance data integration, improve data quality, and enable real-time decision-making by providing flexible frameworks tailored to agile project needs. Conclusion: The study pinpointed significant challenges and actionable solutions in data management for agile software development. Our findings provide practical implications for practitioners and researchers, emphasizing the development of effective data management practices and tools to address those challenges and improve project success.

cs.SE↗

arXiv2023

A/B Testing: A Systematic Literature Review

In A/B testing two variants of a piece of software are compared in the field from an end user's point of view, enabling data-driven decision making. While widely used in practice, no comprehensive study has been conducted on the state-of-the-art in A/B testing. This paper reports the results of a systematic literature review that analyzed 141 primary studies. The results shows that the main targets of A/B testing are algorithms and visual elements. Single classic A/B tests are the dominating type of tests. Stakeholders have three main roles in the design of A/B tests: concept designer, experiment architect, and setup technician. The primary types of data collected during the execution of A/B tests are product/system data and user-centric data. The dominating use of the test results are feature selection, feature rollout, and continued feature development. Stakeholders have two main roles during A/B test execution: experiment coordinator and experiment assessor. The main reported open problems are enhancement of proposed approaches and their usability. Interesting lines for future research include: strengthen the adoption of statistical methods in A/B testing, improving the process of A/B testing, and enhancing the automation of A/B testing.

cs.SE↗

arXiv2023

Human-centric Literature on Trust for SfTI Veracity Spearhead

This article summarizes the literature on trust of digital technologies from a human-centric perspective. We summarize literature on trust in face-to-face interactions from other fields, followed by a discussion of organizational trust, technology-mediated trust, trust of software products, trust of AI, and blockchain. This report was created for the Science for Technological Innovation Veracity Spearhead supported by New Zealand's National Science Challenges.

cs.CY↗

arXiv2023

Barriers for Social Inclusion in Online Software Engineering Communities -- A Study of Offensive Language Use in Gitter Projects

Social inclusion is a fundamental feature of thriving societies. This paper first investigates barriers for social inclusion in online Software Engineering (SE) communities, by identifying a set of 11 attributes and organising them as a taxonomy. Second, by applying the taxonomy and analysing language used in the comments posted by members in 189 Gitter projects (with > 3 million comments), it presents the evidence for the social exclusion problem. It employs a keyword-based search approach for this purpose. Third, it presents a framework for improving social inclusion in SE communities.

cs.SE↗

arXiv2023

Towards Understanding Provenance in Industry

Context: Trustworthiness of software has become a first-class concern of users (e.g., to understand software-made decisions). Also, there is increasing demand to demonstrate regulatory compliance of software and end users want to understand how software-intensive systems make decisions that affect them. Objective: We aim to provide a step towards understanding provenance needs of the software industry to support trustworthy software. Provenance is information about entities, activities, and people involved in producing data, software, or output of software, and used to assess software quality, reliability and trustworthiness of digital products and services. Method: Based on data from in-person and questionnaire-based interviews with professionals in leadership roles we develop an ``influence map'' to analyze who drives provenance, when provenance is relevant, what is impacted by provenance and how provenance can be managed. Results: The influence map helps decision makers navigate concerns related to provenance. It can also act as a checklist for initial provenance analyses of systems. It is empirically-grounded and designed bottom-up (based on perceptions of practitioners) rather than top-down (from regulations or policies). Conclusion: We present an imperfect first step towards understanding provenance based on current perceptions and offer a preliminary view ahead.

cs.SE↗

arXiv2022

Self-Adaptation in Industry: A Survey

Computing systems form the backbone of many areas in our society, from manufacturing to traffic control, healthcare, and financial systems. When software plays a vital role in the design, construction, and operation, these systems are referred as software-intensive systems. Self-adaptation equips a software-intensive system with a feedback loop that either automates tasks that otherwise need to be performed by human operators or deals with uncertain conditions. Such feedback loops have found their way to a variety of practical applications; typical examples are an elastic cloud to adapt computing resources and automated server management to respond quickly to business needs. To gain insight into the motivations for applying self-adaptation in practice, the problems solved using self-adaptation and how these problems are solved, and the difficulties and risks that industry faces in adopting self-adaptation, we performed a large-scale survey. We received 184 valid responses from practitioners spread over 21 countries. Based on the analysis of the survey data, we provide an empirically grounded overview of state-of-the-practice in the application of self-adaptation. From that, we derive insights for researchers to check their current research with industrial needs, and for practitioners to compare their current practice in applying self-adaptation. These insights also provide opportunities for the application of self-adaptation in practice and pave the way for future industry-research collaborations.

cs.SE↗

arXiv2021

An Exploratory Study on Architectural Knowledge in Issue Tracking Systems

Software developers use issue trackers (e.g. Jira) to manage defects, bugs, tasks, change requests, etc. In this paper we explore (a) how architectural knowledge concepts (e.g. architectural component behavior, contextual constraints) are textually represented in issues (e.g. as adjectives), (b) which architectural knowledge concepts commonly occur in issues, and (c) which architectural knowledge concepts appear together. We analyzed issues in the Jira issue trackers of three large Apache projects. To identify ``architecturally relevant'' issues, we linked issues to architecturally relevant source code changes in the studied systems. We then developed a code book by manually labeling a subset of issues. After reaching conceptual saturation, we coded remaining issues. Our findings support empirically-grounded search tools to identify architectural knowledge concepts in issues for future reuse.

cs.SE↗