arXiv subjects

Hajimu Iida

Publications and source records attributed to Hajimu Iida.

17 recordsLinked to original sources

arXiv2026

What Are Developers Actually Discussing When Visual Regression Tests Fail?

Visual Regression Tests (VRTs) are widely adopted as a mechanism for detecting unintended visual changes in user interfaces. By design, VRTs operate on rendered pixel output, and the prevailing assumption is that they catch stylistic regressions such as layout shifts, color mismatches, and font alterations. We conduct an empirical analysis of 307 pull requests (PRs) from 103 GitHub repositories that incorporate VRT results via Chromatic, comparing them against 299 PRs that contain image attachments but no VRT (Visual PRs). Quantitatively, VRT-PRs show no significant acceptance-rate difference, but exhibit a 3.8 times longer median resolution time, 10 times more discussion comments, and 1.75 to 4.5 times larger code changes than Visual PRs. VRT results are typically shared around the midpoint of the review process, sustaining ongoing discussion rather than serving only as a final check. Through a card-sorting analysis of 189 VRT-flagged issues, we identify seven defect categories assigned to the analyzed issues: Layout (39.7\%), Appearance (27.5\%), Color (14.8\%), Text (9.5\%), State (6.9\%), Test (6.3\%), and Image (4.2\%). The three most frequent categories are stylistic, while approximately 18.5\% of analyzed issues (35/189) involve non-stylistic origins, including undefined component state (13 cases), content disappearance (17 cases across multiple categories), and visually imperceptible regressions (5 cases). We further document cases in which VRT detected visual regressions originating from code changes in seemingly unrelated files, exposing non-local effects that no targeted test would have been written to catch. These observations indicate that, in addition to its primary role as a stylistic checker, VRT functions as a secondary detector of unintended consequences of code changes, with implications for how VRT should be integrated into the maintenance toolchain.

cs.SE↗

arXiv2026

To What Extent Does Agent-generated Code Require Maintenance? An Empirical Study

LLM-based autonomous coding agents have reshaped software development. While these agents excel at code generation, open questions persist about the long-term maintainability of AI-generated code. This study empirically investigates the maintenance extent, human involvement, and modification types of AI-generated files versus human-authored code. Using the AIDev dataset of AI-generated pull requests and GitHub, we analyzed over 1,000 files and approximately 3,200 changes from 100 popular repositories. Our findings show that: (i) AI-generated files receive less frequent maintenance than human-authored code, with updates affecting only a small fraction of file size; (ii) the most frequent modifications to AI code are feature extensions, whereas human updates focus on bug fixes, and (iii) human developers perform the large majority of this maintenance.

cs.SE↗

arXiv2026

Do AI Agents Really Improve Code Readability?

Code readability is fundamental to software quality and maintainability. Poor readability extends development time, increases bug-inducing risks, and contributes to technical debt. With the rapid advancement of Large Language Models, AI agent-based approaches have emerged as a promising paradigm for automated refactoring, capable of decomposing complex tasks through autonomous planning and execution. While prior studies have examined refactoring by AI agents, these analyses cover all forms of refactoring, including performance optimization and structural improvement. As a result, the extent to which AI agent-based refactoring specifically improves code readability remains unclear. This study investigates the impact of AI agent-based refactoring on code readability. We extracted commits containing readability-related keywords from the AIDev dataset and analyzed changes in readability metrics before and after each commit, covering 403 commits evaluated using multiple quantitative metrics. Our results indicate that AI agents primarily target logic complexity (42.4%) and documentation improvements (24.2%) rather than surface-level aspects like naming conventions or formatting. However, contrary to expectations, readability-focused commits often degraded traditional quality metrics: the Maintainability Index decreased in 56.1% of commits, while Cyclomatic Complexity increased in 42.7%.

cs.SE↗

arXiv2026

Testing with AI Agents: An Empirical Study of Test Generation Frequency, Quality, and Coverage

Agent-based coding tools have transformed software development practices. Unlike prompt-based approaches that require developers to manually integrate generated code, these agent-based tools autonomously interact with repositories to create, modify, and execute code, including test generation. While many developers have adopted agent-based coding tools, little is known about how these tools generate tests in real-world development scenarios or how AI-generated tests compare to human-written ones. This study presents an empirical analysis of test generation by agent-based coding tools using the AIDev dataset. We extracted 2,232 commits containing test-related changes and investigated three aspects: the frequency of test additions, the structural characteristics of the generated tests, and their impact on code coverage. Our findings reveal that (i) AI authored 16.4% of all commits adding tests in real-world repositories, (ii) AI-generated test methods exhibit distinct structural patterns, featuring longer code and a higher density of assertions while maintaining lower cyclomatic complexity through linear logic, and (iii) AI-generated tests contribute to code coverage comparable to human-written tests, frequently achieving positive coverage gains across several projects.

cs.SE↗

arXiv2026

What to Cut? Predicting Unnecessary Methods in Agentic Code Generation

Agentic Coding, powered by autonomous agents such as GitHub Copilot and Cursor, enables developers to generate code, tests, and pull requests from natural language instructions alone. While this accelerates implementation, it produces larger volumes of code per pull request, shifting the burden from implementers to reviewers. In practice, a notable portion of AI-generated code is eventually deleted during review, yet reviewers must still examine such code before deciding to remove it. No prior work has explored methods to help reviewers efficiently identify code that will be removed.In this paper, we propose a prediction model that identifies functions likely to be deleted during PR review. Our results show that functions deleted for different reasons exhibit distinct characteristics, and our model achieves an AUC of 87.1%. These findings suggest that predictive approaches can help reviewers prioritize their efforts on essential code.

cs.SE↗

arXiv2026

Does Programming Language Matter? An Empirical Study of Fuzzing Bug Detection

Fuzzing has become a popular technique for automatically detecting vulnerabilities and bugs by generating unexpected inputs. In recent years, the fuzzing process has been integrated into continuous integration workflows (i.e., continuous fuzzing), enabling short and frequent testing cycles. Despite its widespread adoption, prior research has not examined whether the effectiveness of continuous fuzzing varies across programming languages. This study conducts a large-scale cross-language analysis to examine how fuzzing bug characteristics and detection efficiency differ among languages. We analyze 61,444 fuzzing bugs and 999,248 builds from 559 OSS-Fuzz projects categorized by primary language. Our findings reveal that (i) C++ and Rust exhibit higher fuzzing bug detection frequencies, (ii) Rust and Python show low vulnerability ratios but tend to expose more critical vulnerabilities, (iii) crash types vary across languages and unreproducible bugs are more frequent in Go but rare in Rust, and (iv) Python attains higher patch coverage but suffers from longer time-to-detection. These results demonstrate that fuzzing behavior and effectiveness are strongly shaped by language design, providing insights for language-aware fuzzing strategies and tool development.

cs.SE↗

arXiv2025

Agent READMEs: An Empirical Study of Context Files for Agentic Coding

Agentic coding tools receive goals written in natural language, break them down into specific tasks, and write or execute code with minimal human intervention. Central to this process are agent context files (e.g., AGENTS.md and CLAUDE.md) that provide persistent, project-level instructions. In this paper, we conduct the first large-scale empirical study of 2,303 agent context files from 1,925 repositories to characterize their structure, maintenance, and content. We find that these files are not static documentation but complex, difficultto-read artifacts that evolve like configuration code through frequent, small additions. Our content analysis of 16 instruction types shows that developers prioritize functional context, such as test procedures (75.9%), implementation details (70.8%), and architecture (68.1%). We also identify a significant gap: non-functional requirements such as security (14.8%) and performance (14.5%) are rarely specified. These findings indicate that while developers use context files to make agents functional, they provide few guardrails to ensure that agent-written code is secure or performant, highlighting the need for improved tools and practices.

cs.SE↗

arXiv2025

Agentic Refactoring: An Empirical Study of AI Coding Agents

Agentic coding tools, such as OpenAI Codex, Claude Code, and Cursor, are transforming the software engineering landscape. These AI-powered systems function as autonomous teammates capable of planning and executing complex development tasks. Agents have become active participants in refactoring, a cornerstone of sustainable software development aimed at improving internal code quality without altering observable behavior. Despite their increasing adoption, there is a critical lack of empirical understanding regarding how agentic refactoring is utilized in practice, how it compares to human-driven refactoring, and what impact it has on code quality. To address this empirical gap, we present a large-scale study of AI agent-generated refactorings in real-world open-source Java projects, analyzing 15,451 refactoring instances across 12,256 pull requests and 14,988 commits derived from the AIDev dataset. Our empirical analysis shows that refactoring is a common and intentional activity in this development paradigm, with agents explicitly targeting refactoring in 26.1% of commits. Analysis of refactoring types reveals that agentic efforts are dominated by low-level, consistency-oriented edits, such as Change Variable Type (11.8%), Rename Parameter (10.4%), and Rename Variable (8.5%), reflecting a preference for localized improvements over the high-level design changes common in human refactoring. Additionally, the motivations behind agentic refactoring focus overwhelmingly on internal quality concerns, with maintainability (52.5%) and readability (28.1%). Furthermore, quantitative evaluation of code quality metrics shows that agentic refactoring yields small but statistically significant improvements in structural metrics, particularly for medium-level changes, reducing class size and complexity (e.g., Class LOC median $\Delta$ = -15.25).

cs.SE↗

arXiv2025

Understanding Self-Admitted Technical Debt in Test Code: An Empirical Study

Developers often opt for easier but non-optimal implementation to meet deadlines or create rapid prototypes, leading to additional effort known as technical debt to improve the code later. Oftentimes, developers explicitly document the technical debt in code comments, referred to as Self-Admitted Technical Debt (SATD). Numerous researchers have investigated the impact of SATD on different aspects of software quality and development processes. However, most of these studies focus on SATD in production code, often overlooking SATD in the test code or assuming that it shares similar characteristics with SATD in production code. In fact, a significant amount of SATD is also present in the test code, with many instances not fitting into existing categories for the production code. This study aims to fill this gap and disclose the nature of SATD in the test code by examining its distribution and types. Moreover, the relation between its presence and test quality is also analyzed. Our empirical study, involving 17,766 SATD comments (14,987 from production code, 2,779 from test code) collected from 50 repositories, demonstrates that while SATD widely exists in test code, it is not directly associated with test smells. Our study also presents comprehensive categories of SATD types in the test code, and machine learning models are developed to automatically classify SATD comments based on their types for easier management. Our results show that the CodeBERT-based model outperforms other machine learning models in terms of recall and F1-score. However, the performance varies on different types of SATD.

cs.SE↗

arXiv2025

Large-Scale Empirical Analysis of Continuous Fuzzing: Insights from 1 Million Fuzzing Sessions

Software vulnerabilities are constantly being reported and exploited in software products, causing significant impacts on society. In recent years, the main approach to vulnerability detection, fuzzing, has been integrated into the continuous integration process to run in short and frequent cycles. This continuous fuzzing allows for fast identification and remediation of vulnerabilities during the development process. Despite adoption by thousands of projects, however, it is unclear how continuous fuzzing contributes to vulnerability detection. This study aims to elucidate the role of continuous fuzzing in vulnerability detection. Specifically, we investigate the coverage and the total number of fuzzing sessions when fuzzing bugs are discovered. We collect issue reports, coverage reports, and fuzzing logs from OSS-Fuzz, an online service provided by Google that performs fuzzing during continuous integration. Through an empirical study of a total of approximately 1.12 million fuzzing sessions from 878 projects participating in OSS-Fuzz, we reveal that (i) a substantial number of fuzzing bugs exist prior to the integration of continuous fuzzing, leading to a high detection rate in the early stages; (ii) code coverage continues to increase as continuous fuzzing progresses; and (iii) changes in coverage contribute to the detection of fuzzing bugs. This study provides empirical insights into how continuous fuzzing contributes to fuzzing bug detection, offering practical implications for future strategies and tool development in continuous fuzzing.

cs.SE↗

arXiv2025

An Empirical Study of Security-Policy Related Issues in Open Source Projects

GitHub recommends that projects adopt a security file that outlines vulnerability reporting procedures. However, the effectiveness and operational challenges of such files are not yet fully understood. This study aims to clarify the challenges that security files face in the vulnerability reporting process within open-source communities. Specifically, we classified and analyzed the content of 711 randomly sampled issues related to security files. We also conducted a quantitative comparative analysis of the close time and number of responses for issues concerning six community health files, including security files. Our analysis revealed that 79.5% of security file-related issues were requests to add the file, and reports that included links were closed, with a median time that was 2 days shorter. These findings offer practical insights for improving security reporting policies and community management, ultimately contributing to a more secure open-source ecosystem.

cs.SE↗

arXiv2025

Detecting and Characterizing Low and No Functionality Packages in the NPM Ecosystem

Trivial packages, small modules with low functionality, are common in the npm ecosystem and can pose security risks despite their simplicity. This paper refines existing definitions and introduce data-only packages that contain no executable logic. A rule-based static analysis method is developed to detect trivial and data-only packages and evaluate their prevalence and associated risks in the 2025 npm ecosystem. The analysis shows that 17.92% of packages are trivial, with vulnerability levels comparable to non-trivial ones, and data-only packages, though rare, also contain risks. The proposed detection tool achieves 94% accuracy (macro-F1 0.87), enabling effective large-scale analysis to reduce security exposure. This findings suggest that trivial and data-only packages warrant greater attention in dependency management to reduce potential technical debt and security exposure.

cs.SE↗

arXiv2025

On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub

Large language models (LLMs) are increasingly being integrated into software development processes. The ability to generate code and submit pull requests with minimal human intervention, through the use of autonomous AI agents, is poised to become a standard practice. However, little is known about the practical usefulness of these pull requests and the extent to which their contributions are accepted in real-world projects. In this paper, we empirically study 567 GitHub pull requests (PRs) generated using Claude Code, an agentic coding tool, across 157 diverse open-source projects. Our analysis reveals that developers tend to rely on agents for tasks such as refactoring, documentation, and testing. The results indicate that 83.8% of these agent-assisted PRs are eventually accepted and merged by project maintainers, with 54.9% of the merged PRs are integrated without further modification. The remaining 45.1% require additional changes benefit from human revisions, especially for bug fixes, documentation, and adherence to project-specific standards. These findings suggest that while agent-assisted PRs are largely acceptable, they still benefit from human oversight and refinement.

cs.SE↗

arXiv2025

On the Use of Agentic Coding Manifests: An Empirical Study of Claude Code

Agentic coding tools receive goals written in natural language as input, break them down into specific tasks, and write/execute the actual code with minimal human intervention. Key to this process are agent manifests, configuration files (such as Claude.md) that provide agents with essential project context, identity, and operational rules. However, the lack of comprehensive and accessible documentation for creating these manifests presents a significant challenge for developers. We analyzed 253 Claude.md files from 242 repositories to identify structural patterns and common content. Our findings show that manifests typically have shallow hierarchies with one main heading and several subsections, with content dominated by operational commands, technical implementation notes, and high-level architecture.

cs.SE↗

arXiv2019

Investigating Compilation Errors of Students Learning Haskell

While functional programming is an efficient way to express complex software, functional programming languages have a steep learning curve. Haskell can be challenging to learn for students who were only introduced to imperative programming. It is important to look for methods and tools that may reduce the difficulty of learning functional programming. Finding methods to help students requires understanding the errors that students make while learning Haskell. There are several previous studies revealing data about Haskell compiler errors, but they do not focus on the analysis of the compiler errors or they only study a certain kind of compiler errors. This study investigates compilation errors of novice Haskell students and make suggestions on how their learning efficiency can be improved. Unlike previous studies we focus on uncovering the root problems with the student solutions by analysing samples of their submissions.

cs.PL↗

arXiv2018

A Manga-Driven System Requirements Development PBL Exercise

We conducted a Project-Based Learning (PBL)-type exercise incorporating Japanese cartoon (manga) techniques into Requirements Development (RD) processes. Manga has established techniques, such as those for character setting and story development, that we thought are also valid for RD processes. Using this manga-driven method, students were able to clarify high-level project goals early in the development life-cycle, and succeeded in defining high quality and unique system ideas.

cs.SE↗

arXiv2015

A Simple Multipath OpenFlow Controller using topology-based algorithm for Multipath TCP

Multipath TCP, or MPTCP, is a widely-researched mechanism that allows a single application-level connection to be split to more than one TCP stream, and consequently more than one network interface, as opposed to the traditional TCP/IP model. Being a transport layer protocol, MPTCP can easily interact between the application using it and the network supporting it. However, MPTCP does not have control of its own route. Default IP routing behavior generally takes all traffic through the shortest or best-metric path. However, this behavior may actually cause paths to collide with each other, creating contention for bandwidth in a number of edges. This can result in a bottleneck which limits the throughput of the network. Therefore, a multipath routing mechanism is necessary to ensure smooth operation of MPTCP. We created smoc, a Simple Multipath OpenFlow Controller, that uses only topology information of the network to avoid collision where possible. Evaluation of smoc in a virtual local-area and a physical wide-area SDNs showed favorable results as smoc provided better performance than simple or spanning-tree routing mechanisms.

cs.NI↗