arXiv subjects

Brittany Reid

Publications and source records attributed to Brittany Reid.

15 recordsLinked to original sources

arXiv2026

What Are Developers Actually Discussing When Visual Regression Tests Fail?

Visual Regression Tests (VRTs) are widely adopted as a mechanism for detecting unintended visual changes in user interfaces. By design, VRTs operate on rendered pixel output, and the prevailing assumption is that they catch stylistic regressions such as layout shifts, color mismatches, and font alterations. We conduct an empirical analysis of 307 pull requests (PRs) from 103 GitHub repositories that incorporate VRT results via Chromatic, comparing them against 299 PRs that contain image attachments but no VRT (Visual PRs). Quantitatively, VRT-PRs show no significant acceptance-rate difference, but exhibit a 3.8 times longer median resolution time, 10 times more discussion comments, and 1.75 to 4.5 times larger code changes than Visual PRs. VRT results are typically shared around the midpoint of the review process, sustaining ongoing discussion rather than serving only as a final check. Through a card-sorting analysis of 189 VRT-flagged issues, we identify seven defect categories assigned to the analyzed issues: Layout (39.7\%), Appearance (27.5\%), Color (14.8\%), Text (9.5\%), State (6.9\%), Test (6.3\%), and Image (4.2\%). The three most frequent categories are stylistic, while approximately 18.5\% of analyzed issues (35/189) involve non-stylistic origins, including undefined component state (13 cases), content disappearance (17 cases across multiple categories), and visually imperceptible regressions (5 cases). We further document cases in which VRT detected visual regressions originating from code changes in seemingly unrelated files, exposing non-local effects that no targeted test would have been written to catch. These observations indicate that, in addition to its primary role as a stylistic checker, VRT functions as a secondary detector of unintended consequences of code changes, with implications for how VRT should be integrated into the maintenance toolchain.

cs.SE↗

arXiv2025

Agent READMEs: An Empirical Study of Context Files for Agentic Coding

Agentic coding tools receive goals written in natural language, break them down into specific tasks, and write or execute code with minimal human intervention. Central to this process are agent context files (e.g., AGENTS.md and CLAUDE.md) that provide persistent, project-level instructions. In this paper, we conduct the first large-scale empirical study of 2,303 agent context files from 1,925 repositories to characterize their structure, maintenance, and content. We find that these files are not static documentation but complex, difficultto-read artifacts that evolve like configuration code through frequent, small additions. Our content analysis of 16 instruction types shows that developers prioritize functional context, such as test procedures (75.9%), implementation details (70.8%), and architecture (68.1%). We also identify a significant gap: non-functional requirements such as security (14.8%) and performance (14.5%) are rarely specified. These findings indicate that while developers use context files to make agents functional, they provide few guardrails to ensure that agent-written code is secure or performant, highlighting the need for improved tools and practices.

cs.SE↗

arXiv2025

An Empirical Study of Security-Policy Related Issues in Open Source Projects

GitHub recommends that projects adopt a security file that outlines vulnerability reporting procedures. However, the effectiveness and operational challenges of such files are not yet fully understood. This study aims to clarify the challenges that security files face in the vulnerability reporting process within open-source communities. Specifically, we classified and analyzed the content of 711 randomly sampled issues related to security files. We also conducted a quantitative comparative analysis of the close time and number of responses for issues concerning six community health files, including security files. Our analysis revealed that 79.5% of security file-related issues were requests to add the file, and reports that included links were closed, with a median time that was 2 days shorter. These findings offer practical insights for improving security reporting policies and community management, ultimately contributing to a more secure open-source ecosystem.

cs.SE↗

arXiv2025

Detecting and Characterizing Low and No Functionality Packages in the NPM Ecosystem

Trivial packages, small modules with low functionality, are common in the npm ecosystem and can pose security risks despite their simplicity. This paper refines existing definitions and introduce data-only packages that contain no executable logic. A rule-based static analysis method is developed to detect trivial and data-only packages and evaluate their prevalence and associated risks in the 2025 npm ecosystem. The analysis shows that 17.92% of packages are trivial, with vulnerability levels comparable to non-trivial ones, and data-only packages, though rare, also contain risks. The proposed detection tool achieves 94% accuracy (macro-F1 0.87), enabling effective large-scale analysis to reduce security exposure. This findings suggest that trivial and data-only packages warrant greater attention in dependency management to reduce potential technical debt and security exposure.

cs.SE↗

arXiv2025

On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub

Large language models (LLMs) are increasingly being integrated into software development processes. The ability to generate code and submit pull requests with minimal human intervention, through the use of autonomous AI agents, is poised to become a standard practice. However, little is known about the practical usefulness of these pull requests and the extent to which their contributions are accepted in real-world projects. In this paper, we empirically study 567 GitHub pull requests (PRs) generated using Claude Code, an agentic coding tool, across 157 diverse open-source projects. Our analysis reveals that developers tend to rely on agents for tasks such as refactoring, documentation, and testing. The results indicate that 83.8% of these agent-assisted PRs are eventually accepted and merged by project maintainers, with 54.9% of the merged PRs are integrated without further modification. The remaining 45.1% require additional changes benefit from human revisions, especially for bug fixes, documentation, and adherence to project-specific standards. These findings suggest that while agent-assisted PRs are largely acceptable, they still benefit from human oversight and refinement.

cs.SE↗

arXiv2025

On the Use of Agentic Coding Manifests: An Empirical Study of Claude Code

Agentic coding tools receive goals written in natural language as input, break them down into specific tasks, and write/execute the actual code with minimal human intervention. Key to this process are agent manifests, configuration files (such as Claude.md) that provide agents with essential project context, identity, and operational rules. However, the lack of comprehensive and accessible documentation for creating these manifests presents a significant challenge for developers. We analyzed 253 Claude.md files from 242 repositories to identify structural patterns and common content. Our findings show that manifests typically have shallow hierarchies with one main heading and several subsections, with content dominated by operational commands, technical implementation notes, and high-level architecture.

cs.SE↗

arXiv2025

Uncovering Intention through LLM-Driven Code Snippet Description Generation

Documenting code snippets is essential to pinpoint key areas where both developers and users should pay attention. Examples include usage examples and other Application Programming Interfaces (APIs), which are especially important for third-party libraries. With the rise of Large Language Models (LLMs), the key goal is to investigate the kinds of description developers commonly use and evaluate how well an LLM, in this case Llama, can support description generation. We use NPM Code Snippets, consisting of 185,412 packages with 1,024,579 code snippets. From there, we use 400 code snippets (and their descriptions) as samples. First, our manual classification found that the majority of original descriptions (55.5%) highlight example-based usage. This finding emphasizes the importance of clear documentation, as some descriptions lacked sufficient detail to convey intent. Second, the LLM correctly identified the majority of original descriptions as "Example" (79.75%), which is identical to our manual finding, showing a propensity for generalization. Third, compared to the originals, the produced description had an average similarity score of 0.7173, suggesting relevance but room for improvement. Scores below 0.9 indicate some irrelevance. Our results show that depending on the task of the code snippet, the intention of the document may differ from being instructions for usage, installations, or descriptive learning examples for any user of a library.

cs.SE↗

arXiv2025

Using LLMs for Security Advisory Investigations: How Far Are We?

Large Language Models (LLMs) are increasingly used in software security, but their trustworthiness in generating accurate vulnerability advisories remains uncertain. This study investigates the ability of ChatGPT to (1) generate plausible security advisories from CVE-IDs, (2) differentiate real from fake CVE-IDs, and (3) extract CVE-IDs from advisory descriptions. Using a curated dataset of 100 real and 100 fake CVE-IDs, we manually analyzed the credibility and consistency of the model's outputs. The results show that ChatGPT generated plausible security advisories for 96% of given input real CVE-IDs and 97% of given input fake CVE-IDs, demonstrating a limitation in differentiating between real and fake IDs. Furthermore, when these generated advisories were reintroduced to ChatGPT to identify their original CVE-ID, the model produced a fake CVE-ID in 6% of cases from real advisories. These findings highlight both the strengths and limitations of ChatGPT in cybersecurity applications. While the model demonstrates potential for automating advisory generation, its inability to reliably authenticate CVE-IDs or maintain consistency upon re-evaluation underscores the risks associated with its deployment in critical security tasks. Our study emphasizes the importance of using LLMs with caution in cybersecurity workflows and suggests the need for further improvements in their design to improve reliability and applicability in security advisory generation.

cs.CR↗

arXiv2025

On Categorizing Open Source Software Security Vulnerability Reporting Mechanisms on GitHub

Open-source projects are essential to software development, but publicly disclosing vulnerabilities without fixes increases the risk of exploitation. The Open Source Security Foundation (OpenSSF) addresses this issue by promoting robust security policies to enhance project security. Current research reveals that many projects perform poorly on OpenSSF criteria, indicating a need for stronger security practices and underscoring the value of SECURITY$.$md files for structured vulnerability reporting. This study aims to provide recommendations for improving security policies. By examining 679 open-source projects, we find that email is still the main source of reporting. Furthermore, we find that projects without SECURITY$.$md files tend to be less secure (lower OpenSSF scores). Our analysis also indicates that, although many maintainers encourage private reporting methods, some contributors continue to disclose vulnerabilities publicly, bypassing established protocols. The results from this preliminary study pave the way for understanding how developers react and communicate a potential security threat. Future challenges include understanding the impact and effectiveness of these mechanisms and what factors may influence how the security threat is addressed.

cs.SE↗

arXiv2024

A Preliminary Study on Self-Contained Libraries in the NPM Ecosystem

The widespread of libraries within modern software ecosystems creates complex networks of dependencies. These dependencies are fragile to breakage, outdated, or redundancy, potentially leading to cascading issues in dependent libraries. One mitigation strategy involves reducing dependencies; libraries with zero dependencies become to self-contained. This paper explores the characteristics of self-contained libraries within the NPM ecosystem. Analyzing a dataset of 2763 NPM libraries, we found that 39.49\% are self-contained. Of these self-contained libraries, 40.42\% previously had dependencies that were later removed. This analysis revealed a significant trend of dependency reduction within the NPM ecosystem. The most frequently removed dependency was babel-runtime. Our investigation indicates that the primary reasons for dependency removal are concerns about the performance and the size of the dependency. Our findings illuminate the nature of self-contained libraries and their origins, offering valuable insights to guide software development practices.

cs.SE↗

arXiv2023

Using the TypeScript compiler to fix erroneous Node.js snippets

Most online code snippets do not run. This means that developers looking to reuse code from online sources must manually find and fix errors. We present an approach for automatically evaluating and correcting errors in Node.js code snippets: Node Code Correction (NCC). NCC leverages the ability of the TypeScript compiler to generate errors and inform code corrections through the combination of TypeScript's built-in codefixes, our own targeted fixes, and deletion of erroneous lines. Compared to existing approaches using linters, our findings suggest that NCC is capable of detecting a larger number of errors per snippet and more error types, and it is more efficient at fixing snippets. We find that 73.7% of the code snippets in NPM documentation have errors; with the use of NCC's corrections, this number was reduced to 25.1%. Our evaluation confirms that the use of the TypeScript compiler to inform code corrections is a promising strategy to aid in the reuse of code snippets from online sources.

cs.SE↗

arXiv2022

Software Engineering User Study Recruitment on Prolific: An Experience Report

Online participant recruitment platforms such as Prolific have been gaining popularity in research, as they enable researchers to easily access large pools of participants. However, participant quality can be an issue; participants may give incorrect information to gain access to more studies, adding unwanted noise to results. This paper details our experience recruiting participants from Prolific for a user study requiring programming skills in Node.js, with the aim of helping other researchers conduct similar studies. We explore a method of recruiting programmer participants using prescreening validation, attention checks and a series of programming knowledge questions. We received 680 responses, and determined that 55 met the criteria to be invited to our user study. We ultimately conducted user study sessions via video calls with 10 participants. We conclude this paper with a series of recommendations for researchers.

cs.SE↗

arXiv2021

What makes a good Node.js package? Investigating Users, Contributors, and Runnability

The Node.js Package Manager (i.e., npm) archive repository serves as a critical part of the JavaScript community and helps support one of the largest developer ecosystems in the world. However, as a developer, selecting an appropriate npm package to use or contribute to can be difficult. To understand what features users and contributors consider important when searching for a good npm package, we conduct a survey asking Node.js developers to evaluate the importance of 30 features derived from existing work, including GitHub activity, software usability, and properties of the repository and documentation. We identify that both user and contributor perspectives share similar views on which features they use to assess package quality. We then extract the 30 features from 104,364 npm packages and analyse the correlations between them, including three software features that measure package ``runnability"; ability to install, build, and execute a unit test. We identify which features are negatively correlated with runnability-related features and find that predicting the runnability of packages is viable. Our study lays the groundwork for future work on understanding how users and contributors select appropriate npm packages.

cs.SE↗

arXiv2021

NCQ: Code reuse support for node.js developers

Code reuse is an important part of software development. The adoption of code reuse practices is especially common among Node.js developers. The Node.js package manager, NPM, indexes over 1 Million packages and developers often seek out packages to solve programming tasks. Due to the vast number of packages, selecting the right package is difficult and time consuming. With the goal of improving productivity of developers that heavily reuse code through third-party packages, we present Node Code Query (NCQ), a Read-Eval-Print-Loop environment that allows developers to 1) search for NPM packages using natural language queries, 2) search for code snippets related to those packages, 3) automatically correct errors in these code snippets, 4) quickly setup new environments for testing those snippets, and 5) transition between search and editing modes. In two user studies with a total of 20 participants, we find that participants begin programming faster and conclude tasks faster with NCQ than with baseline approaches, and that they like, among other features, the search for code snippets and packages. Our results suggest that NCQ makes Node.js developers more efficient in reusing code.

cs.SE↗

arXiv2020

Optimising the Fit of Stack Overflow Code Snippets into Existing Code

Software developers often reuse code from online sources such as Stack Overflow within their projects. However, the process of searching for code snippets and integrating them within existing source code can be tedious. In order to improve efficiency and reduce time spent on code reuse, we present an automated code reuse tool for the Eclipse IDE (Integrated Developer Environment), NLP2TestableCode. NLP2TestableCode can not only search for Java code snippets using natural language tasks, but also evaluate code snippets based on a user's existing code, modify snippets to improve fit and correct errors, before presenting the user with the best snippet, all without leaving the editor. NLP2TestableCode also includes functionality to automatically generate customisable test cases and suggest argument and return types, in order to further evaluate code snippets. In evaluation, NLP2TestableCode was capable of finding compilable code snippets for 82.9% of tasks, and testable code snippets for 42.9%.

cs.SE↗