arXiv subjects

Xavier Devroey

Publications and source records attributed to Xavier Devroey.

17 recordsLinked to original sources

arXiv2026

How Developers Experience Debugging Unfamiliar Codebases with Code Tours Generated and Evaluated by Local LLMs

Code tours are interactive, onboarding documentation to guide developers through a codebase. Large Language Models (LLMs) can automatically synthesize code tours. Prior work on code tour generation has not studied developer experience or trust calibration when debugging unfamiliar codebases with code tours generated and evaluated by open-weight LLMs. This study surveys how the properties of components in open-weight LLM-authored code tours influence developers' experiences when debugging unfamiliar codebases. We built a pipeline that generated and evaluated code tours from real reproducible bugs. 26 developers with varying backgrounds participated in a user study. In total, 26 code tours were authored from real Java bugs mined from 2025 GitHub commits, with each tour independently judged by two different LLMs, resulting in 52 evaluated configurations. Participants thought aloud as they explored each tour. Three authors qualitatively coded the interviews to identify recurring themes. Developers generally preferred tours that scaled detail with the code length, avoided merely restating code, were easily scannable, and adopted a guiding tone. However, some preferences were mutually exclusive, such as the use of imperative mood. Stack traces were often insufficient to identify all steps developers found relevant. Developers also trusted descriptions they perceived as human-written more than those they believed were AI-generated. Finally, LLM-generated annotations of tour quality were unreliable: sycophancy, confabulation, and incoherence were pervasive. This work lays a basis for future research on fine-tuning open-weight models for code tour generation, personalizing generation to accommodate diverging preferences, selecting relevant steps beyond stack traces, calibrating users' trust to avoid both disuse and misuse, and improving open-weight LLMs' ability to be more trustworthy evaluators

cs.SE↗

arXiv2026

System Test Generation for Virtual Reality Applications using Scenario Models

Virtual Reality (VR) applications are increasingly being integrated across a wide range of domains, including surgical training and industrial marketing. However, the long-term adoption and maintenance of VR applications remain limited, particularly due to the lack of effective, systematic, and reproducible software testing approaches tailored to their unique characteristics. To address this issue, we introduce UltraInstinctVR, a novel testing approach for VR applications. Relying on predefined VR models (scenarios), it automates the generation and execution of concrete VR system tests. In our empirical evaluation, we compare UltraInstinctVR with state-of-the-art automated VR testing approaches in terms of coverage and failure detection on 10 open-source VR applications. The results show that UltraInstinctVR outperforms existing automated tools for detecting unique failures and provides valuable insights for identifying real-world bugs in VR applications.

cs.SE↗

arXiv2026

Systematic Detection of Energy Regression and Corresponding Code Patterns in Java Projects

Green software engineering is emerging as a crucial response to information technology's rising energy impact, especially in continuous development. However, there remain challenges in devising automated methods for identifying energy regressions across commits and their associated code change patterns. In particular, little effort has been put into automatically detecting regressions at the commit level by identifying statistically significant changes in energy consumption. In this paper, we introduce EnergyTrackr, an approach designed to detect energy regressions across multiple commits that can then be used to identify code anti-patterns potentially contributing to the increase of software energy consumption over time. We describe our empirical evaluation, including repository mining and source code analysis, made on 3,232 commits from three Java projects, and show the approach's ability to identify significant energy changes. We also highlight recurring anti-patterns such as missing early exits or costly dependency upgrades. We expect EnergyTrackr to assist developers in accurately monitoring energy regressions and improvements within their projects, identifying code anti-patterns, and helping them optimize their source code to reduce software energy consumption.

cs.SE↗

arXiv2026

Real-World Fault Detection for C-Extended Python Projects with Automated Unit Test Generation

Many popular Python libraries use C-extensions for performance-critical operations allowing users to combine the best of the two worlds: The simplicity and versatility of Python and the performance of C. A drawback of this approach is that exceptions raised in C can bypass Python's exception handling and cause the entire interpreter to crash. These crashes are real faults if they occur when calling a public API. While automated test generation should, in principle, detect such faults, crashes in native code can halt the test process entirely, preventing detection or reproduction of the underlying errors and inhibiting coverage of non-crashing parts of the code. To overcome this problem, we propose separating the generation and execution stages of the test-generation process. We therefore adapt Pynguin, an automated test case generation tool for Python, to use subprocess-execution. Executing each generated test in an isolated subprocess prevents a crash from halting the test generation process itself. This allows us to (1) detect such faults, (2) generate reproducible crash-revealing test cases for them, (3) allow studying the underlying faults, and (4) enable test generation for non-crashing parts of the code. To evaluate our approach, we created a dataset consisting of 1648 modules from 21 popular Python libraries with C-extensions. Subprocess-execution allowed automated testing of up to 56.5% more modules and discovered 213 unique crash causes, revealing 32 previously unknown faults.

cs.SE↗

arXiv2025

The Software Infrastructure Attitude Scale (SIAS): A Questionnaire Instrument for Measuring Professionals' Attitudes Toward Technical and Sociotechnical Infrastructure

Context: Recent software engineering (SE) research has highlighted the need for sociotechnical research, implying a demand for customized psychometric scales. Objective: We define the concepts of technical and sociotechnical infrastructure in software engineering, and develop and validate a psychometric scale that measures attitudes toward them. Method: Grounded in theories of infrastructure, attitudes, and prior work on psychometric measurement, we defined the target constructs and generated scale items. The scale was administered to 225 software professionals and evaluated using a split sample. We conducted an exploratory factor analysis (EFA) on one half of the sample to uncover the underlying factor structure and performed a confirmatory factor analysis (CFA) on the other half to validate the structure. Further analyses with the whole sample assessed face, criterion-related, and discriminant validity. Results: EFA supported a two-factor structure (technical and sociotechnical infrastructure), accounting for 65% of the total variance with strong loadings. CFA confirmed excellent model fit. Face and content validity were supported by the item content reflecting cognitive, affective, and behavioral components. Both subscales were correlated with job satisfaction, perceived autonomy, and feedback from the job itself, supporting convergent validity. Regression analysis supported criterion-related validity, while the Heterotrait-Monotrait ratio of correlations (HTMT), the Fornell-Larcker criterion, and model comparison all supported discriminant validity. Discussion: The resulting scale is a valid instrument for measuring attitudes toward technical and sociotechnical infrastructure in software engineering research. Our work contributes to ongoing efforts to integrate psychological measurement rigor into empirical and behavioral software engineering research.

cs.SE↗

arXiv2025

Staying or Leaving? How Job Satisfaction, Embeddedness and Antecedents Predict Turnover Intentions of Software Professionals

Context: Voluntary turnover is common in the software industry, increasing recruitment and onboarding costs and the risk of losing organizational and tacit knowledge. Objective: This study investigates how job satisfaction, work-life balance, job embeddedness, and their antecedents, including job quality, personality traits, attitudes toward technical and sociotechnical infrastructure, and perceptions of organizational justice, relate to software professionals' turnover intentions. Method: We conducted a geographically diverse cross-sectional survey of software professionals (N = 224) and analyzed the data using partial least squares structural equation modeling (PLS-SEM). Our model includes both reflective and formative constructs and tests 15 hypotheses grounded in occupational psychology and software engineering literature. Results: Job satisfaction and embeddedness were significantly negatively associated with software professionals' turnover intentions, while work-life balance showed no direct effect. The strongest antecedents for job satisfaction were work-life balance and job quality, while organizational justice was the strongest predictor of job embeddedness. Discussion: The resulting PLS-SEM model has considerably higher explanatory power for key outcome variables than previous work conducted in the software development context, highlighting the importance of both psychological (e.g., job satisfaction, job embeddedness) and organizational (e.g., organizational justice, job quality) factors in understanding turnover intentions of software professionals. Our results imply that improving job satisfaction and job embeddedness is the key to retaining software professionals. In turn, enhancing job quality, supporting work-life balance, and ensuring high organizational justice can improve job satisfaction and embeddedness, indirectly reducing turnover intentions.

cs.SE↗

arXiv2024

Towards Comprehensive Legislative Requirements for Cyber Physical Systems Testing in the European Union

While procedures prevail on the European market for the greater good of its citizens, it might be daunting when trying to introduce a product, whether innovative or not. In the current world, Cyber-Physical Systems (CPSs) are ubiquitous in our daily lives. Cars can provide intrusive assistance as they can brake or turn wheels on their own, buildings are getting smarter to optimize energy consumption, smart cities are emerging to facilitate information sharing and orchestrate the response to emergency situations, etc. As the presence of such tools will grow in the coming years and people will rely even more on CPSs, we certainly need to ensure that they are safe and reliable for users or everybody else, which is why regulations are so important. However, compliance should not act as a barrier to new actors coming to the European market. Nor should it prevent current actors from keeping systems deemed compliant when introduced while obsolete at the time they are used. While the individual elements we point out might not bring novelty in the various research areas we cover (EU policies, requirements engineering, business engineering, and software engineering), this paper identifies the challenges related to building and testing a CPS with respect to applicable laws and discusses the difficulty of automating the response to those challenges, such as finding a relevant legal text, paying for mentioned materials or identifying the level of compliance to a legal text. Our analysis of the holistic context when considering the compliance testing of CPS provides an overview enabling more effective decision-making as well.

cs.SE↗

arXiv2024

Towards debiasing code review support

Cognitive biases appear during code review. They significantly impact the creation of feedback and how it is interpreted by developers. These biases can lead to illogical reasoning and decision-making, violating one of the main hypotheses supporting code review: developers' accurate and objective code evaluation. This paper explores harmful cases caused by cognitive biases during code review and potential solutions to avoid such cases or mitigate their effects. In particular, we design several prototypes covering confirmation bias and decision fatigue. We rely on a developer-centered design approach by conducting usability tests and validating the prototype with a user experience questionnaire (UEQ) and participants' feedback. We show that some techniques could be implemented in existing code review tools as they are well accepted by reviewers and help prevent behavior detrimental to code review. This work provides a solid first approach to treating cognitive bias in code review.

cs.SE↗

arXiv2024

You Can REST Now: Automated REST API Documentation and Testing via LLM-Assisted Request Mutations

REST APIs are prevalent among web service implementations, easing interoperability through the HTTP protocol. API testers and users exploit the widely adopted OpenAPI Specification (OAS), a machine-readable standard to document REST APIs. However, documenting APIs is a time-consuming and error-prone task, and existing documentation is not always complete, publicly accessible, or up-to-date. This situation limits the efficiency of testing tools and hinders human comprehension. Large Language Models (LLMs) offer the potential to automatically infer API documentation, using their colossal training data. In this paper, we present RESTSpecIT, the first automated approach that infers documentation and performs black-box testing of REST APIs by leveraging LLMs. Our approach requires minimal user input compared to state-of-the-art tools; Given an API name and an LLM access key, RESTSpecIT generates API request seeds and mutates them with data returned by the LLM. The tool then analyzes API responses for documentation inference and testing purposes. RESTSpecIT utilizes an in-context prompt masking strategy, requiring no prior model fine-tuning. We evaluate the quality of our tool with three state-of-the-art LLMs: DeepSeek V3, GPT-4.1, and GPT-3.5. Our evaluation demonstrates that RESTSpecIT can (1) infer documentation with 88.62% of routes and 89.25% of query parameters found on average, (2) discover undocumented API data, (3) operate efficiently (in terms of model costs, requests sent, runtime), and (4) assist REST API testing by uncovering server errors and generating valid OpenAPI Specification inputs for testing tools.

cs.SE↗

arXiv2022

Basic Block Coverage for Search-based Unit Testing and Crash Reproduction

Search-based techniques have been widely used for white-box test generation. Many of these approaches rely on the approach level and branch distance heuristics to guide the search process and generate test cases with high line and branch coverage. Despite the positive results achieved by these two heuristics, they only use the information related to the coverage of explicit branches (e.g., indicated by conditional and loop statements), but ignore potential implicit branchings within basic blocks of code. If such implicit branching happens at runtime (e.g., if an exception is thrown in a branchless-method), the existing fitness functions cannot guide the search process. To address this issue, we introduce a new secondary objective, called Basic Block Coverage (BBC), which takes into account the coverage level of relevant basic blocks in the control flow graph. We evaluated the impact of BBC on search-based unit test generation (using the DynaMOSA algorithm) and search-based crash reproduction (using the STDistance and WeightedSum fitness functions). Our results show that for unit test generation, BBC improves the branch coverage of the generated tests. Although small (around 1.5%), this improvement in the branch coverage is systematic and leads to an increase of the output domain coverage and implicit runtime exception coverage, and of the diversity of runtime states. In terms of crash reproduction, in the combination of STDistance and WeightedSum, BBC helps in reproducing 3 new crashes for each fitness function. BBC significantly decreases the time required to reproduce 43.5% and 45.1% of the crashes using STDistance and WeightedSum, respectively. For these crashes, BBC reduces the consumed time by 71.7% (for STDistance) and 68.7% (for WeightedSum) on average.

cs.SE↗

arXiv2021

JUGE: An Infrastructure for Benchmarking Java Unit Test Generators

Researchers and practitioners have designed and implemented various automated test case generators to support effective software testing. Such generators exist for various languages (e.g., Java, C#, or Python) and for various platforms (e.g., desktop, web, or mobile applications). Such generators exhibit varying effectiveness and efficiency, depending on the testing goals they aim to satisfy (e.g., unit-testing of libraries vs. system-testing of entire applications) and the underlying techniques they implement. In this context, practitioners need to be able to compare different generators to identify the most suited one for their requirements, while researchers seek to identify future research directions. This can be achieved through the systematic execution of large-scale evaluations of different generators. However, the execution of such empirical evaluations is not trivial and requires a substantial effort to collect benchmarks, setup the evaluation infrastructure, and collect and analyse the results. In this paper, we present our JUnit Generation benchmarking infrastructure (JUGE) supporting generators (e.g., search-based, random-based, symbolic execution, etc.) seeking to automate the production of unit tests for various purposes (e.g., validation, regression testing, fault localization, etc.). The primary goal is to reduce the overall effort, ease the comparison of several generators, and enhance the knowledge transfer between academia and industry by standardizing the evaluation and comparison process. Since 2013, eight editions of a unit testing tool competition, co-located with the Search-Based Software Testing Workshop, have taken place and used and updated JUGE. As a result, an increasing amount of tools (over ten) from both academia and industry have been evaluated on JUGE, matured over the years, and allowed the identification of future research directions.

cs.SE↗

arXiv2020

Pandemic Programming: How COVID-19 affects software developers and how their organizations can help

Context. As a novel coronavirus swept the world in early 2020, thousands of software developers began working from home. Many did so on short notice, under difficult and stressful conditions. Objective. This study investigates the effects of the pandemic on developers' wellbeing and productivity. Method. A questionnaire survey was created mainly from existing, validated scales and translated into 12 languages. The data was analyzed using non-parametric inferential statistics and structural equation modeling. Results. The questionnaire received 2225 usable responses from 53 countries. Factor analysis supported the validity of the scales and the structural model achieved a good fit (CFI = 0.961, RMSEA = 0.051, SRMR = 0.067). Confirmatory results include: (1) the pandemic has had a negative effect on developers' wellbeing and productivity; (2) productivity and wellbeing are closely related; (3) disaster preparedness, fear related to the pandemic and home office ergonomics all affect wellbeing or productivity. Exploratory analysis suggests that: (1) women, parents and people with disabilities may be disproportionately affected; (2) different people need different kinds of support. Conclusions. To improve employee productivity, software companies should focus on maximizing employee wellbeing and improving the ergonomics of employees' home offices. Women, parents and disabled persons may require extra support.

cs.SE↗

arXiv2020

Generating Class-Level Integration Tests Using Call Site Information

Search-based approaches have been used in the literature to automate the process of creating unit test cases. However, related work has shown that generated unit-tests with high code coverage could be ineffective, i.e., they may not detect all faults or kill all injected mutants. In this paper, we propose CLING, an integration-level test case generation approach that exploits how a pair of classes, the caller and the callee, interact with each other through method calls. In particular, CLING generates integration-level test cases that maximize the Coupled Branches Criterion (CBC). Coupled branches are pairs of branches containing a branch of the caller and a branch of the callee such that an integration test that exercises the former also exercises the latter. CBC is a novel integration-level coverage criterion, measuring the degree to which a test suite exercises the interactions between a caller and its callee classes. We implemented CLING and evaluated the approach on 140 pairs of classes from five different open-source Java projects. Our results show that (1) CLING generates test suites with high CBC coverage, thanks to the definition of the test suite generation as a many-objectives problem where each couple of branches is an independent objective; (2) such generated suites trigger different class interactions and can kill on average 7.7% (with a maximum of 50%) of mutants that are not detected by tests generated at the unit level; (3) CLING can detect integration faults coming from wrong assumptions about the usage of the callee class (32 for our subject systems) that remain undetected when using automatically generated unit-level test suites.

cs.SE↗

arXiv2019

Search-based Crash Reproduction using Behavioral Model Seeding

Search-based crash reproduction approaches assist developers during debugging by generating a test case which reproduces a crash given its stack trace. One of the fundamental steps of this approach is creating objects needed to trigger the crash. One way to overcome this limitation is seeding: using information about the application during the search process. With seeding, the existing usages of classes can be used in the search process to produce realistic sequences of method calls which create the required objects. In this study, we introduce behavioral model seeding: a new seeding method which learns class usages from both the system under test and existing test cases. Learned usages are then synthesized in a behavioral model (state machine). Then, this model serves to guide the evolutionary process. To assess behavioral model-seeding, we evaluate it against test-seeding (the state-of-the-art technique for seeding realistic objects) and no-seeding (without seeding any class usage). For this evaluation, we use a benchmark of 124 hard-to-reproduce crashes stemming from six open-source projects. Our results indicate that behavioral model-seeding outperforms both test seeding and no-seeding by a minimum of 6% without any notable negative impact on efficiency.

cs.SE↗

arXiv2017

Test them all, is it worth it? Assessing configuration sampling on the JHipster Web development stack

Many approaches for testing configurable software systems start from the same assumption: it is impossible to test all configurations. This motivated the definition of variability-aware abstractions and sampling techniques to cope with large configuration spaces. Yet, there is no theoretical barrier that prevents the exhaustive testing of all configurations by simply enumerating them, if the effort required to do so remains acceptable. Not only this: we believe there is lots to be learned by systematically and exhaustively testing a configurable system. In this case study, we report on the first ever endeavour to test all possible configurations of an industry-strength, open source configurable software system, JHipster, a popular code generator for web applications. We built a testing scaffold for the 26,000+ configurations of JHipster using a cluster of 80 machines during 4 nights for a total of 4,376 hours (182 days) CPU time. We find that 35.70% configurations fail and we identify the feature interactions that cause the errors. We show that sampling strategies (like dissimilarity and 2-wise): (1) are more effective to find faults than the 12 default configurations used in the JHipster continuous integration; (2) can be too costly and exceed the available testing budget. We cross this quantitative analysis with the qualitative assessment of JHipster's lead developers.

cs.SE↗

arXiv2014

State Machine Flattening: Mapping Study and Assessment

State machine formalisms equipped with hierarchy and parallelism allow to compactly model complex system behaviours. Such models can then be transformed into executable code or inputs for model-based testing and verification techniques. Generated artifacts are mostly flat descriptions of system behaviour. \emph{Flattening} is thus an essential step of these transformations. To assess the importance of flattening, we have defined and applied a systematic mapping process and 30 publications were finally selected. However, it appeared that flattening is rarely the sole focus of the publications and that care devoted to the description and validation of flattening techniques varies greatly. Preliminary assessment of associated tool support indicated limited tool availability and scalability on challenging models. We see this initial investigation as a first step towards generic flattening techniques and scalable tool support, cornerstones of reliable model-based behavioural development.

cs.SE↗

arXiv2013

Towards Statistical Prioritization for Software Product Lines Testing

Software Product Lines (SPL) are inherently difficult to test due to the combinatorial explosion of the number of products to consider. To reduce the number of products to test, sampling techniques such as combinatorial interaction testing have been proposed. They usually start from a feature model and apply a coverage criterion (e.g. pairwise feature interaction or dissimilarity) to generate tractable, fault-finding, lists of configurations to be tested. Prioritization can also be used to sort/generate such lists, optimizing coverage criteria or weights assigned to features. However, current sampling/prioritization techniques barely take product behavior into account. We explore how ideas of statistical testing, based on a usage model (a Markov chain), can be used to extract configurations of interest according to the likelihood of their executions. These executions are gathered in featured transition systems, compact representation of SPL behavior. We discuss possible scenarios and give a prioritization procedure illustrated on an example.

cs.SE↗