arXiv subjects

Rijnard van Tonder

Publications and source records attributed to Rijnard van Tonder.

3 recordsLinked to original sources

arXiv2025

Guidelines for Empirical Studies in Software Engineering involving Large Language Models

Large Language Models (LLMs) are widely used in software engineering (SE) research and practice, yet their non-determinism, opaque training data, and rapidly evolving models threaten the reproducibility and replicability of empirical studies. We address this challenge through a collaborative effort of 22 researchers, presenting a taxonomy of seven study types that organizes how LLMs are used in SE research, together with eight guidelines for designing and reporting such studies. Each guideline distinguishes requirements (must) from recommendations (should) and is contextualized by the study types it applies to. Our guidelines recommend that researchers: (1) declare LLM usage and role; (2) report model versions, configurations, and customizations; (3) document the system and prompt design beyond the model; (4) report session traces, i.e., interaction logs and runtime traces; (5) use suitable baselines, benchmarks, and metrics; (6) include an open LLM as a baseline; (7) validate LLM outputs against human judgment; and (8) articulate limitations and mitigations. We complement the guidelines with an applicability matrix mapping guidelines to study types and a reporting checklist for authors and reviewers. We maintain the study types and guidelines online as a living resource for the community to use and shape (llm-guidelines$.$org).

cs.SE↗

arXiv2022

You Don't Know Search: Helping Users Find Code by Automatically Evaluating Alternative Queries

Tens of thousands of engineers use Sourcegraph day-to-day to search for code and rely on it to make progress on software development tasks. We face a key challenge in designing a query language that accommodates the needs of a broad spectrum of users. Our experience shows that users express different and often contradictory preferences for how queries should be interpreted. These preferences stem from users with differing usage contexts, technical experience, and implicit expectations from using prior tools. At the same time, designing a code search query language poses unique challenges because it intersects traditional search engines and full-fledged programming languages. For example, code search queries adopt certain syntactic conventions in the interest of simplicity and terseness but invariably risk encoding implicit semantics that are ambiguous at face-value (a single space in a query could mean three or more semantically different things depending on surrounding terms). Users often need to disambiguate intent with additional syntax so that a query expresses what they actually want to search. This need to disambiguate is one of the primary frustrations we've seen users experience with writing search queries in the last three years. We share our observations that lead us to a fresh perspective where code search behavior can straddle seemingly ambiguous queries. We develop Automated Query Evaluation (AQE), a new technique that automatically generates and adaptively runs alternative query interpretations in frustration-prone conditions. We evaluate AQE with an A/B test across more than 10,000 unique users on our publicly-available code search instance. Our main result shows that relative to the control group, users are on average 22% more likely to click on a search result at all on any given day when AQE is active.

cs.SE↗

arXiv2021

Towards Fully Declarative Program Analysis via Source Code Transformation

Advances in logic programming and increasing industrial uptake of Datalog-inspired approaches demonstrate the emerging need to express powerful code analyses more easily. Declarative program analysis frameworks (e.g., using logic programming like Datalog) significantly ease defining analyses compared to imperative implementations. However, the declarative benefits of these frameworks only materialize after parsing and translating source code to generate facts. Fact generation remains a non-declarative precursor to analysis where imperative implementations first parse and interpret program structures (e.g., abstract syntax trees and control-flow graphs). The procedure of fact generation thus remains opaque and difficult for non-experts to understand or modify. We present a new perspective on this analysis workflow by proposing declarative fact generation to ease specification and exploration of lightweight declarative analyses. Our approach demonstrates the first venture towards fully declarative analysis specification across multiple languages. The key idea is to translate source code directly to Datalog facts in the analysis domain using declarative syntax transformation. We then reuse existing Datalog analyses over generated facts, yielding an end-to-end declarative pipeline. As a first approximation we pursue a syntax-driven approach and demonstrate the feasibility of generating and using lightweight versions of liveness and call graph reachability properties. We then discuss the workability of extending declarative fact generation to also incorporate semantic information.

cs.SE↗