arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2102.06345

A Visual Analysis Approach to Update Systematic Reviews

Katia Romero Felizardo·Elisa Yumi Nakagawa·Stephen G. MacDonell·José Carlos Maldonado
View original source ↗Read source document ↗Export metadata ↓

Abstract

Context: In order to preserve the value of Systematic Reviews (SRs), they should be frequently updated considering new evidence that has been produced since the completion of the previous version of the reviews. However, the update of an SR is a time consuming, manual task. Thus, many SRs have not been updated as they should be and, therefore, they are currently outdated. Objective: The main contribution of this paper is to support the update of SRs. Method: We propose USR-VTM, an approach based on Visual Text Mining (VTM) techniques, to support selection of new evidence in the form of primary studies. We then present a tool, named Revis, which supports our approach. Finally, we evaluate our approach through a comparison of outcomes achieved using USR-VTM versus the traditional (manual) approach. Results: Our results show that USR-VTM increases the number of studies correctly included compared to the traditional approach. Conclusions: USR-VTM effectively supports the update of SRs.

Explore related subjects

cs.SE

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Katia Romero Felizardo, Elisa Yumi Nakagawa, Stephen G. MacDonell, José Carlos Maldonado. 2021-02-12. A Visual Analysis Approach to Update Systematic Reviews. https://doi.org/10.1145/2601248.2601252

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Are AI Coders Snitches? An Empirical Study of Pretraining Data Detection on Code Large Language Models

Recent advances in code large language models (CodeLLMs) have made them indispensable tools in modern software engineering. However, these models occasionally produce outputs that contain proprietary or sensitive code snippets, raising concerns about potential non-compliant use of training data, and posing risks to privacy and intellectual property. To ensure responsible and compliant deployment of CodeLLMs, training data detection (TDD) has become a critical task. While recent TDD methods have shown promise in natural language settings, their effectiveness on code data remains largely underexplored. This gap is particularly important given code's structured syntax and distinct similarity criteria compared to natural language. To address this, we conduct a comprehensive empirical study of seven state-of-the-art TDD methods on source code data, evaluating their performance across eight CodeLLMs. To support this evaluation, we introduce CodeSnitch, a function-level benchmark dataset comprising 9,000 code samples in three programming languages, each explicitly labeled as either included or excluded from CodeLLM training. Beyond evaluation on the original CodeSnitch, we design targeted mutation strategies to test the robustness of TDD methods under three distinct settings. These mutation strategies are grounded in the well-established Type-1 to Type-4 code clone detection taxonomy. Our study provides a systematic assessment of current TDD techniques for code and offers insights to guide the development of more effective and robust detection methods in the future.

cs.SE↗
arXiv2026

On the Illusion of Success: An Empirical Study of Job Reruns and Silent Failures in Industrial CI

Reliability of build outcomes is a cornerstone of effective Continuous Integration (CI). Yet in practice, developers often struggle with non-deterministic issues in the code or CI infrastructure, which undermine trust in build results. When faced with such unexpected outcomes, developers often repeatedly rerun jobs hoping for true success, but this practice is known to increase CI costs and reduce productivity. While recent studies have focused on intermittent job failures, no prior work has investigated silent failures, where build jobs are marked as successful but fail to complete all or part of their tasks. Such silent failures often go unnoticed, creating an illusion of success with detrimental consequences such as bugs escaping into production. This paper presents the first empirical study of silent failures through the practice of rerunning successful jobs. An analysis of 142,387 jobs across 81 industrial projects shows that 11% of successful jobs are rerun, with 35% of these reruns occurring after more than 24 hours. Using mixed-effects models on 32 independent variables (AUC of 85%), we identified key factors associated with reruns of successful jobs, notably testing and static analysis tasks, scripting languages like Shell, and developers prior rerun tendencies. A further analysis of 92 public issues revealed 11 categories of silent failures aligning with these factors, the most frequent being artifact operation errors, caching errors, and ignored exit codes. Overall, our findings provide valuable insights into the circumstances and causes of silent failures to raise awareness among teams, and present solutions to improve CI reliability.

cs.SE↗
arXiv2026

Incentives and Outcomes in Bug Bounties

Bug bounty programs have contributed significantly to security in technology firms in the last decade, but little is known about the role of reward incentives in producing useful outcomes. We analyze incentives and outcomes in Google's Vulnerability Rewards Program (VRP), one of the world's largest bug bounty programs. We analyze the responsiveness of the quality and quantity of bugs received to changes in payments, focusing on a change in Google's reward amounts posted in July, 2024, in which reward amounts increased by up to 200% for the highest impact tier. Our empirical results show an increase in the volume of high-value bugs received after the reward increase, as well as a high positive observed elasticity of labor supply for such bugs. We further break down the sources of this increase between veteran researchers and new researchers, showing that the reward increase both redirected the attention of veteran researchers and attracted new top security researchers into the program.

cs.SE↗