arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 1807.01853

An Insight into the Pull Requests of GitHub

Mohammad Masudur Rahman·Chanchal K. Roy
View original source ↗Read source document ↗Export metadata ↓

Abstract

Given the increasing number of unsuccessful pull requests in GitHub projects, insights into the success and failure of these requests are essential for the developers. In this paper, we provide a comparative study between successful and unsuccessful pull requests made to 78 GitHub base projects by 20,142 developers from 103,192 forked projects. In the study, we analyze pull request discussion texts, project specific information (e.g., domain, maturity), and developer specific information (e.g., experience) in order to report useful insights, and use them to contrast between successful and unsuccessful pull requests. We believe our study will help developers overcome the issues with pull requests in GitHub, and project administrators with informed decision making.

Explore related subjects

cs.SE

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Mohammad Masudur Rahman, Chanchal K. Roy. 2018-07-05. An Insight into the Pull Requests of GitHub. https://doi.org/10.1145/2597073.2597121

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Real Faults in Deep Learning Fault Benchmarks: How Real Are They?

As the adoption of Deep Learning (DL) systems continues to rise, an increasing number of approaches are being proposed to test these systems, localise faults within them, and repair those faults. The best attestation of effectiveness for such techniques is an evaluation that showcases their capability to detect, localise and fix real faults. To facilitate these evaluations, the research community has collected multiple benchmarks of real faults in DL systems. In this work, we perform a manual analysis of 490 faults from five different benchmarks and identify that 305 of them are eligible for our study. Our investigation focuses specifically on how well the bugs correspond to the sources they were extracted from, which fault types are represented, and whether the bugs are reproducible. Our findings indicate that only 15.4% of the faults satisfy our realism conditions. Our attempts to reproduce these faults were successful only in 54% of cases.

cs.SE↗
arXiv2026

LLM-as-a-Judge for Contest-Style Human-AI Co-Creation: A Reliability-Aware Evaluation Framework for Coding

Large language models now judge open-ended outputs and co-create code with programmers, yet judgments of human-AI co-creation must be reliable, comparable across models, and interpretable over multi-turn interaction. This work presents a rubric-driven LLM-as-a-Judge framework for contest-style human-AI co-creation in coding, evaluated on 15 participants, 13 problems, 184 trajectories, and 517 attempts from an AI-permitted track run in parallel with a programming contest. Four judges are evaluated under grouped cross-validation with cluster-bootstrap 95% intervals and probed by injecting single defects into accepted programs against behavior-preserving controls. Three findings concern the judges as instruments and likely extend beyond this setting. First, the judge input is a first-order design variable: supplying the problem statement in place of an identifier rebuilds the judges' ranking (rank correlation 0.2617 [0.123, 0.391] for the three judges whose model was unchanged) and raises their ROC-AUC by 0.130 [0.061, 0.202] to 0.295 [0.214, 0.381]. Second, schema compliance does not imply scale validity: the judges emit as few as 8 distinct probability values, leaving as few as 6 of 99 thresholds effective. Third, accuracy and defect sensitivity are separable: Claude and Gemini are indistinguishable on accuracy (ROC-AUC 0.6985 [0.633, 0.763] against 0.7691 [0.719, 0.817]) yet separate on defect detection (0.8380 [0.806, 0.871] against 0.7512 [0.719, 0.783]). For co-creation, Success@Turn rises from 0.6033 [0.533, 0.674] at the first attempt to 0.8370 [0.783, 0.886] by the tenth, 44 of the 73 trajectories unsolved at first (60.3%) succeed later, and revision size is unrelated to judged improvement (Spearman 0.0007 [-0.085, +0.090]).

cs.SE↗
arXiv2026

Step-Aligned Script Generation with Verification and Recovery for Web Automation Testing

Automated generation of end-to-end web tests remains costly: developers must identify stable selectors, encode multi-step workflows, attach assertions, and maintain scripts as interfaces evolve. Recent LLM-based web agents can complete natural-language tasks in a live browser session, but task completion is not the same as producing a test-ware artifact - code that is executable, replay-able from a reset state, inspectable by developers, and tied to an oracle. We present TESSARA, an adaptive test-script generation framework that converts natural-language test intent into Playwright scripts consisting of executable interaction code, goal-state assertions, and step-level provenance linking each step to its generated code fragment, selector evidence, execution outcome, and repair history. TESSARA grounds each planned step via Step-Aligned Scripting (SAS), enabling fine-grained failure localization, selective regeneration of failing suffixes, and reversibility-aware backtracking that avoids undoing likely commit points (e.g., form submissions). A three-level verification pipeline (syntactic, execution, and semantic/oracle), detects and repairs malformed code, interaction failures, and incorrect landing states. We evaluate TESSARA on three public web benchmarks, measuring script executability, oracle satisfaction, replay stability, and repair efficiency, and show substantial improvements over baselines and underlying VLMs used directly as script generators.

cs.SE↗