Stuck on Red? Understanding GitHub Actions Failing Streaks

Abstract

GitHub Actions is the de facto CI/CD service for GitHub repositories. Its workflows should not fail when they are not expected to, nor keep on failing too long. We empirically study failing streaks, i.e., uninterrupted series of failing runs of a workflow, in 30.9M+ runs of 154K+ workflows of 43K+ GitHub repositories over 238 days. We study how failing streaks begin and end, and what happens during them, based on the trigger and the commit of each run. We also classify all workflows by their purposes, such as building, testing or deploying, and we inspect the logs of a small sample of failing streaks. Failing runs mostly come in streaks: three quarters of them follow another failing run, and half of the workflows with at least three runs have at least one failing streak. Workflows serving multiple purposes have failing streaks more often, even at the same size. Most failing streaks end within hours, but the one in five that last more than a day account for the lion's share of the time that workflows spend failing. A third of the failing streaks start, and two in five end, without any change to the repository. In a sample of 100 failing streaks, those that end this way most often fail because of network issues or an external service. These findings suggest that maintainers should not assume that a failure comes from the last commit. GitHub and tool builders could flag failures that occur without any change to the repository, and alert the active maintainers when a workflow stays failing for more than a day.