From Observation to Intervention: Mastering Bus Holding Control from Historical Uncontrolled Trajectories with Offline Reinforcement Learning

Abstract

Bus operations are inherently unstable---buses initially dispatched at equal headways will inevitably end up bunching up. This issue exacerbates passenger waiting times, creates load imbalances between buses, and undermines service reliability. Reinforcement learning (RL) has proven highly effective for mitigating bus bunching and outperforms traditional model-based approaches. However, existing RL approaches require agents to continuously interact with the environment, which potentially disrupts the bus system before effective policies are learned. This paper proposes an Offline Graph-based Multi-agent Actor-Critic (Offline-GMAC) method that trains bus holding policies using only readily available logged GPS data collected during normal uncontrolled operations, thereby eliminating the need for disruptive online interaction or exploration. The method employs general graph neural networks for multi-agent credit assignment. A key innovation of our method is a data augmentation scheme that synthesizes unperformed holding actions from historical uncontrolled bus trajectories. The scheme exploits the equivalence between stochastic link travel time delays and deliberate holding interventions. Under mild assumptions, we prove that the 1-Wasserstein distance between synthesized and target state transition distributions is bounded above by a term that scales linearly with the standard deviation of link travel times plus a tunable synthesis-dependent term. Evaluations using real-world bus data demonstrate that the proposed offline method achieves performance comparable to, and in some cases, even better than its online counterpart.