Learning Stability of Replay-Based Co-Optimization for Transmission Expansion under Strategic Bidding

Abstract

This paper investigates the behavior of learning-based co-optimization for transmission expansion under strategic bidding in electricity markets. In this framework, transmission capacities are updated while market participants simultaneously learn their bidding strategies through deep reinforcement learning, resulting in coupled and non-stationary learning dynamics. We show that transient policy degradation of bidding agents can generate inconsistent cost-capacity samples, which bias the transmission-capacity update and prevent the co-optimization process from converging to the desired solution. To mitigate this, replay-based capacity updates with nearest-neighbor filtering are introduced to exclude inconsistent samples from the update data. We then analyze a new oscillatory behavior that appears when the replay memory is enlarged. Numerical results on the IEEE 30-bus system demonstrate that enlarged replay memories improve robustness against transient policy degradation but can introduce a temporal lag, leading to oscillations unless the capacity-update learning rate is appropriately reduced. These results reveal the trade-off between replay-memory size and learning rate and provide practical guidelines for stable co-optimization.