ROAM: Reinforcement Learning Optimized Adaptive Moving Target Defense for Smart Grid

Abstract

Smart grids rely on distributed phasor measurement units (PMUs) and wide-area communication networks whose data-acquisition pipelines are vulnerable to stealthy false data injection (FDI) attacks that evade conventional bad-data detection (BDD). Moving target defense (MTD) counters this exposure by continuously remapping sensor identities, but existing schemes apply static or pre-scheduled randomization and are each constrained by a single failure mode: seed-based methods are fast yet fragile to key compromise, while machine learning (ML) based prediction methods are resilient yet incur prohibitive latency. Neither adapts its randomization to live threat evidence. This paper presents ROAM, a deep reinforcement learning (DRL) driven MTD framework in which a deep Q-network (DQN) agent continuously re-optimizes the sensor-to-ID mapping at the aggregation layer. At each decision step, the agent selects the deception coverage, flags the specific sensors to remap, and sets the split of the remapped sensors between seed-based and ML-prediction-based recovery, without modifying PMU hardware and while preserving IEEE C37.118 compatibility. On held-out BDD-compliant FDI scenarios across the IEEE 14-, 57-, and 118- bus systems, ROAM keeps the percentage of stealthy attacks below 3% while sustaining detection rates above 93%. Against an adaptive soft actor-critic (SAC) attacker that observes the defender's configuration, ROAM keeps stealthy attacks below 2.6% with detection above 92%. Hardware-in-the-loop testing on an OPAL-RT platform confirms real-time feasibility, with sub-18ms average shuffle execution and an average reshuffle blind window of 2.6% of the 3-s decision cycle.