arXiv subjects

Aranya Chakrabortty

Publications and source records attributed to Aranya Chakrabortty.

At least 19 recordsLinked to original sources

arXiv2025

PHANTOM: Physics-Aware Adversarial Attacks against Federated Learning-Coordinated EV Charging Management System

The rapid deployment of electric vehicle charging stations (EVCS) within distribution networks necessitates intelligent and adaptive control to maintain the grid's resilience and reliability. In this work, we propose PHANTOM, a physics-aware adversarial network that is trained and optimized through a multi-agent reinforcement learning model. PHANTOM integrates a physics-informed neural network (PINN) enabled by federated learning (FL) that functions as a digital twin of EVCS-integrated systems, ensuring physically consistent modeling of operational dynamics and constraints. Building on this digital twin, we construct a multi-agent RL environment that utilizes deep Q-networks (DQN) and soft actor-critic (SAC) methods to derive adversarial false data injection (FDI) strategies capable of bypassing conventional detection mechanisms. To examine the broader grid-level consequences, a transmission and distribution (T and D) dual simulation platform is developed, allowing us to capture cascading interactions between EVCS disturbances at the distribution level and the operations of the bulk transmission system. Results demonstrate how learned attack policies disrupt load balancing and induce voltage instabilities that propagate across T and D boundaries. These findings highlight the critical need for physics-aware cybersecurity to ensure the resilience of large-scale vehicle-grid integration.

cs.ET↗

arXiv2024

Artificial Intelligence based Approach for Identification and Mitigation of Cyber-Attacks in Wide-Area Control of Power Systems

We propose a generative adversarial network (GAN) based deep learning method that serves the dual role of both identification and mitigation of cyber-attacks in wide-area damping control loops of power systems. Two specific types of attacks considered are false data injection and denial-of-service (DoS). Unlike existing methods, which are either model-based or model-free and yet require two separate learning modules for detection and mitigation leading to longer response times before clearing an attack, our deep learner incorporate both goals within the same integrated framework. A Long Short-Term Memory (LSTM) encoder-decoder based GAN is proposed that captures the temporal dynamics of the power system significantly more accurately than fully-connected GANs, thereby providing better accuracy and faster response for both goals. The method is validated using the IEEE 68-bus power system model.

eess.SY↗

arXiv2024

Co-Optimization of EV Charging Control and Incentivization for Enhanced Power System Stability

We study how high charging rate demands from electric vehicles (EVs) in a power distribution grid may collectively cause poor dynamic performance, and propose a price incentivization strategy to steer customers to settle for lesser charging rate demands so that such performance degradation can be avoided. We pose the problem as a joint optimization and optimal control formulation. The optimization determines the optimal charging setpoints for EVs to minimize the $\mathcal{H}_2$-norm of the transfer function of the grid model, while the optimal control simultaneously develops a linear quadratic regulator (LQR) based state-feedback control signal for the battery currents of those EVs to jointly improve the small-signal dynamic performance of the system states. A subsequent algorithm is developed to determine how much customers may be willing to sacrifice their intended charging rate demands in return for financial incentives. Results are derived for both unidirectional and bidirectional charging, and validated using numerical simulations of multiple EV charging stations (EVCSs) in the IEEE 33-bus power distribution model.

math.OC↗

arXiv2023

Equilibrium-Independent Passivity of Power Systems: A Link Between Classical and Two-Axis Synchronous Generator Models

We study the equilibrium-independent (EI) passivity of a nonlinear power system composed of two-axis generator models. The model of our interest consists of a feedback inter-connection of linear and nonlinear subsystems, called mechanical and electromagnetic subsystems. We mathematically prove the following three facts by analyzing the nonlinear electromagnetic subsystem. First, a lossless transmission network is necessary for the EI passivity of the electromagnetic subsystem. Second, the convexity of a strain energy function characterizes the largest set of equilibria over which the electromagnetic subsystem is EI passive. Finally, we prove that the strain energy function for the network of the two-axis generator models is convex if and only if its flux linkage dynamics is stable, and the strain energy function for the network of the classical generator models derived by singular perturbation approximation of the flux linkage dynamics is convex. Numerical simulation of the IEEE 9-bus power system model demonstrates the practical implications of the various mathematical results. In particular, we validate that the convex domain of the strain energy function over which the electromagnetic subsystem is EI passive is almost identical to the set of all stable equilibria. This result is also generalized to lossy power systems based on our finding that the convexity of the strain energy function is equivalent to the positive semidefiniteness of a synchronizing torque coefficient matrix.

math.DS↗

arXiv2023

Reinforcement Learning-based Control of Nonlinear Systems using Carleman Approximation: Structured and Unstructured Designs

We develop data-driven reinforcement learning (RL) control designs for input-affine nonlinear systems. We use Carleman linearization to express the state-space representation of the nonlinear dynamical model in the Carleman space, and develop a real-time algorithm that can learn nonlinear state-feedback controllers using state and input measurements in the infinite-dimensional Carleman space. Thereafter, we study the practicality of having a finite-order truncation of the control signal, followed by its closed-loop stability analysis. Finally, we develop two additional designs that can learn structured as well as sparse representations of the RL-based nonlinear controller, and provide theoretical conditions for ensuring their closed-loop stability. We present numerical examples to show how our proposed method generates closed-loop responses that are close to the optimal performance of the nonlinear plant. We also compare our designs to other data-driven nonlinear RL control methods such as those based on neural networks, and illustrate their relative advantages and drawbacks.

eess.SY↗

arXiv2022

Robust and Scalable Game-theoretic Security Investment Methods for Voltage Stability of Power Systems

We develop investment approaches to secure electric power systems against load attacks where a malicious intruder (the attacker) covertly changes reactive power setpoints of loads to push the grid towards voltage instability while the system operator (the defender) employs reactive power compensation (RPC) to prevent instability. Extending our previously reported Stackelberg game formulation for this problem, we develop a robust-defense sequential algorithm and a novel genetic algorithm that provides scalability to large-scale power system models. The proposed methods are validated using IEEE prototype power system models with time-varying load uncertainties, demonstrating that reliable and robust defense is feasible unless the operator's RPC investment resources are severely limited relative to the attacker's resources.

eess.SY↗

arXiv2022

Distributed Multi-Agent Reinforcement Learning Based on Graph-Induced Local Value Functions

Achieving distributed reinforcement learning (RL) for large-scale cooperative multi-agent systems (MASs) is challenging because: (i) each agent has access to only limited information; (ii) issues on convergence or computational complexity emerge due to the curse of dimensionality. In this paper, we propose a general computationally efficient distributed framework for cooperative multi-agent reinforcement learning (MARL) by utilizing the structures of graphs involved in this problem. We introduce three coupling graphs describing three types of inter-agent couplings in MARL, namely, the state graph, the observation graph and the reward graph. By further considering a communication graph, we propose two distributed RL approaches based on local value-functions derived from the coupling graphs. The first approach is able to reduce sample complexity significantly under specific conditions on the aforementioned four graphs. The second approach provides an approximate solution and can be efficient even for problems with dense coupling graphs. Here there is a trade-off between minimizing the approximation error and reducing the computational complexity. Simulations show that our RL algorithms have a significantly improved scalability to large-scale MASs compared with centralized and consensus-based distributed RL algorithms.

cs.LG↗

arXiv2022

Distributed Cooperative Multi-Agent Reinforcement Learning with Directed Coordination Graph

Existing distributed cooperative multi-agent reinforcement learning (MARL) frameworks usually assume undirected coordination graphs and communication graphs while estimating a global reward via consensus algorithms for policy evaluation. Such a framework may induce expensive communication costs and exhibit poor scalability due to requirement of global consensus. In this work, we study MARLs with directed coordination graphs, and propose a distributed RL algorithm where the local policy evaluations are based on local value functions. The local value function of each agent is obtained by local communication with its neighbors through a directed learning-induced communication graph, without using any consensus algorithm. A zeroth-order optimization (ZOO) approach based on parameter perturbation is employed to achieve gradient estimation. By comparing with existing ZOO-based RL algorithms, we show that our proposed distributed RL algorithm guarantees high scalability. A distributed resource allocation example is shown to illustrate the effectiveness of our algorithm.

cs.MA↗

arXiv2021

Hierarchical Frequency and Voltage Control using Prioritized Utilization of Inverter Based Resources

We propose a novel hierarchical frequency and voltage control design for multi-area power system integrated with inverter-based resources (IBRs). The design is based on the idea of prioritizing the use of IBRs over conventional generator-based control in compensating for sudden and unpredicted changes in loads and generations, and thereby mitigate any undesired dynamics in the frequency or the voltage by exploiting their fast actuation time constants. A new sequential optimization problem, referred to as Area Prioritized Power Flow (APPF), is formulated to model this prioritization. It is shown that compared to conventional power flow APPF not only leads to a fairer balance between the dispatch of active and reactive power from the IBRs and the synchronous generators, but also limits the impact of any contingency from spreading out beyond its respective control area, thereby guaranteeing a better collective dynamic performance of the grid. This improvement, however, comes at the cost of adding an extra layer of communication needed for executing APPF in a hierarchical way. Results are validated using simulations of a 9-machine, 6-IBR, 33-bus, 3-area power system model, illustrating how APPF can mitigate a disturbance faster and more efficiently by prioritizing the use of local area-resources.

eess.SY↗

arXiv2021

Asynchronous Distributed Reinforcement Learning for LQR Control via Zeroth-Order Block Coordinate Descent

Recently introduced distributed zeroth-order optimization (ZOO) algorithms have shown their utility in distributed reinforcement learning (RL). Unfortunately, in the gradient estimation process, almost all of them require random samples with the same dimension as the global variable and/or require evaluation of the global cost function, which may induce high estimation variance for large-scale networks. In this paper, we propose a novel distributed zeroth-order algorithm by leveraging the network structure inherent in the optimization objective, which allows each agent to estimate its local gradient by local cost evaluation independently, without use of any consensus protocol. The proposed algorithm exhibits an asynchronous update scheme, and is designed for stochastic non-convex optimization with a possibly non-convex feasible domain based on the block coordinate descent method. The algorithm is later employed as a distributed model-free RL algorithm for distributed linear quadratic regulator design, where a learning graph is designed to describe the required interaction relationship among agents in distributed learning. We provide an empirical validation of the proposed algorithm to benchmark its performance on convergence rate and variance against a centralized ZOO algorithm.

eess.SY↗

arXiv2021

Designing Optimal Key Lengths and Control Laws for Encrypted Control Systems based on Sample Identifying Complexity and Deciphering Time

In the state-of-the-art literature on cryptography and control theory, there has been no systematic methodology of constructing cyber-physical systems that can achieve desired control performance while being protected against eavesdropping attacks. In this paper, we tackle this challenging problem. We first propose two novel notions referred to as sample identifying complexity and sample deciphering time in an encrypted-control framework. The former explicitly captures the relation between the dynamical characteristics of control systems and the level of identifiability of the systems while the latter shows the relation between the computation time for the identification and the key length of a cryptosystem. Based on these two tractable new notions, we propose a systematic method for designing the both of an optimal key length to prevent system identification with a given precision within a given life span of systems, and of an optimal controller to maximize both of the control performance and the difficulty of the identification. The efficiency of the proposed method in terms of security level and realtime-ness is investigated through numerical simulations. To the best of our knowledge, this paper first connect the relationship between the security of cryptography and dynamical systems from a control-theoretic perspective.

eess.SY↗

arXiv2021

A Robust Stackelberg Game for Cyber-Security Investment in Networked Control Systems

We present a resource-planning game for cyber-security of networked control systems (NCS). The NCS is assumed to be operating in closed-loop using a linear state-feedback $\mathcal{H}_2$ controller. A zero-sum, two-player Stackelberg game (SG) is developed between an attacker and a defender for this NCS. The attacker aims to disable communication of selected nodes and thereby render the feedback gain matrix to be sparse, leading to degradation of closed-loop performance, while the defender aims to prevent this loss by investing in the protection of targeted nodes. Both players trade their $\mathcal{H}_2$-performance objectives for the costs of their actions. The standard backward induction method is modified to determine a cost-based Stackelberg equilibrium (CBSE) that saves the players' costs without degrading the control performance. We analyze the dependency of a CBSE on the relative budgets of the players as well as on the node "importance" order. Moreover, a robust-defense method is developed for the realistic case when the defender is not informed about the attacker's resources. The proposed algorithms are validated using examples from wide-area control of electric power systems. It is demonstrated that reliable and robust defense is feasible unless the defender's resources are severely limited relative to the attacker's resources. We also show that the proposed methods are robust to time-varying model uncertainties and thus are suitable for long-term security investment in realistic NCSs. Finally, we employ computationally efficient genetic algorithms (GA) to compute the optimal strategies of the attacker and the defender in realistic large power systems.

math.OC↗

arXiv2021

Learning Distributed Stabilizing Controllers for Multi-Agent Systems

We address the problem of model-free distributed stabilization of heterogeneous multi-agent systems using reinforcement learning (RL). Two algorithms are developed. The first algorithm solves a centralized linear quadratic regulator (LQR) problem without knowing any initial stabilizing gain in advance. The second algorithm builds upon the results of the first algorithm, and extends it to distributed stabilization of multi-agent systems with predefined interaction graphs. Rigorous proofs are provided to show that the proposed algorithms achieve guaranteed convergence if specific conditions hold. A simulation example is presented to demonstrate the theoretical results.

eess.SY↗

arXiv2021

Optimal Co-Designs of Communication and Control in Bandwidth-Constrained Cyber-Physical Systems

We address the problem of sparsity-promoting optimal control of cyber-physical systems (CPSs) in the presence of communication delays. The delays are categorized into two types - namely, an inter-layer delay for passing state and control information between the physical layer and the cyber layer, and an intra-layer delay that operates between the computing agents, referred to here as control nodes (CNs), within the cyber-layer. Our objective is to minimize the closed-loop H2-norm of the physical system by co-designing an optimal combination of these two delays and a sparse state-feedback controller while respecting a given bandwidth cost constraint. We propose a two-loop optimization algorithm for this. Based on the alternating directions method of multipliers (ADMM), the inner loop handles the conflicting directions between the decreasing H2-norm and the increasing sparsity level of the controller. The outer loop comprises a semidefinite program (SDP)-based relaxation of non-convex inequalities necessary for closed-loop stability. Moreover, for CPSs where the state and control information assigned to the CNs are not private, we derive an additional algorithm that further sparsifies the communication topology by modifying the row and column structures of the obtained controller, resulting in reassigning the communication map between the cyber and physical layers, and determining which physical agent should send its state information to which CN. Proofs for closed-loop stability and optimality are provided for both algorithms, followed by numerical simulations.

math.OC↗

arXiv2020

Decomposability and Parallel Computation of Multi-Agent LQR

Individual agents in a multi-agent system (MAS) may have decoupled open-loop dynamics, but a cooperative control objective usually results in coupled closed-loop dynamics thereby making the control design computationally expensive. The computation time becomes even higher when a learning strategy such as reinforcement learning (RL) needs to be applied to deal with the situation when the agents dynamics are not known. To resolve this problem, we propose a parallel RL scheme for a linear quadratic regulator (LQR) design in a continuous-time linear MAS. The idea is to exploit the structural properties of two graphs embedded in the $Q$ and $R$ weighting matrices in the LQR objective to define an orthogonal transformation that can convert the original LQR design to multiple decoupled smaller-sized LQR designs. We show that if the MAS is homogeneous then this decomposition retains closed-loop optimality. Conditions for decomposability, an algorithm for constructing the transformation matrix, a parallel RL algorithm, and robustness analysis when the design is applied to non-homogeneous MAS are presented. Simulations show that the proposed approach can guarantee significant speed-up in learning without any loss in the cumulative value of the LQR cost.

eess.SY↗

arXiv2020

Model-Free Optimal Control of Linear Multi-Agent Systems via Decomposition and Hierarchical Approximation

Designing the optimal linear quadratic regulator (LQR) for a large-scale multi-agent system (MAS) is time-consuming since it involves solving a large-size matrix Riccati equation. The situation is further exasperated when the design needs to be done in a model-free way using schemes such as reinforcement learning (RL). To reduce this computational complexity, we decompose the large-scale LQR design problem into multiple smaller-size LQR design problems. We consider the objective function to be specified over an undirected graph, and cast the decomposition as a graph clustering problem. The graph is decomposed into two parts, one consisting of independent clusters of connected components, and the other containing edges that connect different clusters. Accordingly, the resulting controller has a hierarchical structure, consisting of two components. The first component optimizes the performance of each independent cluster by solving the smaller-size LQR design problem in a model-free way using an RL algorithm. The second component accounts for the objective coupling different clusters, which is achieved by solving a least squares problem in one shot. Although suboptimal, the hierarchical controller adheres to a particular structure as specified by inter-agent couplings in the objective function and by the decomposition strategy. Mathematical formulations are established to find a decomposition that minimizes the number of required communication links or reduces the optimality gap. Numerical simulations are provided to highlight the pros and cons of the proposed designs.

eess.SY↗

arXiv2020

Hierarchical Control of Multi-Agent Systems using Online Reinforcement Learning

We propose a new reinforcement learning based approach to designing hierarchical linear quadratic regulator (LQR) controllers for heterogeneous linear multi-agent systems with unknown state-space models and separated control objectives. The separation arises from grouping the agents into multiple non-overlapping groups, and defining the control goal as two distinct objectives. The first objective aims to minimize a group-wise block-decentralized LQR function that models group-level mission. The second objective, on the other hand, tries to minimize an LQR function between the average states (centroids) of the groups. Exploiting this separation, we redefine the weighting matrices of the LQR functions in a way that they allow us to decouple their respective algebraic Riccati equations. Thereafter, we develop a reinforcement learning strategy that uses online measurements of the agent states and the average states to learn the respective controllers based on the approximate Riccati equations. Since the first controller is block-decentralized and, therefore, can be learned in parallel, while the second controller is reduced-dimensional due to averaging, the overall design enjoys a significantly reduced learning time compared to centralized reinforcement learning.

eess.SY↗

arXiv2020

A Stackelberg Security Investment Game for Voltage Stability of Power Systems

We formulate a Stackelberg game between an attacker and a defender of a power system. The attacker attempts to alter the load setpoints of the power system covertly and intelligently, so that the voltage stability margin of the grid is reduced, driving the entire system towards a voltage collapse. The defender, or the system operator, aims to compensate for this reduction by retuning the reactive power injection to the grid by switching on control devices, such as a bank of shunt capacitors. A modified Backward Induction method is proposed to find a cost-based Stackelberg equilibrium (CBSE) of the game, which saves the players' costs while providing the optimal allocation of both players' investment resources under budget and covertness constraints. We analyze the proposed game extensively for the IEEE 9-bus power system model and present an example of its performance for the IEEE 39-bus power system model. It is demonstrated that the defender is able to maintain system stability unless its security budget is much lower than the attacker's budget.

eess.SY↗