Exploring Cooperative and Robust Multi-Agent Reinforcement Learning in Networked Cyber-Physical Systems: Applications in Smart Grids

Significant advances in information and networking technologies have transformed Cyber-Physical Systems (CPS) into networked cyber-physical systems (NCPS). A noteworthy example of such systems is smart grid networks, which include distributed energy resources (DERs), renewable generation, and the widespread adoption of Electric Vehicle (EV). Such complex NCPS require intelligent and autonomous control solutions. For example, the increasing number of EVs introduces significant sources of demand and user behavior uncertainty that can jeopardize the grid stability during peak hours. Traditional model-based demand-supply controls fail to accurately model and capture the complex nature of smart grid systems in the presence of different uncertainties and as the system size grows. To address these challenges, data-driven approaches have emerged as an effective solution for informed decision-making, predictive modeling, and adaptive control to enhance the resiliency of NCPS in uncertain environments.

As a powerful data-driven approach, Multi-Agent Reinforcement Learning (MARL) enables agents to learn and adapt in dynamic and uncertain environments. However, MARL techniques introduce complexities related to communication, coordination, and synchronization among agents. In this PhD research, we investigate autonomous control for smart grid decision networks using MARL. Within this context, first, we examine the issue of imperfect state information, which frequently arises due to the inherent uncertainties and limitations in observing the system state. Secondly, we investigate the challenges associated with distributed MARL techniques, with a special focus on the central training distributed execution (CTDE) methods. Throughout this research, we highlight the significance of cooperation in MARL for achieving autonomous control in smart grid systems and other cyber-physical domains. Thirdly, we propose a novel robust MARL framework using a hierarchical structure. We perform an extensive analysis and evaluation of our proposed hierarchical MARL model for large-scale EV networks, thereby addressing the scalability and robustness challenges as the number of agents within a NCPS increases.