A Hierarchical DRL Approach for Resource Optimization in Multi-RIS Multi-Operator Networks

Haocheng Zhang; Wei Wang; Hao Zhou; Zhiping Lu; Ming Li

Home/Archive/arXiv

arXiv · 2410.12320

A Hierarchical DRL Approach for Resource Optimization in Multi-RIS Multi-Operator Networks

Haocheng Zhang·Wei Wang·Hao Zhou·Zhiping Lu·Ming Li

View original source ↗Read source document ↗Export metadata ↓

Abstract

As reconfigurable intelligent surfaces (RIS) emerge as a pivotal technology in the upcoming sixth-generation (6G) networks, their deployment within practical multiple operator (OP) networks presents significant challenges, including the coordination of RIS configurations among OPs, interference management, and privacy maintenance. A promising strategy is to treat RIS as a public resource managed by an RIS provider (RP), which can enhance resource allocation efficiency by allowing dynamic access for multiple OPs. However, the intricate nature of coordinating management and optimizing RIS configurations significantly complicates the implementation process. In this paper, we propose a hierarchical deep reinforcement learning (HDRL) approach that decomposes the complicated RIS resource optimization problem into several subtasks. Specifically, a top-level RP-agent is responsible for RIS allocation, while low-level OP-agents control their assigned RISs and handle beamforming, RIS phase-shifts, and user association. By utilizing the semi-Markov decision process (SMDP) theory, we establish a sophisticated interaction mechanism between the RP and OPs, and introduce an advanced hierarchical proximal policy optimization (HPPO) algorithm. Furthermore, we propose an improved sequential-HPPO (S-HPPO) algorithm to address the curse of dimensionality encountered with a single RP-agent. Experimental results validate the stability of the HPPO algorithm across various environmental parameters, demonstrating its superiority over other benchmarks for joint resource optimization. Finally, we conduct a detailed comparative analysis between the proposed S-HPPO and HPPO algorithms, showcasing that the S-HPPO algorithm achieves faster convergence and improved performance in large-scale RIS allocation scenarios.

Explore related subjects

eess.SY cs.SY

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Haocheng Zhang, Wei Wang, Hao Zhou, Zhiping Lu, Ming Li. 2025-02-21. A Hierarchical DRL Approach for Resource Optimization in Multi-RIS Multi-Operator Networks. https://arxiv.org/abs/2410.12320

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Constrained Feedback Control of Nonlinear Systems via Approximate HJB and Control Barrier Functions

This paper presents a two-stage framework for constrained feedback control of input-affine nonlinear systems. Offline, an approximate value function for the unconstrained problem is computed, for example using Hamilton--Jacobi--Bellman (HJB)-based policy iteration. Online, the proposed quadratic program (QP) minimizes the pre-Hamiltonian evaluated using the approximate value-function gradient subject to safety constraints enforced by control barrier functions (CBFs). This architecture decouples performance optimization from constraint enforcement, allowing constraints to be modified without recomputing the value function. As in CBF-QP architectures based on control Lyapunov functions (CLFs), safety is enforced as a hard constraint; however, the performance objective targets approximate optimality rather than a prescribed Lyapunov decay. Numerical results on a linear 2-state hovercraft and a nonlinear 9-state spacecraft attitude-control problem show agreement with the constrained open-loop optimal control problem (OCP) benchmark in the linear case, and performance close to the OCP benchmark, improving on CLF-based controllers, in the nonlinear case.

eess.SY↗

arXiv2026

Rao-Blackwellized Stein Gradient Descent for Joint State-Parameter Estimation

We present a filtering framework for online joint state estimation and parameter identification in nonlinear, time-varying systems. The algorithm uses a Rao-Blackwellization technique to infer joint state-parameter posteriors efficiently. In particular, conditional state distributions are computed analytically via Kalman filtering, while model parameters, including the measurement-noise covariance, are approximated using particle-based Stein Variational Gradient Descent (SVGD), enabling stable real-time inference. To handle parameters subject to physical constraints, we further introduce constrained variants that enforce them through an alternating direction method of multipliers (ADMM) splitting of the SVGD update, including nonlinear equality constraints that standard particle filters cannot readily handle. We derive a stability bound that relates the approximation error in the parameter posterior to the resulting error in the marginal state distribution. Performance of the proposed filters is validated on three case studies: a fed-batch bioreactor with Haldane kinetics and a damped pendulum, both under physical constraints, and a neural-network-augmented dynamic system. The examples cover parameter estimation under inequality and equality constraints and online neural-network training within a dynamical model.

eess.SY↗

arXiv2026

Firing Rate Neural Network Implementations of Model Predictive Control

Human and animal brains perform planning to enable complex movements and behaviors, a process that can be effectively described using model predictive control (MPC). How could the brain physically implement MPC? In this work, we translate model predictive controllers into firing rate neural networks, offering insights into the nonlinear neural dynamics that underpin planning. We propose a constructive method; no training is required. This is done first applying the projected gradient method to the dual problem to derive a baseline neural network implementation. We then use factorization and contraction analysis to systematically generate alternative network architectures; in other words, we systematically generate hypotheses for how planning is done in the brain via neural dynamics. Finally, we present numerical simulations to study different neural networks performing MPC to balance an inverted pendulum on a cart (i.e., balancing a stick on a hand), including one example in which imposing sparse connectivity (a property observed in brain networks) does not degrade control performance.

eess.SY↗