POMDPs with Lipschitz-Continuous Belief-Dependent Rewards Have Lipschitz-Continuous Value Functions

Timothy L. Molloy

Home/Archive/arXiv

arXiv · 2610.33454

POMDPs with Lipschitz-Continuous Belief-Dependent Rewards Have Lipschitz-Continuous Value Functions

Timothy L. Molloy

View original source ↗Read source document ↗Export metadata ↓

Abstract

We establish the vector (or elementwise) Lipschitz continuity of value functions for partially observed Markov decision processes (POMDPs) with reward functions that are Lipschitz-continuous functions of the posterior probability distribution of the state (i.e., the belief). Our results significantly tighten existing Lipschitz results for this class of POMDPs on infinite horizons, with previous results only yielding finite constants for discount factors less than 0.5, while our results yield finite Lipschitz constants for all discount factors less than 1. The significance of our Lipschitz constants is illustrated through their use in heuristic search value iteration (HSVI) to construct (finite) piecewise-linear value function approximations.

Explore related subjects

eess.SY cs.SY

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Timothy L. Molloy. 2026-10-24. POMDPs with Lipschitz-Continuous Belief-Dependent Rewards Have Lipschitz-Continuous Value Functions. https://arxiv.org/abs/2610.33454

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Windex: Index-Driven Contextual Scheduler for Scalable Service Guarantees in NextG Cellular Networks

Next-generation cellular networks must support heterogeneous service objectives, ranging from throughput-oriented eMBB to latency- and regularity-constrained URLLC and XR traffic. Enforcing these objectives at the RAN scheduler is difficult because wireless channels, queues, and application arrivals evolve at the millisecond timescale while spectrum resources are shared across users. We formulate service-aware RAN scheduling as a constrained Markov decision process with per-user throughput and service-regularity constraints. A Lagrangian relaxation gives an exact decomposition of the relaxed problem into per-user control problems with a common resource price. We prove fixed-dual indexability for an illustrative time-since-last-service (TSLS) debt arm with a throughput objective and TSLS constraint, note the analogous scalar throughput-debt result, and assume fixed-dual indexability for the full vector state used in deployment. We design Windex, an index-driven scheduler that learns dual-dependent per-user break-even resource prices using compact neural approximators over queue, channel, TSLS, and violation-deficit state. We further design CWindex, a contextual extension that infers service class from queue-history time series when explicit labels are unavailable. We implement Windex as an application compatible with a realtime 5G radio intelligent controller. Inference takes 10-20 $μ$s per user on a general-purpose CPU, well within a 1 ms control budget. Trace-driven emulation using over-the-air channel measurements and larger simulations show that Windex and CWindex substantially reduce per-user service violations while maintaining throughput efficiency.

eess.SY↗

arXiv2026

Minimum Time Consensus of Multi-agent System under Fuel Constraints

This work addresses the problem of finding the minimum time consensus point in the state space for a set of $N$ identical double integrator agents with bounded input and fixed fuel budget constraints. The attainable set of each agent under these constraints is computed, and its boundary is used to characterize the set of initial conditions from which consensus is possible. The minimum time to consensus is the least time at which the attainable sets of all agents intersect, and the corresponding consensus state is their point of intersection. Using convexity of the attainable set and Helly's theorem, this reduces to examining the triplets of agents, for which a closed-form expression for the minimum time and the consensus point is obtained. The triplet computations are independent and are distributed evenly among the $N$ agents. The overall minimum time and consensus point are given by the triplet with the greatest minimum time, and each agent then computes its own control input locally. An example with $12$ agents is used to illustrate the results.

eess.SY↗

arXiv2026

Describing Functions and Phase Response Curves of Excitable Systems

The describing function (DF) and phase response curve (PRC) are classical tools for the analysis of feedback oscillations and rhythmic behaviors, widely used across control engineering, biology, and neuroscience. These tools are known to have limitations in networks of relaxation oscillators and excitable systems. For this reason, the paper proposes a novel approach tailored to excitable systems. Our analysis focuses on the discrete-event operator mapping input trains of events to output trains of events. The methodology is illustrated on the excitability model of Hodgkin-Huxley. The proposed framework provides a basis for designing and analyzing central pattern generators in networks of excitable neurons, with direct relevance to neuromorphic control and neurophysiology.

eess.SY↗