arXiv subjects

Serdar Yüksel

Publications and source records attributed to Serdar Yüksel.

At least 37 records · Page 2Linked to original sources

arXiv2024

Another Look at Partially Observed Optimal Stochastic Control: Existence, Ergodicity, and Approximations without Belief-Reduction

We present an alternative view for the study of optimal control of partially observed Markov Decision Processes (POMDPs). We first revisit the traditional (and by now standard) separated-design method of reducing the problem to fully observed MDPs (belief-MDPs), and present conditions for the existence of optimal policies. Then, rather than working with this standard method, we define a Markov chain taking values in an infinite dimensional product space with the history process serving as the controlled state process and a further refinement in which the control actions and the state process are causally conditionally independent given the measurement/information process. We provide new sufficient conditions for the existence of optimal control policies under the discounted cost and average cost infinite horizon criteria. For the discounted cost setup, we establish near optimality of finite window policies via a direct argument involving near optimality of quantized approximations for MDPs under weak Feller continuity, where finite truncations of memory can be viewed as quantizations of infinite memory with a uniform diameter in each finite window restriction under the product metric. For the average cost setup, we provide new existence conditions and also a general approach on how to initialize the randomness which we show to establish convergence to optimal cost. In the control-free case, our analysis leads to new and weak conditions for the existence and uniqueness of invariant probability measures for nonlinear filter processes.

math.OC↗

arXiv2024

Paths to Equilibrium in Games

In multi-agent reinforcement learning (MARL) and game theory, agents repeatedly interact and revise their strategies as new data arrives, producing a sequence of strategy profiles. This paper studies sequences of strategies satisfying a pairwise constraint inspired by policy updating in reinforcement learning, where an agent who is best responding in one period does not switch its strategy in the next period. This constraint merely requires that optimizing agents do not switch strategies, but does not constrain the non-optimizing agents in any way, and thus allows for exploration. Sequences with this property are called satisficing paths, and arise naturally in many MARL algorithms. A fundamental question about strategic dynamics is such: for a given game and initial strategy profile, is it always possible to construct a satisficing path that terminates at an equilibrium? The resolution of this question has implications about the capabilities or limitations of a class of MARL algorithms. We answer this question in the affirmative for normal-form games. Our analysis reveals a counterintuitive insight that reward deteriorating strategic updates are key to driving play to equilibrium along a satisficing path.

cs.GT↗

arXiv2024

Refined Bounds on Near Optimality Finite Window Policies in POMDPs and Their Reinforcement Learning

Finding optimal policies for Partially Observable Markov Decision Processes (POMDPs) is challenging due to their uncountable state spaces when transformed into fully observable Markov Decision Processes (MDPs) using belief states. Traditional methods such as dynamic programming or policy iteration are difficult to apply in this context, necessitating the use of approximation methods on belief states or other techniques. Recently, in (Journal of Machine Learning Research, vol. 23, pp. 1-46, 2022) and (Mathematics of Operations Research, vol. 48, pp. 2066-2093, Nov. 2023), it was shown that sliding finite window based policies are near-optimal for POMDPs with standard Borel valued hidden state spaces, and can be learned via reinforcement learning, with error bounds explicitly dependent on a uniform filter stability term involving total variation in expectation and sample path-wise, respectively. In this paper, we refine these performance bounds and (i) extend them to bounds via uniform filter stability in expected Wasserstein distance leading to an error bound in expectation, and (ii) complementary conditions bounds via uniform filter stability in sample path-wise total variation distance leading to a uniform error bound. We present explicit examples. Our approach thus provides complementary and more refined bounds on the error terms in both total variation and Wasserstein metrics, offering more relaxed and stronger bounds over the approximation error in POMDP solutions on the performance and near optimality of sliding finite window control policies.

math.OC↗

arXiv2024

Average Cost Optimality of Partially Observed MDPS: Contraction of Non-linear Filters, Optimal Solutions and Approximations

The average cost optimality is known to be a challenging problem for partially observable stochastic control, with few results available beyond the finite state, action, and measurement setup, for which somewhat restrictive conditions are available. In this paper, we present explicit and easily testable conditions for the existence of solutions to the average cost optimality equation where the state space is compact. In particular, we present a new contraction based analysis, which is new to the literature to our knowledge, building on recent regularity results for non-linear filters. Beyond establishing existence, we also present several implications of our analysis that are new to the literature: (i) robustness to incorrect priors (ii) near optimality of policies based on quantized approximations, (iii) near optimality of policies with finite memory, and (iv) convergence in Q-learning. In addition to our main theorem, each of these represents a novel contribution for average cost criteria.

math.OC↗

arXiv2024

Robust Decentralized Control of Coupled Systems via Risk Sensitive Control of Decoupled or Simple Models with Measure Change

Decentralized stochastic control problems with local information involve problems where multiple agents and subsystems which are coupled via dynamics and/or cost are present. Typically, however, the dynamics of such couplings is complex and difficult to precisely model, leading to questions on robustness in control design. Additionally, when such a coupling can be modeled, the problem arrived at is typically challenging and non-convex, due to decentralization of information. In this paper, we develop a robustness framework for optimal decentralized control of interacting agents, where we show that a decentralized control problem with interacting agents can be robustly designed by considering a risk-sensitive version of non-interacting agents/particles. This leads to a tractable robust formulation where we give a bound on the value of the cost function in terms of the risk-sensitive cost function for the non-interacting case plus a term involving the ``strength" of the interaction as measured by relative entropy. We will build on Gaussian measure theory and an associated variational equality. A particular application includes mean-field models consisting of (a generally large number of) interacting agents which are often hard to solve for the case with small or moderate numbers of agents, leading to an interest in effective approximations and robustness. By adapting a risk-sensitivity parameter, we also robustly control a non-symmetrically interacting problem with mean-field cost by one which is symmetric with a risk-sensitive criterion, and in the limit of small interactions, show the stability of optimal solutions to perturbations.

math.OC↗

arXiv2024

Decentralized Exchangeable Stochastic Dynamic Teams in Continuous-time, their Mean-Field Limits and Optimality of Symmetric Policies

We study a class of stochastic exchangeable teams comprising a finite number of decision makers (DMs) as well as their mean-field limits involving infinite numbers of DMs. In the finite population regime, we study exchangeable teams under the centralized information structure. For the infinite population setting, we study exchangeable teams under the decentralized mean-field information sharing. The paper makes the following main contributions: i) For finite population exchangeable teams, we establish the existence of a randomized optimal policy that is exchangeable (permutation invariant) and Markovian; ii) As our main result in the paper, we show that a sequence of exchangeable optimal policies for finite population settings converges to a conditionally symmetric (identical), independent, and decentralized randomized policy for the infinite population problem, which is globally optimal for the infinite population problem. This result establishes the existence of a symmetric, independent, decentralized optimal randomized policy for the infinite population problem. Additionally, this proves the optimality of the limiting measure-valued MDP for the representative DM; iii) Finally, we show that symmetric, independent, decentralized optimal randomized policies are approximately optimal for the corresponding finite-population team with a large number of DMs under the centralized information structure. Our paper thus establishes the relation between the controlled McKean-Vlasov dynamics and the optimal infinite population decentralized stochastic control problem (without an apriori restriction of symmetry in policies of individual agents), for the first time, to our knowledge.

math.OC↗

arXiv2024

Subjective Equilibria under Beliefs of Exogenous Uncertainty: Linear Quadratic Case

We consider a stochastic dynamic game where players have their own linear state dynamics and quadratic cost functions. Players are coupled through some environment variables, generated by another linear system driven by the states and decisions of all players. Each player observes his own states realized up to the current time as well as the past realizations of his own decisions and the environment variables. Each player (incorrectly) believes that the environment variables are generated by an independent exogenous stochastic process. In this setup, we study the notion of ``subjective equilibrium under beliefs of exogenous uncertainty (SEBEU)'' introduced in our recent work arXiv:2005.01640. At an SEBEU, each player's strategy is optimal with respect to his subjective belief; moreover, the objective probability distribution of the environment variables is consistent with players' subjective beliefs. We construct an SEBEU in pure strategies, where each player strategy is an affine function of his own state and his estimate of the system state.

math.OC↗

arXiv2024

Optimal Push and Pull-Based Edge Caching For Dynamic Content

We introduce a framework and optimal `fresh' caching for a content distribution network (CDN) comprising a front-end local cache and a back-end database. The data content is dynamically updated at a back-end database and end-users are interested in the most-recent version of that content. We formulate the average cost minimization problem that captures the system's cost due to the service of aging content as well as the regular cache update cost. We consider the cost minimization problem from two individual perspectives based on the available information to either side of the CDN: the back-end database perspective and the front-end local cache perspective. For the back-end database, the instantaneous version of content is observable but the exact demand is not. Caching decisions made by the back-end database are termed `push-based caching'. For the front-end local cache, the age of content version in the cache is not observable, yet the instantaneous demand is. Caching decisions made by the front-end local cache are termed `pull-based caching'. Our investigations reveal which type of information, updates, or demand dynamic, is of higher value towards achieving the minimum cost based on other network parameters including content popularity, update rate, and demand intensity.

math.OC↗

arXiv2024

On Borkar and Young Relaxed Control Topologies and Continuous Dependence of Invariant Measures on Control Policy

In deterministic and stochastic control theory, relaxed or randomized control policies allow for versatile mathematical analysis (on continuity, compactness, convexity and approximations) to be applicable with no artificial restrictions on the classes of control policies considered, leading to very general existence results on optimal measurable policies under various setups and information structures. On relaxed controls, two studied topologies are the Young and Borkar (weak$^*$) topologies on spaces of functions from a state/measurement space to the space of probability measures on control action spaces; the former via a weak convergence topology on probability measures on a product space with a fixed marginal on the input (state) space, and the latter via a weak$^*$ topology on randomized policies viewed as maps from states/measurements to the space of signed measures with bounded variation. We establish implication and equivalence conditions between the Young and Borkar topologies on control policies. We then show that, under some conditions, for a controlled Markov chain with standard Borel spaces the invariant measure is weakly continuous on the space of stationary control policies defined by either of these topologies. An implication is near optimality of quantized stationary policies in state and actions or continuous stationary and deterministic policies for average cost control under two sets of continuity conditions (with either weak continuity in the state-action pair or strong continuity in the action for each state) on transition kernels.

math.OC↗

arXiv2023

Robustness of Optimal Controlled Diffusions with Near-Brownian Noise via Rough Paths Theory

In this article we show a robustness theorem for controlled stochastic differential equations driven by approximations of Brownian motion. Often, Brownian motion is used as an idealized model of a diffusion where approximations such as Wong-Zakai, Karhnen-Loève or fractional Brownian motion are often seen as more physical. However, there has been extensive literature on solving control problems driven by Brownian motion and little on control problems driven by more realistic models that are only approximately Brownian. The question of robustness naturally arises from such approximations. We show robustness using rough paths theory, which allows for a pathwise theory of stochastic differential equations. To this end, in particular, we show that within the class of Lipschitz continuous control policies, an optimal solution for the Brownian idealized model is near optimal for a true system driven by a non-Brownian (but near-Brownian) noise.

math.OC↗

arXiv2023

Q-Learning for MDPs with General Spaces: Convergence and Near Optimality via Quantization under Weak Continuity

Reinforcement learning algorithms often require finiteness of state and action spaces in Markov decision processes (MDPs) (also called controlled Markov chains) and various efforts have been made in the literature towards the applicability of such algorithms for continuous state and action spaces. In this paper, we show that under very mild regularity conditions (in particular, involving only weak continuity of the transition kernel of an MDP), Q-learning for standard Borel MDPs via quantization of states and actions (called Quantized Q-Learning) converges to a limit, and furthermore this limit satisfies an optimality equation which leads to near optimality with either explicit performance bounds or which are guaranteed to be asymptotically optimal. Our approach builds on (i) viewing quantization as a measurement kernel and thus a quantized MDP as a partially observed Markov decision process (POMDP), (ii) utilizing near optimality and convergence results of Q-learning for POMDPs, and (iii) finally, near-optimality of finite state model approximations for MDPs with weakly continuous kernels which we show to correspond to the fixed point of the constructed POMDP. Thus, our paper presents a very general convergence and approximation result for the applicability of Q-learning for continuous MDPs.

cs.LG↗

arXiv2023

Isomorphism Properties of Optimality and Equilibrium Solutions under Equivalent Information Structure Transformations: Stochastic Dynamic Games and Teams

Static reduction of information structures (ISs) is a method that is commonly adopted in stochastic control, team theory, and game theory. One approach entails change of measure arguments, which has been crucial for stochastic analysis and has been an effective method for establishing existence and approximation results for optimal policies. Another approach entails utilization of invertibility properties of measurements, with further generalizations of equivalent IS reductions being possible. In this paper, we demonstrate the limitations of such approaches for a wide class of stochastic dynamic games and teams, and present a systematic classification of static reductions for which both positive and negative results on equivalence properties of equilibrium solutions can be obtained: (i) those that are policy-independent, (ii) those that are policy-dependent, and (iii) a third type that we will refer to as static measurements with control-sharing reduction (where the measurements are static although control actions are shared according to the partially nested IS). For the first type, we show that there is a bijection between Nash equilibrium (NE) policies under the original IS and their policy-independent static reductions, and establish sufficient conditions under which stationary solutions are also isomorphic between these ISs. For the second type, however, we show that there is generally no isomorphism between NE (or stationary) solutions under the original IS and their policy-dependent static reductions. Sufficient conditions (on the cost functions and policies) are obtained to establish such an isomorphism relationship between Nash equilibria of dynamic non-zero-sum games and their policy-dependent static reductions. For zero-sum games and teams, these sufficient conditions can be further relaxed.

math.OC↗

arXiv2023

Signaling Games in Multiple Dimensions: Geometric Properties of Equilibrium Solutions

Signaling game problems investigate communication scenarios where encoder(s) and decoder(s) have misaligned objectives due to the fact that they either employ different cost functions or have inconsistent priors. This problem has been studied in the literature for scalar sources under various setups. In this paper, we consider multi-dimensional sources under quadratic criteria in the presence of a bias leading to a mismatch in the criteria, where we show that the generalization from the scalar setup is more than technical. We show that the Nash equilibrium solutions lead to structural richness due to the subtle geometric analysis the problem entails, with consequences in both system design, the presence of linear Nash equilibria, and an information theoretic problem formulation. We first provide a set of geometric conditions that must be satisfied in equilibrium considering any multi-dimensional source. Then, we consider independent and identically distributed sources and characterize necessary and sufficient conditions under which an informative linear Nash equilibrium exists. These conditions involve the bias vector that leads to misaligned costs. Depending on certain conditions related to the bias vector, the existence of linear Nash equilibria requires sources with a Gaussian or a symmetric density. Moreover, in the case of Gaussian sources, our results have a rate-distortion theoretic implication that achievable rates and distortions in the considered game theoretic setup can be obtained from its team theoretic counterpart.

cs.IT↗

arXiv2023

Decentralized Multi-Agent Reinforcement Learning for Continuous-Space Stochastic Games

Stochastic games are a popular framework for studying multi-agent reinforcement learning (MARL). Recent advances in MARL have focused primarily on games with finitely many states. In this work, we study multi-agent learning in stochastic games with general state spaces and an information structure in which agents do not observe each other's actions. In this context, we propose a decentralized MARL algorithm and we prove the near-optimality of its policy updates. Furthermore, we study the global policy-updating dynamics for a general class of best-reply based algorithms and derive a closed-form characterization of convergence probabilities over the joint policy space.

cs.LG↗

arXiv2023

Satisficing Paths and Independent Multi-Agent Reinforcement Learning in Stochastic Games

In multi-agent reinforcement learning (MARL), independent learners are those that do not observe the actions of other agents in the system. Due to the decentralization of information, it is challenging to design independent learners that drive play to equilibrium. This paper investigates the feasibility of using satisficing dynamics to guide independent learners to approximate equilibrium in stochastic games. For $ε\geq 0$, an $ε$-satisficing policy update rule is any rule that instructs the agent to not change its policy when it is $ε$-best-responding to the policies of the remaining players; $ε$-satisficing paths are defined to be sequences of joint policies obtained when each agent uses some $ε$-satisficing policy update rule to select its next policy. We establish structural results on the existence of $ε$-satisficing paths into $ε$-equilibrium in both symmetric $N$-player games and general stochastic games with two players. We then present an independent learning algorithm for $N$-player symmetric games and give high probability guarantees of convergence to $ε$-equilibrium under self-play. This guarantee is made using symmetry alone, leveraging the previously unexploited structure of $ε$-satisficing paths.

cs.GT↗

arXiv2023

An Asymptotically Optimal Two-Part Fixed-Rate Coding Scheme for Networked Control with Unbounded Noise

It is known that under fixed-rate information constraints, adaptive quantizers can be used to stabilize an open-loop-unstable linear system on $\mathbb{R}^n$ driven by unbounded noise. These adaptive schemes can be designed so that they have near-optimal rate, and the resulting system will be stable in the sense of having an invariant probability measure, or ergodicity, as well as boundedness of the state second moment. Although structural results and information theoretic bounds of encoders have been studied, the performance of such adaptive fixed-rate quantizers beyond stabilization has not been addressed. In this paper, we propose a two-part adaptive (fixed-rate) coding scheme that achieves state second moment convergence to the classical optimum (i.e., for the fully observed setting) under mild moment conditions on the noise process. The first part, as in prior work, leads to ergodicity (via positive Harris recurrence) and the second part ensures that the state second moment converges to the classical optimum at high rates. These results are established using an intricate analysis which uses random-time state-dependent Lyapunov stochastic drift criteria as a core tool.

math.OC↗

arXiv2022

Subjective Equilibria under Beliefs of Exogenous Uncertainty for Dynamic Games

We present a subjective equilibrium notion (called "subjective equilibrium under beliefs of exogenous uncertainty (SEBEU)" for stochastic dynamic games in which each player chooses her decisions under the (incorrect) belief that a stochastic environment process driving the system is exogenous whereas in actuality this process is a solution of closed-loop dynamics affected by each individual player. Players observe past realizations of the environment variables and their local information. At equilibrium, if players are given the full distribution of the stochastic environment process as if it were an exogenous process, they would have no incentive to unilaterally deviate from their strategies. This notion thus generalizes what is known as the static price-taking behavior in prior literature to a stochastic and dynamic setup. We establish existence of SEBEU, study various properties and present explicit solutions. We obtain the $ε$-Nash equilibrium property of SEBEU when there are many players.

math.OC↗

arXiv2022

Continuity of Cost in Borkar Control Topology and Implications on Discrete Space and Time Approximations for Controlled Diffusions under Several Criteria

We first show that the discounted cost, cost up to an exit time, and ergodic cost involving controlled non-degenerate diffusions are continuous on the space of stationary control policies when the policies are given a topology introduced by Borkar [V. S. Borkar, A topology for Markov controls, Applied Mathematics and Optimization 20 (1989), 55-62]. The same applies for finite horizon problems when the control policies are markov and the topology is revised to include time also as a parameter. We then establish that finite action/piecewise constant stationary policies are dense in the space of stationary Markov policies under this topology. Using the above mentioned continuity and denseness results we establish that finite action/piecewise constant policies approximate optimal stationary policies with arbitrary precision. This gives rise to the applicability of many numerical methods such as policy iteration and stochastic learning methods for discounted cost, cost up to an exit time, and ergodic cost optimal control problems in continuous-time. For the finite-horizon setup, we establish additionally near optimality of time-discretized policies by an analogous argument. We thus present a unified and concise approach for approximations directly applicable under several commonly adopted cost criteria.

math.OC↗