arXiv subjects

Serdar Yüksel

Publications and source records attributed to Serdar Yüksel.

At least 19 recordsLinked to original sources

arXiv2026

Truth Revelation, Information Hiding, or Misinformation: Characterization of Equilibrium Outcomes in Signaling Games

In signaling games where a sender and a receiver have misaligned criteria, equilibrium behavior may lead to fully revealing, quantized, or randomized policies. Notably, the first arises in statistical decision theory and classical communication theoretic problems involving a fully aligned sensor and receiver, the second arises in Nash theoretic simultaneous signaling games, and the last may appear in Stackelberg type (leader-follower) Bayesian signaling games. In this paper, we investigate the Bayesian persuasion problem involving a receiver that tries to estimate the source. We show that for certain payoff structures, the equilibrium solution is such that a source observation is mapped to distinct messages with nonzero probabilities. More specifically, we completely characterize conditions under which the sender requires randomization for the Bayesian persuasion problem involving general sources with finite cardinality. In particular, regardless of whether the equilibrium solution under a deterministic policy restriction is fully revealing, quantized or noninformative, there exists a randomized sender policy that improves the sender's payoff under certain conditions characterized in the paper. Moreover, we provide an algorithmic procedure to obtain the Bayesian persuasion solution, where the algorithm compares the payoffs with finitely many posterior probability combinations. We also consider fully aligned and completely misaligned payoff structures, where the solutions respectively involve a fully revealing sender and a noninformative sender. Then, we unify these results by proving that if the sender's expected payoff with respect to posterior distributions is continuous, then the equilibrium solution involves either a fully revealing sender or a noninformative sender.

cs.GT↗

arXiv2026

Generalization Bounds on Optimal Control for Transformer Training and Wasserstein Distributional Robustness

We derive finite-sample generalization bounds for Transformers trained with dynamic programming recursions. Building on the doubly lifted, measure-valued formulation of Transformer dynamics, we view data sets as probability laws on pairs of empirical input-output measures, allowing us to interpret the training problem as a finite-horizon Markovian control problem. We then analyze a quantized model, derived by quantizing the state, action, and measure-state spaces, and derive explicit finite-sample generalization bounds using concentration inequalities for empirical laws on finite metric spaces together with a Lipschitz stability estimate for the value function. These bounds are transferred to the base model at the cost of an explicit approximation error. Finally, we show that the same machinery yields a distributionally robust control formulation of the training problem, connecting Transformer generalization to Wasserstein distributionally robust optimization.

cs.LG↗

arXiv2026

Mean-Field Systems with Heterogeneous Subteams: Optimality of Cluster-Symmetric Independent Policies and Equivalence with Decentralized McKean-Vlasov Control of Cluster-Representative Agents

Across science and engineering, mean-field methods have been a powerful and versatile approach for the analysis of systems of many interacting elements. However, common arguments used to characterize an infinite population limit can be quite restrictive from a modeling perspective by requiring that all agents be identical (i.e. symmetric, or homogeneous). In this paper, we consider large interactive particle systems under agent heterogeneity for a class of discrete time teams composed of finitely many species of agents, grouped into symmetric subteams, called clusters. In particular, for the class of discounted, partially exchangeable cost criteria considered, we establish the optimality of centralized joint policies which are exchangeable within each cluster and depend on the agent ensemble only up to the state empirical distribution over each cluster. Following this, a generalization of De Finetti's theorem is used to demonstrate the subsequential convergence of these optimal policies to one which is decentralized (depending on only the local state and distribution over each cluster) and symmetric within each subteam as the population size approaches infinity. This solution is shown to induce a sequence of asymptotically optimal policies for the finite population problems which retain their structure and decentralization. Furthermore, our analysis justifies the optimality of a decentralized McKean-Vlasov team representation involving coupled representative agents for each of the clusters, and establishes a verification theorem/value iterations for the mean-field limit. In this way, we provide an avenue for analyzing complex, cooperative systems with finite heterogeneity and set the stage for further research on learning algorithms.

math.OC↗

arXiv2026

SLAM as a Stochastic Control Problem with Partial Information: Optimal Solutions and Rigorous Approximations

Simultaneous localization and mapping (SLAM) is a foundational state estimation problem in robotics in which a robot accurately constructs a map of its environment while also localizing itself within this construction. We study the active SLAM problem through the lens of optimal stochastic control, thereby recasting it as a decision-making problem under partial information. After reviewing several commonly studied models, we present a general stochastic control formulation of active SLAM together with a rigorous treatment of motion, sensing, and map representation. We introduce a new exploration stage cost that encodes the geometry of the state when evaluating information-gathering actions. This formulation, constructed as a nonstandard partially observable Markov decision process (POMDP), is then analyzed to derive rigorously justified approximate solutions that are near-optimal. To enable this analysis, the associated regularity conditions are studied under general assumptions that apply to a wide range of robotics applications. For a particular case, we conduct an extensive numerical study in which standard learning algorithms are used to learn near-optimal policies.

cs.RO↗

arXiv2026

An Optimal Control Approach To Transformer Training

In this paper, we develop a rigorous optimal control-theoretic approach to Transformer training that respects key structural constraints such as (i) realized-input-independence during execution, (ii) the ensemble control nature of the problem, and (iii) positional dependence. We model the Transformer architecture as a discrete-time controlled particle system with shared actions, exhibiting noise-free McKean-Vlasov dynamics. While the resulting dynamics is not Markovian, we show that lifting it to probability measures produces a fully-observed Markov decision process (MDP). Positional encodings are incorporated into the state space to preserve the sequence order under lifting. Using the dynamic programming principle, we establish the existence of globally optimal policies under mild assumptions of compactness. We further prove that closed-loop policies in the lifted is equivalent to an initial-distribution dependent open-loop policy, which are realized-input-independent and compatible with standard Transformer training. To train a Transformer, we propose a triply quantized training procedure for the lifted MDP by quantizing the state space, the space of probability measures, and the action space, and show that any optimal policy for the triply quantized model is near-optimal for the original training problem. Finally, we establish stability and empirical consistency properties of the lifted model by showing that the value function is continuous with respect to the perturbations of the initial empirical measures and convergence of policies as the data size increases. This approach provides a globally optimal and robust alternative to gradient-based training without requiring smoothness or convexity.

cs.LG↗

arXiv2025

Decentralized Detection with Many Sensors: Optimality of Exchangeable and Identical Encoding Policies

We study a class of binary detection problems involving a single fusion center and a large or countably infinite number of sensors. Each sensor acts under a decentralized information structure, accessing only a local noisy observation related to the hypothesis. Based on this observation, sensors select policies to transmit a quantized signal through their actions to the fusion center, which makes the final decision using only these actions. This paper makes the following contributions: i) In the finitely many sensor setting, we provide a formal proof that an optimal encoding policy exists, and such an optimal policy is independent, deterministic, and of threshold type for the sensors and the maximum \emph{a posteriori} probability type for the fusion center; ii) For the finitely many sensor setting, we further show that an optimal encoding policy exhibits an exchangeability (permutation invariance) property; iii) We establish that an optimal encoding policy exists that is symmetric (identical) and independent across sensors in the infinitely many sensor setting under the error exponent cost; iv) Finally, we show that a symmetric optimal policy for the infinite population regime with the error exponent cost is approximately optimal for the large but finite sensor regime under the same cost criterion. We anticipate that the mathematical program used in the paper will find applications in several other massive communications applications.

math.OC↗

arXiv2025

Sensitivity of Filter Kernels and Robustness Bounds to Transition and Measurement Kernel Perturbations in Partially Observable Stochastic Control

Studying the stability of partially observed Markov decision processes (POMDPs) with respect to perturbations in either transition or observation kernels is a significant problem. While asymptotic robustness/stability results as approximate transition kernels and/or measurement kernels converge to the true ones have been previously reported, studies on explicit bounds on value differences and mismatch costs have been limited in scope for POMDPs. In this paper, we provide such explicit bounds under both discounted and average cost criteria. To this end, and also as an independent contribution, we first study the perturbations induced on the filter kernels (that is, the kernels of the belief-MDP reduction of POMDPs) as the transition and measurement kernels are perturbed. The bounds are given in terms of Wasserstein and total variation distances between the original and approximate transition and observation kernels. We then show that control policies optimized for approximate models yield performance guarantees when applied to the true model with explicit bounds. As a particular application, we consider the case where the state space and the measurement spaces are quantized to obtain finite models, and we obtain explicit error bounds which decay to zero as the approximations get finer. This provides explicit performance guarantees for model reduction in POMDPs.

math.OC↗

arXiv2025

Kernel Metrics and Learning for Borel MDPs: Identifiability and Adaptive Control

We consider a Markov decision process with standard Borel spaces and an unknown transition kernel under the average cost criterion. We do not impose any parametrization on the set of possible kernels. To facilitate our analysis, we first develop implication relations between several topologies on kernels defined by pointwise, continuous, or uniform weak convergence; we then review robustness properties on the space of kernels, and finally we establish compactness conditions on the space of kernels. Building on this regularity analysis, we then present two data-driven identifiability results; the first one being Bayesian and the second one empirical. Our conditions for the Bayesian setting are significantly more relaxed compared with prior work which considered either finite or parametric models, though we do not obtain a rate of convergence. Our analysis is asymptotic and builds on measurability in terms of the tail $\sigma$-field of the available information. Identifiability results are then used to design near-optimal adaptive control policies which alternate between periods of exploration, where the controller acts according to a policy which is conducive to the identification of the true kernel, and periods of exploitation where the controller's information on the kernel is utilized. We will establish that such policies are near optimal. In summary, our contribution is with regard to the general standard Borel setup where there is no apriori parametric representation.

math.OC↗

arXiv2024

Reinforcement Learning for Jointly Optimal Coding and Control Policies for a Controlled Markovian System over a Communication Channel

We study the problem of joint optimization involving coding and control policies for a controlled Markovian sytem over a finite-rate noiseless communication channel. While structural results on the optimal encoding and control have been obtained in the literature, their implementation has been prohibitive in general, except for linear models. We develop regularity and existence results on optimal policies. We then obtain rigorous approximation and near optimality results for jointly optimal coding and control policies. To this end, we first develop existence, regularity, and structural properties on optimal policies, followed by rigorous approximations and reinforcement learning results. Notably, we establish near optimality of finite model approximations obtained via predictor quantization as well as sliding finite window approximations, and their reinforcement learning convergence to near optimality. A detailed comparison of the approximation schemes and their reinforcement learning performance is presented.

math.OC↗

arXiv2024

Robustness to Model Approximation, Model Learning From Data, and Sample Complexity in Wasserstein Regular MDPs

The paper studies the robustness properties of discrete-time stochastic optimal control under Wasserstein model approximation for both discounted-cost and average-cost criteria. Specifically, we study the performance loss when applying an optimal policy designed for an approximate model to the true dynamics compared with the optimal cost for the true model under the sup-norm-induced metric, and relate it to the Wasserstein-1 distance between the approximate and true transition kernels. A primary motivation of this analysis is empirical model learning, as well as empirical noise distribution learning, where Wasserstein convergence holds under mild conditions but stronger convergence criteria, such as total variation, may not. We discuss applications of the results to the disturbance estimation problem, where sample complexity bounds are given, and also to a general empirical model learning approach, obtained under either Markov or i.i.d. learning settings.

eess.SY↗

arXiv2024

Refined Bounds on Near Optimality Finite Window Policies in POMDPs and Their Reinforcement Learning

Finding optimal policies for Partially Observable Markov Decision Processes (POMDPs) is challenging due to their uncountable state spaces when transformed into fully observable Markov Decision Processes (MDPs) using belief states. Traditional methods such as dynamic programming or policy iteration are difficult to apply in this context, necessitating the use of approximation methods on belief states or other techniques. Recently, in (Journal of Machine Learning Research, vol. 23, pp. 1-46, 2022) and (Mathematics of Operations Research, vol. 48, pp. 2066-2093, Nov. 2023), it was shown that sliding finite window based policies are near-optimal for POMDPs with standard Borel valued hidden state spaces, and can be learned via reinforcement learning, with error bounds explicitly dependent on a uniform filter stability term involving total variation in expectation and sample path-wise, respectively. In this paper, we refine these performance bounds and (i) extend them to bounds via uniform filter stability in expected Wasserstein distance leading to an error bound in expectation, and (ii) complementary conditions bounds via uniform filter stability in sample path-wise total variation distance leading to a uniform error bound. We present explicit examples. Our approach thus provides complementary and more refined bounds on the error terms in both total variation and Wasserstein metrics, offering more relaxed and stronger bounds over the approximation error in POMDP solutions on the performance and near optimality of sliding finite window control policies.

math.OC↗

arXiv2024

Centralized Reduction of Decentralized Stochastic Control Models and their weak-Feller Regularity

Decentralized stochastic control problems involving general state/measurement/action spaces are intrinsically difficult to study because of the inapplicability of standard tools from centralized (single-agent) stochastic control. In this paper, we address some of these challenges for decentralized stochastic control with standard Borel spaces under two different but tightly related information structures: the one-step delayed information sharing pattern (OSDISP), and the $K$-step periodic information sharing pattern (KSPISP). We will show that the one-step delayed and $K$-step periodic problems can be reduced to a centralized Markov Decision Process (MDP), generalizing prior results which considered finite, linear, or static models, by addressing several measurability and topological questions. We then provide sufficient conditions for the transition kernels of both centralized reductions to be weak-Feller. The existence and separated nature of optimal policies under both information structures are then established. The weak Feller regularity also facilitates rigorous approximation and learning theoretic results, as shown in the paper.

math.OC↗

arXiv2024

Robust Decentralized Control of Coupled Systems via Risk Sensitive Control of Decoupled or Simple Models with Measure Change

Decentralized stochastic control problems with local information involve problems where multiple agents and subsystems which are coupled via dynamics and/or cost are present. Typically, however, the dynamics of such couplings is complex and difficult to precisely model, leading to questions on robustness in control design. Additionally, when such a coupling can be modeled, the problem arrived at is typically challenging and non-convex, due to decentralization of information. In this paper, we develop a robustness framework for optimal decentralized control of interacting agents, where we show that a decentralized control problem with interacting agents can be robustly designed by considering a risk-sensitive version of non-interacting agents/particles. This leads to a tractable robust formulation where we give a bound on the value of the cost function in terms of the risk-sensitive cost function for the non-interacting case plus a term involving the ``strength" of the interaction as measured by relative entropy. We will build on Gaussian measure theory and an associated variational equality. A particular application includes mean-field models consisting of (a generally large number of) interacting agents which are often hard to solve for the case with small or moderate numbers of agents, leading to an interest in effective approximations and robustness. By adapting a risk-sensitivity parameter, we also robustly control a non-symmetrically interacting problem with mean-field cost by one which is symmetric with a risk-sensitive criterion, and in the limit of small interactions, show the stability of optimal solutions to perturbations.

math.OC↗

arXiv2024

Decentralized Exchangeable Stochastic Dynamic Teams in Continuous-time, their Mean-Field Limits and Optimality of Symmetric Policies

We study a class of stochastic exchangeable teams comprising a finite number of decision makers (DMs) as well as their mean-field limits involving infinite numbers of DMs. In the finite population regime, we study exchangeable teams under the centralized information structure. For the infinite population setting, we study exchangeable teams under the decentralized mean-field information sharing. The paper makes the following main contributions: i) For finite population exchangeable teams, we establish the existence of a randomized optimal policy that is exchangeable (permutation invariant) and Markovian; ii) As our main result in the paper, we show that a sequence of exchangeable optimal policies for finite population settings converges to a conditionally symmetric (identical), independent, and decentralized randomized policy for the infinite population problem, which is globally optimal for the infinite population problem. This result establishes the existence of a symmetric, independent, decentralized optimal randomized policy for the infinite population problem. Additionally, this proves the optimality of the limiting measure-valued MDP for the representative DM; iii) Finally, we show that symmetric, independent, decentralized optimal randomized policies are approximately optimal for the corresponding finite-population team with a large number of DMs under the centralized information structure. Our paper thus establishes the relation between the controlled McKean-Vlasov dynamics and the optimal infinite population decentralized stochastic control problem (without an apriori restriction of symmetry in policies of individual agents), for the first time, to our knowledge.

math.OC↗

arXiv2024

Subjective Equilibria under Beliefs of Exogenous Uncertainty: Linear Quadratic Case

We consider a stochastic dynamic game where players have their own linear state dynamics and quadratic cost functions. Players are coupled through some environment variables, generated by another linear system driven by the states and decisions of all players. Each player observes his own states realized up to the current time as well as the past realizations of his own decisions and the environment variables. Each player (incorrectly) believes that the environment variables are generated by an independent exogenous stochastic process. In this setup, we study the notion of ``subjective equilibrium under beliefs of exogenous uncertainty (SEBEU)'' introduced in our recent work arXiv:2005.01640. At an SEBEU, each player's strategy is optimal with respect to his subjective belief; moreover, the objective probability distribution of the environment variables is consistent with players' subjective beliefs. We construct an SEBEU in pure strategies, where each player strategy is an affine function of his own state and his estimate of the system state.

math.OC↗

arXiv2024

Optimality of Symmetric Independent Policies under Decentralized Mean-Field Information Sharing for Stochastic Teams and Equivalence with McKean-Vlasov Control of a Representative Agent

We study a class of stochastic exchangeable teams with a finite number of decision makers (DMs) as well as their mean-field limits with infinitely many DMs. In the finite population regime, we study exchangeable teams under the centralized information structure. The paper makes the following main contributions: i) For finite population exchangeable teams, we establish the existence of an optimal policy that is exchangeable (permutation invariant) and Markovian; ii) As our main result in the paper, we show that a sequence of exchangeable optimal policies for finite population settings (which satisfies a measure valued MDP formulation due to B{\"a}uerle) converges to a decentralized symmetric (identical) and conditionally independent (given the mean-field) policy for the infinite population problem, which is then globally optimal under both the centralized information structure as well as the mean-field sharing information structure. (iii) This result establishes existence of a symmetric, independent, decentralized optimal randomized policy for the infinite population problem and proves the optimality of the limiting measure-valued MDP for the representative DM. Our paper thus establishes the relation between the controlled McKean-Vlasov dynamics and the optimal infinite population decentralized stochastic control problem (without an apriori restriction of symmetry in policies of individual agents), for the first time, to our knowledge (beyond several special cases). We also establish near optimality of a numerical method for solving this problem. iv) Finally, we show that symmetric, independent, decentralized optimal randomized policies are approximately optimal for the corresponding finite-population team with a large number of DMs under the centralized information structure.

math.OC↗

arXiv2024

Paths to Equilibrium in Games

In multi-agent reinforcement learning (MARL) and game theory, agents repeatedly interact and revise their strategies as new data arrives, producing a sequence of strategy profiles. This paper studies sequences of strategies satisfying a pairwise constraint inspired by policy updating in reinforcement learning, where an agent who is best responding in one period does not switch its strategy in the next period. This constraint merely requires that optimizing agents do not switch strategies, but does not constrain the non-optimizing agents in any way, and thus allows for exploration. Sequences with this property are called satisficing paths, and arise naturally in many MARL algorithms. A fundamental question about strategic dynamics is such: for a given game and initial strategy profile, is it always possible to construct a satisficing path that terminates at an equilibrium? The resolution of this question has implications about the capabilities or limitations of a class of MARL algorithms. We answer this question in the affirmative for normal-form games. Our analysis reveals a counterintuitive insight that reward deteriorating strategic updates are key to driving play to equilibrium along a satisficing path.

cs.GT↗

arXiv2024

Satisficing Paths to Equilibrium, Generalized Weakly Acyclic Games, and Learning

Weakly acyclic games generalize potential games and have shown to be fundamental in the study of multi-agent learning as they allow for convergence to an equilibrium via best-responding under inertia. In this paper, we present a generalization of weakly acyclic games, and we demonstrate its importance in multi-agent learning when agents employ experimental strategy updates in periods where they fail to best respond. While weak acyclicity is defined in terms of path connectivity properties of a game's better response graph, our concept is defined using a generalized better response graph under revision dynamics termed as satisficing. We refer to this class of games as generalized weakly acyclic games (GenWAGs). We provide sufficient conditions for this notion of generalized weak acyclicity in both two-player games and n-player games in normal form, including static and dynamic games. Several graph theoretic characterizations of such games are presented together with sufficiency conditions, examples, and counterexamples. Finally, implications on learning via policy revision processes are presented.

cs.GT↗