arXiv subjects

Naci Saldi

Publications and source records attributed to Naci Saldi.

At least 19 recordsLinked to original sources

arXiv2026

When can Optimally Trained Transformers Achieve Near-Lossless Token Prediction for non-Markovian Data?

There exists little mathematical theory explaining why Transformer architectures are empirically successful even for non-Markovian data. To this end, this paper studies next-token prediction via stochastic optimal control theoretically trained Transformers for such sources, where we consider either stationary and ergodic non-Markovian sources or those which are hidden Markov models of some finite order (possibly non-stationary). In either case, we first obtain error bounds via finite models with finite window approximations. For stationary ergodic sources, the resulting error is bounded by an entropy gap; for $N$th order Markovian sources, it decays at an exponential rate $α^{\lfloor M/N\rfloor}$, where $M$ is the window length and $α$ is determined by filter stability (and the Dobrushin coefficients of the transition and measurement kernels). We then construct a Transformer of depth $M+1$, one head per layer, realizing the truncated filter, and we optimally train it with stochastic control developed in earlier work, which rigorously returns a near optimal Transformer. In particular, we show that for a non-Markovian source, either under stationarity and ergodicity for general models or under filter stability for hidden Markov models, an optimally trained Transformer has arbitrarily small performance error with increasing context length, data, and approximation refinement. Numerical studies accompany the theory.

cs.LG↗

arXiv2026

An Optimal Control Approach To Transformer Training

In this paper, we develop a rigorous optimal control-theoretic approach to Transformer training that respects key structural constraints such as (i) realized-input-independence during execution, (ii) the ensemble control nature of the problem, and (iii) positional dependence. We model the Transformer architecture as a discrete-time controlled particle system with shared actions, exhibiting noise-free controlled McKean--Vlasov dynamics, and we show that lifting the dynamics to probability measures produces a fully-observed Markov decision process (MDP). Positional encodings are incorporated into the state space to preserve the sequence order under lifting. Using the dynamic programming principle, we establish the existence of globally optimal policies under mild assumptions of compactness. We further prove that closed-loop policies in the lifted problem is equivalent to an initial-distribution dependent open-loop policy, which are realized-input-independent and compatible with standard Transformer training. To train a Transformer, we propose a triply quantized training procedure for the lifted MDP by quantizing the state space, the space of probability measures, and the action space, and show that any optimal policy for the triply quantized model is near-optimal for the original training problem. Finally, we establish stability and empirical consistency properties of the lifted model by showing that the value function is continuous with respect to the perturbations of the initial empirical measures and convergence of policies as the data size increases. This approach provides a globally optimal and robust theoretical alternative to gradient-based training without requiring smoothness or convexity.

cs.LG↗

arXiv2026

General-Sum Markov Games with Borel Spaces: Existence of Approximate Nash Equilibria via Finite Model Approximations under Setwise or Weak Continuity

Establishing the existence of exact or near Markov or stationary perfect Nash equilibria in nonzero-sum Markov games over Borel spaces is a challenging problem with limited positive results. Motivated by problems in multi-agent and Bayesian learning, this paper demonstrates the existence of approximate Markov and stationary Nash equilibria for such games under relatively mild regularity conditions; notably under setwise continuity. Our approach is constructive: For both compact and non-compact state spaces, we approximate the Borel model with finite state-action models and show that their equilibria correspond to $ε$-equilibria for the original game. Compared with previous results in the literature, which we comprehensively review, we provide more general conditions, along with explicit approximation models whose equilibria are $ε$-equilibria for the original model. For completeness, we also study the approximation of zero-sum Markov games and Markov teams to highlight the key differences between zero-sum and nonzero-sum settings. In particular, while for zero-sum and team games, joint weak (Feller) continuity of the transition kernel is sufficient (as the value function is continuous), this is not the case for general nonzero-sum games.

eess.SY↗

arXiv2026

Entropy-Regularized Mean-Field Games with Average-Reward Criterion: Span-Seminorm Contractions and Model-Free Learning

We study stationary discrete-time mean-field games with finite state and action spaces under the average-reward criterion, with an entropy-regularized objective and population-dependent dynamics and rewards. Under a minorization condition, the soft Q-operator is a span-seminorm contraction for the full transition kernel, so relative value iteration does not require the minorizing measure. When the Lipschitz constant of the kernel is below the minorization mass and the temperature exceeds an explicit threshold, the equilibrium fixed-point operator is a contraction: an equilibrium exists, its population distribution is unique, and a fixed-point iteration converges to it geometrically. We then learn the equilibrium model-free from a simulator, combining soft Q-learning at a fixed population distribution with an empirical transition kernel. We prove a finite-sample bound for soft Q-learning in the span seminorm, with temperature-independent constants, and derive convergence rates for the learned population distribution and policy. A congestion-aware routing game illustrates the results.

math.OC↗

arXiv2026

Infinite Horizon Mean-Field Terminal Value Problem

We introduce a class of finite- and infinite-horizon discrete-time control problems for studying macroscale systems in discrete time, which we refer to as mean-field terminal value problems (MFTVPs). An MFTVP takes a terminal $Q$-function as input and, starting from this terminal $Q$-function, seeks an indefinite backward evolution of optimal policies and state measures, as required for an infinite-horizon non-stationary mean-field equilibrium (MFE) of a discrete-time mean-field game (MFG). We study the existence and uniqueness properties of this problem. As an application, using the uniqueness properties of MFTVPs, we show that, in the average-cost setting (and in the discounted setting for sufficiently small discount factors), MFTVPs can be used as an intermediate step to develop approximation schemes between finite-horizon MFEs and average-cost MFEs (respectively, stationary MFEs). In contrast to approaches that use infinite-horizon MFGs as intermediate problems, MFTVPs can be employed when the discount factor is sufficiently small (or, in the average-cost setting, when the drift factor is sufficiently small), together with a regularizer, to approximate a stationary MFE. Assuming uniqueness of the solution of a MFTVP, the restrictions we impose on the Lipschitz parameters of the system are weaker than the contraction conditions available for regularized stationary MFGs, and our approximation scheme remains applicable even in the presence of multiple stationary MFEs.

math.OC↗

arXiv2026

Monotone Inclusion Approach to Weakly Monotone Discrete-Time Finite-Horizon Mean-Field Games

We revisit the problem of computing mean-field equilibria (MFEs) in discrete-time, monotone, finite-horizon mean-field games (MFGs). We show that, when the transition kernel is independent of the state-measure term and the reward function satisfies the usual weak monotonicity condition and is Lipschitz continuous, anchored proximal gradient descent methods can be used to compute a monotone MFE. We also establish last-iterate convergence results for these methods. Our approach relies on formulating the computation problem as an optimization problem over the space of occupation measures. Using this formulation, we show that the problem is equivalent to a class of constrained Lipschitz monotone inclusion problems. We then apply iterative methods for this monotone inclusion formulation to derive a tractable algorithm. The resulting algorithm achieves a convergence rate of $O(1/\sqrt{T})$ after $T$ iterations, without requiring any regularization. This rate holds even in the absence of a uniqueness assumption for the corresponding MFE.

math.OC↗

arXiv2026

Generalization Bounds on Optimal Control for Transformer Training and Wasserstein Distributional Robustness

We derive finite-sample generalization bounds for Transformers trained with dynamic programming recursions. Building on the doubly lifted, measure-valued formulation of Transformer dynamics, we view data sets as probability laws on pairs of empirical input-output measures, allowing us to interpret the training problem as a finite-horizon Markovian control problem. We then analyze a quantized model, derived by quantizing the state, action, and measure-state spaces, and derive explicit finite-sample generalization bounds using concentration inequalities for empirical laws on finite metric spaces together with a Lipschitz stability estimate for the value function. These bounds are transferred to the base model at the cost of an explicit approximation error. Finally, we show that the same machinery yields a distributionally robust control formulation of the training problem, connecting Transformer generalization to Wasserstein distributionally robust optimization.

cs.LG↗

arXiv2026

Optimality of Symmetric Independent Policies under Decentralized Mean-Field Information Sharing for Stochastic Teams and Equivalence with McKean-Vlasov Control of a Representative Agent

We study a class of stochastic exchangeable teams with a finite number of decision makers (DMs) as well as their mean-field limits with infinitely many DMs. In the finite population regime, we study exchangeable teams under the centralized information structure. The paper makes the following main contributions: i) For finite population exchangeable teams, we establish the existence of an optimal policy that is exchangeable (permutation invariant) and Markovian; ii) As our main result in the paper, we show that a sequence of exchangeable optimal policies for finite population settings (which satisfies a measure valued MDP formulation due to B{ä}uerle) converges to a decentralized symmetric (identical) and conditionally independent (given the mean-field) policy for the infinite population problem, which is then globally optimal under both the centralized information structure as well as the mean-field sharing information structure. (iii) This result establishes existence of a symmetric, independent, decentralized optimal randomized policy for the infinite population problem and proves the optimality of the limiting measure-valued MDP for the representative DM. Our paper thus establishes the relation between the controlled McKean-Vlasov dynamics and the optimal infinite population decentralized stochastic control problem (without an apriori restriction of symmetry in policies of individual agents), for the first time, to our knowledge (beyond several special cases). We also establish near optimality of a numerical method for solving this problem. iv) Finally, we show that symmetric, independent, decentralized optimal randomized policies are approximately optimal for the corresponding finite-population team with a large number of DMs under the centralized information structure.

math.OC↗

arXiv2026

Approximation of Discrete-Time Infinite-Horizon Mean-Field Equilibria via Finite-Horizon Mean-Field Equilibria

We address in this paper a fundamental question that arises in mean-field games (MFGs), namely whether mean-field equilibria (MFE) for discrete-time finite-horizon MFGs can be used to obtain approximate stationary as well as non-stationary MFE for similarly structured infinite-horizon MFGs. We provide a rigorous analysis of this relationship, and show that any accumulation point of MFE of a discounted finite-horizon MFG constitutes, under weak convergence as the time horizon goes to infinity, a non-stationary MFE for the corresponding infinite-horizon MFG. Further, under certain conditions, these non-stationary MFE converge to a stationary MFE, establishing the appealing result that finite-horizon MFE can serve as approximations for stationary MFE. Additionally, we establish improved contraction rates for iterative methods used to compute regularized MFE in finite-horizon settings, extending existing results in the literature. As a byproduct, we obtain that when two MFGs have finite-horizon MFE that are close to each other, the corresponding stationary MFE are also close. As one application of the theoretical results, we show that finite-horizon MFGs can facilitate learning-based approaches to approximate infinite-horizon MFE when system components are unknown. Under further assumptions on the Lipschitz coefficients of the regularized system components (which are stronger than contractivity of finite-horizon MFGs), we obtain exponentially decaying finite-time error bounds -- in the time horizon -- between finite-horizon non-stationary, infinite-horizon non-stationary, and stationary MFE. As a byproduct of our error bounds, we present a new uniqueness criterion for infinite-horizon nonstationary MFE beyond the available contraction results in the literature.

math.OC↗

arXiv2026

Mean-Field Systems with Heterogeneous Subteams: Optimality of Cluster-Symmetric Independent Policies and Equivalence with Decentralized McKean-Vlasov Control of Cluster-Representative Agents

Across science and engineering, mean-field methods have been a powerful and versatile approach for the analysis of systems of many interacting elements. However, common arguments used to characterize an infinite population limit can be quite restrictive from a modeling perspective by requiring that all agents be identical (i.e. symmetric, or homogeneous). In this paper, we consider large interactive particle systems under agent heterogeneity for a class of discrete time teams composed of finitely many species of agents, grouped into symmetric subteams, called clusters. In particular, for the class of discounted, partially exchangeable cost criteria considered, we establish the optimality of centralized joint policies which are exchangeable within each cluster and depend on the agent ensemble only up to the state empirical distribution over each cluster. Following this, a generalization of De Finetti's theorem is used to demonstrate the subsequential convergence of these optimal policies to one which is decentralized (depending on only the local state and distribution over each cluster) and symmetric within each subteam as the population size approaches infinity. This solution is shown to induce a sequence of asymptotically optimal policies for the finite population problems which retain their structure and decentralization. Furthermore, our analysis justifies the optimality of a decentralized McKean-Vlasov team representation involving coupled representative agents for each of the clusters, and establishes a verification theorem/value iterations for the mean-field limit. In this way, we provide an avenue for analyzing complex, cooperative systems with finite heterogeneity and set the stage for further research on learning algorithms.

math.OC↗

arXiv2026

Best Ergodic Averages via Optimal Graph Filters in Reversible Markov Chains

In this paper, we address the problem of finding the best ergodic or Birkhoff averages in the mean ergodic theorem to ensure rapid convergence to a desired value, using graph filters. Our approach begins by representing a function on the state space as a graph signal, where the (directed) graph is formed by the transition probabilities of a reversible Markov chain. We introduce a concept of graph variation, enabling the definition of the graph Fourier transform for graph signals on this directed graph. Viewing the iteration in the mean ergodic theorem as a graph filter, we recognize its non-optimality and propose three optimization problems aimed at determining optimal graph filters. These optimization problems yield the Bernstein, Chebyshev, and Legendre filters. Numerical testing reveals that while the Bernstein filter performs slightly better than the traditional ergodic average, the Chebyshev and Legendre filters significantly outperform the ergodic average, demonstrating rapid convergence to the desired value.

eess.SY↗

arXiv2026

Kernel Based Maximum Entropy Inverse Reinforcement Learning for Mean-Field Games

We consider the maximum causal entropy inverse reinforcement learning (IRL) problem for infinite-horizon stationary mean-field games (MFG), in which we model the unknown reward function within a reproducing kernel Hilbert space (RKHS). This allows the inference of rich and potentially nonlinear reward structures directly from expert demonstrations, in contrast to most existing approaches for MFGs that typically restrict the reward to a linear combination of a fixed finite set of basis functions and rely on finite-horizon formulations. We introduce a Lagrangian relaxation that enables us to reformulate the problem as an unconstrained log-likelihood maximization and obtain a solution via a gradient ascent algorithm. To establish the theoretical consistency of the algorithm, we prove the smoothness of the log-likelihood objective through the Fréchet differentiability of the related soft Bellman operators with respect to the parameters in the RKHS. To illustrate the practical advantages of the RKHS formulation, we validate our framework on a mean-field traffic routing game exhibiting state-dependent preference reversal, where the kernel-based method reduces policy recovery error by over an order of magnitude compared to a linear reward baseline with a comparable parameter count. Furthermore, we extend the framework to the finite-horizon non-stationary setting. We demonstrate that the log-likelihood reformulation is structurally unavailable in this regime and instead develop an alternative gradient descent algorithm on the convex dual via Danskin's theorem, establishing smoothness and convergence guarantees.

cs.LG↗

arXiv2025

Kernel Mean Embedding Topology: Weak and Strong Forms for Stochastic Kernels and Implications for Model Learning

We introduce a novel topology, called Kernel Mean Embedding Topology, for stochastic kernels, in a weak and strong form. This topology, defined on the spaces of Bochner integrable functions from a signal space to a space of probability measures endowed with a Hilbert space structure, allows for a versatile formulation. This construction allows one to obtain both a strong and weak formulation. (i) For its weak formulation, we highlight the utility on relaxed policy spaces, and investigate connections with the Young narrow topology and Borkar (or $ w^* $)-topology, and establish equivalence properties. We report that, while both the $ w^* $-topology and kernel mean embedding topology are relatively compact, they are not closed. Conversely, while the Young narrow topology is closed, it lacks relative compactness. (ii) We show that the strong form provides an appropriate formulation for placing topologies on spaces of models characterized by stochastic kernels with explicit robustness and learning theoretic implications on optimal stochastic control under discounted or average cost criteria. (iii) We thus show that this topology possesses several properties making it ideal to study optimality and approximations (under the weak formulation) and robustness (under the strong formulation) for many applications.

eess.SY↗

arXiv2025

Decentralized Detection with Many Sensors: Optimality of Exchangeable and Identical Encoding Policies

We study a class of binary detection problems involving a single fusion center and a large or countably infinite number of sensors. Each sensor acts under a decentralized information structure, accessing only a local noisy observation related to the hypothesis. Based on this observation, sensors select policies to transmit a quantized signal through their actions to the fusion center, which makes the final decision using only these actions. This paper makes the following contributions: i) In the finitely many sensor setting, we provide a formal proof that an optimal encoding policy exists, and such an optimal policy is independent, deterministic, and of threshold type for the sensors and the maximum \emph{a posteriori} probability type for the fusion center; ii) For the finitely many sensor setting, we further show that an optimal encoding policy exhibits an exchangeability (permutation invariance) property; iii) We establish that an optimal encoding policy exists that is symmetric (identical) and independent across sensors in the infinitely many sensor setting under the error exponent cost; iv) Finally, we show that a symmetric optimal policy for the infinite population regime with the error exponent cost is approximately optimal for the large but finite sensor regime under the same cost criterion. We anticipate that the mathematical program used in the paper will find applications in several other massive communications applications.

math.OC↗

arXiv2025

Maximum Causal Entropy IRL in Mean-Field Games and GNEP Framework for Forward RL

This paper explores the use of Maximum Causal Entropy Inverse Reinforcement Learning (IRL) within the context of discrete-time stationary Mean-Field Games (MFGs) characterized by finite state spaces and an infinite-horizon, discounted-reward setting. Although the resulting optimization problem is non-convex with respect to policies, we reformulate it as a convex optimization problem in terms of state-action occupation measures by leveraging the linear programming framework of Markov Decision Processes. Based on this convex reformulation, we introduce a gradient descent algorithm with a guaranteed convergence rate to efficiently compute the optimal solution. Moreover, we develop a new method that conceptualizes the MFG problem as a Generalized Nash Equilibrium Problem (GNEP), enabling effective computation of the mean-field equilibrium for forward reinforcement learning (RL) problems and marking an advancement in MFG solution techniques. We further illustrate the practical applicability of our GNEP approach by employing this algorithm to generate data for numerical MFG examples.

eess.SY↗

arXiv2025

Quantum Markov Decision Processes: Dynamic and Semi-Definite Programs for Optimal Solutions

In this paper, building on the formulation of quantum Markov decision processes (q-MDPs) presented in our previous work [{\sc N.~Saldi, S.~Sanjari, and S.~Yüksel}, {\em Quantum Markov Decision Processes: General Theory, Approximations, and Classes of Policies}, SIAM Journal on Control and Optimization, 2024], our focus shifts to the development of semi-definite programming approaches for optimal policies and value functions of both open-loop and classical-state-preserving closed-loop policies. First, by using the duality between the dynamic programming and the semi-definite programming formulations of any q-MDP with open-loop policies, we establish that the optimal value function is linear and there exists a stationary optimal policy among open-loop policies. Then, using these results, we establish a method for computing an approximately optimal value function and formulate computation of optimal stationary open-loop policy as a bi-linear program. Next, we turn our attention to classical-state-preserving closed-loop policies. Dynamic programming and semi-definite programming formulations for classical-state-preserving closed-loop policies are established, where duality of these two formulations similarly enables us to prove that the optimal policy is linear and there exists an optimal stationary classical-state-preserving closed-loop policy. Then, similar to the open-loop case, we establish a method for computing the optimal value function and pose computation of optimal stationary classical-state-preserving closed-loop policies as a bi-linear program.

quant-ph↗

arXiv2024

Quantum Markov Decision Processes: General Theory, Approximations, and Classes of Policies

In this paper, the aim is to develop a quantum counterpart to classical Markov decision processes (MDPs). Firstly, we provide a very general formulation of quantum MDPs with state and action spaces in the quantum domain, quantum transitions, and cost functions. Once we formulate the quantum MDP (q-MDP), our focus shifts to establishing the verification theorem that proves the sufficiency of Markovian quantum control policies and provides a dynamic programming principle. Subsequently, a comparison is drawn between our q-MDP model and previously established quantum MDP models (referred to as QOMDPs) found in the literature. Furthermore, approximations of q-MDPs are obtained via finite-action models, which can be formulated as QOMDPs. Finally, classes of open-loop and classical-state-preserving closed-loop policies for q-MDPs are introduced, along with structural results for these policies. In summary, we present a novel quantum MDP model aiming to introduce a new framework, algorithms, and future research avenues. We hope that our approach will pave the way for a new research direction in discrete-time quantum control.

quant-ph↗

arXiv2024

Decentralized Exchangeable Stochastic Dynamic Teams in Continuous-time, their Mean-Field Limits and Optimality of Symmetric Policies

We study a class of stochastic exchangeable teams comprising a finite number of decision makers (DMs) as well as their mean-field limits involving infinite numbers of DMs. In the finite population regime, we study exchangeable teams under the centralized information structure. For the infinite population setting, we study exchangeable teams under the decentralized mean-field information sharing. The paper makes the following main contributions: i) For finite population exchangeable teams, we establish the existence of a randomized optimal policy that is exchangeable (permutation invariant) and Markovian; ii) As our main result in the paper, we show that a sequence of exchangeable optimal policies for finite population settings converges to a conditionally symmetric (identical), independent, and decentralized randomized policy for the infinite population problem, which is globally optimal for the infinite population problem. This result establishes the existence of a symmetric, independent, decentralized optimal randomized policy for the infinite population problem. Additionally, this proves the optimality of the limiting measure-valued MDP for the representative DM; iii) Finally, we show that symmetric, independent, decentralized optimal randomized policies are approximately optimal for the corresponding finite-population team with a large number of DMs under the centralized information structure. Our paper thus establishes the relation between the controlled McKean-Vlasov dynamics and the optimal infinite population decentralized stochastic control problem (without an apriori restriction of symmetry in policies of individual agents), for the first time, to our knowledge.

math.OC↗