arXiv subjects

Renyuan Xu

Publications and source records attributed to Renyuan Xu.

46 records · Page 3Linked to original sources

arXiv2021

Entropy Regularization for Mean Field Games with Learning

Entropy regularization has been extensively adopted to improve the efficiency, the stability, and the convergence of algorithms in reinforcement learning. This paper analyzes both quantitatively and qualitatively the impact of entropy regularization for Mean Field Game (MFG) with learning in a finite time horizon. Our study provides a theoretical justification that entropy regularization yields time-dependent policies and, furthermore, helps stabilizing and accelerating convergence to the game equilibrium. In addition, this study leads to a policy-gradient algorithm for exploration in MFG. Under this algorithm, agents are able to learn the optimal exploration scheduling, with stable and fast convergence to the game equilibrium.

math.OC↗

arXiv2021

A class of stochastic games and moving free boundary problems

In this paper we propose and analyze a class of $N$-player stochastic games that include finite fuel stochastic games as a special case. We first derive sufficient conditions for the Nash equilibrium (NE) in the form of a verification theorem. The associated Quasi-Variational-Inequalities include an essential game component regarding the interactions among players, which may be interpreted as the analytical representation of the conditional optimality for NEs. The derivation of NEs involves solving first a multi-dimensional free boundary problem and then a Skorokhod problem. We call it a "moving free boundary" to highlight the difference between standard control problems and stochastic games. Finally, we present an intriguing connection between these NE strategies and controlled rank-dependent stochastic differential equations.

q-fin.MF↗

arXiv2021

Learning Mean-Field Games

This paper presents a general mean-field game (GMFG) framework for simultaneous learning and decision-making in stochastic games with a large population. It first establishes the existence of a unique Nash Equilibrium to this GMFG, and explains that naively combining Q-learning with the fixed-point approach in classical MFGs yields unstable algorithms. It then proposes a Q-learning algorithm with Boltzmann policy (GMF-Q), with analysis of convergence property and computational complexity. The experiments on repeated Ad auction problems demonstrate that this GMF-Q algorithm is efficient and robust in terms of convergence and learning accuracy. Moreover, its performance is superior in convergence, stability, and learning ability, when compared with existing algorithms for multi-agent reinforcement learning.

math.OC↗

arXiv2021

Mean-Field Controls with Q-learning for Cooperative MARL: Convergence and Complexity Analysis

Multi-agent reinforcement learning (MARL), despite its popularity and empirical success, suffers from the curse of dimensionality. This paper builds the mathematical framework to approximate cooperative MARL by a mean-field control (MFC) approach, and shows that the approximation error is of $\mathcal{O}(\frac{1}{\sqrt{N}})$. By establishing an appropriate form of the dynamic programming principle for both the value function and the Q function, it proposes a model-free kernel-based Q-learning algorithm (MFC-K-Q), which is shown to have a linear convergence rate for the MFC problem, the first of its kind in the MARL literature. It further establishes that the convergence rate and the sample complexity of MFC-K-Q are independent of the number of agents $N$, which provides an $\mathcal{O}(\frac{1}{\sqrt{N}})$ approximation to the MARL problem with $N$ agents in the learning environment. Empirical studies for the network traffic congestion problem demonstrate that MFC-K-Q outperforms existing MARL algorithms when $N$ is large, for instance when $N>50$.

cs.LG↗

arXiv2021

Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizon

We explore reinforcement learning methods for finding the optimal policy in the linear quadratic regulator (LQR) problem. In particular, we consider the convergence of policy gradient methods in the setting of known and unknown parameters. We are able to produce a global linear convergence guarantee for this approach in the setting of finite time horizon and stochastic state dynamics under weak assumptions. The convergence of a projected policy gradient method is also established in order to handle problems with constraints. We illustrate the performance of the algorithm with two examples. The first example is the optimal liquidation of a holding in an asset. We show results for the case where we assume a model for the underlying dynamics and where we apply the method to the data directly. The empirical evidence suggests that the policy gradient method can learn the global optimal solution for a larger class of stochastic systems containing the LQR framework and that it is more robust with respect to model mis-specification when compared to a model-based approach. The second example is an LQR system in a higher dimensional setting with synthetic data.

cs.LG↗

arXiv2021

Interbank lending with benchmark rates: Pareto optima for a class of singular control games

We analyze a class of stochastic differential games of singular control, motivated by the study of a dynamic model of interbank lending with benchmark rates. We describe Pareto optima for this game and show how they may be achieved through the intervention of a regulator, whose policy is a solution to a singular stochastic control problem. Pareto optima are characterized in terms of the solutions to a new class of Skorokhod problems with piecewise-continuous free boundary. Pareto optimal policies are shown to correspond to the enforcement of endogenous bounds on interbank lending rates. Analytical comparison between Pareto optima and Nash equilibria provides insight into the impact of regulatory intervention on the stability of interbank rates.

math.OC↗

arXiv2021

Scaling Properties of Deep Residual Networks

Residual networks (ResNets) have displayed impressive results in pattern recognition and, recently, have garnered considerable theoretical interest due to a perceived link with neural ordinary differential equations (neural ODEs). This link relies on the convergence of network weights to a smooth function as the number of layers increases. We investigate the properties of weights trained by stochastic gradient descent and their scaling with network depth through detailed numerical experiments. We observe the existence of scaling regimes markedly different from those assumed in neural ODE literature. Depending on certain features of the network architecture, such as the smoothness of the activation function, one may obtain an alternative ODE limit, a stochastic differential equation or neither of these. These findings cast doubts on the validity of the neural ODE model as an adequate asymptotic description of deep ResNets and point to an alternative class of differential equations as a better description of the deep network limit.

cs.LG↗

arXiv2020

Delay-Adaptive Learning in Generalized Linear Contextual Bandits

In this paper, we consider online learning in generalized linear contextual bandits where rewards are not immediately observed. Instead, rewards are available to the decision-maker only after some delay, which is unknown and stochastic. We study the performance of two well-known algorithms adapted to this delayed setting: one based on upper confidence bounds, and the other based on Thompson sampling. We describe modifications on how these two algorithms should be adapted to handle delays and give regret characterizations for both algorithms. Our results contribute to the broad landscape of contextual bandits literature by establishing that both algorithms can be made to be robust to delays, thereby helping clarify and reaffirm the empirical success of these two algorithms, which are widely deployed in modern recommendation engines.

cs.LG↗

arXiv2019

Stochastic Games for Fuel Followers Problem: N vs MFG

In this paper we formulate and analyze an $N$-player stochastic game of the classical fuel follower problem and its Mean Field Game (MFG) counterpart. For the $N$-player game, we obtain the Nash Equilibrium (NE) explicitly by deriving and analyzing a system of Hamilton--Jacobi--Bellman (HJB) equations, and by establishing the existence of a unique strong solution to the associated Skorokhod problem on an unbounded polyhedron with an oblique reflection. For the MFG, we derive a bang-bang type NE under some mild technical conditions and by the viscosity solution approach. We also show that this solution is an $ε$-NE to the $N$-player game, with $ε=O(\frac{1}{\sqrt{N}})$. The $N$-player game and the MFG differ in that the NE for the former is state dependent while the NE for the latter is threshold-type bang-bang policy where the threshold is state independent. Our analysis shows that the NE for a stationary MFG may not be the NE for the corresponding MFG.

math.OC↗

arXiv2018

Consistency and Computation of Regularized MLEs for Multivariate Hawkes Processes

This paper proves the consistency property for the regularized maximum likelihood estimators (MLEs) of multivariate Hawkes processes (MHPs). It also develops an alternating minimization type algorithm (AA-iPALM) to compute the MLEs with guaranteed global convergence to the set of stationary points. The performance of this AA-iPALM algorithm on both synthetic and real-world data shows that AA-iPALM consistently improves over iPALM and PALM. Moreover, AA-iPALM is able to identify the causality structure in rental listings on craigslist and herd behavior in the NASDAQ ITCH dataset.

math.PR↗