Likelihood Ratio Gradient Estimation for Steady-State Parameters

Abstract

We consider a discrete-time Markov chain $\boldsymbolΦ$ on a general state-space ${\sf X}$, whose transition probabilities are parameterized by a real-valued vector $\boldsymbolθ$. Under the assumption that $\boldsymbolΦ$ is geometrically ergodic with corresponding stationary distribution $π(\boldsymbolθ)$, we are interested in estimating the gradient $\nabla α(\boldsymbolθ)$ of the steady-state expectation $$α(\boldsymbolθ) = π( \boldsymbolθ) f.$$ To this end, we first give sufficient conditions for the differentiability of $α(\boldsymbolθ)$ and for the calculation of its gradient via a sequence of finite horizon expectations. We then propose two different likelihood ratio estimators and analyze their limiting behavior.