Continuous-Time q-Learning with Nonsmooth Value Functions: a BSDE Approach

Abstract

We study continuous-time q-learning for entropy-regularized diffusion control without assuming classical differentiability of the value function. Rewards and admissible policies need only be Borel measurable, which allows discontinuous rewards. The diffusion coefficient $σ$ is action independent and locally uniformly elliptic, and the drift is $b_0 + σψ$, with $b_0$ action independent and $ψ$ bounded. In this setting the q-function involves the value only through its gradient. For each admissible policy, we represent its value by a backward stochastic differential equation (BSDE) and prove that this value is continuous before the horizon, although the data are only measurable. Replacing the gradient with a measurable representation of the martingale integrand, we define the q-function up to Lebesgue-null sets and recover the classical definition when the value function is smooth. We then prove policy improvement, when the Gibbs update is admissible, and martingale characterizations of the value and the q-function from on-policy and off-policy observations. Here the candidate value need only be continuous before the horizon, and actions are sampled on a deterministic finite grid and held constant between sampling times. Under an integrability condition on the reward, we show that a second BSDE gives an upper bound for the optimal value, attained when its Gibbs policy is admissible, and we characterize its solution in the same way.