On Exact Dual and Mixed-Integer Reformulations for Finite Horizon Markov Decision Processes with Submodular Revenue Maximization

Abstract

We study exact mixed-integer reformulations of a finite horizon Markov decision process (MDP) with transition revenues defined by local submodular revenue maximization problems. We start with the dual linear programming formulation of the MDP, where local optimal revenues appear in the objective function and the constraints describe the state probabilities from the initial state. We combine the dual formulation with the binary variables of the local problems and obtain an exact mixed-integer nonlinear formulation with bilinear terms. We use the submodular inequalities and an exact McCormick linearization to obtain an equivalent mixed-integer linear formulation. We also use the dual structure to conduct a sensitivity analysis of the MDP optimal value with respect to changes in the local revenue coefficients. Based on local incumbent revenues and upper bounds, this sensitivity analysis evaluates the effect of local optimality gaps on the global MDP optimality gap, without solving every local revenue maximization problem to optimality.