Moment Propagation

Abstract

Mean-field variational Bayes is a fast and scalable approach to approximate Bayesian inference, but its independence assumptions often lead to underestimated posterior uncertainty. We introduce moment propagation (MP), a framework for improving marginal posterior approximations by propagating conditional posterior moment information between parameter blocks and matching these moments within convenient approximating families. In conjugate settings, MP identifies variance terms omitted by mean-field approximations and uses them to construct corrected marginal updates. We develop both mean-field and Gaussian variants of MP, the latter using conditional Gaussian structure to obtain low-dimensional local updates for non-conjugate models. We establish consistency and asymptotic variance correctness for two-component models, and show that MP recovers exact marginal posteriors for linear regression and multivariate normal models. We also derive algorithms for multivariate normal models with missing data, probit regression, generalized linear models, and Bayesian Lasso regression. Numerical experiments show that MP, particularly Gaussian MP, can substantially improve marginal posterior accuracy over standard variational approximations while remaining much faster than MCMC.