Eric Janofsky | arXiv Science

arXiv2018

Learning high-dimensional graphical models with regularized quadratic scoring

Pairwise Markov Random Fields (MRFs) or undirected graphical models are parsimonious representations of joint probability distributions. Variables correspond to nodes of a graph, with edges between nodes corresponding to conditional dependencies. Unfortunately, likelihood-based learning and inference is hampered by the intractability of computing the normalizing constant. This paper considers an alternative scoring rule to the log-likelihood, which obviates the need to compute the normalizing constant of the distribution. We show that the rule is a quadratic function of the natural parameters. We optimize the sum of this scoring rule and a sparsity-inducing regularizer. For general continuous-valued exponential families, we provide theoretical results on parameter and edge consistency. As a special case we detail a new approach to sparse precision matrix estimation whose theoretical guarantees match that of the graphical lasso of Yuan and Lin (2007), with faster computational performance than the glasso algorithm of Yuan (2010). We then describe results for model selection in the nonparametric pairwise graphical model using exponential series. The regularized score matching problem is shown to be a convex program; we provide scalable algorithms based on consensus alternating direction method of multipliers (Boyd et al. (2011)) and coordinate-wise descent.

math.ST↗

arXiv2015

Exponential Series Approaches for Nonparametric Graphical Models

This thesis studies high-dimensional, continuous-valued pairwise Markov Random Fields. We are particularly interested in approximating pairwise densities whose logarithm belongs to a Sobolev space. For this problem we propose the method of exponential series [Crain, 1974; Barron and Sheu, 1991], which approximates the log density by a finite- dimensional exponential family with the number of sufficient statistics increasing with the sample size. We consider two approaches to estimating these models. The first is regularized maximum likelihood. This involves optimizing the sum of the log-likelihood of the data and a sparsity-inducing regularizer. We provide consistency and edge selection guarantees for this method. We then propose a variational approximation to the likelihood based on tree- reweighted, nonparametric message passing. We then consider estimation using regularized score matching. This approach uses an alternative scoring rule to the log-likelihood, which obviates the need to compute the normalizing constant of the distribution. For general continuous-valued exponential families, we provide parameter and edge consistency results. We then describe results for model selection in the nonparametric pairwise model using exponential series. The regularized score matching problem is shown to be a convex program; we provide scalable algorithms based on consensus Alternating Direction Method of Multipliers (ADMM, [Boyd et al., 2011]) and Coordinate-wise Descent. We compare our method to others in the literature as well as the aforementioned TRW estimator using simulated data.

math.ST↗