Replica Analysis of Overfitting in Vector Generalized Linear Models

Abstract

We show how the successful recent statistical physics based theory of overfitting in Generalized Linear Models (GLMs) can be generalized to a much larger family of regression models: the so-called Vector Generalized Linear Models (VGLMs), in which not a single linear combination of covariates drives the outcome, but where one has an arbitrary finite number of such linear combinations. The latter more general and complex models can describe more sophisticated statistical patterns in data, but are even more prone to overfitting than GLMs due to their significantly larger number of parameters; this often limits their use. We show that the generalization of the overfitting theory from GLMs to VGLMs causes many mathematical challenges that were absent from the earlier work, mostly related to the increased dimensional complexity of all order parameters in the theory, but that the new problem is once more solvable using the replica method. This opens up the field to many novel and welcome applications, especially in post-genome medical inference and data analysis, where overfitting is and remains a major barrier to progress. After developing the general VGLM overfitting theory, within the replica symmetric (RS) ansatz, we illustrate its merits and its correctness via application to several relevant VGLM regression models.