Quantifying uncertainty and stability among highly correlated predictors: a subspace perspective

Abstract

We study the problem of linear feature selection when features are highly correlated. Such settings pose two fundamental challenges. First, how should model similarity be defined? Simply counting features in common can be misleading: two models may share no features, yet highly correlated features can make the two models very similar in terms of predictive ability. Second, how can feature stability be assessed across runs of a variable selection method? High correlation can yield very different feature sets, so counting how often a feature is selected may label most features as unstable, and selecting stable features would result in models that are too small with poor predictive performance. In essence, these issues arise because existing notions of similarity and stability are "discrete" in nature. To overcome these challenges, we propose a novel framework based on feature subspaces -- the subspaces spanned by selected columns of the feature matrix. This new perspective leads to "continuous" measures of similarity and stability, as well as false positive error, all of which are defined in terms of "closeness" of feature subspaces. Our measures naturally account for feature correlation and reduce to existing discrete notions when features are uncorrelated. To obtain stable models, we propose and theoretically analyze a subspace-based generalization of stability selection (Meinshausen & Bühlmann 2010, Taeb et al. 2020), which combines a discrete model search with a continuous subspace-based assessment of stability. On synthetic and real gene expression data, our method improves on existing stability-based approaches by (i) producing multiple stable models that capture feature interchangeability, and (ii) generating larger models with better predictive performance. Our method is implemented in the R package substab.