Using Markov Boundary Approach for Interpretable and Generalizable Feature Selection

Abstract

The perceived advantage of machine learning (ML) models is that they are flexible and can incorporate a large number of features. However, many of these are typically correlated or dependent, and incorporating all of them can hinder model stability and generalizability. In fact, it is desirable to do some form of feature screening and incorporate only the relevant features. The best approaches should involve subject-matter knowledge and information on causal relationships. This paper deals with an approach called Markov boundary (MB) that is related to causal discovery, using directed acyclic graphs to represent potential relationships and using statistical tests to determine the connections. An MB is the minimum set of features that guarantee that other potential predictors do not affect the target given the boundary while ensuring maximal predictive accuracy. Identifying the Markov boundary is straightforward under assumptions of Gaussianity on the features and linear relationships between them. But these assumptions are not satisfied in practice. This paper outlines common problems associated with identifying the Markov boundary in structured data when relationships are non-linear and the predictors are of mixed data type. We propose a multi-group forward-backward selection strategy that addresses these challenges and demonstrate its capabilities on simulated and real datasets.