Predictive Performance Test based on the Exhaustive Nested Cross-Validation for High-dimensional data

Abstract

It is crucial to assess the predictive performance of a model to establish its practicality and relevance in real-world scenarios, particularly for high-dimensional data analysis. Among data splitting or resampling methods, cross-validation (CV) is extensively used for several tasks such as estimating the prediction error, tuning the regularization parameter, and selecting the most suitable predictive model among competing alternatives. The $K$-fold cross-validation is a popular CV method but its limitation is that the risk estimates are highly dependent on the partitioning of the data (for training and testing). Here, the issues regarding the reproducibility of the $K$-fold CV estimator are demonstrated in hypothesis testing wherein different partitions lead to notably disparate conclusions. This study presents a novel predictive performance test and valid confidence intervals based on exhaustive nested cross-validation for determining the difference in prediction error between two model-fitting algorithms. A naive implementation of the exhaustive nested cross-validation is computationally costly. Here, we address concerns regarding computational complexity by devising a computationally tractable closed-form expression for the proposed cross-validation estimator. Our study also investigates strategies aimed at enhancing statistical power within high-dimensional scenarios while controlling the Type I error rate. \tr{Through comprehensive numerical experiments, we demonstrate that Ridge-based methods using bias to measure uncertainty of CV estimates and adaptive hyperparameter selection provide the most reliable approach for high-dimensional predictive performance testing.} To illustrate the practical utility of our method, we apply it to an RNA sequencing study and demonstrate its effectiveness in the context of biological data analysis.