Optimal Test-Data Piling in HDLSS Classification with Covariance Heterogeneity

Abstract

This work addresses a longstanding question in high-dimensional linear classification: Is perfect classification achievable in heterogeneous covariance structures? We focus on the phenomenon of data piling, where projected data points collapse onto discrete values. We provide a comprehensive characterization of two distinct types of data piling. The first type of data piling refers to the phenomenon where projecting the training data onto a certain direction yields exactly two distinct values-one for each class. This occurs universally when the data dimension $p$ exceeds the sample size $n$. The second type concerns independent test data and arises asymptotically as $p \to \infty$ with fixed $n$. While previous work established the existence of such double data piling under homogeneously spiked covariance structures using negatively ridged classifiers, our analysis extends to the more general and realistic case of heterogeneous covariance. We identify an optimal direction among all piling directions that maximizes the separation between test data piles, which is called the Second Maximal Data Piling direction. An algorithm based on data splitting is proposed to compute this direction using only training data. Our analysis reveals a key insight: the main obstacle to discovering this direction is the imbalance of the tail eigenvalues, rather than differences in spike count, spike magnitude, or the alignment of leading eigenspaces. Extensive simulations confirm our theoretical results and demonstrate the effectiveness of the proposed classifier across a wide range of high-dimensional scenarios.