Shortcut in Generalization: Rethinking Evaluation Protocol for Encrypted Traffic Classification

Abstract

Encrypted traffic classification is commonly evaluated under a range of experimental protocols, with classification performance reported as the primary measure. However, the reported performance may be influenced by shortcuts left uncontrolled in the evaluation protocol, such as shortcut features and cross-partition sharing, rather than reflect generalizable traffic patterns. And test-only feature transformations may further introduce train-test mismatch. In this paper, we study how current evaluation protocols are affected by shortcut-related factors, including field-level shortcuts, cross-partition sharing, and train-test processing. We compare feature importance, retrained feature removal, and field-only split sensitivity. We also vary Source-IP overlap and packets per flow, compare test-only IP transformations with matched train-test transformations. Experiments on three public datasets show that the performance of existing methods are affected by both cross-partition sharing and the consistency of train-test transformations. Specifically, increasing Source-IP overlap from 0% to 100% raises accuracy by about 32-33 points before identifier masking, compared with only about 8 points after masking. For two state-of-the-art pretrained models (i.e., YaTC and NetMamba), the 11-13 points loss caused by test-only IP zeroing is almost fully recovered when the same transformation is applied during both training and testing.