Systematically Optimized CNN-Transformer with Focal Loss for Imbalanced Intrusion Detection on NSL-KDD
Intrusion Detection Systems (IDS) struggle with imbalanced datasets like NSL-KDD, especially in detecting rare R2L and U2R attacks. This work describes a systematically optimized and explainable framework using a CNN-Transformer architecture to improve performance on highly imbalanced data. We decided to use XGBoost for feature selection and Focal Loss as the main mechanism for minority class learning. We used Optuna for end-to-end hyrate=0.00042, batch size=256, and Focal Loss γ), using a thoughtful data splitting process to prevent data leakage. Critically, our ablation studies pointed out that while Focal Loss (optimally at γ = 1.5) substantially enhanced minority recall, adding oversampling methods like SMOTE caused precision degradation via over-correction. Our final model, using only tuned Focal Loss, achieved 98.73% overall accuracy on the NSL-KDD test data, with significantly balanced F1-scores of 84.63% (R2L) and 69.66% (U2R) over baseline approaches. Additionally, we performed SHAP analysis to understand model predictions, understand salient features (e.g., service http, logged in), and explain the persistent U2R precision challenge due to feature overlap. This study presents a comprehensive pipeline for imbalanced IDS, providing methodological insights, and demonstrates that tuned Focal Loss can be a sufficient and effective strategy for class balancing.