Residual-Calibrated Multimodal Spatiotemporal Learning for Future Beam Prediction

Abstract

We propose a residual calibrated multimodal spatiotemporal framework for future beam prediction on DeepSense 6G Scenario 23. The framework integrates visual observations, location features, and historical beam labels to predict the beam label of the next dataset record. A pretrained Swin-Tiny encoder extracts features from four consecutive RGB frames. Bidirectional cross-attention integrates visual and two dimensional Unit 1 location representations, while a temporal Transformer incorporates historical beam label embeddings to model dependencies across the observation window. A supplementary offline residual head adjusts the frozen supervised logits using immediate correctness rewards and an auxiliary classification loss. A fixed probability mixture of the supervised and residual outputs is selected using validation accuracy. From 11,387 aligned records, we construct 11,255 temporal windows and partition them by capture run into 7,854 training, 1,905 validation, and 1,496 test samples. The supervised predictor achieves Top-1, Top-2, Top-3, and Top-5 test accuracies of 62.83%, 74.80%, 79.75%, and 81.08%, respectively. The residual predictor achieves corresponding accuracies of 61.76%, 75.07%, 79.41%, and 81.02%. Residual adjustment increases Top-2 accuracy by 0.27 percentage points, while the supervised predictor retains higher Top-1, Top-3, and Top-5 accuracies. The validation-selected mixture coincides with the residual predictor. The proposed framework combines cross-modal sensing features with explicit beam-history conditioning and a separately trained residual adjustment stage for one-record-ahead beam prediction. Initial evaluation on held-out capture runs characterizes the candidate-ranking performance of both prediction stages and favors the supervised variant for Top-1 selection. These findings motivate further study of historical conditioning and residual refinement.