Radio Echo Representation Learning for Semantic Sensing in Multi-Modal ISAC Systems

Abstract

Semantic sensing extends integrated sensing and communication (ISAC) beyond physical parameter estimation by exploiting task-oriented representations from radio echoes. Nevertheless, existing multimodal sensing remains largely limited to vision-assisted radio physical parameter estimation or typically processes the visual and radio modalities separately, leaving radio representations underutilized for visual task enhancement. Feature-level fusion enables closer cross-modal interaction, while echo interference complicates the learning of user-specific radio representations and its semantic consistency with the visual entities. To address these challenges, this paper proposes a multimodal semantic sensing framework for orthogonal frequency-division multiplexing (OFDM)-ISAC systems. We propose a radio encoder to learn radio representations oriented toward visual localization from superimposed pilot echoes, together with Variational Radio-Visual Grounding (VRVG) to promote semantic consistency between the learned radio representations and visual spatial features. The radio representations are then formulated as semantic probes to highlight the corresponding entities in the visual scene through cross-modal fusion. Experiments using real-world RCooper dataset show that the resulting synergistic radio-visual embeddings enable accurate radio-visual user matching and significantly improve both user-specific visual localization and physical parameter estimation performance. The results substantiate the benefits of radio representation learning and semantically consistent fusion for multimodal semantic sensing.