arXiv · 2611.09916
Semantic-Enhanced Sparse Temporal Modeling for Dynamic Engagement Estimation in Multimodal Interactions
Abstract
Dynamic engagement estimation aims to infer participants' evolving engagement states from continuous multimodal interaction streams. Existing approaches typically combine acoustic, pose, and visual representations with temporal models to capture observable interaction patterns. However, conversational semantics are difficult to align with frame-level engagement annotations in the same manner as continuous audio-visual signals, leaving dialogue context less effectively exploited in existing temporal modeling pipelines. Moreover, engagement annotations often evolve smoothly over time, making dense frame-level modeling computationally redundant and potentially sensitive to local annotation noise. To address these issues, we propose SEST, a Semantically-Enhanced Sparse Temporal framework that models engagement-related information from both semantic and temporal perspectives. SEST leverages Multimodal Large Language Models to extract multimodal semantic embeddings, providing dialogue-aware contextual information beyond conventional frame-level features. In parallel, SEST replaces dense-to-dense temporal modeling with a sparse-to-dense pathway, where continuous frame sequences are first encoded into compact temporal dynamics to reduce redundant computation and broaden temporal coverage, then fused with semantic context and recovered to the original label resolution for dense engagement prediction. Experiments on the MEE2026 benchmark demonstrate the effectiveness of SEST, which achieved first place in the challenge.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Yangchen Yu, Taoliang Wang, Jia Li, Nor Mazlina Ghazali, Mohd Nazim Mohtar, Bingkun Bao, Richang Hong. 2026-08-29. Semantic-Enhanced Sparse Temporal Modeling for Dynamic Engagement Estimation in Multimodal Interactions. https://arxiv.org/abs/2611.09916
Cite the original work for its findings. Save a collection to share your selection of sources.