arXiv · 2611.11214
Disentangling Where and When:Factored Spatio-Temporal Explanations for Video Action Recognition
Abstract
Post-hoc explanation methods for video action recognition typically conflate spatial appearance and temporal dynamics into a single saliency or importance map. We propose a perturbation-based framework that disentangles these contributions into a spatial appearance mask ($M_a$) and a temporal motion mask ($M_m$). The two masks are jointly optimized against a frozen action recognition model through factored perturbation architecture, with nearest frame replacement to identify important temporal dynamics. A separation penalty discourages the appearance mask from spending budget on frames that the motion mask has already discarded. A ReLU-based area constraint penalizes explanations larger than an assigned budget. To evaluate these explanations, we introduce a separation protocol that sweeps spatial pixels and temporal frames independently, backed by proofs characterizing the proposed method. Across EPIC-Kitchens-55, EGTEA Gaze+, and Something-Something V2, our method attains the lowest deletion AUC against leading baselines while producing masks roughly five times sparser.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Bhavan Vasu, Giuseppe Raffa, Prasad Tadepalli. 2026-11-08. Disentangling Where and When:Factored Spatio-Temporal Explanations for Video Action Recognition. https://arxiv.science/abs/2611.11214
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite arXiv Science by Indiciae: BibTeX · RIS.