arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2610.03276

Moving Forward with Video Saliency: A New Dataset and Benchmark where Motion Matters

Susmit Agrawal·Rebecca Wanner·Juliane Verwiebe·Matthias Tangemann·Matthias Bethge·Matthias Kümmerer
View original source ↗Read source document ↗Export metadata ↓

Abstract

Video saliency prediction is inherently harder to model than static image saliency due to the additional temporal dimension. Video saliency benchmarks rest on the premise that predicting gaze on video requires utilizing temporal activity distributed across frames. Prior work has challenged this, showing that static baselines recover a significant fraction of the explainable gaze information on LEDOV, a popular video saliency dataset, and that video saliency models fail in the same places as this static baseline. We verify that this diagnosis still stands: under a more capable gold standard than the original analysis, and an updated panel of recent architectures, the strongest temporal architecture in the panel still does not substantially improve over a fine-tuned static baseline. However, it remains unclear whether the marginal gain reflects limitations of current temporal architectures or a lack of temporal patterns in the benchmark itself. We introduce SalTempto, a video saliency benchmark with greater dynamism: 224 clips of highly dynamic content, sourced from the HACS-Segments dataset so that each clip contains an event together with its lead-up and aftermath, with gaze recordings from up to 16 subjects and a training split for adapting pretrained models. On SalTempto, the static baseline recovers only about 13\% of the headroom above the centerbias, against more than half on LEDOV. A fine-tuned temporal architecture shows a substantial gain in performance over the static baseline, indicating that it does capture meaningfully more temporal information, which LEDOV fails to measure. Yet, even this SoTA model still leaves nearly half of SalTempto's headroom unexplained, indicating room for improvement in video saliency modelling. Examination of SalTempto also lets us describe human tendencies that models miss. SalTempto link: https://huggingface.co/datasets/bethgelab/video_saliency.

Explore related subjects

cs.CV

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Susmit Agrawal, Rebecca Wanner, Juliane Verwiebe, Matthias Tangemann, Matthias Bethge, Matthias Kümmerer. 2026-10-02. Moving Forward with Video Saliency: A New Dataset and Benchmark where Motion Matters. https://arxiv.org/abs/2610.03276

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

VIDiff: Translating Videos via Multi-Modal Instructions with Diffusion Models

Diffusion models have achieved significant success in image and video generation. This motivates a growing interest in video editing tasks, where videos are edited according to provided text descriptions. However, most existing approaches only focus on video editing for short clips and rely on time-consuming tuning or inference. We are the first to propose Video Instruction Diffusion (VIDiff), a unified foundation model designed for a wide range of video tasks. These tasks encompass both understanding tasks (such as language-guided video object segmentation) and generative tasks (video editing and enhancement). Our model can edit and translate the desired results within seconds based on user instructions. Moreover, we design an iterative auto-regressive method to ensure consistency in editing and enhancing long videos. We provide convincing generative results for diverse input videos and written instructions, both qualitatively and quantitatively. More examples can be found at our website https://ChenHsing.github.io/VIDiff.

cs.CV↗
arXiv2026

Textualized and Feature-based Models for Compound Multimodal Emotion Recognition in the Wild

Systems for multimodal emotion recognition (ER) are commonly trained to extract features from different modalities (e.g., visual, audio, and textual) that are combined to predict individual basic emotions. However, compound emotions often occur in real-world scenarios, and the uncertainty of recognizing such complex emotions over diverse modalities is challenging for feature-based models. As an alternative, emerging large language models (LLMs) like BERT and LLaMA can rely on explicit non-verbal cues that may be translated from different non-textual modalities (e.g., audio and visual) into text. Textualization of modalities augments data with emotional cues to help the LLM encode the interconnections between all modalities in a shared text space. In such text-based models, prior knowledge of ER tasks is leveraged to textualize relevant non-verbal cues such as audio tone from vocal expressions, and action unit intensity from facial expressions. Since the pre-trained weights are publicly available for many LLMs, training on large-scale datasets is unnecessary, allowing to fine-tune for downstream tasks such as compound ER (CER). This paper compares the potential of text- and feature-based approaches for compound multimodal ER in videos. Experiments were conducted on the challenging C-EXPR-DB dataset in the wild for CER, and contrasted with results on the MELD dataset for basic ER. Our results indicate that multimodal textualization provides lower accuracy than feature-based models on C-EXPR-DB, where text transcripts are captured in the wild. However, higher accuracy can be achieved when the video data has rich transcripts. Our code is available.

cs.CV↗
arXiv2026

A Sobel-Gradient MLP Baseline for Handwritten Character Recognition

This study examines how much handwritten-character information is retained by a deliberately simple first-order edge representation. Instead of learning spatial filters, each input image is transformed by the fixed Sobel-Feldman operator into signed horizontal and vertical derivative maps, which are independently normalized, flattened, and classified by a multilayer perceptron (MLP). The resulting model therefore separates fixed edge extraction from learned classification and provides a controlled baseline for evaluating the sufficiency of first-order image gradients. In the executed experiments, the Sobel-gradient MLP achieves 98.54 percent test accuracy on MNIST and 92.50 percent on the TensorFlow Datasets (TFDS) EMNIST Letters configuration. Macro F1 scores are 0.9853 and 0.9265, respectively. One-vs-rest ROC analysis further yields micro/macro AUC values of 0.9998/0.9998 on MNIST and 0.9987/0.9982 on EMNIST Letters. Confusion-matrix analysis shows that the remaining errors are concentrated among geometrically similar classes, especially 3/8 and 4/9 for MNIST and I/L and G/Q for EMNIST Letters. These results show that fixed first-order gradients preserve substantial class-discriminative structure, while also revealing the specific ambiguities that remain when recognition is driven by edge geometry alone.

cs.CV↗