arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2607.01813

MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models

Yuanzhi Liu·Shousheng Zhao·Bo Zhou·Kongming Liang·Zhanyu Ma
View original source ↗Read source document ↗Export metadata ↓

Abstract

Evaluation benchmarks are essential for assessing vision-language models (VLMs), but most multimodal benchmarks are static, making them vulnerable to temporal staleness, data contamination, and costly maintenance. We present MMBench-Live, a continuously evolving multimodal benchmark built by a multi-agent-driven automated pipeline. Our framework treats benchmark evolution as task-guided dataset construction, integrating structured benchmark specification, feedback-controlled real-time data acquisition, and verifiable QA generation with executable reasoning. To maintain cross-version comparability, we introduce a distribution-consistent update strategy that extracts task-related visual patterns from the original benchmark to guide data collection and filtering. Instantiated from MMBench, MMBench-Live contains 5.9K newly generated evaluation instances with a high answer correctness rate, while each update costs about USD 30 and takes 1-2 hours. Extensive evaluations show that MMBench-Live preserves stable model rankings, maintains semantic alignment with the original benchmark, and exhibits weaker contamination-related memorization signals, suggesting a practical and scalable paradigm for sustainable multimodal benchmark evolution. The project is available at https://github.com/PRIS-CV/MMBench-Live.

Explore related subjects

cs.CVcs.AI

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Yuanzhi Liu, Shousheng Zhao, Bo Zhou, Kongming Liang, Zhanyu Ma. 2026-07-02. MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models. https://arxiv.org/abs/2607.01813

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Category Level 6D Object Pose Estimation from a Single RGB Image using Diffusion

Estimating the 6D pose and 3D size of an object from visual data is a fundamental task in computer vision. Although single-view geometry is a deeply established domain, contemporary category-level methods frequently rely on rigid prerequisites such as precise object models, ground truth depth, or multi-modal LiDAR integration to achieve robust results. In this work, we introduce a unified generative framework that addresses both single-view category-level pose estimation and temporal sequence tracking using only RGB input. Our method leverages score-based diffusion models to generate a rich multi-hypothesis pose distribution, inherently capturing spatial and geometric uncertainties. While existing diffusion-based estimators typically rely on computationally expensive likelihood models to prune outliers, we propose an efficient alternative utilising Mean Shift to directly isolate the distribution's mode as the final pose estimate. Our approach establishes a new state-of-the-art baseline on the challenging REAL275 benchmark among two-stage, crop-based estimators. Furthermore, by decoupling object detection from pose estimation, our generative framework explicitly avoids the catastrophic domain overfitting inherent to end-to-end single-stage detectors, achieving highly robust zero-shot generalisation on the unseen Wild6D dataset. Finally, we demonstrate that the iterative nature of our score-based sampler enables a seamless transition to video sequences by preserving and propagating the multi-hypothesis distribution across time as a coherent temporal prior.

cs.CV↗
arXiv2026

Interpretable Retinal Disease Prediction Using Biology-Informed Heterogeneous Graph Representations

Interpretability is crucial for utilizing machine learning models as clinical decision support tools for medical diagnostics. However, most state-of-the-art image classifiers based on neural networks are not interpretable. As a result, clinicians often resort to known biomarkers to guide diagnosis, although biomarker-based classification often suffers from drastic information loss compared to raw medical images. This work proposes a method that preserves the rich imaging information while simultaneously enhancing the interpretability of predictions for diabetic retinopathy staging from optical coherence tomography angiography (OCTA) images. The core contribution of our method is a novel biology-informed heterogeneous graph representation that models retinal vessel segments, intercapillary areas, and the foveal avascular zone (FAZ) in a human-interpretable way. This graph representation allows us to frame diabetic retinopathy staging as a graph-level classification task, which we solve using an established, efficient graph neural network architecture. We compare our method against established methods, including classical biomarker-based classifiers, convolutional neural networks (CNNs), and vision transformers in predicting the clinically assigned DR stage based on color fundus photography images. We find stage agreement rates of our method and alternative vision model based classifiers saturating at AUC-ROC values of 84%. Crucially, we use our biology-informed graph to provide explanations of great detail. Our approach surpasses existing methods in precisely localizing and identifying abnormal vessels and non-perfusion areas. Our approach sets the stage for the interpretable identification of patients who require special attention due to their traceable microvascular changes, only observable using the details of OCTA images.

cs.CV↗
arXiv2026

A Unified Hierarchical Framework for Fine-grained Cross-view Geo-localization over Large-scale Scenarios

Cross-view geo-localization is a promising solution for large-scale localization problems, requiring the sequential execution of retrieval and metric localization tasks to achieve fine?grained predictions. However, existing methods typically focus on designing standalone models for these two tasks, resulting in inefficient collaboration and increased training overhead. In this paper, we propose UnifyGeo, a novel unified hierarchical geo-localization framework that integrates retrieval and metric localization tasks into a single network. Specifically, we first em?ploy a unified learning strategy to jointly learn multi-granularity representations, establishing task associations between retrieval and metric localization. Subsequently, we design a re-ranking mechanism guided by a dedicated loss function, which enhances geo-localization performance by improving both retrieval accuracy and metric localization references. Extensive experiments demonstrate that UnifyGeo significantly outperforms state-of-the?art methods in both task-isolated and task-associated settings. On the challenging VIGOR benchmark, UnifyGeo achieves 39.64% and 25.58% 1-meter-level localization recall under same-area and cross-area evaluations, respectively, demonstrating strong fine?grained localization capability in large-scale scenarios. Code will be available at https://github.com/chord-sz/UnifyGeo.

cs.CV↗