arXiv · 2611.15069
DiFiT: Distribution-Fidelity-guided Visual Token Pruning for Multimodal Large Language Models
Abstract
Visual token pruning reduces the inference cost of multimodal large language models, but answer accuracy only partly reflects its effect. Pruning can change the probabilities of alternative response tokens even when greedy answers remain correct and unchanged. We compare full and pruned response distributions under shared reference prefixes. Average predictive drift grows with compression on fixed correctly answered samples. Separate analyses associate larger drift with more confidence-decision disagreements and sampling failures. We propose DiFiT (Distribution-Fidelity-guided Token pruning) to learn discrete selection from reference answers and full-token response distributions. Dual-View Distillation matches probabilities across the vocabulary, while answer supervision supports task quality. A learned scorer and differentiable Top-$K$ learn which visual evidence to retain. A multi-budget mask objective supports different retention ratios through one shared ranking over a frozen model. On Qwen2.5-VL-7B, DiFiT reduces sampling-induced negative flips by 30.3% relative to FastV at 10% retention on four-task greedy-correct controls. It preserves 98.74% of baseline performance on MME at 30% retention.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Jingyuan Zhang, Nihao Niu, Xinyang Lv, Beibei Han, Xiaoyun Wang, Yanning Hou, Ke Xu. 2026-11-11. DiFiT: Distribution-Fidelity-guided Visual Token Pruning for Multimodal Large Language Models. https://arxiv.science/abs/2611.15069
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite arXiv Science by Indiciae: BibTeX · RIS.