Victor Morand | arXiv Science

arXiv2026

MICE: Minimal Interaction Cross-Encoders for efficient Re-ranking

In Information Retrieval (IR), cross-encoders deliver state-of-the-art ranking effectiveness but have a high inference cost, limiting their use to second-stage re-rankers. Prior work has addressed this bottleneck from two largely separate directions: accelerating cross-encoder inference through attention sparsification, or improving first-stage retrieval effectiveness to alleviate the need of a re-ranker, using more complex models, e.g. late-interactions. In this work, we bridge these two directions through an in-depth analysis of cross-encoder internal mechanisms. By identifying and removing superfluous interactions, we derive MICE (Minimal Interaction Cross-Encoders), a new cross-encoder architecture that retains effectiveness while reducing computational overhead. Extensive evaluations show MICE retains most of the performances of its cross-encoder counterparts in-domain and matches or even exceeds it in out-of-domain, while reducing FLOPs down to 2.5 times.

cs.IR↗

arXiv2026

Building Better Encoder-only Cross-Encoders: A Controlled Study of Training Strategies for Neural Re-ranking

Cross-encoders fine-tuned from Transformer backbones remain the standard for second-stage re-ranking, and recent knowledge-distillation strategies have closed much of the gap with LLM re-rankers. However, these strategies have not been compared under controlled conditions. In particular, it remains unclear how distillation from LLM rankers compares to distillation from strong cross-encoder teachers, or to purely supervised objectives. It is also unclear how much newer backbones (RoBERTa, ELECTRA, DeBERTaV3, ModernBERT) contribute compared to the original BERT. We run 162 controlled training runs (9 backbones x 6 objectives x 3 seeds), spanning pointwise, pairwise, and listwise losses with both human labels and two distillation signals, and evaluate on TREC-DL, MSMARCO dev, BEIR, LoTTE, and Robust04. We find that objectives emphasizing relative comparisons - pairwise MarginMSE and listwise InfoNCE - consistently outperform alternative objectives, including more complex listwise LLM distillation, across all backbones, and switching objective yields gains comparable to moving up one backbone size tier. A controlled disentanglement further shows that, once the negative-sampling pool is matched, even a simple pairwise Hinge loss with ColBERTv2 hard negatives matches - and on out of domain beats - listwise LLM distillation, indicating that the quality of the negatives is at least as important as the choice of loss. We release all 162 trained models on HuggingFace (https://huggingface.co/collections/xpmir/reproducing-cross-encoders) and a unified training codebase. (https://github.com/xpmir/cross-encoders)

cs.IR↗