arXiv Science⌕ Search

arXiv · 2611.09680

SPARSEUP: Towards Better Recipes for Training Modern Learned Sparse Retrievers

Abstract

Learned sparse embeddings are a powerful alternative to dense retrieval, yet they have fallen behind single-vector and late interaction models. One reason for their limited adoption is that they are notoriously difficult to train, generally requiring very tight conditions to converge. We present a recipe for training Splade-like encoders on modern backbones and data that achieves a better efficiency-effectiveness trade-off. We discuss the role of logit distribution and its impact on the training stability and the final sparsity of the encoder. We also introduce a new per-position fan-out control - a hard cap on how many vocabulary dimensions each position may emit. We further apply a simple vocabulary merging strategy that removes redundant entries - such as case and word-boundary variants of the same word - which are common in cased BPE-based backbones. We apply our recipe to the LateOn backbone and training data, obtaining SPARSEUP, a sparse encoder that reaches 56.4 average nDCG@10 on BEIR-13, the highest we know of among public vocabulary-based sparse encoders below 150M parameters.

Explore related subjects

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Thibault Formal, Pierre Erbacher, Cosimo Rulli. 2026-11-06. SPARSEUP: Towards Better Recipes for Training Modern Learned Sparse Retrievers. https://arxiv.org/abs/2611.09680

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

E-GEO: A Testbed for Generative Engine Optimization in E-Commerce

With the rise of large language models (LLMs), generative engines have become powerful alternatives to traditional search, creating the need for generative engine optimization (GEO)---improving content visibility and relevance for generative engines. Despite its growing importance, current GEO practices are largely ad hoc, and their impacts remain understudied. In e-commerce, for instance, conversational shopping agents now guide consumers to relevant products, yet it is unclear if GEO can affect how products surface to users. We address this gap by introducing E-GEO, which contains 13,746 realistic, multi-sentence consumer product queries, each paired with 10 retrieved Amazon listings, capturing rich intent, constraints, preferences, and shopping contexts that existing datasets miss. Using this dataset, we conduct a large-scale empirical study of e-commerce GEO across five representative generative engines, seven LLM rewriters, and fifteen rewriting heuristics. We further formulate GEO as an optimization problem and develop a prompt meta-optimization algorithm that significantly improves over heuristic baselines. Notably, prompts independently optimized from different heuristics converge to a stable pattern, suggesting a shared, transferable GEO strategy. Finally, we red-team the GEO system through both static and optimization-based attacks and show that, under a simple in-prompt defense in our setting, gains from GEO reflect genuine content improvement, anchoring GEO as a substantive optimization problem.

cs.IR↗

MCGI: Manifold-Consistent Graph Indexing for Billion-Scale Disk-Resident Vector Search

Graph-based Approximate Nearest Neighbor (ANN) search often suffers from performance degradation in high-dimensional spaces due to the Euclidean-Geodesic mismatch, where greedy routing diverges from the underlying data manifold. To address this challenge, this paper presents Manifold-Consistent Graph Indexing (MCGI), a geometry-aware and disk-resident indexing method that leverages Local Intrinsic Dimensionality (LID) to allocate the search effort of a graph index according to the intrinsic geometry of data. Specifically, MCGI estimates the LID around each query from the distances that its own search has already computed and resizes the candidate list of the query once, so that queries in high-LID regions receive longer lists while queries in low-LID regions terminate earlier; the estimate itself costs no additional disk reads. Theoretical analysis shows that the success probability of any fixed search list decays exponentially with the LID of the query. At matched high recall, MCGI raises the query throughput of DiskANN on the same standard Vamana index by up to $1.43\times$ on three million-scale benchmarks and by up to $1.24\times$ on the billion-scale SIFT1B and Deep1B.

cs.IR↗

omni-macos: On-Device Omni-Modal Search on Apple Silicon

We present omni-macos, a search engine that embeds text, code, documents, images, audio and video into one representation space and runs on the Mac that already holds the files, so the files, the queries and the vectors stay on that machine. A background indexer and an interactive search box share the device. Each distinct chunk is embedded and stored once, and every file becomes searchable from a small first share of its content. Indexer work is split into smaller units while the user types, and one memory setting reaches every allocator found to draw on unified memory. Once the index is large, queries scan a one-bit replica of it, rescore a shortlist exactly and rank every modality in one list on a common score scale. The engine can also leave the GPU entirely, with the encoder on the Neural Engine and search on the CPU. We measure omni-macos on five Macs spanning an eightfold range of accelerator width and a thirty-twofold range of memory, and its ranking on a labeled multilingual and multimodal testbed. omni-macos is open source under Apache 2.0 at https://github.com/hanxiao/omni-macos.

cs.IR↗