Rethinking BPE: Merge Selection as Model Selection

Abstract

Subword tokenization methods such as Byte-Pair Encoding (BPE) are a core component of modern language models; yet, tokenizer capacity remains typically determined by a fixed merge budget, chosen heuristically. Since larger vocabularies increase memory usage and computational cost, selecting compact and effective token inventories is essential for efficient language modeling. We study the optimization of a BPE merge budget as a statistical model selection problem and investigate information-theoretic criteria for \emph{computing optimal merge sets} directly from corpus statistics. To further improve the merge selection in limited data conditions, we augment the BPE training corpora with synthetic texts generated from the original data, providing estimates for the unigram and bigram distributions. Experiments across multiple corpora and downstream tasks show that using a principled vocabulary selection technique reduces model size, with little degradation in performance. We further observe that synthetic data can yield better tokenizers, especially in low-resource settings.