arXiv · 2611.15971
Better Boundary Prediction Does Not Guarantee Better Structural Representation
Abstract
Punctuation marks often mark phrase boundaries, carrying linguistic information. Therefore, punctuation marks' abundance and feasibility as proxies of linguistic structure has led to suggestions that using punctuation marks to optimise on phrasal structure may help language models (LMs) improve their representations of argument, semantic, and syntactic structures. In this paper, we test the above claim in English across three primary LM architectures: decoder-only, encoder-only, and encoder-decoder LMs, by complementing traditional token prediction with punctuation restoration (PR). Our evidence, collected at the <1B parameter scale, suggests that optimising for boundary prediction does not necessarily guarantee higher-quality structural representation in decoder-only, encoder-decoder, and encoder-only LMs, failing to improve performance on downstream tasks that require understanding structure in linguistic input.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Michael Zhou, Markus Frohmann, Junghyun Min, En-Shiun Annie Lee. 2026-11-11. Better Boundary Prediction Does Not Guarantee Better Structural Representation. https://arxiv.science/abs/2611.15971
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite arXiv Science by Indiciae: BibTeX · RIS.