arXiv · 2611.15288
FineSpeaker: Reliable Speaker Representation Learning at Fine Temporal Resolution
Abstract
Speaker representation learning is fundamental to speech processing tasks such as speaker verification and diarization. Although long-context speaker encoders can produce stable and discriminative embeddings, short-duration verification and diarization require finer-grained temporal representations. Short temporal contexts limit the available speaker evidence and make speaker identity difficult to disentangle from phonetic and acoustic variability. This creates a temporal resolution-representation reliability trade-off: speaker representations should maintain local within-speaker consistency, between-speaker discriminability and remain sensitive to speaker changes at fine temporal resolution. Accordingly, we propose a framework for reliable speaker representation learning at fine temporal resolution. FineSpeaker first constructs resolution-controllable frame-wise speaker representations through mean pooling over an adjustable n-frame window, enabling flexible control of representation granularity. It then introduces a speaker encoder with local temporal modeling that exploits short-range speaker context to improve the stability and discriminability of short-duration and frame-wise representations. Furthermore, we develop a generative disentanglement strategy for optimizing frame-level speaker representations, combining VAE-based probabilistic modeling with flow matching to characterize speaker-related variation and suppress non-speaker interference. Experiments demonstrate competitive full-utterance performance, improved fine-grained speaker verification on VoxCeleb and CN-Celeb, and strong zero-shot diarization results, highlighting the benefits of balancing temporal resolution and representation reliability.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Yongkang Yin, Wenhao Yang, Lichun Fan, Jian Luan, Hang Su, Yuexian Zou. 2026-11-11. FineSpeaker: Reliable Speaker Representation Learning at Fine Temporal Resolution. https://arxiv.science/abs/2611.15288
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite arXiv Science by Indiciae: BibTeX · RIS.