arXiv · 2611.20436
Learning from Distribution: Rethinking One-Hot Supervision for ASR and QA in Speech Language Models
Abstract
Speech language models (SLMs) exhibit strong capabilities in automatic speech recognition (ASR) and general-purpose speech understanding, yet ASR-specific training often degrades spoken question answering (QA) performance. We revisit this conflict from the perspective of task-specific input--output mappings and one-hot supervision. Our findings reveal pronounced task-asymmetric effects: ASR-only training severely limits QA capability, whereas QA-only training retains reasonable ASR performance. We hypothesize that the relatively deterministic mapping of ASR, combined with single-reference supervision, biases models toward transcription-specific behavior. Motivated by this perspective, we advocate broadening ASR learning objectives while preserving recognition accuracy. We propose Learning from Distribution (LFD), which aligns speech-conditioned predictions with the conditional output distributions of a pretrained text LLM. Extensive experiments demonstrate that LFD improves both ASR and QA performance, exhibits superior data efficiency, and remains effective under full-parameter fine-tuning. CoT-ASR further supports the broader applicability of our proposed supervision principle.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Yujie Guo, Hongjie Chen, Jiaming Zhou, Jian Kang, Jie Li, Yongxiang Li, Yong Qin. 2026-11-16. Learning from Distribution: Rethinking One-Hot Supervision for ASR and QA in Speech Language Models. https://arxiv.science/abs/2611.20436
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite arXiv Science by Indiciae: BibTeX · RIS.