arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2505.21237

Unfolding A Few Structures for The Many: Memory-Efficient Compression of Conformer and Speech Foundation Models

Zhaoqing Li·Haoning Xu·Xurong Xie·Zengrui Jin·Tianzi Wang·Xunying Liu
View original source ↗Read source document ↗Export metadata ↓

Abstract

This paper presents a novel memory-efficient model compression approach for Conformer ASR and speech foundation systems. Our approach features a unique "small-to-large" design. A compact "seed" model containing a few Conformer or Transformer blocks is trained and unfolded many times to emulate the performance of larger uncompressed models with different logical depths. The seed model and many unfolded paths are jointly trained within a single unfolding cycle. The KL-divergence between the largest unfolded and smallest seed models is used in a self-distillation process to minimize their performance disparity. Experimental results show that our foldable model produces ASR performance comparable to individually constructed Conformer and wav2vec2/HuBERT speech foundation models under various depth configurations, while requiring only minimal memory and storage. Conformer and wav2vec2 models with a reduction of 35% and 30% parameters are obtained without loss of performance, respectively.

Explore related subjects

cs.SDeess.AS

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Zhaoqing Li, Haoning Xu, Xurong Xie, Zengrui Jin, Tianzi Wang, Xunying Liu. 2025-05-27. Unfolding A Few Structures for The Many: Memory-Efficient Compression of Conformer and Speech Foundation Models. https://arxiv.org/abs/2505.21237

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

ParsVoice: A Large-Scale Multi-Speaker Persian Speech Corpus for Text-to-Speech Synthesis

Persian remains substantially underrepresented in open speech-text resources, limiting progress in multi-speaker text-to-speech (TTS), speech-language modelling, and low-resource speech processing. We introduce ParsVoice, the largest publicly available Persian speech-text corpus tailored for training multi-speaker TTS systems, along with a scalable pipeline to construct high-quality speech-text data from long-form audiobook recordings. The pipeline combines a fine-tuned ParsBERT sentence-completion classifier, ASR-based boundary optimization, punctuation restoration, speaker identification, and a multi-dimensional quality assessment that covers both audio and Persian-specific text properties. The resulting release contains a 2,200-hour TTS-ready subset with 1.36 million aligned segments from 1,815 automatically inferred speaker IDs, making it more than 25 times larger than the largest previously available open Persian TTS dataset. To validate the corpus, we fine-tune XTTSv2, a zero-shot multilingual TTS model that operates directly on raw Persian text without phoneme representations. The resulting model achieves a naturalness MOS of 3.6/5 and a speaker-similarity MOS of 4.0/5. ParsVoice, its metadata, and the corpus-construction pipeline are publicly available on Hugging Face: https://huggingface.co/datasets/MohammadJRanjbar/ParsVoice and GitHub: https://github.com/MohammadJRanjbar/ParsVoice, supporting reproducible research on Persian speech synthesis and low-resource speech-language technologies.

cs.SD↗
arXiv2026

Speech Generation Speaker Poisoning: Capability Erasure in Zero-Shot Text-to-Speech

Recent zero-shot Text-to-Speech (TTS) systems can clone previously unseen voices from only a few seconds of audio. We formulate Speech Generation Speaker Poisoning (SGSP), a task that seeks to prevent a model from synthesizing targeted speaker identities while maintaining performance on all other speakers. Unlike conventional machine unlearning, removing training examples is insufficient because modern zero-shot TTS systems can reconstruct identities through learned speaker representations and strong generalization capabilities. We evaluate both inference-time filtering and parameter-modification approaches across settings involving 1, 15, and 100 forget speakers, focusing primarily on speakers seen during training, which we show are harder to suppress than unseen speakers. To characterize the trade-off between utility and privacy, we introduce an evaluation framework based on AUC analysis and a proposed metric, Forget Set Similarity (FSSIM). Our results demonstrate effective speaker suppression for up to 15 forget speakers while revealing that worst-case identity leakage (Max-FSSIM) remains unresolved at multi-speaker scale - establishing this as an open challenge for future work. Together, our work establishes targeted speaker poisoning as a task for zero-shot TTS and, using StyleTTS2 as an initial testbed, provides methods and evaluation protocols for future research, with code and model weights.

cs.SD↗
arXiv2026

Tracing the Origins: Legacy Codec Identification in Neural Audio Transcoding

Residual Vector Quantization (RVQ)-based neural audio codecs (NACs) enable high-fidelity audio distribution at unprecedentedly low bitrates through discrete token-based representations. However, this shift disrupts traditional forensics, as non-linear neural transcoding obscures the underlying traces of legacy compression. This study defines the forensic gap and proposes a Transformer-based framework designed to leverage the hierarchical and temporal dependencies inherent in RVQ sequences. By modeling inter-layer causal relationships and dynamic forensic significance, our model effectively disentangles superimposed artifacts from legacy-to-neural transcoding. Experimental results achieve 97%+ accuracy for codec identification and robust joint identification performance across 32-128 kbps. These results demonstrate that traditional codec traces persist even after neural transcoding, supporting the feasibility and necessity of neural-codec-aware audio forensics.

cs.SD↗