arXiv · 2611.13875
DialInstruct-TTS: Interactive Expressive Speech Generation via Multi-Turn Dialogue with Fine-Grained Temporal Control
Abstract
Existing instruction-based TTS systems primarily support one-shot, utterance-level control, where a single instruction determines the speaking style of an entire utterance. As a result, even minor refinements require rewriting the instruction and regenerating the full utterance, limiting fine-grained interactive control. We propose DialInstruct-TTS, an interactive multi-turn expressive TTS framework that enables users to iteratively refine generated speech through natural-language dialogue until the desired style is achieved. At each dialogue turn, a Qwen-based planner integrates the dialogue history and latest user instruction into a complete TemporalPlan, while a temporal acoustic control module provides fine-grained control over emotion, speaking rate, volume, and phonation. Reference-latent revision preserves non-target regions across turns, allowing users to modify only the requested attributes or temporal spans while retaining previously satisfactory speech. Subjective evaluations demonstrate improved global instruction following, accurate local style realization, and effective non-target preservation. Compared with conventional one-shot interaction requiring full instruction restatement and utterance regeneration, DialInstruct-TTS substantially improves perceived convenience and interaction comfort. Audio samples are available at https://anonymous.4open.science/w/eacl-demo-17C5/.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Wangzixi Zhou, Bagus Tris Atmaja, Sakriani Sakti. 2026-11-10. DialInstruct-TTS: Interactive Expressive Speech Generation via Multi-Turn Dialogue with Fine-Grained Temporal Control. https://arxiv.science/abs/2611.13875
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite arXiv Science by Indiciae: BibTeX · RIS.