arXiv subjects

Jaehong Myung

Publications and source records attributed to Jaehong Myung.

2 recordsLinked to original sources

arXiv2026

CASE: Complementary Articulation-Aware Speech Encoder to Enhance Speech-Driven 3D Facial Animation

General-purpose self-supervised speech encoders provide strong acoustic representations, but their pretraining objectives are not explicitly aligned with the visible articulatory dynamics required by speech-driven 3D facial animation. We propose CASE (Complementary Articulation-aware Speech Encoder), a complementary speech encoder adapted by contrastively aligning short speech clips with continuous 3D lip-mesh VQ-VAE latents. After adaptation, the geometry branch is removed, and the audio-only CASE representation is fused with the baseline SSL speech representation through residual cross-attention, while retaining the motion-decoder architecture and objective. Across four VOCASET baselines, CASE reduces mean lip vertex error by 2.8-6.0%, and mean full-face vertex error decreases in every system. Under the same fusion protocol, CASE yields lower mean LVE than the unadapted AV-SE stream on all four generators, with model-dependent margins. Because adaptation uses Korean speech-geometry pairs and evaluation uses English VOCASET, the result is consistent with cross-language downstream transfer under a shared English-pretrained audio-visual prior.

cs.CV↗

arXiv2026

Local Evidence, Global Failure: Diagnosing Speech-Vision Temporal Composition in Audio-Visual Language Models

Audio-visual large language models (AV-LLMs) can fail at full-video speech-vision decisions even when useful visual evidence is locally recoverable. We introduce Temporal Evidence Decomposition (TED), a training-free diagnostic framework that separates local visual verification from speech-relative aggregation. Automatic speech recognition supplies a transcript and utterance interval. TED queries audio-free windows with the transcript for Temporal Alignment and Forecasting, or answer candidates for Cross-Modal Temporal Binding, then aggregates evidence relative to that interval. Across four frozen AV-LLMs on SVHalluc, Temporal Forecasting accuracy rises from 29.3-33.7% under full-video questioning to 79.3-82.2% with TED. Query replacement reduces recovery; the observed anchor remains beneficial when temporal-region availability is held fixed. Physical-isolation and timing-shortcut controls test alternative explanations. The findings distinguish locally recoverable evidence from successful full-video decisions, motivating explicit separation of narration content, utterance timing, and time-indexed visual evidence.

cs.CV↗