arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2610.09846

DynaConTalk: Wavelet-Constrained Diffusion for Long-Form and Controllable Holistic Co-Speech 3D Motion

Yifei Zhu·Yangyang Cai·Mingyi Shi·Miao Cheng·Lin Gu·Taku Komura·Yoshifumi Kitamura
View original source ↗Read source document ↗Export metadata ↓

Abstract

Holistic co-speech animation is prone to averaging in both motion representation and speech conditioning. In coordinate-space diffusion, slow body posture, mid-frequency gesture strokes, and fast hand or facial details are entangled in one prediction target, often producing low-variance, over-smoothed motion. Meanwhile, dense rhythmic and acoustic cues can dominate sparse content-specific information under fixed multimodal fusion. We present DynaConTalk, a wavelet-constrained diffusion framework for long-form and controllable holistic co-speech motion generation. Diffusion operates in stationary wavelet transform (SWT) coefficient space, whose temporally aligned bands separate coarse posture evolution, gesture strokes, and fine expressive details. Our dynamic gating network preserves HuBERT and speaker identity as a base and selectively adds rhythm, mel, and transcript features through motion-state- and noise-aware residual gates. Attention pooling and learned depth routing deliver complementary conditions to each denoising stage, while a frame-resolution rhythm path preserves precise timing. A signed proposal-consensus update then reconciles these conditions with the evolving motion state. Matched-noise constraint injection uses the same sampling interface for history continuation and localized keypose repair, and extends to reference-guided control. Separate body-hand and facial denoisers, followed by inverse SWT and a pose-driven root regressor, produce holistic motion. Experiments evaluate generation quality, facial accuracy, temporal continuity, and controllable editing. Code, models, and the interactive editing interface are available at https://github.com/zhuyifeiabcd1/DynaConTalk.

Explore related subjects

cs.GR

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Yifei Zhu, Yangyang Cai, Mingyi Shi, Miao Cheng, Lin Gu, Taku Komura, Yoshifumi Kitamura. 2026-10-07. DynaConTalk: Wavelet-Constrained Diffusion for Long-Form and Controllable Holistic Co-Speech 3D Motion. https://arxiv.org/abs/2610.09846

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Trajeform: Embedding Crowd Data

Crowd simulation aims to generate agent behaviours that are rich, natural, and appropriate to their surrounding environment. Recent advances in Machine Learning have enabled such behavioural patterns to be learned directly from trajectory data, allowing simulation models to capture increasingly complex crowd dynamics. However, the capabilities of these systems remain strongly dependent on the type and diversity of the data used during training. When evaluating how well crowd simulators generalise to real-world datasets, most existing metrics focus primarily on trajectory-level differences rather than behaviour-level similarity. In this paper, we present Trajeform, a framework that trains a reconstruction VAE (Variational Autoencoder) on real trajectory data to learn a meaningful continuous embedding that preserves behavioural characteristics of real crowds. The trained encoder can then be applied to synthetic trajectories generated by simulators and used to compare the resulting embedding distributions. Additionally, the learned embedding can be used to analyse supplementary data, such as trajectories corresponding to a simulator's principal behaviours or captured data from controlled environments (e.g., cycling), providing insight into their ability to span the real behavioural embedding space. The framework compares data from three simulators (CCP, modCCP, and RVO) against ground-truth datasets and is evaluated quantitatively and qualitatively using latent-space analysis, distributional metrics, and trajectory generation through sampling from the aggregated posterior distribution. The results indicate that the learned latent space captures meaningful trajectory dependencies found in the real data, making the framework useful both as an additional measure of synthetic data believability and as a tool for informing the training of future trajectory models.

cs.GR↗
arXiv2026

Mastering Multi-State Dynamics: Continuum Damage and Viscoplastic MPM for RL-Based Dexterous Manipulation

Robotic manipulation of multi-state materials needs a simulator that handles large deformation, topology change, and two-way contact between tools and materials while staying fast enough to drive online reinforcement learning. We present MG-CD-MPM, a tool-centered localized extension of CD-MPM. The implicit Ginzburg-Landau phase-field damage system is solved on a local patch by preconditioned conjugate gradients using a geometric-multigrid V-cycle as the preconditioner. The background MPM discretization remains uniform: only the damage solve is localized. Two-way rigid-tool coupling converts MPM contact impulses into forces and torques on the tool, exposing material resistance directly to the policy. The resulting Gym-like simulator covers soft solids, brittle materials, granular media, and viscoplastic or semi-fluid media within a unified per-particle constitutive layer. We demonstrate cutting, scooping, and shoveling in six scenes, combining scripted tool motions with PPO policies trained on two cutting tasks.

cs.GR↗
arXiv2026

Greenscapes: Optimized Vegetation Configurations in Cities

Vegetation plays a central role in cities, serving functional roles such as balancing shade and sky exposure, shaping the microclimate, carbon sequestration, and providing acoustic screening. Furthermore, aesthetic choices in landscape styles (formal, modernist, picturesque) are important for citizens' well-being and should be considered in the design of urban landscapes. In this paper, we present a novel method for placing vegetation, ranging from shrubs to large trees, in 3D urban scenes. Our method uses individual lots as input and employs a conditioned Vision-Language Model (VLM) to generate a top-down rendering of vegetation, placing large trees, small trees, and shrubs on lawn areas. This rendering is then converted into a 3D scene by detecting individual plants and selecting appropriate 3D assets. To refine the proposed configurations, we define an optimization loop that keeps the VLM-generated positions fixed after a deterministic feasibility repair and uses CMA-ES to optimize per-instance scale transformations along with discrete vegetation asset substitutions. To define the objective function for the optimization, we evaluate the generated 3D scenes with efficient surrogate models of shade and path-based acoustic exposure. Furthermore, we render in-scene views at each optimization step to compute view-dependent perceptual descriptors, enabling the generation of both aesthetic and functional vegetation layouts for urban scenes.

cs.GR↗