arXiv subjects

Mingyuan Gao

Publications and source records attributed to Mingyuan Gao.

13 recordsLinked to original sources

arXiv2026

DreamActor-M2: Universal Character Image Animation via Spatiotemporal In-Context Learning

Character image animation aims to synthesize high-fidelity videos by transferring motion from a driving sequence to a static reference image. Despite recent advancements, existing methods suffer from two fundamental challenges: (1) suboptimal motion injection strategies that lead to a trade-off between identity preservation and motion consistency, manifesting as a "see-saw", and (2) an over-reliance on explicit pose priors (e.g., skeletons), which inadequately capture intricate dynamics and hinder generalization to arbitrary, non-humanoid characters. To address these challenges, we present DreamActor-M2, a universal animation framework that reimagines motion conditioning as an in-context learning problem. Our approach follows a two-stage paradigm. First, we bridge the input modality gap by fusing reference appearance and motion cues into a unified latent space, enabling the model to jointly reason about spatial identity and temporal dynamics by leveraging the generative prior of foundational models. Second, we introduce a self-bootstrapped data synthesis pipeline that curates pseudo cross-identity training pairs, facilitating a seamless transition from pose-dependent control to direct, end-to-end RGB-driven animation. This strategy significantly enhances generalization across diverse characters and motion scenarios. To facilitate comprehensive evaluation, we further introduce AW Bench, a versatile benchmark encompassing a wide spectrum of characters types and motion scenarios. Extensive experiments demonstrate that DreamActor-M2 achieves state-of-the-art performance, delivering superior visual fidelity and robust cross-domain generalization. Project Page: https://grisoon.github.io/DreamActor-M2/

cs.CV↗

arXiv2026

OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfer

Videos convey richer information than images or text, capturing both spatial and temporal dynamics. However, most existing video customization methods rely on reference images or task-specific temporal priors, failing to fully exploit the rich spatio-temporal information inherent in videos, thereby limiting flexibility and generalization in video generation. To address these limitations, we propose OmniTransfer, a unified framework for spatio-temporal video transfer. It leverages multi-view information across frames to enhance appearance consistency and exploits temporal cues to enable fine-grained temporal control. To unify various video transfer tasks, OmniTransfer incorporates three key designs: Task-aware Positional Bias that adaptively leverages reference video information to improve temporal alignment or appearance consistency; Reference-decoupled Causal Learning separating reference and target branches to enable precise reference transfer while improving efficiency; and Task-adaptive Multimodal Alignment using multimodal semantic guidance to dynamically distinguish and tackle different tasks. Extensive experiments show that OmniTransfer outperforms existing methods in appearance (ID and style) and temporal transfer (camera movement and video effects), while matching pose-guided methods in motion transfer without using pose, establishing a new paradigm for flexible, high-fidelity video generation.

cs.CV↗

arXiv2026

FlowAct-R1: Towards Interactive Humanoid Video Generation

Interactive humanoid video generation aims to synthesize lifelike visual agents that can engage with humans through continuous and responsive video. Despite recent advances in video synthesis, existing methods often grapple with the trade-off between high-fidelity synthesis and real-time interaction requirements. In this paper, we propose FlowAct-R1, a framework specifically designed for real-time interactive humanoid video generation. Built upon a MMDiT architecture, FlowAct-R1 enables the streaming synthesis of video with arbitrary durations while maintaining low-latency responsiveness. We introduce a chunkwise diffusion forcing strategy, complemented by a novel self-forcing variant, to alleviate error accumulation and ensure long-term temporal consistency during continuous interaction. By leveraging efficient distillation and system-level optimizations, our framework achieves a stable 25fps at 480p resolution with a time-to-first-frame (TTFF) of only around 1.5 seconds. The proposed method provides holistic and fine-grained full-body control, enabling the agent to transition naturally between diverse behavioral states in interactive scenarios. Experimental results demonstrate that FlowAct-R1 achieves exceptional behavioral vividness and perceptual realism, while maintaining robust generalization across diverse character styles.

cs.CV↗

arXiv2025

Two-stage Respiratory Motion-resolved Radial MR Image Reconstruction Using an Interpretable Deep Unrolled Network

Due to the prolonged MRI encoding process, respiratory motion can cause undesired artifacts and image blurring, degrading image quality and limiting clinical applications in abdominal and pulmonary imaging. In this work, we develop a two-stage respiratory motion-resolved radial MR image reconstruction pipeline using an interpretable deep unrolled network (MoraNet), enabling high-quality imaging under free-breathing conditions. Firstly, low-resolution images are reconstructed from the central region of successive golden-angle radial k-space to extract respiratory motion signals. The binned k-space data based on the respiratory signal are then used to reconstruct the motion-resolved high-resolution image for each motion state. The MoraNet applies nonuniform fast Fourier transform (NUFFT) to operate radial encoding and convolutional neural network (CNN) modules to conduct image regularizations. The MoraNet was trained on retrospectively acquired lung MRI images for both fully sampled and undersampled acquisitions. The performance of the proposed method was evaluated on digital CT/MRI breathing XCAT (CoMBAT) phantom data, QUASAR motion phantom data acquired from a 1.0T MRI scanner and volunteer chest data acquired from a 1.5T MRI scanner. The MoraNet pipeline was compared with motion-averaged reconstruction and a conventional compressed sensing (CS)-based method in terms of SSIM, RMSE and computation time. Simulation and experimental results demonstrated that the proposed network could provide accurate respiratory signal estimation and enable effective motion correction. Compared with the CS method, the MoraNet preserved better structural details with lower RMSE and higher SSIM values at acceleration factor of 4, and meanwhile took ten-fold faster inference time.

physics.med-ph↗

arXiv2025

Seedance 1.5 pro: A Native Audio-Visual Joint Generation Foundation Model

Recent strides in video generation have paved the way for unified audio-visual generation. In this work, we present Seedance 1.5 pro, a foundational model engineered specifically for native, joint audio-video generation. Leveraging a dual-branch Diffusion Transformer architecture, the model integrates a cross-modal joint module with a specialized multi-stage data pipeline, achieving exceptional audio-visual synchronization and superior generation quality. To ensure practical utility, we implement meticulous post-training optimizations, including Supervised Fine-Tuning (SFT) on high-quality datasets and Reinforcement Learning from Human Feedback (RLHF) with multi-dimensional reward models. Furthermore, we introduce an acceleration framework that boosts inference speed by over 10X. Seedance 1.5 pro distinguishes itself through precise multilingual and dialect lip-syncing, dynamic cinematic camera control, and enhanced narrative coherence, positioning it as a robust engine for professional-grade content creation. Seedance 1.5 pro is now accessible on Volcano Engine at https://console.volcengine.com/ark/region:ark+cn-beijing/experience/vision?type=GenVideo.

cs.CV↗

arXiv2025

OmniHuman-1.5: Instilling an Active Mind in Avatars via Cognitive Simulation

Existing video avatar models can produce fluid human animations, yet they struggle to move beyond mere physical likeness to capture a character's authentic essence. Their motions typically synchronize with low-level cues like audio rhythm, lacking a deeper semantic understanding of emotion, intent, or context. To bridge this gap, \textbf{we propose a framework designed to generate character animations that are not only physically plausible but also semantically coherent and expressive.} Our model, \textbf{OmniHuman-1.5}, is built upon two key technical contributions. First, we leverage Multimodal Large Language Models to synthesize a structured textual representation of conditions that provides high-level semantic guidance. This guidance steers our motion generator beyond simplistic rhythmic synchronization, enabling the production of actions that are contextually and emotionally resonant. Second, to ensure the effective fusion of these multimodal inputs and mitigate inter-modality conflicts, we introduce a specialized Multimodal DiT architecture with a novel Pseudo Last Frame design. The synergy of these components allows our model to accurately interpret the joint semantics of audio, images, and text, thereby generating motions that are deeply coherent with the character, scene, and linguistic content. Extensive experiments demonstrate that our model achieves leading performance across a comprehensive set of metrics, including lip-sync accuracy, video quality, motion naturalness and semantic consistency with textual prompts. Furthermore, our approach shows remarkable extensibility to complex scenarios, such as those involving multi-person and non-human subjects. Homepage: \href{https://omnihuman-lab.github.io/v1_5/}

cs.CV↗

arXiv2025

DreamVVT: Mastering Realistic Video Virtual Try-On in the Wild via a Stage-Wise Diffusion Transformer Framework

Video virtual try-on (VVT) technology has garnered considerable academic interest owing to its promising applications in e-commerce advertising and entertainment. However, most existing end-to-end methods rely heavily on scarce paired garment-centric datasets and fail to effectively leverage priors of advanced visual models and test-time inputs, making it challenging to accurately preserve fine-grained garment details and maintain temporal consistency in unconstrained scenarios. To address these challenges, we propose DreamVVT, a carefully designed two-stage framework built upon Diffusion Transformers (DiTs), which is inherently capable of leveraging diverse unpaired human-centric data to enhance adaptability in real-world scenarios. To further leverage prior knowledge from pretrained models and test-time inputs, in the first stage, we sample representative frames from the input video and utilize a multi-frame try-on model integrated with a vision-language model (VLM), to synthesize high-fidelity and semantically consistent keyframe try-on images. These images serve as complementary appearance guidance for subsequent video generation. \textbf{In the second stage}, skeleton maps together with fine-grained motion and appearance descriptions are extracted from the input content, and these along with the keyframe try-on images are then fed into a pretrained video generation model enhanced with LoRA adapters. This ensures long-term temporal coherence for unseen regions and enables highly plausible dynamic motions. Extensive quantitative and qualitative experiments demonstrate that DreamVVT surpasses existing methods in preserving detailed garment content and temporal stability in real-world scenarios. Our project page https://virtu-lab.github.io/

cs.CV↗

arXiv2025

DreamActor-H1: High-Fidelity Human-Product Demonstration Video Generation via Motion-designed Diffusion Transformers

In e-commerce and digital marketing, generating high-fidelity human-product demonstration videos is important for effective product presentation. However, most existing frameworks either fail to preserve the identities of both humans and products or lack an understanding of human-product spatial relationships, leading to unrealistic representations and unnatural interactions. To address these challenges, we propose a Diffusion Transformer (DiT)-based framework. Our method simultaneously preserves human identities and product-specific details, such as logos and textures, by injecting paired human-product reference information and utilizing an additional masked cross-attention mechanism. We employ a 3D body mesh template and product bounding boxes to provide precise motion guidance, enabling intuitive alignment of hand gestures with product placements. Additionally, structured text encoding is used to incorporate category-level semantics, enhancing 3D consistency during small rotational changes across frames. Trained on a hybrid dataset with extensive data augmentation strategies, our approach outperforms state-of-the-art techniques in maintaining the identity integrity of both humans and products and generating realistic demonstration motions. Project page: https://lizhenwangt.github.io/DreamActor-H1/.

cs.CV↗

arXiv2025

InterActHuman: Multi-Concept Human Animation with Layout-Aligned Audio Conditions

End-to-end human animation with rich multi-modal conditions, e.g., text, image and audio has achieved remarkable advancements in recent years. However, most existing methods could only animate a single subject and inject conditions in a global manner, ignoring scenarios where multiple concepts could appear in the same video with rich human-human interactions and human-object interactions. Such a global assumption prevents precise and per-identity control of multiple concepts including humans and objects, therefore hinders applications. In this work, we discard the single-entity assumption and introduce a novel framework that enforces strong, region-specific binding of conditions from modalities to each identity's spatiotemporal footprint. Given reference images of multiple concepts, our method could automatically infer layout information by leveraging a mask predictor to match appearance cues between the denoised video and each reference appearance. Furthermore, we inject local audio condition into its corresponding region to ensure layout-aligned modality matching in an iterative manner. This design enables the high-quality generation of human dialogue videos between two to three people or video customization from multiple reference images. Empirical results and ablation studies validate the effectiveness of our explicit layout control for multi-modal conditions compared to implicit counterparts and other existing methods. Video demos are available at https://zhenzhiwang.github.io/interacthuman/

cs.CV↗

arXiv2024

Quantum walks of correlated photons in non-Hermitian photonic lattices

Entanglement entropy characterizes the correlation of multi-particles and unveils the crucial features of open quantum systems. However, the experimental realization of exploring entanglement in non-Hermitian systems remains a challenge. In parallel, quantum walks have offered the possibility of studying the underlying mechanisms of non-Hermitian physics, which includes exceptional points, the non-Hermitian skin effect, and non-Bloch phase transitions. Unfortunately, these studies have only involved and prevailingly focused on the behavior of a single particle. Here, we propose and experimentally realize quantum walks of two indistinguishable photons in engineered non-Hermitian photonic lattices. We have successfully observed the unidirectional behavior of quantum walks in the bulk far from the edges induced by the skin effect. Moreover, we experimentally reveal the suppression of entanglement that is caused by the skin effect in non-Hermitian systems. Our study may facilitate a deep understanding of entanglement in open quantum many-body systems that are far from thermal equilibrium.

quant-ph↗

arXiv2024

Image Reconstruction with B0 Inhomogeneity using an Interpretable Deep Unrolled Network on an Open-bore MRI-Linac

MRI-Linac systems require fast image reconstruction with high geometric fidelity to localize and track tumours for radiotherapy treatments. However, B0 field inhomogeneity distortions and slow MR acquisition potentially limit the quality of the image guidance and tumour treatments. In this study, we develop an interpretable unrolled network, referred to as RebinNet, to reconstruct distortion-free images from B0 inhomogeneity-corrupted k-space for fast MRI-guided radiotherapy applications. RebinNet includes convolutional neural network (CNN) blocks to perform image regularizations and nonuniform fast Fourier Transform (NUFFT) modules to incorporate B0 inhomogeneity information. The RebinNet was trained on a publicly available MR dataset from eleven healthy volunteers for both fully sampled and subsampled acquisitions. Grid phantom and human brain images acquired from an open-bore 1T MRI-Linac scanner were used to evaluate the performance of the proposed network. The RebinNet was compared with the conventional regularization algorithm and our recently developed UnUNet method in terms of root mean squared error (RMSE), structural similarity (SSIM), residual distortions, and computation time. Imaging results demonstrated that the RebinNet reconstructed images with lowest RMSE (<0.05) and highest SSIM (>0.92) at four-time acceleration for simulated brain images. The RebinNet could better preserve structural details and substantially improve the computational efficiency (ten-fold faster) compared to the conventional regularization methods, and had better generalization ability than the UnUNet method. The proposed RebinNet can achieve rapid image reconstruction and overcome the B0 inhomogeneity distortions simultaneously, which would facilitate accurate and fast image guidance in radiotherapy treatments.

physics.med-ph↗

arXiv2023

Twisted van der Waals Quantum Materials: Fundamentals, Tunability and Applications

Twisted vdW quantum materials have emerged as a rapidly developing field of 2D semiconductors. These materials establish a new central research area and provide a promising platform for studying quantum phenomena and investigating the engineering of novel optoelectronic properties such as single-photon emission, non-linear optical response, magnon physics, and topological superconductivity. These captivating electronic and optical properties result from, and can be tailored by, the interlayer coupling using moir\'e patterns formed by vertically stacking atomic layers with controlled angle misorientation or lattice mismatch. Their outstanding properties and the high degree of tunability position them as compelling building blocks for both compact quantum-enabled devices and classical optoelectronics. This article offers a comprehensive review of recent advancements in the understanding and manipulation of twisted van der Waals structures and presents a survey of the state-of-the-art research on moir\'e superlattices, encompassing interdisciplinary interests. It delves into fundamental theories, synthesis and fabrication, and visualization techniques, and the wide range of novel physical phenomena exhibited by these structures, with a focus on their potential for practical device integration in applications ranging from quantum information to biosensors, and including classical optoelectronics such as modulators, light emitting diodes (LEDs), lasers, and photodetectors. It highlights the unique ability of moir\'e superlattices to connect multiple disciplines, covering chemistry, electronics, optics, photonics, magnetism, topological and quantum physics. This comprehensive review provides a valuable resource for researchers interested in moir\'e superlattices, shedding light on their fundamental characteristics and their potential for transformative applications in various fields.

cond-mat.mtrl-sci↗

arXiv2022

Mixed Fault Tolerance Protocols with Trusted Execution Environment

Blockchain systems are designed, built and operated in the presence of failures. There are two dominant failure models, namely crash fault and Byzantine fault. Byzantine fault tolerance (BFT) protocols offer stronger security guarantees, and thus are widely used in blockchain systems. However, their security guarantees come at a dear cost to their performance and scalability. Several works have improved BFT protocols, and Trusted Execution Environment (TEE) has been shown to be an effective solution. However, existing such works typically assume that each participating node is equipped with TEE. For blockchain systems wherein participants typically have different hardware configurations, i.e., some nodes feature TEE while others do not, existing TEE-based BFT protocols are not applicable. This work studies the setting wherein not all participating nodes feature TEE, under which we propose a new fault model called mixed fault. We explore a new approach to designing efficient distributed fault-tolerant protocols under the mixed fault model. In general, mixed fault tolerance (MFT) protocols assume a network of $n$ nodes, among which up to $f = \frac{n-2}{3}$ can be subject to mixed faults. We identify two key principles for designing efficient MFT protocols, namely, (i) prioritizing non-equivocating nodes in leading the protocol, and (ii) advocating the use of public-key cryptographic primitives that allow authenticated messages to be aggregated. We showcase these design principles by prescribing an MFT protocol, namely MRaft. We implemented a prototype of MRaft using Intel SGX, integrated it into the CCF blockchain framework, conducted experiments, and showed that MFT protocols can obtain the same security guarantees as their BFT counterparts while still providing better performance (both transaction throughput and latency) and scalability.

cs.DC↗