arXiv subjects

Yuxuan Wang

Publications and source records attributed to Yuxuan Wang.

At least 217 records · Page 12Linked to original sources

arXiv2024

Medical Dialogue: A Survey of Categories, Methods, Evaluation and Challenges

This paper surveys and organizes research works on medical dialog systems, which is an important yet challenging task. Although these systems have been surveyed in the medical community from an application perspective, a systematic review from a rigorous technical perspective has to date remained noticeably absent. As a result, an overview of the categories, methods, and evaluation of medical dialogue systems remain limited and underspecified, hindering the further improvement of this area. To fill this gap, we investigate an initial pool of 325 papers from well-known computer science, and natural language processing conferences and journals, and make an overview. Recently, large language models have shown strong model capacity on downstream tasks, which also reshaped medical dialog systems' foundation. Despite the alluring practical application value, current medical dialogue systems still suffer from problems. To this end, this paper lists the grand challenges of medical dialog systems, especially of large language models.

cs.CL↗

arXiv2024

AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining

Although audio generation shares commonalities across different types of audio, such as speech, music, and sound effects, designing models for each type requires careful consideration of specific objectives and biases that can significantly differ from those of other types. To bring us closer to a unified perspective of audio generation, this paper proposes a framework that utilizes the same learning method for speech, music, and sound effect generation. Our framework introduces a general representation of audio, called "language of audio" (LOA). Any audio can be translated into LOA based on AudioMAE, a self-supervised pre-trained representation learning model. In the generation process, we translate any modalities into LOA by using a GPT-2 model, and we perform self-supervised audio generation learning with a latent diffusion model conditioned on LOA. The proposed framework naturally brings advantages such as in-context learning abilities and reusable self-supervised pretrained AudioMAE and latent diffusion models. Experiments on the major benchmarks of text-to-audio, text-to-music, and text-to-speech demonstrate state-of-the-art or competitive performance against previous approaches. Our code, pretrained model, and demo are available at https://audioldm.github.io/audioldm2.

cs.SD↗

arXiv2024

FIMP-HGA: A Novel Approach to Addressing the Partitioning Min-Max Weighted Matching Problem

The Partitioning Min-Max Weighted Matching (PMMWM) problem, being a practical NP-hard problem, integrates the task of partitioning the vertices of a bipartite graph into disjoint sets of limited size with the classical Maximum-Weight Perfect Matching (MPWM) problem. Initially introduced in 2015, the state-of-the-art method for addressing PMMWM is the MP$_{\text{LS}}$. In this paper, we present a novel approach, the Fast Iterative Match-Partition Hybrid Genetic Algorithm (FIMP-HGA), for addressing PMMWM. Similar to MP$_{\text{LS}}$, FIMP-HGA divides the solving into match and partition stages, iteratively refining the solution. In the match stage, we propose the KM-M algorithm, which reduces matching complexity through incremental adjustments, significantly enhancing runtime efficiency. For the partition stage, we introduce a Hybrid Genetic Algorithm (HGA) incorporating an elite strategy and design a Greedy Partition Crossover (GPX) operator alongside a Multilevel Local Search (MLS) to optimize individuals in the population. Population initialization employs various methods, including the multi-way Karmarkar-Karp (KK) algorithm, ensuring both quality and diversity. At each iteration, the bipartite graph is adjusted based on the current solution, aiming for continuous improvement. To conduct comprehensive experiments, we develop a new instance generation method compatible with existing approaches, resulting in four benchmark groups. Extensive experiments evaluate various algorithm modules, accurately assessing each module's impact on improvement. Evaluation results on our benchmarks demonstrate that the proposed FIMP-HGA significantly enhances solution quality compared to MP$_{\text{LS}}$, meanwhile reducing runtime by 3 to 20 times.

cs.NE↗

arXiv2024

Enhancing Prosthetic Safety and Environmental Adaptability: A Visual-Inertial Prosthesis Motion Estimation Approach on Uneven Terrains

Environment awareness is crucial for enhancing walking safety and stability of amputee wearing powered prosthesis when crossing uneven terrains such as stairs and obstacles. However, existing environmental perception systems for prosthesis only provide terrain types and corresponding parameters, which fails to prevent potential collisions when crossing uneven terrains and may lead to falls and other severe consequences. In this paper, a visual-inertial motion estimation approach is proposed for prosthesis to perceive its movement and the changes of spatial relationship between the prosthesis and uneven terrain when traversing them. To achieve this, we estimate the knee motion by utilizing a depth camera to perceive the environment and align feature points extracted from stairs and obstacles. Subsequently, an error-state Kalman filter is incorporated to fuse the inertial data into visual estimations to reduce the feature extraction error and obtain a more robust estimation. The motion of prosthetic joint and toe are derived using the prosthesis model parameters. Experiment conducted on our collected dataset and stair walking trials with a powered prosthesis shows that the proposed method can accurately tracking the motion of the human leg and prosthesis with an average root-mean-square error of toe trajectory less than 5 cm. The proposed method is expected to enable the environmental adaptive control for prosthesis, thereby enhancing amputee's safety and mobility in uneven terrains.

cs.RO↗

arXiv2024

How Many Grid-Forming Converters Do We Need? A Perspective From Small Signal Stability and Power Grid Strength

Grid-forming (GFM) control has been considered a promising solution for accommodating large-scale power electronics converters into modern power grids thanks to its grid-friendly dynamics, in particular, voltage source behavior on the AC side. The voltage source behavior of GFM converters can provide voltage support for the power grid, and therefore enhance the power grid (voltage) strength. However, grid-following (GFL) converters can also perform constant AC voltage magnitude control by properly regulating their reactive current, which may also behave like a voltage source. Currently, it still remains unclear what are the essential differences between the voltage source behaviors of GFL and GFM converters, and which type of voltage source behavior can enhance the power grid strength. In this paper, we will demonstrate that only GFM converters can provide effective voltage source behavior and enhance the power grid strength in terms of small signal dynamics. Based on our analysis, we further study the problem of how to configure GFM converters in the grid and how many GFM converters we will need. We investigate how the capacity ratio between GFM and GFL converters affects the equivalent power grid strength and thus the small signal stability of the system. We give guidelines on how to choose this ratio to achieve a desired stability margin. We validate our analysis using high-fidelity simulations.

eess.SY↗

arXiv2024

VoiceShop: A Unified Speech-to-Speech Framework for Identity-Preserving Zero-Shot Voice Editing

We present VoiceShop, a novel speech-to-speech framework that can modify multiple attributes of speech, such as age, gender, accent, and speech style, in a single forward pass while preserving the input speaker's timbre. Previous works have been constrained to specialized models that can only edit these attributes individually and suffer from the following pitfalls: the magnitude of the conversion effect is weak, there is no zero-shot capability for out-of-distribution speakers, or the synthesized outputs exhibit undesirable timbre leakage. Our work proposes solutions for each of these issues in a simple modular framework based on a conditional diffusion backbone model with optional normalizing flow-based and sequence-to-sequence speaker attribute-editing modules, whose components can be combined or removed during inference to meet a wide array of tasks without additional model finetuning. Audio samples are available at \url{https://voiceshopai.github.io}.

cs.SD↗

arXiv2024

A unified front-end framework for English text-to-speech synthesis

The front-end is a critical component of English text-to-speech (TTS) systems, responsible for extracting linguistic features that are essential for a text-to-speech model to synthesize speech, such as prosodies and phonemes. The English TTS front-end typically consists of a text normalization (TN) module, a prosody word prosody phrase (PWPP) module, and a grapheme-to-phoneme (G2P) module. However, current research on the English TTS front-end focuses solely on individual modules, neglecting the interdependence between them and resulting in sub-optimal performance for each module. Therefore, this paper proposes a unified front-end framework that captures the dependencies among the English TTS front-end modules. Extensive experiments have demonstrated that the proposed method achieves state-of-the-art (SOTA) performance in all modules.

cs.CL↗

arXiv2024

HawkEye: Training Video-Text LLMs for Grounding Text in Videos

Video-text Large Language Models (video-text LLMs) have shown remarkable performance in answering questions and holding conversations on simple videos. However, they perform almost the same as random on grounding text queries in long and complicated videos, having little ability to understand and reason about temporal information, which is the most fundamental difference between videos and images. In this paper, we propose HawkEye, one of the first video-text LLMs that can perform temporal video grounding in a fully text-to-text manner. To collect training data that is applicable for temporal video grounding, we construct InternVid-G, a large-scale video-text corpus with segment-level captions and negative spans, with which we introduce two new time-aware training objectives to video-text LLMs. We also propose a coarse-grained method of representing segments in videos, which is more robust and easier for LLMs to learn and follow than other alternatives. Extensive experiments show that HawkEye is better at temporal video grounding and comparable on other video-text tasks with existing video-text LLMs, which verifies its superior video-text multi-modal understanding abilities.

cs.CV↗

arXiv2024

A-SDM: Accelerating Stable Diffusion through Redundancy Removal and Performance Optimization

The Stable Diffusion Model (SDM) is a popular and efficient text-to-image (t2i) generation and image-to-image (i2i) generation model. Although there have been some attempts to reduce sampling steps, model distillation, and network quantization, these previous methods generally retain the original network architecture. Billion scale parameters and high computing requirements make the research of model architecture adjustment scarce. In this work, we first explore the computational redundancy part of the network, and then prune the redundancy blocks of the model and maintain the network performance through a progressive incubation strategy. Secondly, in order to maintaining the model performance, we add cross-layer multi-expert conditional convolution (CLME-Condconv) to the block pruning part to inherit the original convolution parameters. Thirdly, we propose a global-regional interactive (GRI) attention to speed up the computationally intensive attention part. Finally, we use semantic-aware supervision (SAS) to align the outputs of the teacher model and student model at the semantic level. Experiments show that this method can effectively train a lightweight model close to the performance of the original SD model, and effectively improve the model speed under limited resources. Experiments show that the proposed method can effectively train a light-weight model close to the performance of the original SD model, and effectively improve the model speed under limited resources. After acceleration, the UNet part of the model is 22% faster and the overall speed is 19% faster.

cs.CV↗

arXiv2024

Teaching Text-to-Image Models to Communicate in Dialog

A picture is worth a thousand words, thus, it is crucial for conversational agents to understand, perceive, and effectively respond with pictures. However, we find that directly employing conventional image generation techniques is inadequate for conversational agents to produce image responses effectively. In this paper, we focus on the innovative dialog-to-image generation task, where the model synthesizes a high-resolution image aligned with the given dialog context as a response. To tackle this problem, we design a tailored fine-tuning approach on the top of state-of-the-art text-to-image generation models to fully exploit the structural and semantic features in dialog context during image generation. Concretely, we linearize the dialog context with specific indicators to maintain the dialog structure, and employ in-domain data to alleviate the style mismatch between dialog-to-image and conventional image generation tasks. Empirical results on PhotoChat and MMDialog Corpus show that our approach brings consistent and remarkable improvement with 3 state-of-the-art pre-trained text-to-image generation backbones.

cs.CL↗

arXiv2024

On Generalized Sub-Gaussian Canonical Processes and Their Applications

We obtain the tail probability of generalized sub-Gaussian canonical processes. It can be viewed as a variant of the Bernstein-type inequality in the i.i.d case, and we further get a tighter bound of concentration inequality through uniformly randomized techniques. A concentration inequality for general functions involving independent random variables is also derived as an extension. As for applications, we derive convergence results for principal component analysis and the Rademacher complexities method.

math.PR↗

arXiv2024

STAIR: Spatial-Temporal Reasoning with Auditable Intermediate Results for Video Question Answering

Recently we have witnessed the rapid development of video question answering models. However, most models can only handle simple videos in terms of temporal reasoning, and their performance tends to drop when answering temporal-reasoning questions on long and informative videos. To tackle this problem we propose STAIR, a Spatial-Temporal Reasoning model with Auditable Intermediate Results for video question answering. STAIR is a neural module network, which contains a program generator to decompose a given question into a hierarchical combination of several sub-tasks, and a set of lightweight neural modules to complete each of these sub-tasks. Though neural module networks are already widely studied on image-text tasks, applying them to videos is a non-trivial task, as reasoning on videos requires different abilities. In this paper, we define a set of basic video-text sub-tasks for video question answering and design a set of lightweight modules to complete them. Different from most prior works, modules of STAIR return intermediate outputs specific to their intentions instead of always returning attention maps, which makes it easier to interpret and collaborate with pre-trained models. We also introduce intermediate supervision to make these intermediate outputs more accurate. We conduct extensive experiments on several video question answering datasets under various settings to show STAIR's performance, explainability, compatibility with pre-trained models, and applicability when program annotations are not available. Code: https://github.com/yellow-binary-tree/STAIR

cs.CV↗

arXiv2024

Many-body higher-order topological invariant for $C_n$-symmetric insulators

Higher-order topological insulators in two spatial dimensions display fractional corner charges. While fractional charges in one dimension are known to be captured by a many-body bulk invariant, computed by the Resta formula, a many-body bulk invariant for higher-order topology and the corresponding fractional corner charges remains elusive despite several attempts. Inspired by recent work by Tada and Oshikawa, we propose a well-defined many-body bulk invariant for $C_n$ symmetric higher-order topological insulators, which is valid for both non-interacting and interacting systems. Instead of relating them to the bulk quadrupole moment as was previously done, we show that in the presence of $C_n$ rotational symmetry, this bulk invariant can be directly identified with quantized fractional corner charges. In particular, we prove that the corner charge is quantized as $e/n$ with $C_n$ symmetry, leading to a $\mathbb{Z}_n$ classification for higher-order topological insulators in two dimensions.

cond-mat.str-el↗

arXiv2023

InstructME: An Instruction Guided Music Edit And Remix Framework with Latent Diffusion Models

Music editing primarily entails the modification of instrument tracks or remixing in the whole, which offers a novel reinterpretation of the original piece through a series of operations. These music processing methods hold immense potential across various applications but demand substantial expertise. Prior methodologies, although effective for image and audio modifications, falter when directly applied to music. This is attributed to music's distinctive data nature, where such methods can inadvertently compromise the intrinsic harmony and coherence of music. In this paper, we develop InstructME, an Instruction guided Music Editing and remixing framework based on latent diffusion models. Our framework fortifies the U-Net with multi-scale aggregation in order to maintain consistency before and after editing. In addition, we introduce chord progression matrix as condition information and incorporate it in the semantic space to improve melodic harmony while editing. For accommodating extended musical pieces, InstructME employs a chunk transformer, enabling it to discern long-term temporal dependencies within music sequences. We tested InstructME in instrument-editing, remixing, and multi-round editing. Both subjective and objective evaluations indicate that our proposed method significantly surpasses preceding systems in music quality, text relevance and harmony. Demo samples are available at https://musicedit.github.io/

cs.SD↗

arXiv2023

Nodal topological superconductivity in nodal-line semimetals

We analyze possible nodal superconducting phases that emerge from a doped nodal-line semimetal. We show that nodal-line superconducting phases are favored by interactions mediated by short-range ferromagnetic fluctuations or Hund's coupling. It is found that the leading pairing channels are momentum-independent, orbital-singlet and spin-triplet. In the pairing state, we show that the Bogoliubov-de Gennes (BdG) Hamiltonian hosts a pair of topologically protected nodal rings on the equators of the torus Fermi surface (FS). Using a topological classification for gapless systems with inversion symmetry, we find that these nodal rings are topologically nontrivial and protected by integer-valued monopole charges $ν= \pm 2$. In the scenario of pairing driven by ferromagnetic fluctuations, we analyze the fate of superconductivity in the magnetically ordered phase. Based on Ginzburg-Landau free energy analysis, we find the energetically favored superconducting state is characterized by the coexistence of two pairing orders whose $\bf d$-vectors are perpendicular to the magnetization axis $\bf M$ with their phases unfixed. In this case, each nodal loop in the pairing state splits into two, carrying a $\pm 1$ monopole charge. For bulk-boundary correspondence, these nodal rings enclose flat-band Majorana zero modes on top and bottom surface Brillouin Zones with distinct $\mathbb{Z}$-valued topological invariants.

cond-mat.supr-con↗

arXiv2023

Audio Prompt Tuning for Universal Sound Separation

Universal sound separation (USS) is a task to separate arbitrary sounds from an audio mixture. Existing USS systems are capable of separating arbitrary sources, given a few examples of the target sources as queries. However, separating arbitrary sounds with a single system is challenging, and the robustness is not always guaranteed. In this work, we propose audio prompt tuning (APT), a simple yet effective approach to enhance existing USS systems. Specifically, APT improves the separation performance of specific sources through training a small number of prompt parameters with limited audio samples, while maintaining the generalization of the USS model by keeping its parameters frozen. We evaluate the proposed method on MUSDB18 and ESC-50 datasets. Compared with the baseline model, APT can improve the signal-to-distortion ratio performance by 0.67 dB and 2.06 dB using the full training set of two datasets. Moreover, APT with only 5 audio samples even outperforms the baseline systems utilizing full training data on the ESC-50 dataset, indicating the great potential of few-shot APT.

eess.AS↗

arXiv2023

A calculation method to estimate thermal conductivity of high entropy ceramic for thermal barrier coatings

High entropy ceramics are highly promising as next generation thermal barrier coatings due to their unique disorder structure, which imparts ultra-low thermal conductivity and good high temperature stability. Unlike traditional ceramic materials, the thermal resistance in high entropy ceramics predominantly arises from phonon-disorder scattering rather than phonon-phonon interactions. In this study, we propose a calculation method based on the supercell phonon unfolding (SPU) technique to predict the thermal conductivity of high entropy ceramics, specially focusing on rocksalt oxides structures. Our prediction method relies on using the reciprocal value of SPU phonon spectra linewidth as an indicator of phonon lifetime. The obtained results demonstrate a strong agreement between the predicted thermal conductivities and the experimental measurements, validating the feasibility of our calculation method. Furthermore, we extensively investigate and discuss the atomic relaxation and lattice distortion effects in 5-dopants and 6-dopants rocksalt structures during the process.

cond-mat.mtrl-sci↗

arXiv2023

LLaMA Rider: Spurring Large Language Models to Explore the Open World

Recently, various studies have leveraged Large Language Models (LLMs) to help decision-making and planning in environments, and try to align the LLMs' knowledge with the world conditions. Nonetheless, the capacity of LLMs to continuously acquire environmental knowledge and adapt in an open world remains uncertain. In this paper, we propose an approach to spur LLMs to explore the open world, gather experiences, and learn to improve their task-solving capabilities. In this approach, a multi-round feedback-revision mechanism is utilized to encourage LLMs to actively select appropriate revision actions guided by feedback information from the environment. This facilitates exploration and enhances the model's performance. Besides, we integrate sub-task relabeling to assist LLMs in maintaining consistency in sub-task planning and help the model learn the combinatorial nature between tasks, enabling it to complete a wider range of tasks through training based on the acquired exploration experiences. By evaluation in Minecraft, an open-ended sandbox world, we demonstrate that our approach LLaMA-Rider enhances the efficiency of the LLM in exploring the environment, and effectively improves the LLM's ability to accomplish more tasks through fine-tuning with merely 1.3k instances of collected data, showing minimal training costs compared to the baseline using reinforcement learning.

cs.LG↗