arXiv subjects

Zhichao Wang

Publications and source records attributed to Zhichao Wang.

At least 109 records · Page 6Linked to original sources

arXiv2022

Expressive-VC: Highly Expressive Voice Conversion with Attention Fusion of Bottleneck and Perturbation Features

Voice conversion for highly expressive speech is challenging. Current approaches struggle with the balancing between speaker similarity, intelligibility and expressiveness. To address this problem, we propose Expressive-VC, a novel end-to-end voice conversion framework that leverages advantages from both neural bottleneck feature (BNF) approach and information perturbation approach. Specifically, we use a BNF encoder and a Perturbed-Wav encoder to form a content extractor to learn linguistic and para-linguistic features respectively, where BNFs come from a robust pre-trained ASR model and the perturbed wave becomes speaker-irrelevant after signal perturbation. We further fuse the linguistic and para-linguistic features through an attention mechanism, where speaker-dependent prosody features are adopted as the attention query, which result from a prosody encoder with target speaker embedding and normalized pitch and energy of source speech as input. Finally the decoder consumes the integrated features and the speaker-dependent prosody feature to generate the converted speech. Experiments demonstrate that Expressive-VC is superior to several state-of-the-art systems, achieving both high expressiveness captured from the source speech and high speaker similarity with the target speaker; meanwhile intelligibility is well maintained.

eess.AS↗

arXiv2022

AccentSpeech: Learning Accent from Crowd-sourced Data for Target Speaker TTS with Accents

Learning accent from crowd-sourced data is a feasible way to achieve a target speaker TTS system that can synthesize accent speech. To this end, there are two challenging problems to be solved. First, direct use of the poor acoustic quality crowd-sourced data and the target speaker data in accent transfer will apparently lead to synthetic speech with degraded quality. To mitigate this problem, we take a bottleneck feature (BN) based TTS approach, in which TTS is decomposed into a Text-to-BN (T2BN) module to learn accent and a BN-to-Mel (BN2Mel) module to learn speaker timbre, where neural network based BN feature serves as the intermediate representation that are robust to noise interference. Second, direct training T2BN using the crowd-sourced data in the two-stage system will produce accent speech of target speaker with poor prosody. This is because the the crowd-sourced recordings are contributed from the ordinary unprofessional speakers. To tackle this problem, we update the two-stage approach to a novel three-stage approach, where T2BN and BN2Mel are trained using the high-quality target speaker data and a new BN-to-BN module is plugged in between the two modules to perform accent transfer. To train the BN2BN module, the parallel unaccented and accented BN features are obtained by a proposed data augmentation procedure. Finally the proposed three-stage approach manages to produce accent speech for the target speaker with good prosody, as the prosody pattern is inherited from the professional target speaker and accent transfer is achieved by the BN2BN module at the same time. The proposed approach, named as AccentSpeech, is validated in a Mandarin TTS accent transfer task.

cs.SD↗

arXiv2022

Streaming Voice Conversion Via Intermediate Bottleneck Features And Non-streaming Teacher Guidance

Streaming voice conversion (VC) is the task of converting the voice of one person to another in real-time. Previous streaming VC methods use phonetic posteriorgrams (PPGs) extracted from automatic speech recognition (ASR) systems to represent speaker-independent information. However, PPGs lack the prosody and vocalization information of the source speaker, and streaming PPGs contain undesired leaked timbre of the source speaker. In this paper, we propose to use intermediate bottleneck features (IBFs) to replace PPGs. VC systems trained with IBFs retain more prosody and vocalization information of the source speaker. Furthermore, we propose a non-streaming teacher guidance (TG) framework that addresses the timbre leakage problem. Experiments show that our proposed IBFs and the TG framework achieve a state-of-the-art streaming VC naturalness of 3.85, a content consistency of 3.77, and a timbre similarity of 3.77 under a future receptive field of 160 ms which significantly outperform previous streaming VC systems.

eess.AS↗

arXiv2022

Detecting the oscillation and propagation of the nascent dynamic solar wind structure at 2.6 solar radii using VLBI radio telescopes

Probing the solar corona is crucial to study the coronal heating and solar wind acceleration. However, the transient and inhomogeneous solar wind flows carry large-amplitude inherent Alfven waves and turbulence, which make detection more difficult. We report the oscillation and propagation of the solar wind at 2.6 solar radii (Rs) by observation of China Tianwen and ESA Mars Express with radio telescopes. The observations were carried out on Oct.9 2021, when one coronal mass ejection (CME) passed across the ray paths of the telescope beams. We obtain the frequency fluctuations (FF) of the spacecraft signals from each individual telescope. Firstly, we visually identify the drift of the frequency spikes at a high spatial resolution of thousands of kilometers along the projected baselines. They are used as traces to estimate the solar wind velocity. Then we perform the cross-correlation analysis on the time series of FF from different telescopes. The velocity variations of solar wind structure along radial and tangential directions during the CME passage are obtained. The oscillation of tangential velocity confirms the detection of streamer wave. Moreover, at the tail of the CME, we detect the propagation of an accelerating fast field-aligned density structure indicating the presence of magnetohydrodynamic waves. This study confirm that the ground station-pairs are able to form particular spatial projection baselines with high resolution and sensitivity to study the detailed propagation of the nascent dynamic solar wind structure.

astro-ph.SR↗

arXiv2022

An Improved Structured Mesh Generation Method Based on Physics-informed Neural Networks

Mesh generation remains a key technology in many areas where numerical simulations are required. As numerical algorithms become more efficient and computers become more powerful, the percentage of time devoted to mesh generation becomes higher. In this paper, we present an improved structured mesh generation method. The method formulates the meshing problem as a global optimization problem related to a physics-informed neural network. The mesh is obtained by intelligently solving the physical boundary-constrained partial differential equations. To improve the prediction accuracy of the neural network, we also introduce a novel auxiliary line strategy and an efficient network model during meshing. The strategy first employs a priori auxiliary lines to provide ground truth data and then uses these data to construct a loss term to better constrain the convergence of the subsequent training. The experimental results indicate that the proposed method is effective and robust. It can accurately approximate the mapping (transformation) from the computational domain to the physical domain and enable fast high-quality structured mesh generation.

cs.GR↗

arXiv2022

Cross-speaker Emotion Transfer Based On Prosody Compensation for End-to-End Speech Synthesis

Cross-speaker emotion transfer speech synthesis aims to synthesize emotional speech for a target speaker by transferring the emotion from reference speech recorded by another (source) speaker. In this task, extracting speaker-independent emotion embedding from reference speech plays an important role. However, the emotional information conveyed by such emotion embedding tends to be weakened in the process to squeeze out the source speaker's timbre information. In response to this problem, a prosody compensation module (PCM) is proposed in this paper to compensate for the emotional information loss. Specifically, the PCM tries to obtain speaker-independent emotional information from the intermediate feature of a pre-trained ASR model. To this end, a prosody compensation encoder with global context (GC) blocks is introduced to obtain global emotional information from the ASR model's intermediate feature. Experiments demonstrate that the proposed PCM can effectively compensate the emotion embedding for the emotional information loss, and meanwhile maintain the timbre of the target speaker. Comparisons with state-of-the-art models show that our proposed method presents obvious superiority on the cross-speaker emotion transfer task.

cs.SD↗

arXiv2022

Large cavitation bubbles in the tube with a conical-frustum shaped closed end during a transient process

The transient process accompanied by extreme acceleration in the conical sections of hydraulic systems (e.g., draft tube, diffuser) can induce large cavitation bubbles both at the closed ends and in the bulk liquid. The collapses of the large cavitation bubbles can cause severe damage to the solid walls. We conduct experiments in the tubes with different conical-frustum shaped closed ends with the `tube-arrest' method and observe bubbles generated at these two locations. For the bubbles generated at the close end of the tube, we propose the onset criteria, consisting of two universal non-dimensional parameters $Ca_1$ and $Ca_2$, of large cavitation bubbles separating the water column. We investigate their dynamics including the collapse time and speed. The results indicate that the larger the conical angle, the faster the bubbles collapse. For the bubbles generated in the bulk liquid, we numerically study the collapse time, the jet characteristics and the pressure pulse at bubble collapse. We observe a much stronger jet and pressure pulse of bubbles in tubes, comparing with a bubble near an infinite plate. Our results can provide guidance in the design and safe operation of hydraulic machinery with complex geometries, considering the cavitation during the transient process.

physics.flu-dyn↗

arXiv2022

TorchNTK: A Library for Calculation of Neural Tangent Kernels of PyTorch Models

We introduce torchNTK, a python library to calculate the empirical neural tangent kernel (NTK) of neural network models in the PyTorch framework. We provide an efficient method to calculate the NTK of multilayer perceptrons. We compare the explicit differentiation implementation against autodifferentiation implementations, which have the benefit of extending the utility of the library to any architecture supported by PyTorch, such as convolutional networks. A feature of the library is that we expose the user to layerwise NTK components, and show that in some regimes a layerwise calculation is more memory efficient. We conduct preliminary experiments to demonstrate use cases for the software and probe the NTK.

cs.LG↗

arXiv2022

High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representation

We study the first gradient descent step on the first-layer parameters $\boldsymbol{W}$ in a two-layer neural network: $f(\boldsymbol{x}) = \frac{1}{\sqrt{N}}\boldsymbol{a}^\topσ(\boldsymbol{W}^\top\boldsymbol{x})$, where $\boldsymbol{W}\in\mathbb{R}^{d\times N}, \boldsymbol{a}\in\mathbb{R}^{N}$ are randomly initialized, and the training objective is the empirical MSE loss: $\frac{1}{n}\sum_{i=1}^n (f(\boldsymbol{x}_i)-y_i)^2$. In the proportional asymptotic limit where $n,d,N\to\infty$ at the same rate, and an idealized student-teacher setting, we show that the first gradient update contains a rank-1 "spike", which results in an alignment between the first-layer weights and the linear component of the teacher model $f^*$. To characterize the impact of this alignment, we compute the prediction risk of ridge regression on the conjugate kernel after one gradient step on $\boldsymbol{W}$ with learning rate $η$, when $f^*$ is a single-index model. We consider two scalings of the first step learning rate $η$. For small $η$, we establish a Gaussian equivalence property for the trained feature map, and prove that the learned kernel improves upon the initial random features model, but cannot defeat the best linear model on the input. Whereas for sufficiently large $η$, we prove that for certain $f^*$, the same ridge estimator on trained features can go beyond this "linear regime" and outperform a wide range of random features and rotationally invariant kernels. Our results demonstrate that even one gradient step can lead to a considerable advantage over random features, and highlight the role of learning rate scaling in the initial phase of training.

stat.ML↗

arXiv2022

Cross-speaker emotion disentangling and transfer for end-to-end speech synthesis

The cross-speaker emotion transfer task in text-to-speech (TTS) synthesis particularly aims to synthesize speech for a target speaker with the emotion transferred from reference speech recorded by another (source) speaker. During the emotion transfer process, the identity information of the source speaker could also affect the synthesized results, resulting in the issue of speaker leakage. This paper proposes a new method with the aim to synthesize controllable emotional expressive speech and meanwhile maintain the target speaker's identity in the cross-speaker emotion TTS task. The proposed method is a Tacotron2-based framework with emotion embedding as the conditioning variable to provide emotion information. Two emotion disentangling modules are contained in our method to 1) get speaker-irrelevant and emotion-discriminative embedding, and 2) explicitly constrain the emotion and speaker identity of synthetic speech to be that as expected. Moreover, we present an intuitive method to control the emotion strength in the synthetic speech for the target speaker. Specifically, the learned emotion embedding is adjusted with a flexible scalar value, which allows controlling the emotion strength conveyed by the embedding. Extensive experiments have been conducted on a Mandarin disjoint corpus, and the results demonstrate that the proposed method is able to synthesize reasonable emotional speech for the target speaker. Compared to the state-of-the-art reference embedding learned methods, our method gets the best performance on the cross-speaker emotion transfer task, indicating that our method achieves the new state-of-the-art performance on learning the speaker-irrelevant emotion embedding. Furthermore, the strength ranking test and pitch trajectories plots demonstrate that the proposed method can effectively control the emotion strength, leading to prosody-diverse synthetic speech.

cs.SD↗

arXiv2022

One-shot Voice Conversion For Style Transfer Based On Speaker Adaptation

One-shot style transfer is a challenging task, since training on one utterance makes model extremely easy to over-fit to training data and causes low speaker similarity and lack of expressiveness. In this paper, we build on the recognition-synthesis framework and propose a one-shot voice conversion approach for style transfer based on speaker adaptation. First, a speaker normalization module is adopted to remove speaker-related information in bottleneck features extracted by ASR. Second, we adopt weight regularization in the adaptation process to prevent over-fitting caused by using only one utterance from target speaker as training data. Finally, to comprehensively decouple the speech factors, i.e., content, speaker, style, and transfer source style to the target, a prosody module is used to extract prosody representation. Experiments show that our approach is superior to the state-of-the-art one-shot VC systems in terms of style and speaker similarity; additionally, our approach also maintains good speech quality.

eess.AS↗

arXiv2022

Causal Disentanglement for Semantics-Aware Intent Learning in Recommendation

Traditional recommendation models trained on observational interaction data have generated large impacts in a wide range of applications, it faces bias problems that cover users' true intent and thus deteriorate the recommendation effectiveness. Existing methods tracks this problem as eliminating bias for the robust recommendation, e.g., by re-weighting training samples or learning disentangled representation. The disentangled representation methods as the state-of-the-art eliminate bias through revealing cause-effect of the bias generation. However, how to design the semantics-aware and unbiased representation for users true intents is largely unexplored. To bridge the gap, we are the first to propose an unbiased and semantics-aware disentanglement learning called CaDSI (Causal Disentanglement for Semantics-Aware Intent Learning) from a causal perspective. Particularly, CaDSI explicitly models the causal relations underlying recommendation task, and thus produces semantics-aware representations via disentangling users true intents aware of specific item context. Moreover, the causal intervention mechanism is designed to eliminate confounding bias stemmed from context information, which further to align the semantics-aware representation with users true intent. Extensive experiments and case studies both validate the robustness and interpretability of our proposed model.

cs.IR↗

arXiv2022

Min-max minimal hypersurfaces with higher multiplicity

We exhibit the first set of examples of non-bumpy metrics on the $(n+1)$-sphere ($2\leq n\leq 6$) in which the varifold associated with the two-parameter min-max construction must be a multiplicity-two minimal $n$-sphere. This is proved by a new area-and-separation estimate for certain minimal hypersurfaces with Morse index two inspired by an early work of Colding-Minicozzi. We also construct non-bumpy projective spaces in which the first min-max hypersurfaces are one-sided, and non-bumpy balls in which the free boundary min-max hypersurfaces are improper.

math.DG↗

arXiv2022

IQDUBBING: Prosody modeling based on discrete self-supervised speech representation for expressive voice conversion

Prosody modeling is important, but still challenging in expressive voice conversion. As prosody is difficult to model, and other factors, e.g., speaker, environment and content, which are entangled with prosody in speech, should be removed in prosody modeling. In this paper, we present IQDubbing to solve this problem for expressive voice conversion. To model prosody, we leverage the recent advances in discrete self-supervised speech representation (DSSR). Specifically, prosody vector is first extracted from pre-trained VQ-Wav2Vec model, where rich prosody information is embedded while most speaker and environment information are removed effectively by quantization. To further filter out the redundant information except prosody, such as content and partial speaker information, we propose two kinds of prosody filters to sample prosody from the prosody vector. Experiments show that IQDubbing is superior to baseline and comparison systems in terms of speech quality while maintaining prosody consistency and speaker similarity.

eess.AS↗

arXiv2021

Deep Treatment-Adaptive Network for Causal Inference

Causal inference is capable of estimating the treatment effect (i.e., the causal effect of treatment on the outcome) to benefit the decision making in various domains. One fundamental challenge in this research is that the treatment assignment bias in observational data. To increase the validity of observational studies on causal inference, representation based methods as the state-of-the-art have demonstrated the superior performance of treatment effect estimation. Most representation based methods assume all observed covariates are pre-treatment (i.e., not affected by the treatment), and learn a balanced representation from these observed covariates for estimating treatment effect. Unfortunately, this assumption is often too strict a requirement in practice, as some covariates are changed by doing an intervention on treatment (i.e., post-treatment). By contrast, the balanced representation learned from unchanged covariates thus biases the treatment effect estimation.

cs.LG↗

arXiv2021

Multi-speaker Multi-style Text-to-speech Synthesis With Single-speaker Single-style Training Data Scenarios

In the existing cross-speaker style transfer task, a source speaker with multi-style recordings is necessary to provide the style for a target speaker. However, it is hard for one speaker to express all expected styles. In this paper, a more general task, which is to produce expressive speech by combining any styles and timbres from a multi-speaker corpus in which each speaker has a unique style, is proposed. To realize this task, a novel method is proposed. This method is a Tacotron2-based framework but with a fine-grained text-based prosody predicting module and a speaker identity controller. Experiments demonstrate that the proposed method can successfully express a style of one speaker with the timber of another speaker bypassing the dependency on a single speaker's multi-style corpus. Moreover, the explicit prosody features used in the prosody predicting module can increase the diversity of synthetic speech by adjusting the value of prosody features.

eess.AS↗

arXiv2021

Uryson width of three dimensional mean convex domain with non-negative Ricci curvature

We prove that for every three dimensional manifold with nonnegative Ricci curvature and strictly mean convex boundary, there exists a Morse function so that each connected component of its level sets has a uniform diameter bound, which depends only on the lower bound of mean curvature. This gives an upper bound of Uryson 1-width for those three manifolds with boundary.

math.DG↗

arXiv2021

Existence of infinitely many free boundary minimal hypersurfaces

In this paper, we prove that in any compact Riemannian manifold with smooth boundary, of dimension at least 3 and at most 7, there exist infinitely many almost properly embedded free boundary minimal hypersurfaces. This settles the free boundary version of Yau's conjecture. The proof uses adaptions of A. Song's work and the early works by Marques-Neves in their resolution to Yau's conjecture, together with Li-Zhou's regularity theorem for free boundary min-max minimal hypersurfaces.

math.DG↗