arXiv subjects

Yi Luo

Publications and source records attributed to Yi Luo.

At least 145 records · Page 8Linked to original sources

arXiv2020

Ultra-Lightweight Speech Separation via Group Communication

Model size and complexity remain the biggest challenges in the deployment of speech enhancement and separation systems on low-resource devices such as earphones and hearing aids. Although methods such as compression, distillation and quantization can be applied to large models, they often come with a cost on the model performance. In this paper, we provide a simple model design paradigm that explicitly designs ultra-lightweight models without sacrificing the performance. Motivated by the sub-band frequency-LSTM (F-LSTM) architectures, we introduce the group communication (GroupComm), where a feature vector is split into smaller groups and a small processing block is used to perform inter-group communication. Unlike standard F-LSTM models where the sub-band outputs are concatenated, an ultra-small module is applied on all the groups in parallel, which allows a significant decrease on the model size. Experiment results show that comparing with a strong baseline model which is already lightweight, GroupComm can achieve on par performance with 35.6 times fewer parameters and 2.3 times fewer operations.

eess.AS↗

arXiv2020

Rethinking the Separation Layers in Speech Separation Networks

Modules in all existing speech separation networks can be categorized into single-input-multi-output (SIMO) modules and single-input-single-output (SISO) modules. SIMO modules generate more outputs than input, and SISO modules keep the numbers of input and output the same. While the majority of separation models only contain SIMO architectures, it has also been shown that certain two-stage separation systems integrated with a post-enhancement SISO module can improve the separation quality. Why performance improvements can be achieved by incorporating the SISO modules? Are SIMO modules always necessary? In this paper, we empirically examine those questions by designing models with varying configurations in the SIMO and SISO modules. We show that comparing with the standard SIMO-only design, a mixed SIMO-SISO design with a same model size is able to improve the separation performance especially under low-overlap conditions. We further validate the necessity of SIMO modules and show that SISO-only models are still able to perform separation without sacrificing the performance. The observations allow us to rethink the model design paradigm and present different views on how the separation is performed.

eess.AS↗

arXiv2020

Implicit Filter-and-sum Network for Multi-channel Speech Separation

Various neural network architectures have been proposed in recent years for the task of multi-channel speech separation. Among them, the filter-and-sum network (FaSNet) performs end-to-end time-domain filter-and-sum beamforming and has shown effective in both ad-hoc and fixed microphone array geometries. In this paper, we investigate multiple ways to improve the performance of FaSNet. From the problem formulation perspective, we change the explicit time-domain filter-and-sum operation which involves all the microphones into an implicit filter-and-sum operation in the latent space of only the reference microphone. The filter-and-sum operation is applied on a context around the frame to be separated. This allows the problem formulation to better match the objective of end-to-end separation. From the feature extraction perspective, we modify the calculation of sample-level normalized cross correlation (NCC) features into feature-level NCC (fNCC) features. This makes the model better matches the implicit filter-and-sum formulation. Experiment results on both ad-hoc and fixed microphone array geometries show that the proposed modification to the FaSNet, which we refer to as iFaSNet, is able to significantly outperform the benchmark FaSNet across all conditions with an on par model complexity.

eess.AS↗

arXiv2020

Distortion-controlled Training for End-to-end Reverberant Speech Separation with Auxiliary Autoencoding Loss

The performance of speech enhancement and separation systems in anechoic environments has been significantly advanced with the recent progress in end-to-end neural network architectures. However, the performance of such systems in reverberant environments is yet to be explored. A core problem in reverberant speech separation is about the training and evaluation metrics. Standard time-domain metrics may introduce unexpected distortions during training and fail to properly evaluate the separation performance due to the presence of the reverberations. In this paper, we first introduce the "equal-valued contour" problem in reverberant separation where multiple outputs can lead to the same performance measured by the common metrics. We then investigate how "better" outputs with lower target-specific distortions can be selected by auxiliary autoencoding training (A2T). A2T assumes that the separation is done by a linear operation on the mixture signal, and it adds an loss term on the autoencoding of the direct-path target signals to ensure that the distortion introduced on the direct-path signals is controlled during separation. Evaluations on separation signal quality and speech recognition accuracy show that A2T is able to control the distortion on the direct-path signals and improve the recognition accuracy.

eess.AS↗

arXiv2020

Integration of speech separation, diarization, and recognition for multi-speaker meetings: System description, comparison, and analysis

Multi-speaker speech recognition of unsegmented recordings has diverse applications such as meeting transcription and automatic subtitle generation. With technical advances in systems dealing with speech separation, speaker diarization, and automatic speech recognition (ASR) in the last decade, it has become possible to build pipelines that achieve reasonable error rates on this task. In this paper, we propose an end-to-end modular system for the LibriCSS meeting data, which combines independently trained separation, diarization, and recognition components, in that order. We study the effect of different state-of-the-art methods at each stage of the pipeline, and report results using task-specific metrics like SDR and DER, as well as downstream WER. Experiments indicate that the problem of overlapping speech for diarization and ASR can be effectively mitigated with the presence of a well-trained separation module. Our best system achieves a speaker-attributed WER of 12.7%, which is close to that of a non-overlapping ASR.

eess.AS↗

arXiv2020

Soliton dynamics in microresonators with XPM induced negative thermo-optic effect

Optical frequency comb generation in microresonators has attracted significant attention over the past decade, as it offers the promising potential for chip-scale optical frequency synthesis, optical clocks and precise optical spectroscopy. However, accessing temporal dissipative Kerr soliton (DKSs) is known to be severely hampered by thermal effects. Furthermore, due to the degeneracy of soliton existence range with respect to soliton number, deterministically accessing single soliton state is another challenge. Here, we demonstrate stable and deterministic single soliton generation in AlN-on-sapphire platform via auxiliary laser pumping scheme without the requirement of fast control of the pump power and detuning. Moreover, we reveal the underlying physics of soliton switching in a dual-pumped microcomb, which is fully described by the Lugiato - Lefever equation. The switching process is attributed to cross-phase modulation (XPM) induced degeneracy lifting of the soliton existence range, corresponding to an effective negative thermo-optic effect.

physics.optics↗

arXiv2020

Integrated Gallium Nitride Nonlinear Photonics

Gallium nitride (GaN) as a wide-band gap material has been widely used in solid-state lighting. Thanks to its high nonlinearity and high refractive index contrast, GaN-on-insulator (GaNOI) is also a promising platform for nonlinear optical applications. Despite its intriguing optical proprieties, nonlinear applications of GaN have rarely been studied due to the relatively high optical loss of GaN waveguides (2 dB/cm). In this letter, we report GaNOI microresonator with intrinsic quality factor over 2 million, corresponding to an optical loss of 0.26 dB/cm. Parametric oscillation threshold power as low as 8.8 mW is demonstrated, and the experimentally extracted nonlinear index of GaN at telecom wavelengths is estimated to be n2 = 1.2*10 -18 m2W-1, which is comparable with silicon. Single soliton generation in GaN is implemented by an auxiliary laser pumping scheme, so as to mitigate the high thermorefractive effect in GaN. The large intrinsic nonlinear refractive index, together with its broadband transparency window and high refractive index contrast, make GaNOI a most balanced platform for chip-scale nonlinear applications.

physics.optics↗

arXiv2020

Low Energy Magneto-optics of Tb$_{2}$Ti$_{2}$O$_{7}$ in [111] Magnetic Field

The pyrochlore magnet Tb$_{2}$Ti$_{2}$O$_{7}$ shows a lack of magnetic order to low temperatures and is considered to be a quantum spin liquid candidate. We perform time-domain THz spectroscopy on high quality Tb$_{2}$Ti$_{2}$O$_{7}$ crystal and study the low energy excitations as a function of [111] magnetic field with high energy resolution. The low energy crystal field excitations change their energies anomalously under magnetic field. Despite several sharp field dependent changes, we show that the material's spectrum can be described not by a phase transitions, but by field dependent hybridization between the low energy crystal field levels. We highlight the strong coupling between spin and lattice degrees of freedom in Tb$_{2}$Ti$_{2}$O$_{7}$ as evidenced by the magnetic field tunable crystal field environment. Calculations based on single ion physics with field induced symmetry reduction of the crystal field environment can reproduce our data.

cond-mat.str-el↗

arXiv2020

An End-to-end Architecture of Online Multi-channel Speech Separation

Multi-speaker speech recognition has been one of the keychallenges in conversation transcription as it breaks the singleactive speaker assumption employed by most state-of-the-artspeech recognition systems. Speech separation is consideredas a remedy to this problem. Previously, we introduced a sys-tem, calledunmixing,fixed-beamformerandextraction(UFE),that was shown to be effective in addressing the speech over-lap problem in conversation transcription. With UFE, an inputmixed signal is processed by fixed beamformers, followed by aneural network post filtering. Although promising results wereobtained, the system contains multiple individually developedmodules, leading potentially sub-optimum performance. In thiswork, we introduce an end-to-end modeling version of UFE. Toenable gradient propagation all the way, an attentional selectionmodule is proposed, where an attentional weight is learnt foreach beamformer and spatial feature sampled over space. Ex-perimental results show that the proposed system achieves com-parable performance in an offline evaluation with the originalseparate processing-based pipeline, while producing remark-able improvements in an online evaluation.

eess.AS↗

arXiv2020

Separating Varying Numbers of Sources with Auxiliary Autoencoding Loss

Many recent source separation systems are designed to separate a fixed number of sources out of a mixture. In the cases where the source activation patterns are unknown, such systems have to either adjust the number of outputs or to identify invalid outputs from the valid ones. Iterative separation methods have gain much attention in the community as they can flexibly decide the number of outputs, however (1) they typically rely on long-term information to determine the stopping time for the iterations, which makes them hard to operate in a causal setting; (2) they lack a "fault tolerance" mechanism when the estimated number of sources is different from the actual number. In this paper, we propose a simple training method, the auxiliary autoencoding permutation invariant training (A2PIT), to alleviate the two issues. A2PIT assumes a fixed number of outputs and uses auxiliary autoencoding loss to force the invalid outputs to be the copies of the input mixture, and detects invalid outputs in a fully unsupervised way during inference phase. Experiment results show that A2PIT is able to improve the separation performance across various numbers of speakers and effectively detect the number of speakers in a mixture.

eess.AS↗

arXiv2020

Deep learning-based holographic polarization microscopy

Polarized light microscopy provides high contrast to birefringent specimen and is widely used as a diagnostic tool in pathology. However, polarization microscopy systems typically operate by analyzing images collected from two or more light paths in different states of polarization, which lead to relatively complex optical designs, high system costs or experienced technicians being required. Here, we present a deep learning-based holographic polarization microscope that is capable of obtaining quantitative birefringence retardance and orientation information of specimen from a phase recovered hologram, while only requiring the addition of one polarizer/analyzer pair to an existing holographic imaging system. Using a deep neural network, the reconstructed holographic images from a single state of polarization can be transformed into images equivalent to those captured using a single-shot computational polarized light microscope (SCPLM). Our analysis shows that a trained deep neural network can extract the birefringence information using both the sample specific morphological features as well as the holographic amplitude and phase distribution. To demonstrate the efficacy of this method, we tested it by imaging various birefringent samples including e.g., monosodium urate (MSU) and triamcinolone acetonide (TCA) crystals. Our method achieves similar results to SCPLM both qualitatively and quantitatively, and due to its simpler optical design and significantly larger field-of-view, this method has the potential to expand the access to polarization microscopy and its use for medical diagnosis in resource limited settings.

physics.optics↗

arXiv2020

Continuous speech separation: dataset and analysis

This paper describes a dataset and protocols for evaluating continuous speech separation algorithms. Most prior studies on speech separation use pre-segmented signals of artificially mixed speech utterances which are mostly \emph{fully} overlapped, and the algorithms are evaluated based on signal-to-distortion ratio or similar performance metrics. However, in natural conversations, a speech signal is continuous, containing both overlapped and overlap-free components. In addition, the signal-based metrics have very weak correlations with automatic speech recognition (ASR) accuracy. We think that not only does this make it hard to assess the practical relevance of the tested algorithms, it also hinders researchers from developing systems that can be readily applied to real scenarios. In this paper, we define continuous speech separation (CSS) as a task of generating a set of non-overlapped speech signals from a \textit{continuous} audio stream that contains multiple utterances that are \emph{partially} overlapped by a varying degree. A new real recorded dataset, called LibriCSS, is derived from LibriSpeech by concatenating the corpus utterances to simulate a conversation and capturing the audio replays with far-field microphones. A Kaldi-based ASR evaluation protocol is also established by using a well-trained multi-conditional acoustic model. By using this dataset, several aspects of a recently proposed speaker-independent CSS algorithm are investigated. The dataset and evaluation scripts are available to facilitate the research in this direction.

cs.SD↗

arXiv2020

Low energy magnons in the chiral ferrimagnet $\text{Cu}_2\text{OSeO}_3$: a coarse-grained approach

We report a comprehensive neutron scattering study of low energy magnetic excitations in the breathing pyrochlore helimagnetic $\text{Cu}_2\text{OSeO}_3$. Fully documenting the four lowest energy magnetic modes that leave the ferrimagnetic configuration of the "strong tetrahedra" intact ($|\hbarω|<13$ meV), we find gapless quadratic dispersion at the $Γ$ point for energies above 0.2 meV, two doublets separated by 1.6(2) meV at the $R$ point, and a bounded continuum at the $X$ point. Our constrained rigid spin cluster model relates these features to Dzyaloshinskii-Moriya (DM) interactions and the incommensurate helical ground state. Combining conventional spin wave theory with a spin cluster form-factor accurately reproduces the measured equal time structure factor through multiple Brillouin zones. An effective spin Hamiltonian describing the complex anisotropic inter-cluster interactions is obtained.

cond-mat.str-el↗

arXiv2020

Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation

Recent studies in deep learning-based speech separation have proven the superiority of time-domain approaches to conventional time-frequency-based methods. Unlike the time-frequency domain approaches, the time-domain separation systems often receive input sequences consisting of a huge number of time steps, which introduces challenges for modeling extremely long sequences. Conventional recurrent neural networks (RNNs) are not effective for modeling such long sequences due to optimization difficulties, while one-dimensional convolutional neural networks (1-D CNNs) cannot perform utterance-level sequence modeling when its receptive field is smaller than the sequence length. In this paper, we propose dual-path recurrent neural network (DPRNN), a simple yet effective method for organizing RNN layers in a deep structure to model extremely long sequences. DPRNN splits the long sequential input into smaller chunks and applies intra- and inter-chunk operations iteratively, where the input length can be made proportional to the square root of the original sequence length in each operation. Experiments show that by replacing 1-D CNN with DPRNN and apply sample-level modeling in the time-domain audio separation network (TasNet), a new state-of-the-art performance on WSJ0-2mix is achieved with a 20 times smaller model than the previous best system.

eess.AS↗

arXiv2020

End-to-end Microphone Permutation and Number Invariant Multi-channel Speech Separation

An important problem in ad-hoc microphone speech separation is how to guarantee the robustness of a system with respect to the locations and numbers of microphones. The former requires the system to be invariant to different indexing of the microphones with the same locations, while the latter requires the system to be able to process inputs with varying dimensions. Conventional optimization-based beamforming techniques satisfy these requirements by definition, while for deep learning-based end-to-end systems those constraints are not fully addressed. In this paper, we propose transform-average-concatenate (TAC), a simple design paradigm for channel permutation and number invariant multi-channel speech separation. Based on the filter-and-sum network (FaSNet), a recently proposed end-to-end time-domain beamforming system, we show how TAC significantly improves the separation performance across various numbers of microphones in noisy reverberant separation tasks with ad-hoc arrays. Moreover, we show that TAC also significantly improves the separation performance with fixed geometry array configuration, further proving the effectiveness of the proposed paradigm in the general problem of multi-microphone speech separation.

eess.AS↗

arXiv2020

Real-time binaural speech separation with preserved spatial cues

Deep learning speech separation algorithms have achieved great success in improving the quality and intelligibility of separated speech from mixed audio. Most previous methods focused on generating a single-channel output for each of the target speakers, hence discarding the spatial cues needed for the localization of sound sources in space. However, preserving the spatial information is important in many applications that aim to accurately render the acoustic scene such as in hearing aids and augmented reality (AR). Here, we propose a speech separation algorithm that preserves the interaural cues of separated sound sources and can be implemented with low latency and high fidelity, therefore enabling a real-time modification of the acoustic scene. Based on the time-domain audio separation network (TasNet), a single-channel time-domain speech separation system that can be implemented in real-time, we propose a multi-input-multi-output (MIMO) end-to-end extension of TasNet that takes binaural mixed audio as input and simultaneously separates target speakers in both channels. Experimental results show that the proposed end-to-end MIMO system is able to significantly improve the separation performance and keep the perceived location of the modified sources intact in various acoustic scenes.

eess.AS↗

arXiv2019

Fragmentation and isomerization of polycyclic aromatic hydrocarbons in the interstellar medium: coronene as a case study

Aims. Due to the limitations of current computational technology, the fragmentation and isomerization products of vibrationally-excited polycyclic aromatic hydrocarbon (PAH) molecules and their derivatives are poorly studied. In this work, we investigate the intermediate products of PAHs and their derivatives as well as the gas-phase reactions relevant to the interstellar medium, with coronene as a case study. Methods. Based on the semi-empirical method of PM3 as implemented in the CP2K program, molecular dynamics simulations are performed to model the major processes (e.g., vibrations, fragmentations, and isomerizations) of coronene and its derivatives (e.g., methylated coronene, hydrogenated coronene, dehydrogenated coronene, nitrogen-substituted coronene, and oxygen-substituted coronene) at temperatures of 3000 K and 4000 K. Results. We find that the anharmonic effects are crucial for the simulation of vibrational excitation. For the molecules studied here, H2, CO, HCN, and CH2 are the major fragments. Following the dissociation of these small units, most of the molecules could maintain their ring structures, but a few molecules would break completely into carbon chains. The transformation from hexagon to pentagon or heptagon may occur and the heteroatomic substitutions (e.g., N- or O-substitutions) facilitate the transformation.

astro-ph.GA↗

arXiv2019

Visually Constructing the Chemical Structure of a Single Molecule by Scanning Raman Picoscopy

The strong spatial confinement of a nanocavity plasmonic field has made it possible to visualize the inner structure of a single molecule and even to distinguish its vibrational modes in real space. With such ever-improved spatial resolution, it is anticipated that full vibrational imaging of a molecule could be achieved to reveal molecular structural details. Here we demonstrate full Raman images of individual vibrational modes on the Ångström level for a single Mg-porphine molecule, revealing distinct characteristics of each vibrational mode in real space. Furthermore, by exploiting the underlying interference effect and Raman fingerprint database, we propose a new methodology for structural determination, coined as scanning Raman picoscopy, to show how such ultrahigh-resolution spectromicroscopic vibrational images can be used to visually assemble the chemical structure of a single molecule through a simple Lego-like building process.

physics.chem-ph↗