arXiv subjects

Li Yang

Publications and source records attributed to Li Yang.

At least 235 records · Page 13Linked to original sources

arXiv2021

Big Bird: Transformers for Longer Sequences

Transformers-based models, such as BERT, have been one of the most successful deep learning models for NLP. Unfortunately, one of their core limitations is the quadratic dependency (mainly in terms of memory) on the sequence length due to their full attention mechanism. To remedy this, we propose, BigBird, a sparse attention mechanism that reduces this quadratic dependency to linear. We show that BigBird is a universal approximator of sequence functions and is Turing complete, thereby preserving these properties of the quadratic, full attention model. Along the way, our theoretical analysis reveals some of the benefits of having $O(1)$ global tokens (such as CLS), that attend to the entire sequence as part of the sparse attention mechanism. The proposed sparse attention can handle sequences of length up to 8x of what was previously possible using similar hardware. As a consequence of the capability to handle longer context, BigBird drastically improves performance on various NLP tasks such as question answering and summarization. We also propose novel applications to genomics data.

cs.LG↗

arXiv2020

MetaGater: Fast Learning of Conditional Channel Gated Networks via Federated Meta-Learning

While deep learning has achieved phenomenal successes in many AI applications, its enormous model size and intensive computation requirements pose a formidable challenge to the deployment in resource-limited nodes. There has recently been an increasing interest in computationally-efficient learning methods, e.g., quantization, pruning and channel gating. However, most existing techniques cannot adapt to different tasks quickly. In this work, we advocate a holistic approach to jointly train the backbone network and the channel gating which enables dynamical selection of a subset of filters for more efficient local computation given the data input. Particularly, we develop a federated meta-learning approach to jointly learn good meta-initializations for both backbone networks and gating modules, by making use of the model similarity across learning tasks on different nodes. In this way, the learnt meta-gating module effectively captures the important filters of a good meta-backbone network, based on which a task-specific conditional channel gated network can be quickly adapted, i.e., through one-step gradient descent, from the meta-initializations in a two-stage procedure using new samples of that task. The convergence of the proposed federated meta-learning algorithm is established under mild conditions. Experimental results corroborate the effectiveness of our method in comparison to related work.

cs.LG↗

arXiv2020

Scalable variational Monte Carlo with graph neural ansatz

Deep neural networks have been shown as a potentially powerful ansatz in variational Monte Carlo for solving quantum many-body problems. We propose two improvements in this direction. The first is graph neural ansatz (GNA), which is a variational wavefunction universal to arbitrary geometry. GNA results in accurate ground-state energies on 2D Kagome lattices, triangular lattices, and randomly connected graphs. Secondly, we design a distributed workflow on multiple accelerators to scale up the computation. We compute Kagome lattices with sizes up to 432 sites on 128 TPU cores. The parameter sharing nature of the GNA also leads to transferability across different system sizes and geometries.

cond-mat.str-el↗

arXiv2020

Averaging principle and normal deviations for multi-scale stochastic hyperbolic-parabolic equations

We study the asymptotic behavior of stochastic hyperbolic parabolic equations with slow and fast time scales. Both the strong and weak convergence in the averaging principe are established, which can be viewed as a functional law of large numbers. Then we study the stochastic fluctuations of the original system around its averaged equation. We show that the normalized difference converges weakly to the solution of a linear stochastic wave equation, which is a form of functional central limit theorem. We provide a unified proof for the above convergence by using the Poisson equation in Hilbert spaces. Moreover, sharp rates of convergence are obtained, which are shown not to depend on the regularity of the coefficients in the equation for the fast variable.

math.PR↗

arXiv2020

ETC: Encoding Long and Structured Inputs in Transformers

Transformer models have advanced the state of the art in many Natural Language Processing (NLP) tasks. In this paper, we present a new Transformer architecture, Extended Transformer Construction (ETC), that addresses two key challenges of standard Transformer architectures, namely scaling input length and encoding structured inputs. To scale attention to longer inputs, we introduce a novel global-local attention mechanism between global tokens and regular input tokens. We also show that combining global-local attention with relative position encodings and a Contrastive Predictive Coding (CPC) pre-training objective allows ETC to encode structured inputs. We achieve state-of-the-art results on four natural language datasets requiring long and/or structured inputs.

cs.LG↗

arXiv2020

Study on $η_{c2}(η_{b2})$ electromagnetic decay into double photons

Within the framework of nonrelativistic QCD (NRQCD) factorization formalism, we compute the helicity amplitude as well as the decay width of $η_{Q2}$ ($Q=c,b$) electromagnetic decay into two photons up to next-to-next-to-leading order (NNLO) in $α_s$ expansion. For the first time, we verify the validity of NRQCD factorization for the D-wave quarkonium decay at NNLO. We find that the $\mathcal{O}(α_s)$ and $\mathcal{O}(α_s^2)$ corrections to the helicity amplitude are negative and moderate, nevertheless both corrections combine to suppress the leading-order prediction for the decay width significantly. By approximating the total decay width of $η_{Q2}$ as the sum of those for the hadronic decay and the electric $E1$ transition, we obtain the branching ratios ${\rm Br}(η_{c2}\to 2γ)\approx 5\times10^{-6}$ and ${\rm Br}(η_{b2}\to 2γ)\approx 4\times10^{-7}$. To explore the potential measurement on $η_{Q2}$, we further evaluate the production cross section of $η_{Q2}$ at LHCb at the lowest order in $α_s$ expansion. With the kinematic constraint on the longitudinal rapidity $4.5>y>2$ and transverse momentum $P_T>(2-4)m_Q$ for $η_{Q2}$, we find the cross section can reach $2-50$ nb for $η_{c2}$, and $1-22$ pb for $η_{b2}$. Considering the integrated luminosity $\mathcal{L}=10\, {\rm fb}^{-1}$ at $\sqrt{s}=7$ TeV and $\sqrt{s}=13$ TeV, we estimate that there are several hundreds events of $pp\to η_{c2}\to 2γ$. Since the background is relatively clean, it is promising to reconstruct $η_{c2}$ through its electromagnetic decay. On the contrary, due to small branching ratio and production cross section, it is quite challenging to detect $η_{b2}\to 2γ$ at LHCb.

hep-ph↗

arXiv2020

Meron-Like Topological Spin Defects in Monolayer CrCl3

Noncollinear spin textures in low-dimensional magnetic systems have been studied for decades because of their extraordinary properties and promising applications derived from the chirality and topological nature. However, material realizations of topological spin states are still limited. Employing first-principles and Monte Carlo simulations, we propose that monolayer chromium trichloride (CrCl3) can be a promising candidate for observing the vortex/antivortex type of topological defects, so-called merons. The numbers of vortices and antivortices are found to be the same, maintaining an overall integer topological unit. By perturbing with external magnetic fields, we show the robustness of these meron pairs and reveal a rich phase space to tune the hybridization between the ferromagnetic order and meron-like defects. The signatures of topological excitations under external magnetic field also provide crucial information for experimental justifications. Our study predicts that two-dimensional magnets with weak spin-orbit coupling can be a promising family for realizing meron-like spin textures.

cond-mat.mes-hall↗

arXiv2020

KSM: Fast Multiple Task Adaption via Kernel-wise Soft Mask Learning

Deep Neural Networks (DNN) could forget the knowledge about earlier tasks when learning new tasks, and this is known as \textit{catastrophic forgetting}. While recent continual learning methods are capable of alleviating the catastrophic problem on toy-sized datasets, some issues still remain to be tackled when applying them in real-world problems. Recently, the fast mask-based learning method (e.g. piggyback \cite{mallya2018piggyback}) is proposed to address these issues by learning only a binary element-wise mask in a fast manner, while keeping the backbone model fixed. However, the binary mask has limited modeling capacity for new tasks. A more recent work \cite{hung2019compacting} proposes a compress-grow-based method (CPG) to achieve better accuracy for new tasks by partially training backbone model, but with order-higher training cost, which makes it infeasible to be deployed into popular state-of-the-art edge-/mobile-learning. The primary goal of this work is to simultaneously achieve fast and high-accuracy multi task adaption in continual learning setting. Thus motivated, we propose a new training method called \textit{kernel-wise Soft Mask} (KSM), which learns a kernel-wise hybrid binary and real-value soft mask for each task, while using the same backbone model. Such a soft mask can be viewed as a superposition of a binary mask and a properly scaled real-value tensor, which offers a richer representation capability without low-level kernel support to meet the objective of low hardware overhead. We validate KSM on multiple benchmark datasets against recent state-of-the-art methods (e.g. Piggyback, Packnet, CPG, etc.), which shows good improvement in both accuracy and training cost.

cs.CV↗

arXiv2020

A Progressive Sub-Network Searching Framework for Dynamic Inference

Many techniques have been developed, such as model compression, to make Deep Neural Networks (DNNs) inference more efficiently. Nevertheless, DNNs still lack excellent run-time dynamic inference capability to enable users trade-off accuracy and computation complexity (i.e., latency on target hardware) after model deployment, based on dynamic requirements and environments. Such research direction recently draws great attention, where one realization is to train the target DNN through a multiple-term objective function, which consists of cross-entropy terms from multiple sub-nets. Our investigation in this work show that the performance of dynamic inference highly relies on the quality of sub-net sampling. With objective to construct a dynamic DNN and search multiple high quality sub-nets with minimal searching cost, we propose a progressive sub-net searching framework, which is embedded with several effective techniques, including trainable noise ranking, channel group and fine-tuning threshold setting, sub-nets re-selection. The proposed framework empowers the target DNN with better dynamic inference capability, which outperforms prior works on both CIFAR-10 and ImageNet dataset via comprehensive experiments on different network structures. Taken ResNet18 as an example, our proposed method achieves much better dynamic inference accuracy compared with prior popular Universally-Slimmable-Network by 4.4%-maximally and 2.3%-averagely in ImageNet dataset with the same model size.

cs.CV↗

arXiv2020

Dynamical Fermionization in One Dimensional Spinor Gases

Dynamical fermionization refers to the phenomenon in Tonks-Girardeau (TG) gases where, upon release from harmonic confinement, the gas's momentum density profile evolves asymptotically to that of an ideal Fermi gas in the initial trap. This phenomenon has been demonstrated theoretically in hardcore and anyonic TG gases, and recently experimentally observed in a strongly interacting Bose gas. We extend this study to a one dimensional (1D) spinor gas of arbitrary spin in the strongly interacting regime, and analytically prove that the total momentum distribution after the harmonic trap is turned off approaches that of a spinless ideal Fermi gas, while the asymptotic momentum distribution of each spin component takes the same shape of the initial real space density profile of that spin component. Our work demonstrates the rich physics arising from the interplay between the spin and the charge degrees of freedom in a spinor system.

cond-mat.quant-gas↗

arXiv2020

Modulation Doping via a 2d Atomic Crystalline Acceptor

Two-dimensional (2d) nano-electronics, plasmonics, and emergent phases require clean and local charge control, calling for layered, crystalline acceptors or donors. Our Raman, photovoltage, and electrical conductance measurements combined with \textit{ab initio} calculations establish the large work function and narrow bands of $α$-RuCl$_3$ enable modulation doping of exfoliated, chemical vapor deposition (CVD), and molecular beam epitaxy (MBE) materials. Short-ranged lateral doping (${\leq}65\ \text{nm}$) and high homogeneity are achieved in proximate materials with a single layer of \arucl. This leads to the highest monolayer graphene (mlg) mobilities ($4,900\ \text{cm}^2/ \text{Vs}$) at these high hole densities ($3\times10^{13}\ \text{cm}^{-2}$); and yields larger charge transfer to bilayer graphene (blg) ($6\times10^{13}\ \text{cm}^{-2}$). We further demonstrate proof of principle optical sensing, control via twist angle, and charge transfer through hexagonal boron nitride (hBN).

cond-mat.mtrl-sci↗

arXiv2020

Intrinsic Spin Photogalvanic Effect in Nonmagnetic Insulator

We show that with the help of spin-orbit coupling, nonlinear light-matter interactions can efficiently couple with spin and valley degrees of freedom. This revealed spin photogalvanic effect can generate the long-time pursued intrinsic pure spin current (PSC) in non-centrosymmetric nonmagnetic insulators. Different from the spin and valley Hall effect, such a photo-driven spin current is universal and can be generated without external bias field. Using first-principles simulation, we study monolayer transition metal dichalcogenides (TMDs) to demonstrate this effect and confirm an enhanced PSC under linearly polarized photoexcitation. The amplitude of the PSC is one order larger than that of the charge current observed in monolayer TMDs. This exotic nonlinear light-spin interaction indicates that light can be utilized as a rapid fashion to manipulate the spin-polarized current, which is crucial for future low-dissipation nanodevices.

cond-mat.mtrl-sci↗

arXiv2020

Quasiparticle Energies and Excitonic Effects of Chromium Trichloride: from Two Dimensions to Bulk

Layered van der Waals (vdW) magnetic materials have attracted significant research interest to date. In this work, we employ the first-principles many-body perturbation theory to calculate excited-state properties of a prototype vdW magnet, chromium trichloride (CrCl3), covering monolayer, bilayer, and bulk structures. Unlike usual non-magnetic vdW semiconductors, in which many-electron interactions and excited states are sensitive to dimensionality, many-electron interactions are always enhanced and dominate quasiparticle energies and optical responses of both two-dimensional and bulk CrCl3. The electron-hole (e-h) binding energy can reach 3 eV in monolayer and remains as high as 2 eV in bulk. Because of the cancellation effect between self-energy corrections and e-h binding energies, the lowest-energy exciton (optical gap) is almost not affected by the change of dimensionality. Besides, for the excitons with similar e-h binding energies, their dipole oscillator strength can differ by a few orders of magnitude.Our analysis shows that such a big difference is from a unique interference effect between complex exciton wavefunctions and interband transitions. Finally, we find that the interlayer stacking sequence and magnetic coupling barely change quasiparticle band gaps and optical absorption spectra of CrCl3. Our calculated low-energy exciton peak positions agree with available measurements. These findings give insight into the understanding of many-electron interactions and the interplay between magnetic orders and optical excitations in vdW magnetic materials.

cond-mat.mtrl-sci↗

arXiv2020

Photo-degradation Protection in 2D In-Plane Heterostructures Revealed by Hyperspectral Nanoimaging: the Role of Nano-Interface 2D Alloys

Single-layer heterostructures exhibit striking quasiparticle properties and many-body interaction effects that hold promise for a range of applications. However, their properties can be altered by intrinsic and extrinsic defects, thus diminishing their applicability. Therefore, it is of paramount importance to identify defects and understand 2D materials' degradation over time using advanced multimodal imaging techniques as well as stabilize degradation via built-in interface protection. Here we implemented a liquid-phase precursor approach to synthesize 2D in-plane MoS2-WS2 heterostructures exhibiting nanoscale alloyed interfaces and map exotic interface effects during photo-degradation using a novel combination of hyperspectral tip-enhanced photoluminescence, Raman and near-field nanoscopy. Surprisingly, 2D alloyed regions exhibit remarkable thermal and photo-degradation stability providing protection against oxidation. Coupled with surface and interface strain, 2D alloy regions create localized potential wells that concentrate excitonic species via a charge carrier funneling effect. These results provide a clear understanding of the importance of 2D alloys as systems able to withstand degradation effects over time, and could be now used to stabilize optoelectronic devices based on 2D materials.

cond-mat.mes-hall↗

arXiv2020

A Fast Radio Burst discovered in FAST drift scan survey

We report the discovery of a highly dispersed fast radio burst, FRB~181123, from an analysis of $\sim$1500~hr of drift-scan survey data taken using the Five-hundred-meter Aperture Spherical radio Telescope (FAST). The pulse has three distinct emission components, which vary with frequency across our 1.0--1.5~GHz observing band. We measure the peak flux density to be $>0.065$~Jy and the corresponding fluence $>0.2$~Jy~ms. Based on the observed dispersion measure of 1812~cm$^{-3}$~pc, we infer a redshift of $\sim 1.9$. From this, we estimate the peak luminosity and isotropic energy to be $\lesssim 2\times10^{43}$~erg~s$^{-1}$ and $\lesssim 2\times10^{40}$~erg, respectively. With only one FRB from the survey detected so far, our constraints on the event rate are limited. We derive a 95\% confidence lower limit for the event rate of 900 FRBs per day for FRBs with fluences $>0.025$~Jy~ms. We performed follow-up observations of the source with FAST for four hours and have not found a repeated burst. We discuss the implications of this discovery for our understanding of the physical mechanisms of FRBs.

astro-ph.HE↗

arXiv2020

Nonreciprocal Second-Harmonic Generation in Few-Layer Chromium Triiodide

It is of fundamental importance but challenging to simultaneously identify atomic and magnetic configurations of two-dimensional van der Waals materials. In this work, we show that the nonreciprocal second-harmonic generation (SHG) can be a powerful tool to answer this challenge. Despite the preserved lattice inversion symmetry, the interlayer antiferromagnetic order and spin-orbit coupling generate enhanced SHG in PT-symmetric bilayer chromium triiodide (CrI3). Importantly, the in-plane polarization-resolved SHG is sensitive to subtly different interlayer structures that cannot be told by linear optical spectra. Beyond bilayer, we further predict that the intensity and angle-resolved SHG can be employed to identify both interlayer atomic and magnetic configurations of trilayer CrI3. Our first-principles results agree with available measurements and show the potential of SHG as a non-contacting approach to explore correlations between interlayer structures and magnetic orders of emerging ultra-thin magnetic materials.

cond-mat.mtrl-sci↗

arXiv2020

Giant linearly-polarized photogalvanic effect and second harmonic generation in two-dimensional axion insulators

The second-order nonlinear optical (NLO) processes, such as the photogalvanic effect and second-order harmonic generation (SHG), play crucial roles in probing and controlling light-matter interactions for energy and device applications. To date, most studies of second-order NLO processes focus on materials with broken spatial inversion symmetry, such as proper ferroelectrics and noncentrosymmetric Weyl semimetals. Nevertheless, inversion symmetry of Shubnikov groups can be broken via spin-ordering in centrosymmetric crystals. Unfortunately, these materials are less common, and their NLO responses are usually weak. Combining quantum perturbation theory and first-principles simulations, we predict a giant injection-current photogalvanic effect and SHG in a family of emerging axion insulators, the even septuple layers of MnBi2Te4 (MBT) materials that exhibit the zero-plateau quantum anomalous Hall (QAH) effect. Their amplitudes of injection current and SHG are about two orders of magnitude larger than those of widely used ferroelectrics, such as BiFeO3 and LiNbO3. Moreover, unlike the usual injection current observed under circularly-polarized light, the injection photocurrent of MBTs only emerges under linearly polarized light, making it convenient for device applications. These unique characters are from a combination effect of parity-time symmetry, three-fold rotation symmetry, and significant spin-orbit coupling. These enhanced NLO effects are valuable for characterizing subtle topological orders in QAH systems and also shed light on novel infrared photo-detector and photovoltaic applications based on magnetic topological materials.

cond-mat.mtrl-sci↗

arXiv2020

Artificial Multiferroics and Enhanced Magnetoelectric Effect in van der Waals Heterostructures

Multiferroic materials with coupled ferroelectric and ferromagnetic properties are important for multifunctional devices due to their potential ability of controlling magnetism via electric field, and vice versa. The recent discoveries of two-dimensional ferromagnetic and ferroelectric materials have ignited tremendous research interest and aroused hope to search for two-dimensional multiferroics. However, intrinsic two-dimensional multiferroic materials and, particularly, those with strong magnetoelectric couplings are still rare to date. In this paper, using first-principles simulations, we propose artificial two-dimensional multiferroics via a van der Waals heterostructure formed by ferromagnetic bilayer chromium triiodide (CrI3) and ferroelectric monolayer Sc2CO2. In addition to the coexistence of ferromagnetism and ferroelectricity, our calculations show that, by switching the electric polarization of Sc2CO2, we can tune the interlayer magnetic couplings of bilayer CrI3 between ferromagnetic and antiferromagnetic states. We further reveal that such a strong magnetoelectric effect is from a dramatic change of the band alignment induced by the strong build-in electric polarization in Sc2CO2 and the subsequent change of the interlayer magnetic coupling of bilayer CrI3. These artificial multiferroics and enhanced magnetoelectric effect give rise to realizing multifunctional nanoelectronics by van der Waals heterostructures.

cond-mat.mes-hall↗