arXiv subjects

Peng Li

Publications and source records attributed to Peng Li.

At least 379 records · Page 21Linked to original sources

arXiv2021

Path-Enhanced Multi-Relational Question Answering with Knowledge Graph Embeddings

The multi-relational Knowledge Base Question Answering (KBQA) system performs multi-hop reasoning over the knowledge graph (KG) to achieve the answer. Recent approaches attempt to introduce the knowledge graph embedding (KGE) technique to handle the KG incompleteness but only consider the triple facts and neglect the significant semantic correlation between paths and multi-relational questions. In this paper, we propose a Path and Knowledge Embedding-Enhanced multi-relational Question Answering model (PKEEQA), which leverages multi-hop paths between entities in the KG to evaluate the ambipolar correlation between a path embedding and a multi-relational question embedding via a customizable path representation mechanism, benefiting for achieving more accurate answers from the perspective of both the triple facts and the extra paths. Experimental results illustrate that PKEEQA improves KBQA models' performance for multi-relational question answering with explainability to some extent derived from paths.

cs.CL↗

arXiv2021

RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Models

Backdoor attacks, which maliciously control a well-trained model's outputs of the instances with specific triggers, are recently shown to be serious threats to the safety of reusing deep neural networks (DNNs). In this work, we propose an efficient online defense mechanism based on robustness-aware perturbations. Specifically, by analyzing the backdoor training process, we point out that there exists a big gap of robustness between poisoned and clean samples. Motivated by this observation, we construct a word-based robustness-aware perturbation to distinguish poisoned samples from clean samples to defend against the backdoor attacks on natural language processing (NLP) models. Moreover, we give a theoretical analysis about the feasibility of our robustness-aware perturbation-based defense method. Experimental results on sentiment analysis and toxic detection tasks show that our method achieves better defending performance and much lower computational costs than existing online defense methods. Our code is available at https://github.com/lancopku/RAP.

cs.CL↗

arXiv2021

Topology-Imbalance Learning for Semi-Supervised Node Classification

The class imbalance problem, as an important issue in learning node representations, has drawn increasing attention from the community. Although the imbalance considered by existing studies roots from the unequal quantity of labeled examples in different classes (quantity imbalance), we argue that graph data expose a unique source of imbalance from the asymmetric topological properties of the labeled nodes, i.e., labeled nodes are not equal in terms of their structural role in the graph (topology imbalance). In this work, we first probe the previously unknown topology-imbalance issue, including its characteristics, causes, and threats to semi-supervised node classification learning. We then provide a unified view to jointly analyzing the quantity- and topology- imbalance issues by considering the node influence shift phenomenon with the Label Propagation algorithm. In light of our analysis, we devise an influence conflict detection -- based metric Totoro to measure the degree of graph topology imbalance and propose a model-agnostic method ReNode to address the topology-imbalance issue by re-weighting the influence of labeled nodes adaptively based on their relative positions to class boundaries. Systematic experiments demonstrate the effectiveness and generalizability of our method in relieving topology-imbalance issue and promoting semi-supervised node classification. The further analysis unveils varied sensitivity of different graph neural networks (GNNs) to topology imbalance, which may serve as a new perspective in evaluating GNN architectures.

cs.LG↗

arXiv2021

Dynamic Knowledge Distillation for Pre-trained Language Models

Knowledge distillation~(KD) has been proved effective for compressing large-scale pre-trained language models. However, existing methods conduct KD statically, e.g., the student model aligns its output distribution to that of a selected teacher model on the pre-defined training dataset. In this paper, we explore whether a dynamic knowledge distillation that empowers the student to adjust the learning procedure according to its competency, regarding the student performance and learning efficiency. We explore the dynamical adjustments on three aspects: teacher model adoption, data selection, and KD objective adaptation. Experimental results show that (1) proper selection of teacher model can boost the performance of student model; (2) conducting KD with 10% informative instances achieves comparable performance while greatly accelerates the training; (3) the student performance can be boosted by adjusting the supervision contribution of different alignment objective. We find dynamic knowledge distillation is promising and provide discussions on potential future directions towards more efficient KD methods. Our code is available at https://github.com/lancopku/DynamicKD.

cs.CL↗

arXiv2021

Demonstration of Panda: A Weakly Supervised Entity Matching System

Entity matching (EM) refers to the problem of identifying tuple pairs in one or more relations that refer to the same real world entities. Supervised machine learning (ML) approaches, and deep learning based approaches in particular, typically achieve state-of-the-art matching results. However, these approaches require many labeled examples, in the form of matching and non-matching pairs, which are expensive and time-consuming to label. In this paper, we introduce Panda, a weakly supervised system specifically designed for EM. Panda uses the same labeling function abstraction as Snorkel, where labeling functions (LF) are user-provided programs that can generate large amounts of (somewhat noisy) labels quickly and cheaply, which can then be combined via a labeling model to generate accurate final predictions. To support users developing LFs for EM, Panda provides an integrated development environment (IDE) that lives in a modern browser architecture. Panda's IDE facilitates the development, debugging, and life-cycle management of LFs in the context of EM tasks, similar to how IDEs such as Visual Studio or Eclipse excel in general-purpose programming. Panda's IDE includes many novel features purpose-built for EM, such as smart data sampling, a builtin library of EM utility functions, automatically generated LFs, visual debugging of LFs, and finally, an EM-specific labeling model. We show in this demo that Panda IDE can greatly accelerate the development of high-quality EM solutions using weak supervision.

cs.DB↗

arXiv2021

Evaluation of mechanical and energy properties for the phase field modeling of failure

In recent years, various phase field models have been developed in variational methods to simulate the failure of brittle solids. However, there is a lack of objective evaluation of the existing results, and in particular, there are few studies on model nonhomogeneous resolution, stress-strain linear elastic properties, and failure stress estimation. To compensate for the above gaps, the commonly used variational phase field model is systematically analyzed to solve the problem of evaluating the mechanics and energy properties of the model in this paper. The unified expression of the analytical solution and the nonhomogeneous solution under specific boundary conditions is analyzed and verified. Additionally, we theoretically analyze the energy properties of the phase field model and study the influence of the critical strain energy on the damage field, stress and strain of different models. Finally, the effect of different effective material parameters on the material failure stress is analyzed, the stress estimation formula for the failure of mode I of various phase field models is given, and the accuracy of the theory is verified by some examples.

math.NA↗

arXiv2021

Multimodal Incremental Transformer with Visual Grounding for Visual Dialogue Generation

Visual dialogue is a challenging task since it needs to answer a series of coherent questions on the basis of understanding the visual environment. Previous studies focus on the implicit exploration of multimodal co-reference by implicitly attending to spatial image features or object-level image features but neglect the importance of locating the objects explicitly in the visual content, which is associated with entities in the textual content. Therefore, in this paper we propose a {\bf M}ultimodal {\bf I}ncremental {\bf T}ransformer with {\bf V}isual {\bf G}rounding, named MITVG, which consists of two key parts: visual grounding and multimodal incremental transformer. Visual grounding aims to explicitly locate related objects in the image guided by textual entities, which helps the model exclude the visual content that does not need attention. On the basis of visual grounding, the multimodal incremental transformer encodes the multi-turn dialogue history combined with visual scene step by step according to the order of the dialogue and then generates a contextually and visually coherent response. Experimental results on the VisDial v0.9 and v1.0 datasets demonstrate the superiority of the proposed model, which achieves comparable performance.

cs.CL↗

arXiv2021

Facilitating Parallel Fuzzing with mutually-exclusive Task Distribution

Fuzz testing, or fuzzing, has become one of the de facto standard techniques for bug finding in the software industry. In general, fuzzing provides various inputs to the target program to discover unhandled exceptions and crashes. In business sectors where the time budget is limited, software vendors often launch many fuzzing instances in parallel as common means of increasing code coverage. However, most of the popular fuzzing tools in their parallel mode-naively run multiple instances concurrently, without elaborate distribution of workload. This can lead different instances to explore overlapped code regions, eventually reducing the benefits of concurrency. In this paper, we propose a general model to describe parallel fuzzing. This model distributes mutually-exclusive but similarly-weighted tasks to different instances, facilitating concurrency and also fairness across instances. Following this model, we develop a solution, called AFL-EDGE, to improve the parallel mode of AFL, considering a round of mutations to a unique seed as a task and adopting edge coverage to define the uniqueness of a seed. We have implemented AFL-EDGE on top of AFL and evaluated the implementation with AFL on 9 widely used benchmark programs. It shows that AFL-EDGE can benefit the edge coverage of AFL. In a 24-hour test, the increase of edge coverage brought by AFL-EDGE to AFL ranges from 9.49% to 10.20%, depending on the number of instances. As a side benefit, we discovered 14 previously unknown bugs.

cs.SE↗

arXiv2021

Magnetism and Spin Dynamics in Room-Temperature van der Waals Magnet Fe$_5$GeTe$_2$

Two-dimensional (2D) van der Waals (vdWs) materials have gathered a lot of attention recently. However, the majority of these materials have Curie temperatures that are well below room temperature, making it challenging to incorporate them into device applications. In this work, we synthesized a room-temperature vdW magnetic crystal Fe$_5$GeTe$_2$ with a Curie temperature T$_c = 332$ K, and studied its magnetic properties by vibrating sample magnetometry (VSM) and broadband ferromagnetic resonance (FMR) spectroscopy. The experiments were performed with external magnetic fields applied along the c-axis (H$\parallel$c) and the ab-plane (H$\parallel$ab), with temperatures ranging from 300 K to 10 K. We have found a sizable Landé g-factor difference between the H$\parallel$c and H$\parallel$ab cases. In both cases, the Landé g-factor values deviated from g = 2. This indicates contribution of orbital angular momentum to the magnetic moment. The FMR measurements reveal that Fe$_5$GeTe$_2$ has a damping constant comparable to Permalloy. With reducing temperature, the linewidth was broadened. Together with the VSM data, our measurements indicate that Fe$_5$GeTe$_2$ transitions from ferromagnetic to ferrimagnetic at lower temperatures. Our experiments highlight key information regarding the magnetic state and spin scattering processes in Fe$_5$GeTe$_2$, which promote the understanding of magnetism in Fe$_5$GeTe$_2$, leading to implementations of Fe$_5$GeTe$_2$ based room-temperature spintronic devices.

cond-mat.mtrl-sci↗

arXiv2021

WeChat Neural Machine Translation Systems for WMT21

This paper introduces WeChat AI's participation in WMT 2021 shared news translation task on English->Chinese, English->Japanese, Japanese->English and English->German. Our systems are based on the Transformer (Vaswani et al., 2017) with several novel and effective variants. In our experiments, we employ data filtering, large-scale synthetic data generation (i.e., back-translation, knowledge distillation, forward-translation, iterative in-domain knowledge transfer), advanced finetuning approaches, and boosted Self-BLEU based model ensemble. Our constrained systems achieve 36.9, 46.9, 27.8 and 31.3 case-sensitive BLEU scores on English->Chinese, English->Japanese, Japanese->English and English->German, respectively. The BLEU scores of English->Chinese, English->Japanese and Japanese->English are the highest among all submissions, and that of English->German is the highest among all constrained submissions.

cs.CL↗

arXiv2021

CascadeBERT: Accelerating Inference of Pre-trained Language Models via Calibrated Complete Models Cascade

Dynamic early exiting aims to accelerate the inference of pre-trained language models (PLMs) by emitting predictions in internal layers without passing through the entire model. In this paper, we empirically analyze the working mechanism of dynamic early exiting and find that it faces a performance bottleneck under high speed-up ratios. On one hand, the PLMs' representations in shallow layers lack high-level semantic information and thus are not sufficient for accurate predictions. On the other hand, the exiting decisions made by internal classifiers are unreliable, leading to wrongly emitted early predictions. We instead propose a new framework for accelerating the inference of PLMs, CascadeBERT, which dynamically selects proper-sized and complete models in a cascading manner, providing comprehensive representations for predictions. We further devise a difficulty-aware objective, encouraging the model to output the class probability that reflects the real difficulty of each instance for a more reliable cascading mechanism. Experimental results show that CascadeBERT can achieve an overall 15\% improvement under 4$\times$ speed-up compared with existing dynamic early exiting methods on six classification tasks, yielding more calibrated and accurate predictions.

cs.CL↗

arXiv2021

Quantum cluster kink and ring frustration

In this paper, we work on the pure and mixed cluster models with periodic boundary condition. The first purpose is to establish the concept of quantum cluster kink. We clarify that there are two types of cluster kinks since there are two types of ground states depending on the choice of cluster length, of which the first type exhibits symmetry breaking order and the second one string order. Simple pictures are constructed according to the pure cluster model, which facilitates us to introduce the quantity, cluster kink number. As we demonstrate, cluster kinks deriving from different sources can coexist, compete with each other, and lead to quantum phase transition in a mixed cluster model. The second purpose is to elucidate that the effect of ring frustration can be realized in the cluster model with symmetry breaking order, instead of the one with string order. The reason lies in that ring frustration can induce a huge ground-state degeneracy or a special extended-kink phase with gapless excitations in the former case. And, although ring frustration does not change the phase transition point, it can produce unusual ground state in the extended-kink phase, whose special properties are uncovered by the nonlocal scaling factor in the correlation function and the doubled degeneracies of the eigenvalues of the entanglement spectrum.

cond-mat.str-el↗

arXiv2021

Giant Topological Hall Effect in van der Waals Heterostructures of CrTe2/Bi2Te3

Discoveries of interfacial topological Hall effect (THE) provide an ideal platform for exploring physics arising from the interplay between topology and magnetism. The interfacial topological Hall effect is closely related to the Dzyaloshinskii-Moriya interaction (DMI) at interface and topological spin textures. However, it is difficult to achieve a sizable THE in heterostructures due to the stringent constraints on the constituents of THE heterostructures such as strong spin-orbit coupling (SOC). Here we report the observation of a giant THE signal of 1.39 $μΩ\cdot$cm in the van der Waals heterostructures of CrTe2/Bi2Te3 fabricated by molecular beam epitaxy, a prototype of two-dimensional (2D) ferromagnet (FM)/topological insulator (TI). This large magnitude of THE is attributed to an optimized combination of 2D ferromagnetism in CrTe2, strong SOC in Bi2Te3, and an atomically sharp interface. Our work reveals CrTe2/Bi2Te3 as a convenient platform for achieving large interfacial THE in hybrid systems, which could be utilized to develop quantum science and high-density information storage.

cond-mat.mes-hall↗

arXiv2021

EasyTransfer -- A Simple and Scalable Deep Transfer Learning Platform for NLP Applications

The literature has witnessed the success of leveraging Pre-trained Language Models (PLMs) and Transfer Learning (TL) algorithms to a wide range of Natural Language Processing (NLP) applications, yet it is not easy to build an easy-to-use and scalable TL toolkit for this purpose. To bridge this gap, the EasyTransfer platform is designed to develop deep TL algorithms for NLP applications. EasyTransfer is backended with a high-performance and scalable engine for efficient training and inference, and also integrates comprehensive deep TL algorithms, to make the development of industrial-scale TL applications easier. In EasyTransfer, the built-in data and model parallelism strategies, combined with AI compiler optimization, show to be 4.0x faster than the community version of distributed training. EasyTransfer supports various NLP models in the ModelZoo, including mainstream PLMs and multi-modality models. It also features various in-house developed TL algorithms, together with the AppZoo for NLP applications. The toolkit is convenient for users to quickly start model training, evaluation, and online deployment. EasyTransfer is currently deployed at Alibaba to support a variety of business scenarios, including item recommendation, personalized search, conversational question answering, etc. Extensive experiments on real-world datasets and online applications show that EasyTransfer is suitable for online production with cutting-edge performance for various applications. The source code of EasyTransfer is released at Github (https://github.com/alibaba/EasyTransfer).

cs.CL↗

arXiv2021

H2Learn: High-Efficiency Learning Accelerator for High-Accuracy Spiking Neural Networks

Although spiking neural networks (SNNs) take benefits from the bio-plausible neural modeling, the low accuracy under the common local synaptic plasticity learning rules limits their application in many practical tasks. Recently, an emerging SNN supervised learning algorithm inspired by backpropagation through time (BPTT) from the domain of artificial neural networks (ANNs) has successfully boosted the accuracy of SNNs and helped improve the practicability of SNNs. However, current general-purpose processors suffer from low efficiency when performing BPTT for SNNs due to the ANN-tailored optimization. On the other hand, current neuromorphic chips cannot support BPTT because they mainly adopt local synaptic plasticity rules for simplified implementation. In this work, we propose H2Learn, a novel architecture that can achieve high efficiency for BPTT-based SNN learning which ensures high accuracy of SNNs. At the beginning, we characterized the behaviors of BPTT-based SNN learning. Benefited from the binary spike-based computation in the forward pass and the weight update, we first design lookup table (LUT) based processing elements in Forward Engine and Weight Update Engine to make accumulations implicit and to fuse the computations of multiple input points. Second, benefited from the rich sparsity in the backward pass, we design a dual-sparsity-aware Backward Engine which exploits both input and output sparsity. Finally, we apply a pipeline optimization between different engines to build an end-to-end solution for the BPTT-based SNN learning. Compared with the modern NVIDIA V100 GPU, H2Learn achieves 7.38x area saving, 5.74-10.20x speedup, and 5.25-7.12x energy saving on several benchmark datasets.

cs.NE↗

arXiv2021

Revealing the Heavy Quasiparticles in the Heavy-Fermion Superconductor CeCu2Si2

The superconducting order parameter of the first heavy-fermion superconductor CeCu2Si2 is currently under debate. A key ingredient to understand its superconductivity and physical properties is the quasiparticle dispersion and Fermi surface, which remains elusive experimentally. Here we present measurements from angle-resolved photoemission spectroscopy. Our results emphasize the key role played by the Ce 4f electrons for the low-temperature Fermi surface, highlighting a band-dependent conduction-f electron hybridization. In particular, we find a very heavy quasi-two-dimensional electron band near the bulk X point and moderately heavy three-dimensional hole pockets near the Z point. Comparison with theoretical calculations reveals the strong local correlation in this compound, calling for further theoretical studies. Our results provide the electronic basis to understand the heavy fermion behavior and superconductivity; implications for the enigmatic superconductivity of this compound are also discussed.

cond-mat.str-el↗

arXiv2021

Backpropagated Neighborhood Aggregation for Accurate Training of Spiking Neural Networks

While backpropagation (BP) has been applied to spiking neural networks (SNNs) achieving encouraging results, a key challenge involved is to backpropagate a continuous-valued loss over layers of spiking neurons exhibiting discontinuous all-or-none firing activities. Existing methods deal with this difficulty by introducing compromises that come with their own limitations, leading to potential performance degradation. We propose a novel BP-like method, called neighborhood aggregation (NA), which computes accurate error gradients guiding weight updates that may lead to discontinuous modifications of firing activities. NA achieves this goal by aggregating finite differences of the loss over multiple perturbed membrane potential waveforms in the neighborhood of the present membrane potential of each neuron while utilizing a new membrane potential distance function. Our experiments show that the proposed NA algorithm delivers the state-of-the-art performance for SNN training on several datasets.

cs.NE↗

arXiv2021

Sample Recycling Method -- A New Approach to Efficient Nested Monte Carlo Simulations

Nested stochastic modeling has been on the rise in many fields of the financial industry. Such modeling arises whenever certain components of a stochastic model are stochastically determined by other models. There are at least two main areas of applications, including (1) portfolio risk management in the banking sector and (2) principle-based reserving and capital requirements in the insurance sector. As financial instrument values often change with economic fundamentals, the risk management of a portfolio (outer loop) often requires the assessment of financial positions subject to changes in risk factors in the immediate future. The valuation of financial position (inner loop) is based on projections of cashflows and risk factors into the distant future. The nesting of such stochastic modeling can be computationally challenging. Most of existing techniques to speed up nested simulations are based on curve fitting. The main idea is to establish a functional relationship between inner loop estimator and risk factors by running a limited set of economic scenarios, and, instead of running inner loop simulations, inner loop estimations are made by feeding other scenarios into the fitted curve. This paper presents a non-conventional approach based on the concept of sample recycling. Its essence is to run inner loop estimation for a small set of outer loop scenarios and to find inner loop estimates under other outer loop scenarios by recycling those known inner loop paths. This new approach can be much more efficient when traditional techniques are difficult to implement in practice.

q-fin.CP↗