arXiv subjects

Shuai Zhang

Publications and source records attributed to Shuai Zhang.

At least 307 records · Page 17Linked to original sources

arXiv2022

SKDBERT: Compressing BERT via Stochastic Knowledge Distillation

In this paper, we propose Stochastic Knowledge Distillation (SKD) to obtain compact BERT-style language model dubbed SKDBERT. In each iteration, SKD samples a teacher model from a pre-defined teacher ensemble, which consists of multiple teacher models with multi-level capacities, to transfer knowledge into student model in an one-to-one manner. Sampling distribution plays an important role in SKD. We heuristically present three types of sampling distributions to assign appropriate probabilities for multi-level teacher models. SKD has two advantages: 1) it can preserve the diversities of multi-level teacher models via stochastically sampling single teacher model in each iteration, and 2) it can also improve the efficacy of knowledge distillation via multi-level teacher models when large capacity gap exists between the teacher model and the student model. Experimental results on GLUE benchmark show that SKDBERT reduces the size of a BERT$_{\rm BASE}$ model by 40% while retaining 99.5% performances of language understanding and being 100% faster.

cs.CL↗

arXiv2022

A Fine-grained Interpretability Evaluation Benchmark for Neural NLP

While there is increasing concern about the interpretability of neural models, the evaluation of interpretability remains an open problem, due to the lack of proper evaluation datasets and metrics. In this paper, we present a novel benchmark to evaluate the interpretability of both neural models and saliency methods. This benchmark covers three representative NLP tasks: sentiment analysis, textual similarity and reading comprehension, each provided with both English and Chinese annotated data. In order to precisely evaluate the interpretability, we provide token-level rationales that are carefully annotated to be sufficient, compact and comprehensive. We also design a new metric, i.e., the consistency between the rationales before and after perturbations, to uniformly evaluate the interpretability on different types of tasks. Based on this benchmark, we conduct experiments on three typical models with three saliency methods, and unveil their strengths and weakness in terms of interpretability. We will release this benchmark https://www.luge.ai/#/luge/task/taskDetail?taskId=15 and hope it can facilitate the research in building trustworthy systems.

cs.CL↗

arXiv2022

Neural Methods for Logical Reasoning Over Knowledge Graphs

Reasoning is a fundamental problem for computers and deeply studied in Artificial Intelligence. In this paper, we specifically focus on answering multi-hop logical queries on Knowledge Graphs (KGs). This is a complicated task because, in real-world scenarios, the graphs tend to be large and incomplete. Most previous works have been unable to create models that accept full First-Order Logical (FOL) queries, which include negative queries, and have only been able to process a limited set of query structures. Additionally, most methods present logic operators that can only perform the logical operation they are made for. We introduce a set of models that use Neural Networks to create one-point vector embeddings to answer the queries. The versatility of neural networks allows the framework to handle FOL queries with Conjunction ($\wedge$), Disjunction ($\vee$) and Negation ($\neg$) operators. We demonstrate experimentally the performance of our model through extensive experimentation on well-known benchmarking datasets. Besides having more versatile operators, the models achieve a 10\% relative increase over the best performing state of the art and more than 30\% over the original method based on single-point vector embeddings.

cs.AI↗

arXiv2022

Non-abelian gauge fields in circuit systems

Circuits can provide a platform to study novel physics and have been used, for example, to explore various topological phases. Gauge fields-particularly, non-Abelian gauge fields-can play a pivotal role in the design and modulation of novel physical states, but their circuit implementation has so far been limited. Here we show that non-Abelian gauge fields can be synthesized in circuits created from building blocks that consist of capacitors, inductors and resistors. With these building blocks, we create circuit designs for the spin-orbit interaction and the topological Chern state, which are phenomena that represent non-Abelian gauge fields in momentum space. We also use the approach to design non-reciprocal circuits that can be used to implement the non-Abelian Aharonov-Bohm effect in real space.

cond-mat.mes-hall↗

arXiv2022

Profiling Television Watching Behaviour Using Bayesian Hierarchical Joint Models for Time-to-Event and Count Data

Customer churn prediction is a valuable task in many industries. In telecommunications it presents great challenges, given the high dimensionality of the data, and how difficult it is to identify underlying frustration signatures, which may represent an important driver regarding future churn behaviour. Here, we propose a novel Bayesian hierarchical joint model that is able to characterise customer profiles based on how many events take place within different television watching journeys, and how long it takes between events. The model drastically reduces the dimensionality of the data from thousands of observations per customer to 11 customer-level parameter estimates and random effects. We test our methodology using data from 40 BT customers (20 active and 20 who eventually cancelled their subscription) whose TV watching behaviours were recorded from October to December 2019, totalling approximately half a million observations. Employing different machine learning techniques using the parameter estimates and random effects from the Bayesian hierarchical model as features yielded up to 92\% accuracy predicting churn, associated with 100\% true positive rates and false positive rates as low as 14\% on a validation set. Our proposed methodology represents an efficient way of reducing the dimensionality of the data, while at the same time maintaining high descriptive and predictive capabilities. We provide code to implement the Bayesian model at https://github.com/rafamoral/profiling_tv_watching_behaviour.

stat.ME↗

arXiv2022

Giant and Reversible Electronic Structure Evolution in a Magnetic Topological Material EuCd2As2

The electronic structure and the physical properties of quantum materials can be significantly altered by charge carrier doping and magnetic state transition. Here we report a discovery of a giant and reversible electronic structure evolution with doping in a magnetic topological material. By performing high-resolution angle-resolved photoemission measurements on EuCd2As2,we found that a huge amount of hole doping can be introduced into the sample surface due to surface absorption. The electronic structure exhibits a dramatic change with the hole doping which can not be described by a rigid band shift. Prominent band splitting is observed at high doping which corresponds to a doping-induced magnetic transition at low temperature (below -15 K) from an antiferromagnetic state to a ferromagnetic state. These results have established a detailed electronic phase diagram of EuCd2As2 where the electronic structure and the magnetic structure change systematically and dramatically with the doping level. They further suggest that the transport, magnetic and topological properties of EuCd2As2 can be greatly modified by doping. These work will stimulate further investigations to explore for new phenomena and properties in doping this magnetic topological material.

cond-mat.mtrl-sci↗

arXiv2022

Nanoscale three-dimensional magnetic sensing with a probabilistic nanomagnet driven by spin-orbit torque

Detection of vector magnetic fields at nanoscale dimensions is critical in applications ranging from basic material science, to medical diagnostic. Meanwhile, an all-electric operation is of great significance for achieving a simple and compact sensing system. Here, we propose and experimentally demonstrate a simple approach to sensing a vector magnetic field at nanoscale dimensions, by monitoring a probabilistic nanomagnet's transition probability from a metastable state, excited by a driving current due to SOT, to a settled state. We achieve sensitivities for Hx, Hy, and Hz of 1.02%/Oe, 1.09%/Oe and 3.43%/Oe, respectively, with a 200 x 200 nm^2 nanomagnet. The minimum detectable field is dependent on the driving pulse events N, and is expected to be as low as 1 uT if N = 3 x 10^6.

cond-mat.mtrl-sci↗

arXiv2022

SLAM-TKA: Real-time Intra-operative Measurement of Tibial Resection Plane in Conventional Total Knee Arthroplasty

Total knee arthroplasty (TKA) is a common orthopaedic surgery to replace a damaged knee joint with artificial implants. The inaccuracy of achieving the planned implant position can result in the risk of implant component aseptic loosening, wear out, and even a joint revision, and those failures most of the time occur on the tibial side in the conventional jig-based TKA (CON-TKA). This study aims to precisely evaluate the accuracy of the proximal tibial resection plane intra-operatively in real-time such that the evaluation processing changes very little on the CON-TKA operative procedure. Two X-ray radiographs captured during the proximal tibial resection phase together with a pre-operative patient-specific tibia 3D mesh model segmented from computed tomography (CT) scans and a trocar pin 3D mesh model are used in the proposed simultaneous localisation and mapping (SLAM) system to estimate the proximal tibial resection plane. Validations using both simulation and in-vivo datasets are performed to demonstrate the robustness and the potential clinical value of the proposed algorithm.

cs.RO↗

arXiv2022

Topological States in Chevrel Phase Materials from First-principle Calculations

Chevrel phase materials form a family of ternary molybdenum chalcogenides with a general chemical formula $A_x{\rm Mo}_6X_8$ ($A$ = metal elements, $X$ = chalcogen). The variety of $A$ atoms makes a large number of family members and leads to many tunable physical properties, such as the superconductivity, thermoelectricity and the ionic conductivity. In this work, we have further found various nontrivial band topological states in these materials by using first-principle calculations. The compounds having time-reversal symmetry, such as ${\rm BaMo}_6{\rm S}_8$, ${\rm SrMo}_6{\rm S}_8$, and ${\rm Mo}_6{\rm S}_8$, are topological insulators in both of the $R\bar{3}$ and $P\bar{1}$ phases, whereas ${\rm EuMo}_6{\rm S}_8$ within ferromagnetic state, it is an axion insulator in the $R\bar{3}$ phase and a trivial one in the $P\bar{1}$ phase. This indicates that the change of $A$ ions can modify the chemical potential, lattice distortion, and magnetic orders, which offers a unique way to influence the topological states and other properties. We hope this work can stimulate further studies of Chevrel phase materials to find more intriguing phenomena, such as topological superconducting states and Majorana modes.

cond-mat.mtrl-sci↗

arXiv2022

Smart Multi-tenant Federated Learning

Federated learning (FL) is an emerging distributed machine learning method that empowers in-situ model training on decentralized edge devices. However, multiple simultaneous training activities could overload resource-constrained devices. In this work, we propose a smart multi-tenant FL system, MuFL, to effectively coordinate and execute simultaneous training activities. We first formalize the problem of multi-tenant FL, define multi-tenant FL scenarios, and introduce a vanilla multi-tenant FL system that trains activities sequentially to form baselines. Then, we propose two approaches to optimize multi-tenant FL: 1) activity consolidation merges training activities into one activity with a multi-task architecture; 2) after training it for rounds, activity splitting divides it into groups by employing affinities among activities such that activities within a group have better synergy. Extensive experiments demonstrate that MuFL outperforms other methods while consuming 40% less energy. We hope this work will inspire the community to further study and optimize multi-tenant FL.

cs.LG↗

arXiv2022

Chat-to-Design: AI Assisted Personalized Fashion Design

In this demo, we present Chat-to-Design, a new multimodal interaction system for personalized fashion design. Compared to classic systems that recommend apparel based on keywords, Chat-to-Design enables users to design clothes in two steps: 1) coarse-grained selection via conversation and 2) fine-grained editing via an interactive interface. It encompasses three sub-systems to deliver an immersive user experience: A conversation system empowered by natural language understanding to accept users' requests and manages dialogs; A multimodal fashion retrieval system empowered by a large-scale pretrained language-image network to retrieve requested apparel; A fashion design system empowered by emerging generative techniques to edit attributes of retrieved clothes.

cs.AI↗

arXiv2022

Reducing language context confusion for end-to-end code-switching automatic speech recognition

Code-switching deals with alternative languages in communication process. Training end-to-end (E2E) automatic speech recognition (ASR) systems for code-switching is especially challenging as code-switching training data are always insufficient to combat the increased multilingual context confusion due to the presence of more than one language. We propose a language-related attention mechanism to reduce multilingual context confusion for the E2E code-switching ASR model based on the Equivalence Constraint (EC) Theory. The linguistic theory requires that any monolingual fragment that occurs in the code-switching sentence must occur in one of the monolingual sentences. The theory establishes a bridge between monolingual data and code-switching data. We leverage this linguistics theory to design the code-switching E2E ASR model. The proposed model efficiently transfers language knowledge from rich monolingual data to improve the performance of the code-switching ASR model. We evaluate our model on ASRU 2019 Mandarin-English code-switching challenge dataset. Compared to the baseline model, our proposed model achieves a 17.12% relative error reduction.

cs.CL↗

arXiv2022

xFraud: Explainable Fraud Transaction Detection

At online retail platforms, it is crucial to actively detect the risks of transactions to improve customer experience and minimize financial loss. In this work, we propose xFraud, an explainable fraud transaction prediction framework which is mainly composed of a detector and an explainer. The xFraud detector can effectively and efficiently predict the legitimacy of incoming transactions. Specifically, it utilizes a heterogeneous graph neural network to learn expressive representations from the informative heterogeneously typed entities in the transaction logs. The explainer in xFraud can generate meaningful and human-understandable explanations from graphs to facilitate further processes in the business unit. In our experiments with xFraud on real transaction networks with up to 1.1 billion nodes and 3.7 billion edges, xFraud is able to outperform various baseline models in many evaluation metrics while remaining scalable in distributed settings. In addition, we show that xFraud explainer can generate reasonable explanations to significantly assist the business analysis via both quantitative and qualitative evaluations.

cs.LG↗

arXiv2022

Optimizing Performance of Federated Person Re-identification: Benchmarking and Analysis

The increasingly stringent data privacy regulations limit the development of person re-identification (ReID) because person ReID training requires centralizing an enormous amount of data that contains sensitive personal information. To address this problem, we introduce federated person re-identification (FedReID) -- implementing federated learning, an emerging distributed training method, to person ReID. FedReID preserves data privacy by aggregating model updates, instead of raw data, from clients to a central server. Furthermore, we optimize the performance of FedReID under statistical heterogeneity via benchmark analysis. We first construct a benchmark with an enhanced algorithm, two architectures, and nine person ReID datasets with large variances to simulate the real-world statistical heterogeneity. The benchmark results present insights and bottlenecks of FedReID under statistical heterogeneity, including challenges in convergence and poor performance on datasets with large volumes. Based on these insights, we propose three optimization approaches: (1) We adopt knowledge distillation to facilitate the convergence of FedReID by better transferring knowledge from clients to the server; (2) We introduce client clustering to improve the performance of large datasets by aggregating clients with similar data distributions; (3) We propose cosine distance weight to elevate performance by dynamically updating the weights for aggregation depending on how well models are trained in clients. Extensive experiments demonstrate that these approaches achieve satisfying convergence with much better performance on all datasets. We believe that FedReID will shed light on implementing and optimizing federated learning on more computer vision applications.

cs.CV↗

arXiv2022

Two-dimensional Obstructed Atomic Insulators with Fractional Corner Charge in MA$_2$Z$_4$ Family

According to topological quantum chemistry, a class of electronic materials have been called obstructed atomic insulators (OAIs), in which a portion of valence electrons necessarily have their centers located on some empty $\textit{Wyckoff}$ positions without atoms occupation in the lattice. The obstruction of centering these electrons coinciding with their host atoms is nontrivial and results in metallic boundary states when the boundary is properly cut. Here, on basis of first-principles calculations in combination with topological quantum chemistry analysis, we propose two dimensional MA$_2$Z$_4$ (M = Cr, Mo and W; A = Si and Ge, Z = N, P and As) monolayer family are all OAIs. A typical case is the recently synthesized MoSi$_2$N$_4$. Although it is a topological trivial insulator with the occupied electronic states being integer combination of elementary band representations, it has valence electrons centering empty $\textit{Wyckoff}$ positions. It exhibits unique OAI-induced metallic edge states along the (1$\bar{1}$0) edge of MoSi$_2$N$_4$ monolayer and the in-gap corner states at three vertices of certain hexagonal nanodisk samples respecting C$_3$ rotation symmetry. The readily synthesized MoSi$_2$N$_4$ is quite stable and has a large bulk band gap of 1.94 eV, which makes the identification of these edge and corner states most possible for experimental clarification.

cond-mat.mtrl-sci↗

arXiv2022

Towards compact phase-matched and waveguided nonlinear optics in atomically layered semiconductors

Nonlinear frequency conversion provides essential tools for light generation, photon entanglement, and manipulation. Transition metal dichalcogenides (TMDs) possess huge nonlinear susceptibilities and 3R-stacked TMD crystals further combine broken inversion symmetry and aligned layering, representing ideal candidates to boost the nonlinear optical gain with minimal footprint. Here, we report on the efficient frequency conversion of 3R-MoS2, revealing the evolution of its exceptional second-order nonlinear processes along the ordinary (in-plane) and extraordinary (out-of-plane) directions. By measuring second harmonic generation (SHG) of 3R-MoS2 with various thickness - from monolayer (~0.65 nm) to bulk (~1 μm) - we present the first measurement of the in-plane SHG coherence length (~530 nm) at 1520 nm and achieve record nonlinear optical enhancement from a van der Waals material, >10^4 stronger than a monolayer. It is found that 3R-MoS2 slabs exhibit similar conversion efficiencies of lithium niobate, but within propagation lengths >100-fold shorter at telecom wavelengths. Furthermore, along the extraordinary axis, we achieve broadly tunable SHG from 3R-MoS2 in a waveguide geometry, revealing the coherence length in such structure for the first time. We characterize the full refractive index spectrum and quantify both birefringence components in anisotropic 3R-MoS2 crystals with near-field nano-imaging. Empowered with these data we assess the intrinsic limits of the conversion efficiency and nonlinear optical processes in 3R-MoS2 attainable in waveguide geometries. Our analysis highlights the potential of 3R-stacked TMDs for integrated photonics, providing critical parameters for designing highly efficient on-chip nonlinear optical devices including periodically poled structures, resonators, compact optical parametric oscillators and amplifiers, and optical quantum circuits.

physics.optics↗

arXiv2022

Modelling graph dynamics in fraud detection with "Attention"

At online retail platforms, detecting fraudulent accounts and transactions is crucial to improve customer experience, minimize loss, and avoid unauthorized transactions. Despite the variety of different models for deep learning on graphs, few approaches have been proposed for dealing with graphs that are both heterogeneous and dynamic. In this paper, we propose DyHGN (Dynamic Heterogeneous Graph Neural Network) and its variants to capture both temporal and heterogeneous information. We first construct dynamic heterogeneous graphs from registration and transaction data from eBay. Then, we build models with diachronic entity embedding and heterogeneous graph transformer. We also use model explainability techniques to understand the behaviors of DyHGN-* models. Our findings reveal that modelling graph dynamics with heterogeneous inputs need to be conducted with "attention" depending on the data structure, distribution, and computation cost.

cs.LG↗

arXiv2022

Federated Unsupervised Domain Adaptation for Face Recognition

Given labeled data in a source domain, unsupervised domain adaptation has been widely adopted to generalize models for unlabeled data in a target domain, whose data distributions are different. However, existing works are inapplicable to face recognition under privacy constraints because they require sharing of sensitive face images between domains. To address this problem, we propose federated unsupervised domain adaptation for face recognition, FedFR. FedFR jointly optimizes clustering-based domain adaptation and federated learning to elevate performance on the target domain. Specifically, for unlabeled data in the target domain, we enhance a clustering algorithm with distance constrain to improve the quality of predicted pseudo labels. Besides, we propose a new domain constraint loss (DCL) to regularize source domain training in federated learning. Extensive experiments on a newly constructed benchmark demonstrate that FedFR outperforms the baseline and classic methods on the target domain by 3% to 14% on different evaluation metrics.

cs.CV↗