arXiv subjects

Shuai Zhang

Publications and source records attributed to Shuai Zhang.

At least 343 records · Page 19Linked to original sources

arXiv2021

Intrinsically Honeycomb-patterned Hydrogenated Graphene with Spin Polarized Edge-states

Since the advent of graphene ushered the era of two-dimensional materials, many forms of hydrogenated graphene have been reported, exhibiting diverse properties ranging from a tunable band gap to ferromagnetic ordering. Patterned hydrogenated graphene with micron-scale patterns has been fabricated by lithographic means. Here we report successful millimeter-scale synthesis of an intrinsically honeycomb patterned form of hydrogenated graphene on Ru(0001) by epitaxial growth followed by hydrogenation. Combining scanning tunneling microscopy observations with density-functional-theory (DFT) calculations, we reveal that an atomic-hydrogen layer intercalates between graphene and Ru(0001). The result is a hydrogen honeycomb structure that serves as a template for the final hydrogenation, which converts the graphene into graphane only over the template, yielding honeycomb-patterned hydrogenated graphene (HPHG). In effect, HPHG is a form of patterned graphane. DFT calculations find that the unhydrogenated graphene regions embedded in the patterned graphane exhibit spin-polarized edge states. This type of growth mechanism provides new pathways for the fabrication of intrinsically patterned graphene-based materials.

cond-mat.mtrl-sci↗

arXiv2021

A Sequence-to-Set Network for Nested Named Entity Recognition

Named entity recognition (NER) is a widely studied task in natural language processing. Recently, a growing number of studies have focused on the nested NER. The span-based methods, considering the entity recognition as a span classification task, can deal with nested entities naturally. But they suffer from the huge search space and the lack of interactions between entities. To address these issues, we propose a novel sequence-to-set neural network for nested NER. Instead of specifying candidate spans in advance, we provide a fixed set of learnable vectors to learn the patterns of the valuable spans. We utilize a non-autoregressive decoder to predict the final set of entities in one pass, in which we are able to capture dependencies between entities. Compared with the sequence-to-sequence method, our model is more suitable for such unordered recognition task as it is insensitive to the label order. In addition, we utilize the loss function based on bipartite matching to compute the overall training loss. Experimental results show that our proposed model achieves state-of-the-art on three nested NER corpora: ACE 2004, ACE 2005 and KBP 2017. The code is available at https://github.com/zqtan1024/sequence-to-set.

cs.CL↗

arXiv2021

Towards Unsupervised Domain Adaptation for Deep Face Recognition under Privacy Constraints via Federated Learning

Unsupervised domain adaptation has been widely adopted to generalize models for unlabeled data in a target domain, given labeled data in a source domain, whose data distributions differ from the target domain. However, existing works are inapplicable to face recognition under privacy constraints because they require sharing sensitive face images between two domains. To address this problem, we propose a novel unsupervised federated face recognition approach (FedFR). FedFR improves the performance in the target domain by iteratively aggregating knowledge from the source domain through federated learning. It protects data privacy by transferring models instead of raw data between domains. Besides, we propose a new domain constraint loss (DCL) to regularize source domain training. DCL suppresses the data volume dominance of the source domain. We also enhance a hierarchical clustering algorithm to predict pseudo labels for the unlabeled target domain accurately. To this end, FedFR forms an end-to-end training pipeline: (1) pre-train in the source domain; (2) predict pseudo labels by clustering in the target domain; (3) conduct domain-constrained federated learning across two domains. Extensive experiments and analysis on two newly constructed benchmarks demonstrate the effectiveness of FedFR. It outperforms the baseline and classic methods in the target domain by over 4% on the more realistic benchmark. We believe that FedFR will shed light on applying federated learning to more computer vision tasks under privacy constraints.

cs.CV↗

arXiv2021

Exploring particle dynamics during self-organization processes via rotationally invariant latent representations

The dynamic of complex ordering systems with active rotational degrees of freedom exemplified by protein self-assembly is explored using a machine learning workflow that combines deep learning-based semantic segmentation and rotationally invariant variational autoencoder-based analysis of orientation and shape evolution. The latter allows for disentanglement of the particle orientation from other degrees of freedom and compensates for shifts. The disentangled representations in the latent space encode the rich spectrum of local transitions that can now be visualized and explored via continuous variables. The time dependence of ensemble averages allows insight into the time dynamics of the system, and in particular, illustrates the presence of the potential ordering transition. Finally, analysis of the latent variables along the single-particle trajectory allows tracing these parameters on a single particle level. The proposed approach is expected to be universally applicable for the description of the imaging data in optical, scanning probe, and electron microscopy seeking to understand the dynamics of complex systems where rotations are a significant part of the process.

cond-mat.soft↗

arXiv2021

FSR: Accelerating the Inference Process of Transducer-Based Models by Applying Fast-Skip Regularization

Transducer-based models, such as RNN-Transducer and transformer-transducer, have achieved great success in speech recognition. A typical transducer model decodes the output sequence conditioned on the current acoustic state and previously predicted tokens step by step. Statistically, The number of blank tokens in the prediction results accounts for nearly 90\% of all tokens. It takes a lot of computation and time to predict the blank tokens, but only the non-blank tokens will appear in the final output sequence. Therefore, we propose a method named fast-skip regularization, which tries to align the blank position predicted by a transducer with that predicted by a CTC model. During the inference, the transducer model can predict the blank tokens in advance by a simple CTC project layer without many complicated forward calculations of the transducer decoder and then skip them, which will reduce the computation and improve the inference speed greatly. All experiments are conducted on a public Chinese mandarin dataset AISHELL-1. The results show that the fast-skip regularization can indeed help the transducer model learn the blank position alignments. Besides, the inference with fast-skip can be speeded up nearly 4 times with only a little performance degradation.

eess.AS↗

arXiv2021

One In A Hundred: Select The Best Predicted Sequence from Numerous Candidates for Streaming Speech Recognition

The RNN-Transducers and improved attention-based encoder-decoder models are widely applied to streaming speech recognition. Compared with these two end-to-end models, the CTC model is more efficient in training and inference. However, it cannot capture the linguistic dependencies between the output tokens. Inspired by the success of two-pass end-to-end models, we introduce a transformer decoder and the two-stage inference method into the streaming CTC model. During inference, the CTC decoder first generates many candidates in a streaming fashion. Then the transformer decoder selects the best candidate based on the corresponding acoustic encoded states. The second-stage transformer decoder can be regarded as a conditional language model. We assume that a large enough number and enough diversity of candidates generated in the first stage can compensate the CTC model for the lack of language modeling ability. All the experiments are conducted on a Chinese Mandarin dataset AISHELL-1. The results show that our proposed model can implement streaming decoding in a fast and straightforward way. Our model can achieve up to a 20% reduction in the character error rate than the baseline CTC model. In addition, our model can also perform non-streaming inference with only a little performance degradation.

eess.AS↗

arXiv2021

TSNAT: Two-Step Non-Autoregressvie Transformer Models for Speech Recognition

The autoregressive (AR) models, such as attention-based encoder-decoder models and RNN-Transducer, have achieved great success in speech recognition. They predict the output sequence conditioned on the previous tokens and acoustic encoded states, which is inefficient on GPUs. The non-autoregressive (NAR) models can get rid of the temporal dependency between the output tokens and predict the entire output tokens in at least one step. However, the NAR model still faces two major problems. On the one hand, there is still a great gap in performance between the NAR models and the advanced AR models. On the other hand, it's difficult for most of the NAR models to train and converge. To address these two problems, we propose a new model named the two-step non-autoregressive transformer(TSNAT), which improves the performance and accelerating the convergence of the NAR model by learning prior knowledge from a parameters-sharing AR model. Furthermore, we introduce the two-stage method into the inference process, which improves the model performance greatly. All the experiments are conducted on a public Chinese mandarin dataset ASIEHLL-1. The results show that the TSNAT can achieve a competitive performance with the AR model and outperform many complicated NAR models.

eess.AS↗

arXiv2021

Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting

Many real-world applications require the prediction of long sequence time-series, such as electricity consumption planning. Long sequence time-series forecasting (LSTF) demands a high prediction capacity of the model, which is the ability to capture precise long-range dependency coupling between output and input efficiently. Recent studies have shown the potential of Transformer to increase the prediction capacity. However, there are several severe issues with Transformer that prevent it from being directly applicable to LSTF, including quadratic time complexity, high memory usage, and inherent limitation of the encoder-decoder architecture. To address these issues, we design an efficient transformer-based model for LSTF, named Informer, with three distinctive characteristics: (i) a $ProbSparse$ self-attention mechanism, which achieves $O(L \log L)$ in time complexity and memory usage, and has comparable performance on sequences' dependency alignment. (ii) the self-attention distilling highlights dominating attention by halving cascading layer input, and efficiently handles extreme long input sequences. (iii) the generative style decoder, while conceptually simple, predicts the long time-series sequences at one forward operation rather than a step-by-step way, which drastically improves the inference speed of long-sequence predictions. Extensive experiments on four large-scale datasets demonstrate that Informer significantly outperforms existing methods and provides a new solution to the LSTF problem.

cs.LG↗

arXiv2021

Switch Spaces: Learning Product Spaces with Sparse Gating

Learning embedding spaces of suitable geometry is critical for representation learning. In order for learned representations to be effective and efficient, it is ideal that the geometric inductive bias aligns well with the underlying structure of the data. In this paper, we propose Switch Spaces, a data-driven approach for learning representations in product space. Specifically, product spaces (or manifolds) are spaces of mixed curvature, i.e., a combination of multiple euclidean and non-euclidean (hyperbolic, spherical) manifolds. To this end, we introduce sparse gating mechanisms that learn to choose, combine and switch spaces, allowing them to be switchable depending on the input data with specialization. Additionally, the proposed method is also efficient and has a constant computational complexity regardless of the model size. Experiments on knowledge graph completion and item recommendations show that the proposed switch space achieves new state-of-the-art performances, outperforming pure product spaces and recently proposed task-specific models.

cs.LG↗

arXiv2021

Integrating Knowledge into End-to-End Speech Recognition from External Text-Only Data

Attention-based encoder-decoder (AED) models have achieved promising performance in speech recognition. However, because of the end-to-end training, an AED model is usually trained with speech-text paired data. It is challenging to incorporate external text-only data into AED models. Another issue of the AED model is that it does not use the right context of a text token while predicting the token. To alleviate the above two issues, we propose a unified method called LST (Learn Spelling from Teachers) to integrate knowledge into an AED model from the external text-only data and leverage the whole context in a sentence. The method is divided into two stages. First, in the representation stage, a language model is trained on the text. It can be seen as that the knowledge in the text is compressed into the LM. Then, at the transferring stage, the knowledge is transferred to the AED model via teacher-student learning. To further use the whole context of the text sentence, we propose an LM called causal cloze completer (COR), which estimates the probability of a token, given both the left context and the right context of it. Therefore, with LST training, the AED model can leverage the whole context in the sentence. Different from fusion based methods, which use LM during decoding, the proposed method does not increase any extra complexity at the inference stage. We conduct experiments on two scales of public Chinese datasets AISHELL-1 and AISHELL-2. The experimental results demonstrate the effectiveness of leveraging external text-only data and the whole context in a sentence with our proposed method, compared with baseline hybrid systems and AED model based systems.

eess.AS↗

arXiv2021

MicroRec: Efficient Recommendation Inference by Hardware and Data Structure Solutions

Deep neural networks are widely used in personalized recommendation systems. Unlike regular DNN inference workloads, recommendation inference is memory-bound due to the many random memory accesses needed to lookup the embedding tables. The inference is also heavily constrained in terms of latency because producing a recommendation for a user must be done in about tens of milliseconds. In this paper, we propose MicroRec, a high-performance inference engine for recommendation systems. MicroRec accelerates recommendation inference by (1) redesigning the data structures involved in the embeddings to reduce the number of lookups needed and (2) taking advantage of the availability of High-Bandwidth Memory (HBM) in FPGA accelerators to tackle the latency by enabling parallel lookups. We have implemented the resulting design on an FPGA board including the embedding lookup step as well as the complete inference process. Compared to the optimized CPU baseline (16 vCPU, AVX2-enabled), MicroRec achieves 13.8~14.7x speedup on embedding lookup alone and 2.5$~5.4x speedup for the entire recommendation inference in terms of throughput. As for latency, CPU-based engines needs milliseconds for inferring a recommendation while MicroRec only takes microseconds, a significant advantage in real-time recommendation systems.

cs.AR↗

arXiv2021

Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with $1/n$ Parameters

Recent works have demonstrated reasonable success of representation learning in hypercomplex space. Specifically, "fully-connected layers with Quaternions" (4D hypercomplex numbers), which replace real-valued matrix multiplications in fully-connected layers with Hamilton products of Quaternions, both enjoy parameter savings with only 1/4 learnable parameters and achieve comparable performance in various applications. However, one key caveat is that hypercomplex space only exists at very few predefined dimensions (4D, 8D, and 16D). This restricts the flexibility of models that leverage hypercomplex multiplications. To this end, we propose parameterizing hypercomplex multiplications, allowing models to learn multiplication rules from data regardless of whether such rules are predefined. As a result, our method not only subsumes the Hamilton product, but also learns to operate on any arbitrary nD hypercomplex space, providing more architectural flexibility using arbitrarily $1/n$ learnable parameters compared with the fully-connected layer counterpart. Experiments of applications to the LSTM and Transformer models on natural language inference, machine translation, text style transfer, and subject verb agreement demonstrate architectural flexibility and effectiveness of the proposed approach.

cs.LG↗

arXiv2021

Mimicing the Kane-Mele type spin orbit interaction by spin-flexual phonon coupling in graphene devices

On the efforts of enhancing the spin orbit interaction (SOI) of graphene for seeking the dissipationless quantum spin Hall devices, unique Kane-Mele type SOI and high mobility samples are desired. However, common external decoration often introduces extrinsic Rashba-type SOI and simultaneous impurity scattering. Here we show, by the EDTA-Dy molecule decorating, the Kane-Mele type SOI is mimicked with even improved carrier mobility. It is evidenced by the suppressed weak localization at equal carrier densities and simultaneous Elliot-Yafet spin relaxation. The extracted spin scattering time is monotonically dependent on the carrier elastic scattering time, where the Elliot-Yafet plot gives the interaction strength of 3.3 meV. Improved quantum Hall plateaus can be even seen after the external operation. This is attributed to the spin-flexural phonon coupling induced by the enhanced graphene ripples, as revealed by the in-plane magnetotransport measurement.

cond-mat.mes-hall↗

arXiv2021

Experimental evidence on the dissipationless transport of chiral edge state of the high-field Chern insulator in MnBi2Te4 nanodevices

We demonstrate the dissipationless transport of the chiral edge state (CES) in the nanodevices of quantum anomalous Hall insulator candidate MnBi2Te4. The device presents a near-zero longitudinal resistance together with a quantized Hall plateau in excess of 0.97 h/e2 over a range of temperatures from very low up to the Neel temperature of 22 K. Each of four-probe nonlocal measurements gives near-zero resistance and two-probe measurements exhibit a plateau of +1 h/e2, while the results of three-probe nonlocal measurements depend on the magnetic field. This indicates non-dissipation as well as the chirality of the edge state. The CES shows three regimes of temperature dependence, i.e., well-preserved dissipationless transport below 6 K, variable range hopping while increasing the temperature and thermal activation at higher than 22 K. Even at the lowest temperature, a current of over 1.4 μA breaks the dissipationless transport. These form a complete set of evidences of the Chern insulator state in the MnBi2Te4 systems.

cond-mat.mes-hall↗

arXiv2021

Topology Aware Deep Learning for Wireless Network Optimization

Data-driven machine learning approaches have recently been proposed to facilitate wireless network optimization by learning latent knowledge from historical optimization instances. However, existing methods do not well handle the topology information that directly impacts the network optimization results. Directly operating on simple representations, e.g., adjacency matrices, results in poor generalization performance as the learned results depend on specific ordering of the network elements in the training data. To address this issue, we propose a two-stage topology-aware machine learning framework (TALF), which trains a graph embedding unit and a deep feed-forward network (DFN) jointly. By propagating and summarizing the underlying graph topological information, TALF encodes the topology in the vector representation of the optimization instance, which is used by the later DFN to infer critical structures of an optimal or near-optimal solution. The proposed approach is evaluated on a canonical wireless network flow problem with diverse network typologies and flow deployments. In-depth study on trade-off between efficiency and effectiveness of the inference results is also conducted, and we show that our approach is better at differentiate links by saving up to 60% computation time at over 90% solution quality.

cs.NI↗

arXiv2021

Crystal and Electronic Structure of GaTa$_4$Se$_8$ From First-Principle Calculations

GaTa$_4$Se$_8$ belongs to the lacunar spinel family. Its crystal structures is still a puzzle though there have been intensive studies on its novel properties, such as the Mott insulator phase and superconductivity under pressure. In this work, we investigate its phonon spectra through first-principle calculations and proposed it most probably has crystal structure phase transition, which is consistent with several experimental observations. For the prototype lacunar spinel with cubic symmetry of space group $F\bar{4}3m$, its phonon spectra have three soft modes in the whole Brillouin zone, indicating the strong dynamical instability of such crystal structure. In order to find the dynamically stable crystal structure, further calculations indicate two new structures of GaTa$_4$Se$_8$, corresponding to $R3m$ and $P\bar{4}2_{1}m$, verifying that at the ambient pressure, there does exist structure phase transition of GaTa$_4$Se$_8$ from $F\bar{4}3m$ to other structures when the temperature is lowered. We also performed electronic structure calculation for $R3m$ and $P\bar{4}2_{1}m$ structure, showing that $P\bar{4}2_{1}m$ structure GaTa$_4$Se$_8$ is band insulator, and obtained Mott insulator state for $R3m$ structure by DMFT calculation under single-band Hubbard model picture when interaction parameter U is larger than 0.40 eV vs. band width of 0.25 eV. It is reasonable to assume that while lowering the temperature, $F\bar{4}3m$ structure GaTa$_4$Se$_8$ becomes $R3m$ structure GaTa$_4$Se$_8$ first, then $P\bar{4}2_{1}m$ structure GaTa$_4$Se$_8$, because of the symmetry of $P\bar{4}2_{1}m$ is lower than $R3m$ after Jahn-Teller distortion. The structure transition may explain the magnetic susceptibility anomalous at low temperature.

cond-mat.mtrl-sci↗

arXiv2021

Structural Motifs and Bonding in Two Families of Boron Structures Predicted at Megabar Pressures

The complex crystal chemistry of elemental boron has led to numerous proposed structures with distinctive motifs as well as contradictory findings. Herein, evolutionary structure searches performed at 100 GPa have uncovered a series of potential new metastable phases of boron, and bonding analyses were carried out to elucidate their electronic structure. These polymorphs, dynamically stable at 100 GPa, were grouped into two families. The first was derived from the thermodynamic minimum at these conditions, $α$-Ga, whereas channels comprised the second. Two additional intergrowth structures were uncovered, and it was shown they could be constructed by stacking layers of $α$-Ga-like and channel-like allotropes on top of each other. A detailed bonding analysis revealed networks of four-center $σ$-bonding functions linked by two-center B-B bonds in the $α$-Ga based structures, and networks that were largely composed of three-center $σ$-bonding functions in the channel-based structures. Seven of these high pressure phases were found to be metastable at atmospheric conditions, and their Vickers hardnesses were estimated to be $\sim$36 GPa.

cond-mat.mtrl-sci↗

arXiv2020

Comparison of ablators for the polar direct drive exploding pusher platform

We examine the performance of pure boron, boron carbide, high density carbon, and boron nitride ablators in the polar direct drive exploding pusher (PDXP) platform. The platform uses the polar direct drive configuration at the National Ignition Facility to drive high ion temperatures in a room temperature capsule and has potential applications for plasma physics studies and as a neutron source. The higher tensile strength of these materials compared to plastic enables a thinner ablator to support higher gas pressures, which could help optimize its performance for plasma physics experiments, while ablators containing boron enable the possiblity of collecting addtional data to constrain models of the platform. Applying recently developed and experimentally validated equation of state models for the boron materials, we examine the performance of these materials as ablators in 2D simulations, with particular focus on changes to the ablator and gas areal density, as well as the predicted symmetry of the inherently 2D implosion.

physics.comp-ph↗