arXiv subjects

Shuai Zhang

Publications and source records attributed to Shuai Zhang.

At least 325 records · Page 18Linked to original sources

arXiv2022

Divergence-aware Federated Self-Supervised Learning

Self-supervised learning (SSL) is capable of learning remarkable representations from centrally available data. Recent works further implement federated learning with SSL to learn from rapidly growing decentralized unlabeled images (e.g., from cameras and phones), often resulted from privacy constraints. Extensive attention has been paid to SSL approaches based on Siamese networks. However, such an effort has not yet revealed deep insights into various fundamental building blocks for the federated self-supervised learning (FedSSL) architecture. We aim to fill in this gap via in-depth empirical study and propose a new method to tackle the non-independently and identically distributed (non-IID) data problem of decentralized data. Firstly, we introduce a generalized FedSSL framework that embraces existing SSL methods based on Siamese networks and presents flexibility catering to future methods. In this framework, a server coordinates multiple clients to conduct SSL training and periodically updates local models of clients with the aggregated global model. Using the framework, our study uncovers unique insights of FedSSL: 1) stop-gradient operation, previously reported to be essential, is not always necessary in FedSSL; 2) retaining local knowledge of clients in FedSSL is particularly beneficial for non-IID data. Inspired by the insights, we then propose a new approach for model update, Federated Divergence-aware Exponential Moving Average update (FedEMA). FedEMA updates local models of clients adaptively using EMA of the global model, where the decay rate is dynamically measured by model divergence. Extensive experiments demonstrate that FedEMA outperforms existing methods by 3-4% on linear evaluation. We hope that this work will provide useful insights for future research.

cs.LG↗

arXiv2022

Secure two-way fiber-optic time transfer against sub-ns asymmetric delay attack

Two-way fiber-optic time transfer is a promising precise time synchronization technique with sub-nanosecond accuracy. However, asymmetric delay attack is a serious threat which cannot be prevent by any encryption method. In this paper, a dynamic model based scheme is proposed to defense the sub-nanosecond asymmetric delay attack. A threshold is set according to the estimated time difference by a two-state clock model where the fixed frequency difference is excluded from the time difference to detect the asymmetric delay attack which is smaller than the time difference induced by the fixed frequency difference. Theoretical simulation and experimental demonstration are implemented to prove the feasibility of the scheme. A two-way fiber-optic time transfer system with time stability with 24.5ps, 3.98ps, and 2.95ps at 1s, 10s, and 100s averaging time is shown under sub-ns asymmetric time delay attack experimentally. The proposed method provides a promising secure sub-ns precise time synchronization technique against asymmetric delay attack.

quant-ph↗

arXiv2022

How does unlabeled data improve generalization in self-training? A one-hidden-layer theoretical analysis

Self-training, a semi-supervised learning algorithm, leverages a large amount of unlabeled data to improve learning when the labeled data are limited. Despite empirical successes, its theoretical characterization remains elusive. To the best of our knowledge, this work establishes the first theoretical analysis for the known iterative self-training paradigm and proves the benefits of unlabeled data in both training convergence and generalization ability. To make our theoretical analysis feasible, we focus on the case of one-hidden-layer neural networks. However, theoretical understanding of iterative self-training is non-trivial even for a shallow neural network. One of the key challenges is that existing neural network landscape analysis built upon supervised learning no longer holds in the (semi-supervised) self-training paradigm. We address this challenge and prove that iterative self-training converges linearly with both convergence rate and generalization accuracy improved in the order of $1/\sqrt{M}$, where $M$ is the number of unlabeled samples. Experiments from shallow neural networks to deep neural networks are also provided to justify the correctness of our established theoretical insights on self-training.

cs.LG↗

arXiv2022

A Tight Three-parameter Correlation and Related Classification on Gamma-Ray Bursts

Gamma-ray bursts (GRBs) are widely believed to be from massive collapsars and/or compact binary mergers, which accordingly, would generate long and short GRBs, respectively. The details on this classification scheme have been in constant debate given more and more observational data available to us. In this work, we apply a series of data mining methods to studying the potential classification information contained in the prompt emission of GRBs detected by the Fermi Gamma-ray Burst Monitor. A tight global correlation is found between fluence ($f$), peak flux ($F$) and prompt duration ($T_{90}$) which takes the form of $ \log {\it f}= 0.75 \log T_{90} +0.92 \log F -7.14$. Based on this correlation, we can define a new parameter $L = 1.66\log T_{90} + 0.84 \log {\it f} - 0.46 \log F + 3.24$ by linear discriminant analysis that would distinguish between long and short GRBs with much less ambiguity than $T_{90}$. We also discussed the three subclasses scheme of GRB classification derived from clusters analysis based on a Gaussian mixture model, and suggest that, besides SGRBs, LGRBs may be divided into long-bright gamma-ray bursts (LBGRBs) and long-faint gamma-ray bursts (LFGRBs), LBGRBs have statistical higher $f$ and $F$ than LFGRBs; further statistical analysis found that LBGRBs also have higher number of GRB pulses than LFGRBs.

astro-ph.HE↗

arXiv2022

Charge Carrier Mediation and Ferromagnetism induced in MnBi6Te10 Magnetic Topological Insulators by antimony doping

A new kind of intrinsic magnetic topological insulators (MTI) MnBi2Te4 family have shed light on the observation of novel topological quantum effect such as quantum anomalous Hall effect (QAHE). However, the strong anti-ferromagnetic (AFM) coupling and high carrier concentration in the bulk hinder the practical applications. In closely related materials MnBi4Te7 and MnBi6Te10, the interlayer magnetic coupling is greatly suppressed by Bi2Te3 layer intercalation. However, AFM is still the ground state in these compounds. Here by magnetic and transport measurements, we demonstrate that Sb substitutional dopant plays a dual role in MnBi6Te10, which can not only adjust the charge carrier type and the concentration, but also induce the solid into a ferromagnetic (FM) ground state. AFM ground state region which is also close to the charge neutral point can be found in the phase diagram of Mn(SbxBi1-x)6Te10 when x ~ 0.25. An intrinsic FM-MTI candidate is thus demonstrated, and it may take a step further for the realization of high-quality and high-temperature QAHE and the related topological quantum effects in the future.

cond-mat.mtrl-sci↗

arXiv2022

EasyFL: A Low-code Federated Learning Platform For Dummies

Academia and industry have developed several platforms to support the popular privacy-preserving distributed learning method -- Federated Learning (FL). However, these platforms are complex to use and require a deep understanding of FL, which imposes high barriers to entry for beginners, limits the productivity of researchers, and compromises deployment efficiency. In this paper, we propose the first low-code FL platform, EasyFL, to enable users with various levels of expertise to experiment and prototype FL applications with little coding. We achieve this goal while ensuring great flexibility and extensibility for customization by unifying simple API design, modular design, and granular training flow abstraction. With only a few lines of code, EasyFL empowers them with many out-of-the-box functionalities to accelerate experimentation and deployment. These practical functionalities are heterogeneity simulation, comprehensive tracking, distributed training optimization, and seamless deployment. They are proposed based on challenges identified in the proposed FL life cycle. Compared with other platforms, EasyFL not only requires just three lines of code (at least 10x lesser) to build a vanilla FL application but also incurs lower training overhead. Besides, our evaluations demonstrate that EasyFL expedites distributed training by 1.5x. It also improves the efficiency of deployment. We believe that EasyFL will increase the productivity of researchers and democratize FL to wider audiences.

cs.DC↗

arXiv2021

Nature of the bonded-to-atomic transition in liquid silica to TPa pressures

First-principles calculations and analysis of the thermodynamic, structural, and electronic properties of liquid SiO$_2$ characterize the bonded-to-atomic transition at 0.1--1.6 TPa and 10$^4$--10$^5$ K (1--7 eV), the high-energy-density regime relevant to understanding planetary interiors. We find strong ionic bonds that become short-lived due to high kinetics during the transition, with sensitivity of the transition temperature to pressure, and our calculated Hugoniots agree with past experimental data. These results reconcile previous experimental and theoretical findings by clarifying the nature of the bond dissociation process in early Earth and "rocky" (oxide) constituents of large planets.

cond-mat.mtrl-sci↗

arXiv2021

Clustering COVID-19 Lung Scans

With the ongoing COVID-19 pandemic, understanding the characteristics of the virus has become an important and challenging task in the scientific community. While tests do exist for COVID-19, the goal of our research is to explore other methods of identifying infected individuals. Our group applied unsupervised clustering techniques to explore a dataset of lungscans of COVID-19 infected, Viral Pneumonia infected, and healthy individuals. This is an important area to explore as COVID-19 is a novel disease that is currently being studied in detail. Our methodology explores the potential that unsupervised clustering algorithms have to reveal important hidden differences between COVID-19 and other respiratory illnesses. Our experiments use: Principal Component Analysis (PCA), K-Means++ (KM++) and the recently developed Robust Continuous Clustering algorithm (RCC). We evaluate the performance of KM++ and RCC in clustering COVID-19 lung scans using the Adjusted Mutual Information (AMI) score.

cs.CV↗

arXiv2021

Nanoscale Optical Imaging of 2D Semiconductor Stacking Orders by Exciton-Enhanced Second Harmonic Generation

Second harmonic generation (SHG) is a nonlinear optical response arising exclusively from broken inversion symmetry in the electric-dipole limit. Recently, SHG has attracted widespread interest as a versatile and noninvasive tool for characterization of crystal symmetry and emerging ferroic or topological orders in quantum materials. However, conventional far-field optics is unable to probe local symmetry at the deep subwavelength scale. Here, we demonstrate near-field SHG imaging of 2D semiconductors and heterostructures with the spatial resolution down to 20 nm using a scattering-type nano-optical apparatus. We show that near-field SHG efficiency is greatly enhanced by excitons in atomically thin transition metal dichalcogenides. Furthermore, by correlating nonlinear and linear scattering-type nano-imaging, we resolve nanoscale variations of interlayer stacking order in bilayer WSe2, and reveal the stacking-tuned excitonic light-matter-interactions. Our work demonstrates nonlinear optical interrogation of crystal symmetry and structure-property relationships at the nanometer length scales relevant to emerging properties in quantum materials.

physics.optics↗

arXiv2021

Why Lottery Ticket Wins? A Theoretical Perspective of Sample Complexity on Pruned Neural Networks

The \textit{lottery ticket hypothesis} (LTH) states that learning on a properly pruned network (the \textit{winning ticket}) improves test accuracy over the original unpruned network. Although LTH has been justified empirically in a broad range of deep neural network (DNN) involved applications like computer vision and natural language processing, the theoretical validation of the improved generalization of a winning ticket remains elusive. To the best of our knowledge, our work, for the first time, characterizes the performance of training a pruned neural network by analyzing the geometric structure of the objective function and the sample complexity to achieve zero generalization error. We show that the convex region near a desirable model with guaranteed generalization enlarges as the neural network model is pruned, indicating the structural importance of a winning ticket. Moreover, when the algorithm for training a pruned neural network is specified as an (accelerated) stochastic gradient descent algorithm, we theoretically show that the number of samples required for achieving zero generalization error is proportional to the number of the non-pruned weights in the hidden layer. With a fixed number of samples, training a pruned neural network enjoys a faster convergence rate to the desired model than training the original unpruned one, providing a formal justification of the improved generalization of the winning ticket. Our theoretical results are acquired from learning a pruned neural network of one hidden layer, while experimental results are further provided to justify the implications in pruning multi-layer neural networks.

cs.LG↗

arXiv2021

Quantitative analysis of diffraction by liquids using a pink-spectrum X-ray source

We describes a new approach for performing quantitative structure-factor analysis and density measurements of liquids using x-ray diffraction with a pink-spectrum x-ray source. The methodology corrects for the pink beam effect by performing a Taylor series expansion of the diffraction signal. The mean density, background scale factor, peak x-ray energy about which the expansion is performed, and the cutoff radius for density measurement are estimated using the derivative-free optimization scheme. The formalism is demonstrated for a simulated radial distribution function for tin. Finally, the proposed methodology is applied to experimental data on shock compressed tin recorded at the Dynamic Compression Sector at the Advanced Photon Source, with derived densities comparing favorably to other experimental results and the equations of state of tin.

physics.app-ph↗

arXiv2021

Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERT

Attention-based encoder-decoder (AED) models have achieved promising performance in speech recognition. However, because the decoder predicts text tokens (such as characters or words) in an autoregressive manner, it is difficult for an AED model to predict all tokens in parallel. This makes the inference speed relatively slow. We believe that because the encoder already captures the whole speech utterance, which has the token-level relationship implicitly, we can predict a token without explicitly autoregressive language modeling. When the prediction of a token does not rely on other tokens, the parallel prediction of all tokens in the sequence is realizable. Based on this idea, we propose a non-autoregressive speech recognition model called LASO (Listen Attentively, and Spell Once). The model consists of an encoder, a decoder, and a position dependent summarizer (PDS). The three modules are based on basic attention blocks. The encoder extracts high-level representations from the speech. The PDS uses positional encodings corresponding to tokens to convert the acoustic representations into token-level representations. The decoder further captures token-level relationships with the self-attention mechanism. At last, the probability distribution on the vocabulary is computed for each token position. Therefore, speech recognition is re-formulated as a position-wise classification problem. Further, we propose a cross-modal transfer learning method to refine semantics from a large-scale pre-trained language model BERT for improving the performance.

cs.CL↗

arXiv2021

Onset of low-frequency capillary waves driven by high-frequency ultrasound

High frequency thickness mode ultrasound is an energy-efficient way to atomize high-viscosity fluid at high flow rate into fine aerosol mists of micron-sized droplet distributions. However the complex physics of the atomization process is not well understood. It is found that with low power the droplet vibrates at low frequency (O[100 Hz]) when driven by high-frequency ultrasound (O[1 MHz] and above). To study the mechanism of the energy transfer that spans these vastly different timescales, we measure the droplet's interfacial response to 6.6~MHz ultrasound excitation using high-speed digital holography. We show that the onset of low-frequency capillary waves is driven by feedback interplay between the acoustic radiation pressure distribution and the droplet surface. These dynamics are mediated by the Young-Laplace boundary between the droplet interior and ambient environment. Numerical simulations are performed via global optimization against the rigorously defined interfacial physics. The proposed pressure-interface feedback model is explicitly based on the pressure distribution hypothesis. For low power acoustic excitation, the simulations reveal a stable oscillatory feedback that induces capillary wave formation. The simulation results are confirmed with direct observations of the microscale droplet interface dynamics as provided by the high resolution holographic measurements. The pressure-interface feedback model accurately predicts the on-source vibration amplitude required to initiate capillary waves, and interfacial oscillation amplitude and frequency. The radiation pressure distribution is likewise confirmed with particle migration observations. Viscous effects on wave attenuation are also studied by comparing experimental and simulated results for a pure water droplet and 90% wt.- 10% wt. glycerol-water solution droplet.

physics.flu-dyn↗

arXiv2021

Improving Network Slimming with Nonconvex Regularization

Convolutional neural networks (CNNs) have developed to become powerful models for various computer vision tasks ranging from object detection to semantic segmentation. However, most of the state-of-the-art CNNs cannot be deployed directly on edge devices such as smartphones and drones, which need low latency under limited power and memory bandwidth. One popular, straightforward approach to compressing CNNs is network slimming, which imposes $\ell_1$ regularization on the channel-associated scaling factors via the batch normalization layers during training. Network slimming thereby identifies insignificant channels that can be pruned for inference. In this paper, we propose replacing the $\ell_1$ penalty with an alternative nonconvex, sparsity-inducing penalty in order to yield a more compressed and/or accurate CNN architecture. We investigate $\ell_p (0 < p < 1)$, transformed $\ell_1$ (T$\ell_1$), minimax concave penalty (MCP), and smoothly clipped absolute deviation (SCAD) due to their recent successes and popularity in solving sparse optimization problems, such as compressed sensing and variable selection. We demonstrate the effectiveness of network slimming with nonconvex penalties on three neural network architectures -- VGG-19, DenseNet-40, and ResNet-164 -- on standard image classification datasets. Based on the numerical experiments, T$\ell_1$ preserves model accuracy against channel pruning, $\ell_{1/2, 3/4}$ yield better compressed models with similar accuracies after retraining as $\ell_1$, and MCP and SCAD provide more accurate models after retraining with similar compression as $\ell_1$. Network slimming with T$\ell_1$ regularization also outperforms the latest Bayesian modification of network slimming in compressing a CNN architecture in terms of memory storage while preserving its model accuracy after channel pruning.

cs.CV↗

arXiv2021

Collaborative Unsupervised Visual Representation Learning from Decentralized Data

Unsupervised representation learning has achieved outstanding performances using centralized data available on the Internet. However, the increasing awareness of privacy protection limits sharing of decentralized unlabeled image data that grows explosively in multiple parties (e.g., mobile phones and cameras). As such, a natural problem is how to leverage these data to learn visual representations for downstream tasks while preserving data privacy. To address this problem, we propose a novel federated unsupervised learning framework, FedU. In this framework, each party trains models from unlabeled data independently using contrastive learning with an online network and a target network. Then, a central server aggregates trained models and updates clients' models with the aggregated model. It preserves data privacy as each party only has access to its raw data. Decentralized data among multiple parties are normally non-independent and identically distributed (non-IID), leading to performance degradation. To tackle this challenge, we propose two simple but effective methods: 1) We design the communication protocol to upload only the encoders of online networks for server aggregation and update them with the aggregated encoder; 2) We introduce a new module to dynamically decide how to update predictors based on the divergence caused by non-IID. The predictor is the other component of the online network. Extensive experiments and ablations demonstrate the effectiveness and significance of FedU. It outperforms training with only one party by over 5% and other methods by over 14% in linear and semi-supervised evaluation on non-IID data.

cs.DC↗

arXiv2021

Joint Optimization in Edge-Cloud Continuum for Federated Unsupervised Person Re-identification

Person re-identification (ReID) aims to re-identify a person from non-overlapping camera views. Since person ReID data contains sensitive personal information, researchers have adopted federated learning, an emerging distributed training method, to mitigate the privacy leakage risks. However, existing studies rely on data labels that are laborious and time-consuming to obtain. We present FedUReID, a federated unsupervised person ReID system to learn person ReID models without any labels while preserving privacy. FedUReID enables in-situ model training on edges with unlabeled data. A cloud server aggregates models from edges instead of centralizing raw data to preserve data privacy. Moreover, to tackle the problem that edges vary in data volumes and distributions, we personalize training in edges with joint optimization of cloud and edge. Specifically, we propose personalized epoch to reassign computation throughout training, personalized clustering to iteratively predict suitable labels for unlabeled data, and personalized update to adapt the server aggregated model to each edge. Extensive experiments on eight person ReID datasets demonstrate that FedUReID not only achieves higher accuracy but also reduces computation cost by 29%. Our FedUReID system with the joint optimization will shed light on implementing federated learning to more multimedia tasks without data labels.

cs.DC↗

arXiv2021

Locate and Label: A Two-stage Identifier for Nested Named Entity Recognition

Named entity recognition (NER) is a well-studied task in natural language processing. Traditional NER research only deals with flat entities and ignores nested entities. The span-based methods treat entity recognition as a span classification task. Although these methods have the innate ability to handle nested NER, they suffer from high computational cost, ignorance of boundary information, under-utilization of the spans that partially match with entities, and difficulties in long entity recognition. To tackle these issues, we propose a two-stage entity identifier. First we generate span proposals by filtering and boundary regression on the seed spans to locate the entities, and then label the boundary-adjusted span proposals with the corresponding categories. Our method effectively utilizes the boundary information of entities and partially matched spans during training. Through boundary regression, entities of any length can be covered theoretically, which improves the ability to recognize long entities. In addition, many low-quality seed spans are filtered out in the first stage, which reduces the time complexity of inference. Experiments on nested NER datasets demonstrate that our proposed method outperforms previous state-of-the-art models.

cs.CL↗

arXiv2021

An Efficient and Balanced Graph Partition Algorithm for the Subgraph-Centric Programming Model on Large-scale Power-law Graphs

The subgraph-centric programming model is a promising approach and has been applied in many state-of-the-art distributed graph computing frameworks. However, traditional graph partition algorithms have significant difficulties in processing large-scale power-law graphs. The major problem is the communication bottleneck found in many subgraph-centric frameworks. Detailed analysis indicates that the communication bottleneck is caused by the huge communication volume or the extreme message imbalance among partitioned subgraphs. The traditional partition algorithms do not consider both factors at the same time, especially on power-law graphs. In this paper, we propose a novel efficient and balanced vertex-cut graph partition algorithm (EBV) which grants appropriate weights to the overall communication cost and communication balance. We observe that the number of replicated vertices and the balance of edge and vertex assignment have a great influence on communication patterns of distributed subgraph-centric frameworks, which further affect the overall performance. Based on this insight, We design an evaluation function that quantifies the proportion of replicated vertices and the balance of edges and vertices assignments as important parameters. Besides, we sort the order of edge processing by the sum of end-vertices' degrees from small to large. Experiments show that EBV reduces replication factor and communication by at least 21.8% and 23.7% respectively than other self-based partition algorithms. When deployed in the subgraph-centric framework, it reduces the running time on power-law graphs by an average of 16.8% compared with the state-of-the-art partition algorithm. Our results indicate that EBV has a great potential in improving the performance of subgraph-centric frameworks for the parallel large-scale power-law graph processing.

cs.DC↗