arXiv subjects

Xudong Wang

Publications and source records attributed to Xudong Wang.

At least 145 records · Page 8Linked to original sources

arXiv2022

Clipped Hyperbolic Classifiers Are Super-Hyperbolic Classifiers

Hyperbolic space can naturally embed hierarchies, unlike Euclidean space. Hyperbolic Neural Networks (HNNs) exploit such representational power by lifting Euclidean features into hyperbolic space for classification, outperforming Euclidean neural networks (ENNs) on datasets with known semantic hierarchies. However, HNNs underperform ENNs on standard benchmarks without clear hierarchies, greatly restricting HNNs' applicability in practice. Our key insight is that HNNs' poorer general classification performance results from vanishing gradients during backpropagation, caused by their hybrid architecture connecting Euclidean features to a hyperbolic classifier. We propose an effective solution by simply clipping the Euclidean feature magnitude while training HNNs. Our experiments demonstrate that clipped HNNs become super-hyperbolic classifiers: They are not only consistently better than HNNs which already outperform ENNs on hierarchical data, but also on-par with ENNs on MNIST, CIFAR10, CIFAR100 and ImageNet benchmarks, with better adversarial robustness and out-of-distribution detection.

cs.LG↗

arXiv2022

Long-tailed Recognition by Routing Diverse Distribution-Aware Experts

Natural data are often long-tail distributed over semantic classes. Existing recognition methods tackle this imbalanced classification by placing more emphasis on the tail data, through class re-balancing/re-weighting or ensembling over different data groups, resulting in increased tail accuracies but reduced head accuracies. We take a dynamic view of the training data and provide a principled model bias and variance analysis as the training data fluctuates: Existing long-tail classifiers invariably increase the model variance and the head-tail model bias gap remains large, due to more and larger confusion with hard negatives for the tail. We propose a new long-tailed classifier called RoutIng Diverse Experts (RIDE). It reduces the model variance with multiple experts, reduces the model bias with a distribution-aware diversity loss, reduces the computational cost with a dynamic expert routing module. RIDE outperforms the state-of-the-art by 5% to 7% on CIFAR100-LT, ImageNet-LT and iNaturalist 2018 benchmarks. It is also a universal framework that is applicable to various backbone networks, long-tailed algorithms, and training mechanisms for consistent performance gains. Our code is available at: https://github.com/frank-xwang/RIDE-LongTailRecognition.

cs.CV↗

arXiv2022

Unsupervised Hierarchical Semantic Segmentation with Multiview Cosegmentation and Clustering Transformers

Unsupervised semantic segmentation aims to discover groupings within and across images that capture object and view-invariance of a category without external supervision. Grouping naturally has levels of granularity, creating ambiguity in unsupervised segmentation. Existing methods avoid this ambiguity and treat it as a factor outside modeling, whereas we embrace it and desire hierarchical grouping consistency for unsupervised segmentation. We approach unsupervised segmentation as a pixel-wise feature learning problem. Our idea is that a good representation shall reveal not just a particular level of grouping, but any level of grouping in a consistent and predictable manner. We enforce spatial consistency of grouping and bootstrap feature learning with co-segmentation among multiple views of the same image, and enforce semantic consistency across the grouping hierarchy with clustering transformers between coarse- and fine-grained features. We deliver the first data-driven unsupervised hierarchical semantic segmentation method called Hierarchical Segment Grouping (HSG). Capturing visual similarity and statistical co-occurrences, HSG also outperforms existing unsupervised segmentation methods by a large margin on five major object- and scene-centric benchmarks. Our code is publicly available at https://github.com/twke18/HSG .

cs.CV↗

arXiv2022

Debiased Learning from Naturally Imbalanced Pseudo-Labels

Pseudo-labels are confident predictions made on unlabeled target data by a classifier trained on labeled source data. They are widely used for adapting a model to unlabeled data, e.g., in a semi-supervised learning setting. Our key insight is that pseudo-labels are naturally imbalanced due to intrinsic data similarity, even when a model is trained on balanced source data and evaluated on balanced target data. If we address this previously unknown imbalanced classification problem arising from pseudo-labels instead of ground-truth training labels, we could remove model biases towards false majorities created by pseudo-labels. We propose a novel and effective debiased learning method with pseudo-labels, based on counterfactual reasoning and adaptive margins: The former removes the classifier response bias, whereas the latter adjusts the margin of each class according to the imbalance of pseudo-labels. Validated by extensive experimentation, our simple debiased learning delivers significant accuracy gains over the state-of-the-art on ImageNet-1K: 26% for semi-supervised learning with 0.2% annotations and 9% for zero-shot learning. Our code is available at: https://github.com/frank-xwang/debiased-pseudo-labeling.

cs.LG↗

arXiv2022

Random diffusivity processes in an external force field

Brownian yet non-Gaussian processes have recently been observed in numerous biological systems and the corresponding theories have been built based on random diffusivity models. Considering the particularity of random diffusivity, this paper studies the effect of an external force acting on two kinds of random diffusivity models whose difference is embodied in whether the fluctuation-dissipation theorem is valid. Based on the two random diffusivity models, we derive the Fokker-Planck equations with an arbitrary external force, and analyse various observables in the case with a constant force, including the Einstein relation, the moments, the kurtosis, and the asymptotic behaviors of the probability density function of particle's displacement at different time scales. Both the theoretical results and numerical simulations of these observables show significant difference between the two kinds of random diffusivity models, which implies the important role of the fluctuation-dissipation theorem in random diffusivity systems.

cond-mat.stat-mech↗

arXiv2022

Hybrid integration of deterministic quantum dots-based single-photon sources with CMOS-compatible silicon carbide photonics

Thin film 4H-silicon carbide (4H-SiC) is emerging as a contender for realizing large-scale optical quantum circuits due to its high CMOS technology compatibility and large optical nonlinearities. Though, challenges remain in producing wafer-scale 4H-SiC thin film on insulator (4H-SiCOI) for dense integration of photonic circuits, and in efficient coupling of deterministic quantum emitters that are essential for scalable quantum photonics. Here we demonstrate hybrid integration of self-assembled InGaAs quantum dots (QDs) based single-photon sources (SPSs) with wafer-scale 4H-SiC photonic chips prepared by ion slicing technique. By designing a bilayer vertical coupler, we realize generation and highly efficient routing of single-photon emission in the hybrid quantum photonic chip. Furthermore, we realize a chip-integrated beamsplitter operation for triggered single photons through fabricating a 1x2 multi-mode interferometer (MMI) with a symmetric power splitting ratio of 50:50. The successful demonstration of heterogeneously integrating QDs-based SPSs on 4H-SiC photonic chip prepared by ion slicing technique constitutes an important step toward CMOS-compatible, fast reconfigurable quantum photonic circuits with deterministic SPSs.

physics.optics↗

arXiv2021

Synthesis and thermal expansion of chalcogenide MAX phase Hf2SeC

Thermal expansion of MAX phases along different directions tended to be different because of the anisotropy of hexagonal crystals. Herein, a new Hf2SeC phase was synthesized and confirmed to be relatively isotropic, whose coefficients of thermal expansion (CTEs) were determined to be 9.73 μK-1 and 10.18 μK-1 along a and c directions. The strong M-S bond endowed Hf2SC and Zr2SC lower CTEs than Hf2SeC and Zr2SeC. A good relationship between the thermal expansion anisotropy and the ratio of elastic stiffness constant c11 and c33 was established. This straightforward approximation could be used to roughly predict the thermal expansion anisotropy of MAX phases.

cond-mat.mtrl-sci↗

arXiv2021

Novel anomalous diffusion phenomena of underdamped Langevin equation with random parameters

The diffusion behavior of particles moving in complex heterogeneous environment is a very topical issue. We characterize particle's trajectory via an underdamped Langevin system driven by a Gaussian white noise with a time dependent diffusivity of velocity, together with a random relaxation timescale $τ$ to parameterize the effect of complex medium. We mainly concern how the random parameter $τ$ influences the diffusion behavior and ergodic property of this Langevin system. Besides, the comparison between the fixed and random initial velocity $v_0$ is conducted to show the effect of different initial ensembles. The heavy-tailed distribution of $τ$ with finite mean is found to suppress the decay rate of the velocity correlation function and promote the diffusion behavior, playing a competition role to the time dependent diffusivity. More interestingly, a random $v_0$ with a specific distribution depending on random $τ$ also enhances the diffusion. Both the random parameters $τ$ and $v_0$ influence the dynamics of the Langevin system in an non-obvious way, which cannot be ignored even they has finite moments.

cond-mat.stat-mech↗

arXiv2021

Hankel-structured Tensor Robust PCA for Multivariate Traffic Time Series Anomaly Detection

Spatiotemporal traffic data (e.g., link speed/flow) collected from sensor networks can be organized as multivariate time series with additional spatial attributes. A crucial task in analyzing such data is to identify and detect anomalous observations and events from the data with complex spatial and temporal dependencies. Robust Principal Component Analysis (RPCA) is a widely used tool for anomaly detection. However, the traditional RPCA purely relies on the global low-rank assumption while ignoring the local temporal correlations. In light of this, this study proposes a Hankel-structured tensor version of RPCA for anomaly detection in spatiotemporal data. We treat the raw data with anomalies as a multivariate time series matrix (location $\times$ time) and assume the denoised matrix has a low-rank structure. Then we transform the low-rank matrix to a third-order tensor by applying temporal Hankelization. In the end, we decompose the corrupted matrix into a low-rank Hankel tensor and a sparse matrix. With the Hankelization operation, the model can simultaneously capture the global and local spatiotemporal correlations and exhibit more robust performance. We formulate the problem as an optimization problem and use tensor nuclear norm (TNN) to approximate the tensor rank and $l_1$ norm to approximate the sparsity. We develop an efficient solution algorithm based on the Alternating Direction Method of Multipliers (ADMM). Despite having three hyper-parameters, the model is easy to set in practice. We evaluate the proposed method by synthetic data and metro passenger flow time series and the results demonstrate the accuracy of anomaly detection.

cs.LG↗

arXiv2021

Density dependent local structures in InTe phase-change materials

Chalcogenide phase-change materials (PCMs) based random access memory (PCRAM) is one of the leading candidates for the development of non-volatile memory and neuro-inspired computing technologies. Recent work shows Indium to be an important alloying element for PCRAM, while a thorough understanding of the parent compound InTe, in particular, its amorphous phase, is still lacking. In this work, we carry out ab initio simulations and chemical bonding analyses on amorphous and various crystalline polymorphs of InTe. We reveal that the local geometries are highly density dependent in amorphous structures, forming In-centered tetrahedral motifs under ambient conditions but defective octahedral motifs under pressure, which stems from the bonding characters of its crystalline polymorphs. In addition, our ab initio molecular dynamics simulations predict rapid crystallization capability of InTe under pressure. At last, we make a suggestion for better use of Indium and propose an "active" device design to utilize both thermal and mechanical effects for phase-change applications.

cond-mat.mtrl-sci↗

arXiv2021

Ergodic property of random diffusivity system with trapping events

Brownian yet non-Gaussian phenomenon has recently been observed in many biological and active matter systems. The main idea of explaining this phenomenon is to introduce a random diffusivity for particles moving in inhomogeneous environment. This paper considers a Langevin system containing a random diffusivity and an $α$-stable subordinator with $α<1$. This model describes the particle's motion in complex media where both the long trapping events and random diffusivity exist. We derive the general expressions of ensemble- and time-averaged mean-squared displacements which only contain the values of the inverse subordinator and diffusivity. Further taking specific time-dependent diffusivity, we obtain the analytic expressions of ergodicity breaking parameter and probability density function of the time-averaged mean-squared displacement. The results imply the nonergodicity of the random diffusivity model for any kind of diffusivity, including the critical case where the model presenting normal diffusion.

cond-mat.stat-mech↗

arXiv2021

Bonding nature and optical contrast of $TiTe_2$/$Sb_2Te_3$ phase-change heterostructure

Chalcogenide phase-change materials (PCMs) are regarded as the leading candidate for storage-class non-volatile memory and neuro-inspired computing. Recently, using the $TiTe_2$/$Sb_2Te_3$ material combination, a new framework - phase-change heterostructure (PCH), has been developed and proved to effectively suppress the noise and drift in electrical resistance upon memory programming, largely reducing the inter-device variability. However, the atomic-scale structural and chemical nature of PCH remains to be fully understood. In this work, we carry out thorough ab initio simulations to assess the bonding characteristics of the PCH. We show that the $TiTe_2$ crystalline nanolayers do not chemically interact with the surrounding $Sb_2Te_3$, and are stabilized by strong covalent and electrostatic Ti-Te interactions, which create a prohibitively high barrier for atomic migrations along the pulsing direction. We also find significant contrast in computed dielectric functions in the PCH, suggesting possible optical applications of this class of devices. With the more confined space and therefore constrained phase transition compared to traditional PCM devices, the recently introduced class of PCH-based devices may lead to improvements in phase-change photonic and optoelectronic applications with much lower stochasticity during programming.

cond-mat.mtrl-sci↗

arXiv2021

FedNILM: Applying Federated Learning to NILM Applications at the Edge

Non-intrusive load monitoring (NILM) helps disaggregate the household's main electricity consumption to energy usages of individual appliances, thus greatly cutting down the cost in fine-grained household load monitoring. To address the arisen privacy concern in NILM applications, federated learning (FL) could be leveraged for NILM model training and sharing. When applying the FL paradigm in real-world NILM applications, however, we are faced with the challenges of edge resource restriction, edge model personalization and edge training data scarcity. In this paper we present FedNILM, a practical FL paradigm for NILM applications at the edge client. Specifically, FedNILM is designed to deliver privacy-preserving and personalized NILM services to large-scale edge clients, by leveraging i) secure data aggregation through federated learning, ii) efficient cloud model compression via filter pruning and multi-task learning, and iii) personalized edge model building with unsupervised transfer learning. Our experiments on real-world energy data show that, FedNILM is able to achieve personalized energy disaggregation with the state-of-the-art accuracy, while ensuring privacy preserving at the edge client.

cs.LG↗

arXiv2021

Unraveling the optical contrast in Sb2Te and AgInSbTe phase-change materials

Chalcogenide phase-change materials (PCMs) show a significant contrast in optical reflectivity and electrical resistivity upon crystallization from the amorphous phase and are leading candidates for non-volatile photonic and electronic applications. In addition to the flagship Ge2Sb2Te5 phase-change alloy, doped Sb2Te alloys, in particular AgInSbTe used in rewritable optical discs, have been widely investigated for decades, and nevertheless the theoretical insights on the optical properties of this important family of PCMs are scarce. Here, we carry out thorough ab initio simulations to gain an atomistic understanding of the optical properties of Sb2Te and AgInSbTe. We show that the large optical contrast between the amorphous and crystalline phase stems from the change in bond type in the parent compound Sb2Te. Ag and In impurities serve mostly the purpose of stabilization of the amorphous phase, and have marginal impact on the large variation in the dielectric function upon the phase transitions.

cond-mat.mtrl-sci↗

arXiv2021

Unsupervised Feature Learning by Cross-Level Instance-Group Discrimination

Unsupervised feature learning has made great strides with contrastive learning based on instance discrimination and invariant mapping, as benchmarked on curated class-balanced datasets. However, natural data could be highly correlated and long-tail distributed. Natural between-instance similarity conflicts with the presumed instance distinction, causing unstable training and poor performance. Our idea is to discover and integrate between-instance similarity into contrastive learning, not directly by instance grouping, but by cross-level discrimination (CLD) between instances and local instance groups. While invariant mapping of each instance is imposed by attraction within its augmented views, between-instance similarity could emerge from common repulsion against instance groups. Our batch-wise and cross-view comparisons also greatly improve the positive/negative sample ratio of contrastive learning and achieve better invariant mapping. To effect both grouping and discrimination objectives, we impose them on features separately derived from a shared representation. In addition, we propose normalized projection heads and unsupervised hyper-parameter tuning for the first time. Our extensive experimentation demonstrates that CLD is a lean and powerful add-on to existing methods such as NPID, MoCo, InfoMin, and BYOL on highly correlated, long-tail, or balanced datasets. It not only achieves new state-of-the-art on self-supervision, semi-supervision, and transfer learning benchmarks, but also beats MoCo v2 and SimCLR on every reported performance attained with a much larger compute. CLD effectively brings unsupervised learning closer to natural data and real-world applications. Our code is publicly available at: https://github.com/frank-xwang/CLD-UnsupervisedLearning.

cs.CV↗

arXiv2021

Unsupervised Visual Attention and Invariance for Reinforcement Learning

Vision-based reinforcement learning (RL) is successful, but how to generalize it to unknown test environments remains challenging. Existing methods focus on training an RL policy that is universal to changing visual domains, whereas we focus on extracting visual foreground that is universal, feeding clean invariant vision to the RL policy learner. Our method is completely unsupervised, without manual annotations or access to environment internals. Given videos of actions in a training environment, we learn how to extract foregrounds with unsupervised keypoint detection, followed by unsupervised visual attention to automatically generate a foreground mask per video frame. We can then introduce artificial distractors and train a model to reconstruct the clean foreground mask from noisy observations. Only this learned model is needed during test to provide distraction-free visual input to the RL policy learner. Our Visual Attention and Invariance (VAI) method significantly outperforms the state-of-the-art on visual domain generalization, gaining 15 to 49% (61 to 229%) more cumulative rewards per episode on DeepMind Control (our DrawerWorld Manipulation) benchmarks. Our results demonstrate that it is not only possible to learn domain-invariant vision without any supervision, but freeing RL from visual distractions also makes the policy more focused and thus far better.

cs.RO↗

arXiv2020

Ergodic property of Langevin systems with superstatistical, uncorrelated or correlated diffusivity

Brownian yet non-Gaussian diffusion has recently been observed in numerous biological and active matter system. The cause of the non-Gaussian distribution have been elaborately studied in the idea of a superstatistical dynamics or a diffusing diffusivity. Based on a random diffusivity model, we here focus on the ergodic property and the scatter of the amplitude of time-averaged mean-squared displacement (TAMSD). Further, we individually investigate this model with three categories of diffusivities, including diffusivity being a random variable $D$, a time-dependent but uncorrelated diffusivity $D(t)$, and a correlated stochastic process $D(t)$. We find that ensemble-averaged TAMSDs are always normal while ensemble-averaged mean-squared displacement can be anomalous. Further, the scatter of dimensionless amplitude is determined by the time average of diffusivity $D(t)$. Our results are valid for arbitrary diffusivities.

cond-mat.stat-mech↗

arXiv2020

Tied Block Convolution: Leaner and Better CNNs with Shared Thinner Filters

Convolution is the main building block of convolutional neural networks (CNN). We observe that an optimized CNN often has highly correlated filters as the number of channels increases with depth, reducing the expressive power of feature representations. We propose Tied Block Convolution (TBC) that shares the same thinner filters over equal blocks of channels and produces multiple responses with a single filter. The concept of TBC can also be extended to group convolution and fully connected layers, and can be applied to various backbone networks and attention modules. Our extensive experimentation on classification, detection, instance segmentation, and attention demonstrates TBC's significant across-the-board gain over standard convolution and group convolution. The proposed TiedSE attention module can even use 64 times fewer parameters than the SE module to achieve comparable performance. In particular, standard CNNs often fail to accurately aggregate information in the presence of occlusion and result in multiple redundant partial object proposals. By sharing filters across channels, TBC reduces correlation and can effectively handle highly overlapping instances. TBC increases the average precision for object detection on MS-COCO by 6% when the occlusion ratio is 80%. Our code will be released.

cs.CV↗