arXiv subjects

Yan Yan

Publications and source records attributed to Yan Yan.

At least 199 records · Page 11Linked to original sources

arXiv2021

Saying the Unseen: Video Descriptions via Dialog Agents

Current vision and language tasks usually take complete visual data (e.g., raw images or videos) as input, however, practical scenarios may often consist the situations where part of the visual information becomes inaccessible due to various reasons e.g., restricted view with fixed camera or intentional vision block for security concerns. As a step towards the more practical application scenarios, we introduce a novel task that aims to describe a video using the natural language dialog between two agents as a supplementary information source given incomplete visual data. Different from most existing vision-language tasks where AI systems have full access to images or video clips, which may reveal sensitive information such as recognizable human faces or voices, we intentionally limit the visual input for AI systems and seek a more secure and transparent information medium, i.e., the natural language dialog, to supplement the missing visual information. Specifically, one of the intelligent agents - Q-BOT - is given two semantic segmented frames from the beginning and the end of the video, as well as a finite number of opportunities to ask relevant natural language questions before describing the unseen video. A-BOT, the other agent who has access to the entire video, assists Q-BOT to accomplish the goal by answering the asked questions. We introduce two different experimental settings with either a generative (i.e., agents generate questions and answers freely) or a discriminative (i.e., agents select the questions and answers from candidates) internal dialog generation process. With the proposed unified QA-Cooperative networks, we experimentally demonstrate the knowledge transfer process between the two dialog agents and the effectiveness of using the natural language dialog as a supplement for incomplete implicit visions.

cs.CV↗

arXiv2021

Simon Says: Evaluating and Mitigating Bias in Pruned Neural Networks with Knowledge Distillation

In recent years the ubiquitous deployment of AI has posed great concerns in regards to algorithmic bias, discrimination, and fairness. Compared to traditional forms of bias or discrimination caused by humans, algorithmic bias generated by AI is more abstract and unintuitive therefore more difficult to explain and mitigate. A clear gap exists in the current literature on evaluating and mitigating bias in pruned neural networks. In this work, we strive to tackle the challenging issues of evaluating, mitigating, and explaining induced bias in pruned neural networks. Our paper makes three contributions. First, we propose two simple yet effective metrics, Combined Error Variance (CEV) and Symmetric Distance Error (SDE), to quantitatively evaluate the induced bias prevention quality of pruned models. Second, we demonstrate that knowledge distillation can mitigate induced bias in pruned neural networks, even with unbalanced datasets. Third, we reveal that model similarity has strong correlations with pruning induced bias, which provides a powerful method to explain why bias occurs in pruned neural networks. Our code is available at https://github.com/codestar12/pruning-distilation-bias

cs.LG↗

arXiv2021

Learning to Aggregate and Personalize 3D Face from In-the-Wild Photo Collection

Non-parametric face modeling aims to reconstruct 3D face only from images without shape assumptions. While plausible facial details are predicted, the models tend to over-depend on local color appearance and suffer from ambiguous noise. To address such problem, this paper presents a novel Learning to Aggregate and Personalize (LAP) framework for unsupervised robust 3D face modeling. Instead of using controlled environment, the proposed method implicitly disentangles ID-consistent and scene-specific face from unconstrained photo set. Specifically, to learn ID-consistent face, LAP adaptively aggregates intrinsic face factors of an identity based on a novel curriculum learning approach with relaxed consistency loss. To adapt the face for a personalized scene, we propose a novel attribute-refining network to modify ID-consistent face with target attribute and details. Based on the proposed method, we make unsupervised 3D face modeling benefit from meaningful image facial structure and possibly higher resolutions. Extensive experiments on benchmarks show LAP recovers superior or competitive face shape and texture, compared with state-of-the-art (SOTA) methods with or without prior and supervision.

cs.CV↗

arXiv2021

The chiral restored quark stars can exist in the universe

In this paper, the equation of state (EOS) of deconfined quark stars is studied in the framework of the two-flavor NJL model, and the self-consistent mean field approximation is employed by introducing a parameter $α$ combining the original Lagrangian and the Fierz-transformed Lagrangian, $\mathcal{L}_R= (1-α)\mathcal{L}+α\mathcal{L}_F$, to measure the weights of different interaction channels. It is believed that the deconfinement of phase transition happens along with the chiral phase transition. Thus, due to the lack of description of confinement in the NJL model, the vacuum pressure is set to confine quarks at low densities, which is the pressure corresponding to the critical point of chiral phase transition. We find that deconfined quark stars can reach over two-solar-mass, and the bag constant therefore shifts from $(130 ~\mathrm{MeV})^4$ to $(150 ~\mathrm{MeV})^4$ as $α$ grows. In addition, the tidal deformability $Λ$ is yielded ranging from 253 to 482 along with the decrease of $~α$, which satisfies the astronomical constraint of $Λ<800$ for 1.4-solar-mass neutron stars.

nucl-th↗

arXiv2021

Hybrid stars can be self-bound

Based on the properties of uniform nuclear matter at the nuclear saturation density and basic thermodynamic relations, we first re-study the composition of matter on the surface of normal neutron stars and hybrid stars. It is found that the hybrid stars are composed of uniform hadronic matter on the surface rather than heavy nuclei. Then we use the Walceka model and the self-consistent NJL model to describe the equation of state of low-density hadrons and quark matter at high densities respectively, and the P--interpolation method is employed to connect the equation of state at the extreme densities to study hybrid stars. As a result, we find that the obtained hybrid star mass-radius relation and tidal deformability meet the latest astronomical data. More importantly, we find that the hybrid stars we obtained can be self-bound rather than gravitationally bound, which is completely different from previous related studies.

nucl-th↗

arXiv2021

Feature Decomposition and Reconstruction Learning for Effective Facial Expression Recognition

In this paper, we propose a novel Feature Decomposition and Reconstruction Learning (FDRL) method for effective facial expression recognition. We view the expression information as the combination of the shared information (expression similarities) across different expressions and the unique information (expression-specific variations) for each expression. More specifically, FDRL mainly consists of two crucial networks: a Feature Decomposition Network (FDN) and a Feature Reconstruction Network (FRN). In particular, FDN first decomposes the basic features extracted from a backbone network into a set of facial action-aware latent features to model expression similarities. Then, FRN captures the intra-feature and inter-feature relationships for latent features to characterize expression-specific variations, and reconstructs the expression feature. To this end, two modules including an intra-feature relation modeling module and an inter-feature relation modeling module are developed in FRN. Experimental results on both the in-the-lab databases (including CK+, MMI, and Oulu-CASIA) and the in-the-wild databases (including RAF-DB and SFEW) show that the proposed FDRL method consistently achieves higher recognition accuracy than several state-of-the-art methods. This clearly highlights the benefit of feature decomposition and reconstruction for classifying expressions.

cs.CV↗

arXiv2021

Stable Nonlinear and IQ Imbalance RF Fingerprint for Wireless OFDM Devices

An estimation method of Radio Frequency fingerprint (RFF) based on the physical hardware properties of the nonlinearity and in-phase and quadrature (IQ) imbalance of the transmitter is proposed for the authentication of wireless orthogonal frequency division multiplexing (OFDM) devices. Firstly, the parameters of the nonlinearity of the transmitter and finite impulse response (FIR) of the wireless multipath channel are estimated with a Hammerstein system parameter separation technique. Secondly, the best IQ imbalance parameter combination estimation is obtained with a searching algorithm and the applied conjugate anti-symmetric pilots. Thirdly, the estimations of the nonlinear coefficients and IQ imbalance parameter combination are considered as a novel RFF, the features are extracted from the novel RFF, and a k-Nearest Neighbor (k-NN) classifier is used to classify the communication devices with the features. It is demonstrated with the numerical experiments of five transmitters that the novel RFF eliminates the adverse effect of the wireless channel and is therefore stable. The proposed RF fingerprinting method is helpful for the high-strength authentication of the OFDM communication devices with subtle differences from the same model and same series.

cs.IT↗

arXiv2021

Nonlinear RF Fingerprints Authentication for OFDM Wireless Devices based on Demodulated Symbols

Radio Frequency fingerprints (RFF) authentication is one of the methods for the physical-layer information security, which uses the hardware characteristics of the transmitter to identify its real identity. In order to improve the performance of RFF based on preamble with fixed duration, a nonlinear RFF authentication method based on payload symbols is proposed for the wireless OFDM devices with the bit mapping scheme of QPSK. The communication system is modeled as a Hammerstein system containing the nonlinear transmitter and wireless multipath fading channel. A parameter separation technique based on orthogonal polynomial is presented for the estimation of the parameters of the Hammerstein system. The Hammerstein system parameter separation technique is firstly used to estimate the linear parameter with the training signal, which is used to compensate the adverse effect of the linear channel for the demodulation of the successive payload symbols. The demodulated payload symbols are further used to estimate the nonlinear coefficients of the transmitter with the Hammerstein system parameter separation technique again, which is used as the novel RFF for the authentication of the QPSK-OFDM devices. Numerical experiments demonstrate that the proposed method is feasible. The novel method can also be extended to the OFDM signals with other bit mapping schemes.

cs.IT↗

arXiv2021

Learning Audio-Visual Correlations from Variational Cross-Modal Generation

People can easily imagine the potential sound while seeing an event. This natural synchronization between audio and visual signals reveals their intrinsic correlations. To this end, we propose to learn the audio-visual correlations from the perspective of cross-modal generation in a self-supervised manner, the learned correlations can be then readily applied in multiple downstream tasks such as the audio-visual cross-modal localization and retrieval. We introduce a novel Variational AutoEncoder (VAE) framework that consists of Multiple encoders and a Shared decoder (MS-VAE) with an additional Wasserstein distance constraint to tackle the problem. Extensive experiments demonstrate that the optimized latent representation of the proposed MS-VAE can effectively learn the audio-visual correlations and can be readily applied in multiple audio-visual downstream tasks to achieve competitive performance even without any given label information during training.

cs.CV↗

arXiv2021

A Metamodel and Framework for Artificial General Intelligence From Theory to Practice

This paper introduces a new metamodel-based knowledge representation that significantly improves autonomous learning and adaptation. While interest in hybrid machine learning / symbolic AI systems leveraging, for example, reasoning and knowledge graphs, is gaining popularity, we find there remains a need for both a clear definition of knowledge and a metamodel to guide the creation and manipulation of knowledge. Some of the benefits of the metamodel we introduce in this paper include a solution to the symbol grounding problem, cumulative learning, and federated learning. We have applied the metamodel to problems ranging from time series analysis, computer vision, and natural language understanding and have found that the metamodel enables a wide variety of learning mechanisms ranging from machine learning, to graph network analysis and learning by reasoning engines to interoperate in a highly synergistic way. Our metamodel-based projects have consistently exhibited unprecedented accuracy, performance, and ability to generalize. This paper is inspired by the state-of-the-art approaches to AGI, recent AGI-aspiring work, the granular computing community, as well as Alfred Korzybski's general semantics. One surprising consequence of the metamodel is that it not only enables a new level of autonomous learning and optimal functioning for machine intelligences, but may also shed light on a path to better understanding how to improve human cognition.

cs.AI↗

arXiv2020

A Metastable CaSH$_3$ Phase Composed of HS Honeycomb Sheets that is Superconducting Under Pressure

Evolutionary searches predicted a number of ternary phases that could be synthesized at pressures of 100-300~GPa. $P6_3/mmc$ CaSH$_2$, $Pnma$ CaSH$_2$, $Cmc2_1$ CaSH$_6$, and $I\bar{4}$ CaSH$_{20}$ were composed of a Ca-S lattice along with H$_2$ molecules coordinated in a ``side-on'' fashion to Ca. The H-H bond lengths in these semiconducting phases were elongated because of H$_2$ $σ\rightarrow$ Ca d donation, and Ca d$\rightarrow$ H$_2$ $σ^*$ back-donation, via a Kubas-like mechanism. $P\bar{6}m2$ CaSH$_3$, consisting of two-dimensional HS and CaH$_2$ sheets, was metastable and metallic above \edit{128~GPa}. The presence of van Hove singularities increased its density of states at the Fermi level, and concomitantly the superconducting critical temperature, which was estimated to be as high as \edit{$\sim$100~K at 128~GPa}. This work will inspire the search for superconductivity in materials based upon \edit{honeycomb HX (X=S, Se, Te), and MH$_2$ (M=Mg, Ca, Sr, Ba) layers} under pressure.

cond-mat.supr-con↗

arXiv2020

Hierarchical Representation via Message Propagation for Robust Model Fitting

In this paper, we propose a novel hierarchical representation via message propagation (HRMP) method for robust model fitting, which simultaneously takes advantages of both the consensus analysis and the preference analysis to estimate the parameters of multiple model instances from data corrupted by outliers, for robust model fitting. Instead of analyzing the information of each data point or each model hypothesis independently, we formulate the consensus information and the preference information as a hierarchical representation to alleviate the sensitivity to gross outliers. Specifically, we firstly construct a hierarchical representation, which consists of a model hypothesis layer and a data point layer. The model hypothesis layer is used to remove insignificant model hypotheses and the data point layer is used to remove gross outliers. Then, based on the hierarchical representation, we propose an effective hierarchical message propagation (HMP) algorithm and an improved affinity propagation (IAP) algorithm to prune insignificant vertices and cluster the remaining data points, respectively. The proposed HRMP can not only accurately estimate the number and parameters of multiple model instances, but also handle multi-structural data contaminated with a large number of outliers. Experimental results on both synthetic data and real images show that the proposed HRMP significantly outperforms several state-of-the-art model fitting methods in terms of fitting accuracy and speed.

cs.CV↗

arXiv2020

Parallel Blockwise Knowledge Distillation for Deep Neural Network Compression

Deep neural networks (DNNs) have been extremely successful in solving many challenging AI tasks in natural language processing, speech recognition, and computer vision nowadays. However, DNNs are typically computation intensive, memory demanding, and power hungry, which significantly limits their usage on platforms with constrained resources. Therefore, a variety of compression techniques (e.g. quantization, pruning, and knowledge distillation) have been proposed to reduce the size and power consumption of DNNs. Blockwise knowledge distillation is one of the compression techniques that can effectively reduce the size of a highly complex DNN. However, it is not widely adopted due to its long training time. In this paper, we propose a novel parallel blockwise distillation algorithm to accelerate the distillation process of sophisticated DNNs. Our algorithm leverages local information to conduct independent blockwise distillation, utilizes depthwise separable layers as the efficient replacement block architecture, and properly addresses limiting factors (e.g. dependency, synchronization, and load balancing) that affect parallelism. The experimental results running on an AMD server with four Geforce RTX 2080Ti GPUs show that our algorithm can achieve 3x speedup plus 19% energy savings on VGG distillation, and 3.5x speedup plus 29% energy savings on ResNet distillation, both with negligible accuracy loss. The speedup of ResNet distillation can be further improved to 3.87 when using four RTX6000 GPUs in a distributed cluster.

cs.LG↗

arXiv2020

Photometric and Spectroscopic Study of Flares on Ross 15

We conducted photometric and spectroscopic observations for Ross 15 in order to further study the flare properties of this less observed flare star. A total of 28 B-band flares are detected in 128 hours of photometric observations, leading to a total flare rate of 0.22(+-0.04) hour^-1, more accurate than that provided by previous work. We give the energy range of the B-band flare (10^29.5 - 10^31.5 erg) and the FFD for the star. Within the same energy range, the FFD are lower than that of GJ 1243 (M4) and YZ CMi (M4.5), roughly in the middle of those of three M5-type stars and higher than the average FFDs of spectral types >= M6. We performed, for the first time to Ross 15, simultaneous high-cadence spectroscopic and photometric observations, resulting in detection of the most energetic flare in our sample. The intensity enhancements of the continuum and Balmer lines with significant correlations between them are detected during the flare, which is same with that of the other deeply studied flare stars of the similar spectral type.

astro-ph.SR↗

arXiv2020

Robust Visual Tracking via Statistical Positive Sample Generation and Gradient Aware Learning

In recent years, Convolutional Neural Network (CNN) based trackers have achieved state-of-the-art performance on multiple benchmark datasets. Most of these trackers train a binary classifier to distinguish the target from its background. However, they suffer from two limitations. Firstly, these trackers cannot effectively handle significant appearance variations due to the limited number of positive samples. Secondly, there exists a significant imbalance of gradient contributions between easy and hard samples, where the easy samples usually dominate the computation of gradient. In this paper, we propose a robust tracking method via Statistical Positive sample generation and Gradient Aware learning (SPGA) to address the above two limitations. To enrich the diversity of positive samples, we present an effective and efficient statistical positive sample generation algorithm to generate positive samples in the feature space. Furthermore, to handle the issue of imbalance between easy and hard samples, we propose a gradient sensitive loss to harmonize the gradient contributions between easy and hard samples. Extensive experiments on three challenging benchmark datasets including OTB50, OTB100 and VOT2016 demonstrate that the proposed SPGA performs favorably against several state-of-the-art trackers.

cs.CV↗

arXiv2020

Revisiting Optical Flow Estimation in 360 Videos

Nowadays 360 video analysis has become a significant research topic in the field since the appearance of high-quality and low-cost 360 wearable devices. In this paper, we propose a novel LiteFlowNet360 architecture for 360 videos optical flow estimation. We design LiteFlowNet360 as a domain adaptation framework from perspective video domain to 360 video domain. We adapt it from simple kernel transformation techniques inspired by Kernel Transformer Network (KTN) to cope with inherent distortion in 360 videos caused by the sphere-to-plane projection. First, we apply an incremental transformation of convolution layers in feature pyramid network and show that further transformation in inference and regularization layers are not important, hence reducing the network growth in terms of size and computation cost. Second, we refine the network by training with augmented data in a supervised manner. We perform data augmentation by projecting the images in a sphere and re-projecting to a plane. Third, we train LiteFlowNet360 in a self-supervised manner using target domain 360 videos. Experimental results show the promising results of 360 video optical flow estimation using the proposed novel architecture.

cs.CV↗

arXiv2020

Egok360: A 360 Egocentric Kinetic Human Activity Video Dataset

Recently, there has been a growing interest in wearable sensors which provides new research perspectives for 360 ° video analysis. However, the lack of 360 ° datasets in literature hinders the research in this field. To bridge this gap, in this paper we propose a novel Egocentric (first-person) 360° Kinetic human activity video dataset (EgoK360). The EgoK360 dataset contains annotations of human activity with different sub-actions, e.g., activity Ping-Pong with four sub-actions which are pickup-ball, hit, bounce-ball and serve. To the best of our knowledge, EgoK360 is the first dataset in the domain of first-person activity recognition with a 360° environmental setup, which will facilitate the egocentric 360 ° video understanding. We provide experimental results and comprehensive analysis of variants of the two-stream network for 360 egocentric activity recognition. The EgoK360 dataset can be downloaded from https://egok360.github.io/.

cs.CV↗

arXiv2020

Describing Unseen Videos via Multi-Modal Cooperative Dialog Agents

With the arising concerns for the AI systems provided with direct access to abundant sensitive information, researchers seek to develop more reliable AI with implicit information sources. To this end, in this paper, we introduce a new task called video description via two multi-modal cooperative dialog agents, whose ultimate goal is for one conversational agent to describe an unseen video based on the dialog and two static frames. Specifically, one of the intelligent agents - Q-BOT - is given two static frames from the beginning and the end of the video, as well as a finite number of opportunities to ask relevant natural language questions before describing the unseen video. A-BOT, the other agent who has already seen the entire video, assists Q-BOT to accomplish the goal by providing answers to those questions. We propose a QA-Cooperative Network with a dynamic dialog history update learning mechanism to transfer knowledge from A-BOT to Q-BOT, thus helping Q-BOT to better describe the video. Extensive experiments demonstrate that Q-BOT can effectively learn to describe an unseen video by the proposed model and the cooperative learning method, achieving the promising performance where Q-BOT is given the full ground truth history dialog.

cs.CV↗