arXiv subjects

Lei Huang

Publications and source records attributed to Lei Huang.

At least 181 records · Page 10Linked to original sources

arXiv2023

WHC: Weighted Hybrid Criterion for Filter Pruning on Convolutional Neural Networks

Filter pruning has attracted increasing attention in recent years for its capacity in compressing and accelerating convolutional neural networks. Various data-independent criteria, including norm-based and relationship-based ones, were proposed to prune the most unimportant filters. However, these state-of-the-art criteria fail to fully consider the dissimilarity of filters, and thus might lead to performance degradation. In this paper, we first analyze the limitation of relationship-based criteria with examples, and then introduce a new data-independent criterion, Weighted Hybrid Criterion (WHC), to tackle the problems of both norm-based and relationship-based criteria. By taking the magnitude of each filter and the linear dependence between filters into consideration, WHC can robustly recognize the most redundant filters, which can be safely pruned without introducing severe performance degradation to networks. Extensive pruning experiments in a simple one-shot manner demonstrate the effectiveness of the proposed WHC. In particular, WHC can prune ResNet-50 on ImageNet with more than 42% of floating point operations reduced without any performance loss in top-5 accuracy.

cs.CV↗

arXiv2022

Scale-Semantic Joint Decoupling Network for Image-text Retrieval in Remote Sensing

Image-text retrieval in remote sensing aims to provide flexible information for data analysis and application. In recent years, state-of-the-art methods are dedicated to ``scale decoupling'' and ``semantic decoupling'' strategies to further enhance the capability of representation. However, these previous approaches focus on either the disentangling scale or semantics but ignore merging these two ideas in a union model, which extremely limits the performance of cross-modal retrieval models. To address these issues, we propose a novel Scale-Semantic Joint Decoupling Network (SSJDN) for remote sensing image-text retrieval. Specifically, we design the Bidirectional Scale Decoupling (BSD) module, which exploits Salience Feature Extraction (SFE) and Salience-Guided Suppression (SGS) units to adaptively extract potential features and suppress cumbersome features at other scales in a bidirectional pattern to yield different scale clues. Besides, we design the Label-supervised Semantic Decoupling (LSD) module by leveraging the category semantic labels as prior knowledge to supervise images and texts probing significant semantic-related information. Finally, we design a Semantic-guided Triple Loss (STL), which adaptively generates a constant to adjust the loss function to improve the probability of matching the same semantic image and text and shorten the convergence time of the retrieval model. Our proposed SSJDN outperforms state-of-the-art approaches in numerical experiments conducted on four benchmark remote sensing datasets.

cs.CV↗

arXiv2022

Impact of the nonthermal electron radiation effects on the horizon scale image structure of Sagittarius A*

The Event Horizon Telescope (EHT), with $\sim$20 $μ$as high angular resolution, recently resolved the millimeter image of the suppermassive black hole in the Galaxy, Sagittarius A*. This opens a new window to study the plasma on horizon scales. The accreting disk probably contains a small fraction of nonthermal electrons and their emissions should contribute to the observed image. We study if such contributions are sufficient to cause structural differences detectable by current and future observational capabilities. We introduce nonthermal electrons in a semi-analytical accretion disk, which considers viscosity-leading heating processes, and adopt a continued hybrid electron energy distribution of thermal distribution and power-law tail. We generate the black hole images and extract the structural features as crescent parameters. We find the existence of nonthermal electron radiation makes the crescent much brighter, slightly larger, moderately thicker, and much more symmetric. When the nonthermal connecting Lorentz factor $γ_c=65$, which is equivalent to the nonthermal electrons accounting for $\sim1.5$% of the totals, nonthermal effects cause $\sim2$% size difference at 230 GHz. Comparing with the structural changes caused by other physical factors, including inclination between the system and the observer, black hole spin, and interstellar medium scattering effects, we find that although nonthermal electron radiation takes the most unimportant role at 230 GHz, it becomes more significant at 345 GHz.

astro-ph.HE↗

arXiv2022

Magnetic activities and parameters of 43 flare stars in the GWAC archive

In the archive of the Ground Wide Angle Camera (GWAC), we found 43 white light flares from 43 stars, among which, three are sympathetic or homologous flares, and one of them also has a quasi-periodic pulsation with a period of $13.0\pm1.5$ minutes. Among these 43 flare stars, there are 19 new active stars and 41 stars that have available TESS and/or K2 light curves, from which we found 931 stellar flares. We also obtained rotational or orbital periods of 34 GWAC flare stars, of which 33 are less than 5.4 days, and ephemerides of three eclipsing binaries from these light curves. Combining with low resolution spectra from LAMOST and the Xinglong 2.16m telescope, we found that $L_{\rm Hα}/L_{\rm bol}$ are in the saturation region in the rotation-activity diagram. From the LAMOST medium-resolution spectrum, we found that Star \#3 (HAT 178-02667) has double H$α$ emissions which imply it is a binary, and two components are both active stars. Thirteen stars have flare frequency distributions (FFDs) from TESS and/or K2 light curves. These FFDs show that the flares detected by GWAC can occur at a frequency of 0.5 to 9.5 yr$^{-1}$. The impact of flares on habitable planets was also studied based on these FFDs, and flares from some GWAC flare stars may produce enough energetic flares to destroy ozone layers, but none can trigger prebiotic chemistry on their habitable planets.

astro-ph.SR↗

arXiv2022

Multi-frequency VLBI Observations of the M 84 Inner Jet/Counterjet

Observational studies of inner-most regions of the edge-on jets in nearby active galactic nuclei (AGN) are crucial to understand their kinematics and morphology. For the inner jet of the nearby low luminosity AGN in M 84, we present new high-sensitivity observations with very long baseline interferometry since 2019, as well as archival Very Long Baseline Array observations in 2014. We find that the compact core in M 84 has an inverted-to-flat spectrum from 1.5 to 88 GHz. Based on the turnover frequency of $4.2\pm 0.2$ GHz in the spectrum, we estimated a magnetic field strength of 1-10mG and an electron number density of $\sim 10^5 cm^{-3}$ in the core region. Three inner jet components within $\sim 3$ mas from the core are identified and traced in the images at 22 GHz, whose apparent speeds are 0.11 c, 0.27 c, and 0.32 c, respectively. We calculate the viewing angle of $\sim58$ degree for the inner jet based on the proper motion and the flux ratio of jet-to-counterjet. A propagating sinusoidal model with a wavelength of $\sim 3.4$ mas is used to fit the helical morphology of the jet extended to 20 mas ($\sim 2.2\times 10^4$ Schwarzschild Radii).

astro-ph.HE↗

arXiv2022

Joint Matrix Decomposition for Deep Convolutional Neural Networks Compression

Deep convolutional neural networks (CNNs) with a large number of parameters require intensive computational resources, and thus are hard to be deployed in resource-constrained platforms. Decomposition-based methods, therefore, have been utilized to compress CNNs in recent years. However, since the compression factor and performance are negatively correlated, the state-of-the-art works either suffer from severe performance degradation or have relatively low compression factors. To overcome this problem, we propose to compress CNNs and alleviate performance degradation via joint matrix decomposition, which is different from existing works that compressed layers separately. The idea is inspired by the fact that there are lots of repeated modules in CNNs. By projecting weights with the same structures into the same subspace, networks can be jointly compressed with larger ranks. In particular, three joint matrix decomposition schemes are developed, and the corresponding optimization approaches based on Singular Value Decomposition are proposed. Extensive experiments are conducted across three challenging compact CNNs for different benchmark data sets to demonstrate the superior performance of our proposed algorithms. As a result, our methods can compress the size of ResNet-34 by 22X with slighter accuracy degradation compared with several state-of-the-art methods.

cs.CV↗

arXiv2022

Sub-network Multi-objective Evolutionary Algorithm for Filter Pruning

Filter pruning is a common method to achieve model compression and acceleration in deep neural networks (DNNs).Some research regarded filter pruning as a combinatorial optimization problem and thus used evolutionary algorithms (EA) to prune filters of DNNs. However, it is difficult to find a satisfactory compromise solution in a reasonable time due to the complexity of solution space searching. To solve this problem, we first formulate a multi-objective optimization problem based on a sub-network of the full model and propose a Sub-network Multiobjective Evolutionary Algorithm (SMOEA) for filter pruning. By progressively pruning the convolutional layers in groups, SMOEA can obtain a lightweight pruned result with better performance.Experiments on VGG-14 model for CIFAR-10 verify the effectiveness of the proposed SMOEA. Specifically, the accuracy of the pruned model with 16.56% parameters decreases by 0.28% only, which is better than the widely used popular filter pruning criteria.

cs.NE↗

arXiv2022

Understanding the Failure of Batch Normalization for Transformers in NLP

Batch Normalization (BN) is a core and prevalent technique in accelerating the training of deep neural networks and improving the generalization on Computer Vision (CV) tasks. However, it fails to defend its position in Natural Language Processing (NLP), which is dominated by Layer Normalization (LN). In this paper, we are trying to answer why BN usually performs worse than LN in NLP tasks with Transformer models. We find that the inconsistency between training and inference of BN is the leading cause that results in the failure of BN in NLP. We define Training Inference Discrepancy (TID) to quantitatively measure this inconsistency and reveal that TID can indicate BN's performance, supported by extensive experiments, including image classification, neural machine translation, language modeling, sequence labeling, and text classification tasks. We find that BN can obtain much better test performance than LN when TID keeps small through training. To suppress the explosion of TID, we propose Regularized BN (RBN) that adds a simple regularization term to narrow the gap between batch statistics and population statistics of BN. RBN improves the performance of BN consistently and outperforms or is on par with LN on 17 out of 20 settings, involving ten datasets and two common variants of Transformer Our code is available at https://github.com/wjxts/RegularizedBN.

cs.CL↗

arXiv2022

An Investigation into Whitening Loss for Self-supervised Learning

A desirable objective in self-supervised learning (SSL) is to avoid feature collapse. Whitening loss guarantees collapse avoidance by minimizing the distance between embeddings of positive pairs under the conditioning that the embeddings from different views are whitened. In this paper, we propose a framework with an informative indicator to analyze whitening loss, which provides a clue to demystify several interesting phenomena as well as a pivoting point connecting to other SSL methods. We reveal that batch whitening (BW) based methods do not impose whitening constraints on the embedding, but they only require the embedding to be full-rank. This full-rank constraint is also sufficient to avoid dimensional collapse. Based on our analysis, we propose channel whitening with random group partition (CW-RGP), which exploits the advantages of BW-based methods in preventing collapse and avoids their disadvantages requiring large batch size. Experimental results on ImageNet classification and COCO object detection reveal that the proposed CW-RGP possesses a promising potential for learning good representations. The code is available at https://github.com/winci-ai/CW-RGP.

cs.CV↗

arXiv2022

Optimality conditions for homogeneous polynomial optimization on the unit sphere

In this note, we prove that for homogeneous polynomial optimization on the sphere, if the objective $f$ is generic in the input space, all feasible points satisfying the first order and second order necessary optimality conditions are local minimizers, which addresses an issue raised in the recent work by Lasserre (Optimization Letters, 2021). As a corollary, this implies that Lasserre's hierarchy has finite convergence when $f$ is generic.

math.OC↗

arXiv2022

A machine learning based column-and-row generation approach for integrated air cargo recovery problem

Freighter airlines need to recover both aircraft and cargo schedules when disruptions happen. This process is usually divided into three sequential decisions to recovery flights, aircraft, and cargoes. This study focuses on the integrated recovery problem that makes aircraft and cargo recovery decisions simultaneously. We formulate two integrated models based on the flight connection network, one is the arc-based model, and the other is the string-based model. The arc-based model makes the flight delay decisions by duplicating flight copies, and is solved directly by commercial solvers such as Cplex. The string-based model makes the flight delay decisions in the variable generation process. The main difficulty of the string-based model is that the number of constraints grows with the newly generated flight delay decisions. Therefore, the traditional column generation method can not be applied directly. To tackle this challenge, we propose a machine learning based column-and-row generation approach. The machine learning method is used to uncover the critical delay decisions of short through connections in each column-and-row generation iteration by eliminating the poor flight delay decisions. We also propose a set of valid inequality constraints which can greatly improve the objective of LP relaxation solution and reduce the integral gap. The effectiveness and efficiency of our model is tested by simulated scenarios based on real operational data from the largest Chinese freighter airlines. The computational results show that a significant cost reduction can be achieved with the proposed string-based model in reasonable time.

math.OC↗

arXiv2022

MDM: Molecular Diffusion Model for 3D Molecule Generation

Molecule generation, especially generating 3D molecular geometries from scratch (i.e., 3D \textit{de novo} generation), has become a fundamental task in drug designs. Existing diffusion-based 3D molecule generation methods could suffer from unsatisfactory performances, especially when generating large molecules. At the same time, the generated molecules lack enough diversity. This paper proposes a novel diffusion model to address those two challenges. First, interatomic relations are not in molecules' 3D point cloud representations. Thus, it is difficult for existing generative models to capture the potential interatomic forces and abundant local constraints. To tackle this challenge, we propose to augment the potential interatomic forces and further involve dual equivariant encoders to encode interatomic forces of different strengths. Second, existing diffusion-based models essentially shift elements in geometry along the gradient of data density. Such a process lacks enough exploration in the intermediate steps of the Langevin dynamics. To address this issue, we introduce a distributional controlling variable in each diffusion/reverse step to enforce thorough explorations and further improve generation diversity. Extensive experiments on multiple benchmarks demonstrate that the proposed model significantly outperforms existing methods for both unconditional and conditional generation tasks. We also conduct case studies to help understand the physicochemical properties of the generated molecules.

cs.LG↗

arXiv2022

Homogenization for polynomial optimization with unbounded sets

This paper considers polynomial optimization with unbounded sets. We give a homogenization formulation and propose a hierarchy of Moment-SOS relaxations to solve it. Under the assumptions that the feasible set is closed at infinity and the ideal of homogenized equality constraining polynomials is real radical, we show that this hierarchy of Moment-SOS relaxations has finite convergence, if some optimality conditions (i.e., the linear independence constraint qualification, strict complementarity and second order sufficiency conditions) hold at every minimizer, including the one at infinity. Moreover, we prove extended versions of Putinar-Vasilescu type Positivstellensatz for polynomials that are nonnegative on unbounded sets. The classical Moment-SOS hierarchy with denominators is also studied. In particular, we give a positive answer to a conjecture of Mai, Lasserre and Magron in their recent work.

math.OC↗

arXiv2022

Generalized truncated moment problems with unbounded sets

This paper studies generalized truncated moment problems with unbounded sets. First, we study geometric properties of the truncated moment cone and its dual cone of nonnegative polynomials. By the technique of homogenization, we give a convergent hierarchy of Moment-SOS relaxations for approximating these cones. With them, we give a Moment-SOS method for solving generalized truncated moment problems with unbounded sets. Finitely atomic representing measures, or certificates for their nonexistence, can be obtained by the proposed method. Numerical experiments and applications are also given.

math.NA↗

arXiv2022

Moire quasi-bound states in the continuum

The novel physics of twisted bilayer graphene has motivated extensive studies of magic-angle flat bands hosted by moiré structures in electronic, photonic and acoustic systems. On the other hand, bound states in the continuum (BICs) have also attracted great attention in recent years because of their potential applications in the field of designing superior optical devices. Here, we combine these two independent concepts to construct a new optical state in a twisted bilayer photonic crystal slab, which is called as moiré quasi-BIC, and numerically demonstrate that such an exotic optical state possesses dual characteristics of moiré flat bands and quasi-BICs. To illustrate the mechanism for the formation of moiré flat bands, we develop an effective model at the center of the Brillouin zone and show that moiré flat bands could be fulfilled by balancing the interlayer coupling strength and the twist angle around the band edge above the light line. Moreover, by decreasing the twist angle of moiré photonic crystal slabs with flat bands, it is shown that the moiré flat-band mode at the Brillouin center gradually approaches a perfect BIC, where the total radiation loss from all diffraction channels is significantly suppressed. To clarify the advantage of moiré quasi-BICs, enhanced second-harmonic generation (SHG) is numerically proven with a wide-angle optical source. The efficiency of SHG assisted by designed moiré quasi-BICs can be greatly improved compared with that based on dispersive quasi-BICs with similar quality factors.

physics.optics↗

arXiv2022

Towards Better Understanding with Uniformity and Explicit Regularization of Embeddings in Embedding-based Neural Topic Models

Embedding-based neural topic models could explicitly represent words and topics by embedding them to a homogeneous feature space, which shows higher interpretability. However, there are no explicit constraints for the training of embeddings, leading to a larger optimization space. Also, a clear description of the changes in embeddings and the impact on model performance is still lacking. In this paper, we propose an embedding regularized neural topic model, which applies the specially designed training constraints on word embedding and topic embedding to reduce the optimization space of parameters. To reveal the changes and roles of embeddings, we introduce \textbf{uniformity} into the embedding-based neural topic model as the evaluation metric of embedding space. On this basis, we describe how embeddings tend to change during training via the changes in the uniformity of embeddings. Furthermore, we demonstrate the impact of changes in embeddings in embedding-based neural topic models through ablation studies. The results of experiments on two mainstream datasets indicate that our model significantly outperforms baseline models in terms of the harmony between topic quality and document modeling. This work is the first attempt to exploit uniformity to explore changes in embeddings of embedding-based neural topic models and their impact on model performance to the best of our knowledge.

cs.CL↗

arXiv2022

Bi-level Doubly Variational Learning for Energy-based Latent Variable Models

Energy-based latent variable models (EBLVMs) are more expressive than conventional energy-based models. However, its potential on visual tasks are limited by its training process based on maximum likelihood estimate that requires sampling from two intractable distributions. In this paper, we propose Bi-level doubly variational learning (BiDVL), which is based on a new bi-level optimization framework and two tractable variational distributions to facilitate learning EBLVMs. Particularly, we lead a decoupled EBLVM consisting of a marginal energy-based distribution and a structural posterior to handle the difficulties when learning deep EBLVMs on images. By choosing a symmetric KL divergence in the lower level of our framework, a compact BiDVL for visual tasks can be obtained. Our model achieves impressive image generation performance over related works. It also demonstrates the significant capacity of testing image reconstruction and out-of-distribution detection.

cs.CV↗

arXiv2022

Delving into the Estimation Shift of Batch Normalization in a Network

Batch normalization (BN) is a milestone technique in deep learning. It normalizes the activation using mini-batch statistics during training but the estimated population statistics during inference. This paper focuses on investigating the estimation of population statistics. We define the estimation shift magnitude of BN to quantitatively measure the difference between its estimated population statistics and expected ones. Our primary observation is that the estimation shift can be accumulated due to the stack of BN in a network, which has detriment effects for the test performance. We further find a batch-free normalization (BFN) can block such an accumulation of estimation shift. These observations motivate our design of XBNBlock that replace one BN with BFN in the bottleneck block of residual-style networks. Experiments on the ImageNet and COCO benchmarks show that XBNBlock consistently improves the performance of different architectures, including ResNet and ResNeXt, by a significant margin and seems to be more robust to distribution shift.

cs.CV↗