arXiv subjects

Zhe Wang

Publications and source records attributed to Zhe Wang.

At least 415 records · Page 23Linked to original sources

arXiv2022

PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detection

3D object detection is receiving increasing attention from both industry and academia thanks to its wide applications in various fields. In this paper, we propose Point-Voxel Region-based Convolution Neural Networks (PV-RCNNs) for 3D object detection on point clouds. First, we propose a novel 3D detector, PV-RCNN, which boosts the 3D detection performance by deeply integrating the feature learning of both point-based set abstraction and voxel-based sparse convolution through two novel steps, i.e., the voxel-to-keypoint scene encoding and the keypoint-to-grid RoI feature abstraction. Second, we propose an advanced framework, PV-RCNN++, for more efficient and accurate 3D object detection. It consists of two major improvements: sectorized proposal-centric sampling for efficiently producing more representative keypoints, and VectorPool aggregation for better aggregating local point features with much less resource consumption. With these two strategies, our PV-RCNN++ is about $3\times$ faster than PV-RCNN, while also achieving better performance. The experiments demonstrate that our proposed PV-RCNN++ framework achieves state-of-the-art 3D detection performance on the large-scale and highly-competitive Waymo Open Dataset with 10 FPS inference speed on the detection range of 150m * 150m.

cs.CV↗

arXiv2022

Decentralized Federated Reinforcement Learning for User-Centric Dynamic TFDD Control

The explosive growth of dynamic and heterogeneous data traffic brings great challenges for 5G and beyond mobile networks. To enhance the network capacity and reliability, we propose a learning-based dynamic time-frequency division duplexing (D-TFDD) scheme that adaptively allocates the uplink and downlink time-frequency resources of base stations (BSs) to meet the asymmetric and heterogeneous traffic demands while alleviating the inter-cell interference. We formulate the problem as a decentralized partially observable Markov decision process (Dec-POMDP) that maximizes the long-term expected sum rate under the users' packet dropping ratio constraints. In order to jointly optimize the global resources in a decentralized manner, we propose a federated reinforcement learning (RL) algorithm named federated Wolpertinger deep deterministic policy gradient (FWDDPG) algorithm. The BSs decide their local time-frequency configurations through RL algorithms and achieve global training via exchanging local RL models with their neighbors under a decentralized federated learning framework. Specifically, to deal with the large-scale discrete action space of each BS, we adopt a DDPG-based algorithm to generate actions in a continuous space, and then utilize Wolpertinger policy to reduce the mapping errors from continuous action space back to discrete action space. Simulation results demonstrate the superiority of our proposed algorithm to benchmark algorithms with respect to system sum rate.

cs.LG↗

arXiv2022

Deep Learning Based Audio-Visual Multi-Speaker DOA Estimation Using Permutation-Free Loss Function

In this paper, we propose a deep learning based multi-speaker direction of arrival (DOA) estimation with audio and visual signals by using permutation-free loss function. We first collect a data set for multi-modal sound source localization (SSL) where both audio and visual signals are recorded in real-life home TV scenarios. Then we propose a novel spatial annotation method to produce the ground truth of DOA for each speaker with the video data by transformation between camera coordinate and pixel coordinate according to the pin-hole camera model. With spatial location information served as another input along with acoustic feature, multi-speaker DOA estimation could be solved as a classification task of active speaker detection. Label permutation problem in multi-speaker related tasks will be addressed since the locations of each speaker are used as input. Experiments conducted on both simulated data and real data show that the proposed audio-visual DOA estimation model outperforms audio-only DOA estimation model by a large margin.

eess.AS↗

arXiv2022

ASD: Towards Attribute Spatial Decomposition for Prior-Free Facial Attribute Recognition

Representing the spatial properties of facial attributes is a vital challenge for facial attribute recognition (FAR). Recent advances have achieved the reliable performances for FAR, benefiting from the description of spatial properties via extra prior information. However, the extra prior information might not be always available, resulting in the restricted application scenario of the prior-based methods. Meanwhile, the spatial ambiguity of facial attributes caused by inherent spatial diversities of facial parts is ignored. To address these issues, we propose a prior-free method for attribute spatial decomposition (ASD), mitigating the spatial ambiguity of facial attributes without any extra prior information. Specifically, assignment-embedding module (AEM) is proposed to enable the procedure of ASD, which consists of two operations: attribute-to-location assignment and location-to-attribute embedding. The attribute-to-location assignment first decomposes the feature map based on latent factors, assigning the magnitude of attribute components on each spatial location. Then, the assigned attribute components from all locations to represent the global-level attribute embeddings. Furthermore, correlation matrix minimization (CMM) is introduced to enlarge the discriminability of attribute embeddings. Experimental results demonstrate the superiority of ASD compared with state-of-the-art prior-based methods, while the reliable performance of ASD for the case of limited training data is further validated.

cs.CV↗

arXiv2022

Towards Trustworthy Multi-label Sewer Defect Classification via Evidential Deep Learning

An automatic vision-based sewer inspection plays a key role of sewage system in a modern city. Recent advances focus on utilizing deep learning model to realize the sewer inspection system, benefiting from the capability of data-driven feature representation. However, the inherent uncertainty of sewer defects is ignored, resulting in the missed detection of serious unknown sewer defect categories. In this paper, we propose a trustworthy multi-label sewer defect classification (TMSDC) method, which can quantify the uncertainty of sewer defect prediction via evidential deep learning. Meanwhile, a novel expert base rate assignment (EBRA) is proposed to introduce the expert knowledge for describing reliable evidences in practical situations. Experimental results demonstrate the effectiveness of TMSDC and the superior capability of uncertainty estimation is achieved on the latest public benchmark.

cs.CV↗

arXiv2022

Towards Efficient 3D Object Detection with Knowledge Distillation

Despite substantial progress in 3D object detection, advanced 3D detectors often suffer from heavy computation overheads. To this end, we explore the potential of knowledge distillation (KD) for developing efficient 3D object detectors, focusing on popular pillar- and voxel-based detectors.In the absence of well-developed teacher-student pairs, we first study how to obtain student models with good trade offs between accuracy and efficiency from the perspectives of model compression and input resolution reduction. Then, we build a benchmark to assess existing KD methods developed in the 2D domain for 3D object detection upon six well-constructed teacher-student pairs. Further, we propose an improved KD pipeline incorporating an enhanced logit KD method that performs KD on only a few pivotal positions determined by teacher classification response, and a teacher-guided student model initialization to facilitate transferring teacher model's feature extraction ability to students through weight inheritance. Finally, we conduct extensive experiments on the Waymo dataset. Our best performing model achieves $65.75\%$ LEVEL 2 mAPH, surpassing its teacher model and requiring only $44\%$ of teacher flops. Our most efficient model runs 51 FPS on an NVIDIA A100, which is $2.2\times$ faster than PointPillar with even higher accuracy. Code is available at \url{https://github.com/CVMI-Lab/SparseKD}.

cs.CV↗

arXiv2022

FreGAN: Exploiting Frequency Components for Training GANs under Limited Data

Training GANs under limited data often leads to discriminator overfitting and memorization issues, causing divergent training. Existing approaches mitigate the overfitting by employing data augmentations, model regularization, or attention mechanisms. However, they ignore the frequency bias of GANs and take poor consideration towards frequency information, especially high-frequency signals that contain rich details. To fully utilize the frequency information of limited data, this paper proposes FreGAN, which raises the model's frequency awareness and draws more attention to producing high-frequency signals, facilitating high-quality generation. In addition to exploiting both real and generated images' frequency information, we also involve the frequency signals of real images as a self-supervised constraint, which alleviates the GAN disequilibrium and encourages the generator to synthesize adequate rather than arbitrary frequency signals. Extensive results demonstrate the superiority and effectiveness of our FreGAN in ameliorating generation quality in the low-data regime (especially when training data is less than 100). Besides, FreGAN can be seamlessly applied to existing regularization and attention mechanism models to further boost the performance.

cs.CV↗

arXiv2022

Uplink Performance of RIS-aided Cell-Free Massive MIMO System Over Spatially Correlated Channels

We consider a practical spatially correlated reconfigurable intelligent surface (RIS)-aided cell-free (CF) massive multiple-input-multiple-output (mMIMO) system with multi-antenna access points (APs) over spatially correlated Rician fading channels. The minimum mean square error (MMSE) channel estimator is adopted to estimate the aggregated RIS channels. Then, we investigate the uplink spectral efficiency (SE) with the maximum ratio (MR) and the local minimum mean squared error (L-MMSE) combining at the APs and obtain the closed-form expression for characterizing the performance of the former. The accuracy of our derived analytical results has been verified by extensive Monte-Carlo simulations. Our results show that increasing the number of RIS elements is always beneficial, but with diminishing returns when the number of RIS elements is sufficiently large. Furthermore, the effect of the number of AP antennas on system performance is more pronounced under a small number of RIS elements, while the spatial correlation of RIS elements imposes a more severe negative impact on the system performance than that of the AP antennas.

cs.IT↗

arXiv2022

Going Further With Winograd Convolutions: Tap-Wise Quantization for Efficient Inference on 4x4 Tile

Most of today's computer vision pipelines are built around deep neural networks, where convolution operations require most of the generally high compute effort. The Winograd convolution algorithm computes convolutions with fewer MACs compared to the standard algorithm, reducing the operation count by a factor of 2.25x for 3x3 convolutions when using the version with 2x2-sized tiles $F_2$. Even though the gain is significant, the Winograd algorithm with larger tile sizes, i.e., $F_4$, offers even more potential in improving throughput and energy efficiency, as it reduces the required MACs by 4x. Unfortunately, the Winograd algorithm with larger tile sizes introduces numerical issues that prevent its use on integer domain-specific accelerators and higher computational overhead to transform input and output data between spatial and Winograd domains. To unlock the full potential of Winograd $F_4$, we propose a novel tap-wise quantization method that overcomes the numerical issues of using larger tiles, enabling integer-only inference. Moreover, we present custom hardware units that process the Winograd transformations in a power- and area-efficient way, and we show how to integrate such custom modules in an industrial-grade, programmable DSA. An extensive experimental evaluation on a large set of state-of-the-art computer vision benchmarks reveals that the tap-wise quantization algorithm makes the quantized Winograd $F_4$ network almost as accurate as the FP32 baseline. The Winograd-enhanced DSA achieves up to 1.85x gain in energy efficiency and up to 1.83x end-to-end speed-up for state-of-the-art segmentation and detection networks.

cs.AR↗

arXiv2022

Localized elasticity governs the nonlinear rheology of colloidal supercooled liquids

We propose a microscopic picture for understanding the nonlinear rheology of supercooled liquids with soft-repulsive potentials. Based on Brownian dynamics simulations of supercooled charge-stabilized colloidal suspensions, our analysis shows that the shear thinning of viscosity (eta) at large enough shear rates (sr), expressed as eta~sr^(-lambda), originates from the evolution of the localized elastic region (LER). An LER is a transient region composed of the first several coordination shells of a reference particle. In response to the external shear, particles within LER undergo nearly affine displacement before the yielding of LER. The characteristic strain (gamma) and size (xi) of LER respectively depend on the shear rate by gamma~sr^epsilon and xi~sr^(-nu). Three exponents, lambda, epsilon, and nu, are related by lambda=1-epsilon=4*nu. This simple relation connects the nonlinear rheology to the elastic properties and the microscopic configurational distortion of the system. The relaxation of the LER is promoted by the large-step nonaffine particle displacement along the extensional direction of the shear geometry with the step length of 0.4 particle diameter. The elastic deformation and the relaxation of the LER are ubiquitous and succesive in the flow, which compose the fundamental process governing the bulk nonlinear viscoelasticity. We apply this model to analyze the Rheo-Small Angle Neutron Scattering data of sheared charge-stabilized colloidal suspensions. It is seen that our model well explains the neutron spectra and the rheological data.

cond-mat.soft↗

arXiv2022

Developing, Evaluating and Scaling Learning Agents in Multi-Agent Environments

The Game Theory & Multi-Agent team at DeepMind studies several aspects of multi-agent learning ranging from computing approximations to fundamental concepts in game theory to simulating social dilemmas in rich spatial environments and training 3-d humanoids in difficult team coordination tasks. A signature aim of our group is to use the resources and expertise made available to us at DeepMind in deep reinforcement learning to explore multi-agent systems in complex environments and use these benchmarks to advance our understanding. Here, we summarise the recent work of our team and present a taxonomy that we feel highlights many important open challenges in multi-agent research.

cs.MA↗

arXiv2022

Reconstruction of Point Events in Liquid-Scintillator Detectors Subjected to Total Reflection

The outer water buffer is an economic option to shield the external radiative backgrounds for liquid-scintillator neutrino detectors. However, the consequential total reflection of scintillation light at the media boundary introduces extra complexity to the detector optics. This paper develops a precise detector-response model by investigating how total reflection complicates photon propagation and degrades reconstruction. We first parameterize the detector response by regression, providing an unbiased energy and vertex reconstruction in the total reflection region while keeping the number of parameters under control. From the experience of event degeneracy at the Jinping prototype, we then identify the root cause as the multimodality in the reconstruction likelihood function, determined by the refractive index of the buffer, detector scale and PMT coverage. To avoid multimodality, we propose a straightforward criterion based on the expected photo-electron-count ratios between neighboring PMTs. The criterion will be used to ensure success in future liquid-scintillator detectors by guaranteeing the effectiveness of event reconstruction.

physics.ins-det↗

arXiv2022

Magnetic Resonance Fingerprinting with compressed sensing and distance metric learning

Magnetic Resonance Fingerprinting (MRF) is a novel technique that simultaneously estimates multiple tissue-related parameters, such as the longitudinal relaxation time T1, the transverse relaxation time T2, off resonance frequency B0 and proton density, from a scanned object in just tens of seconds. However, the MRF method suffers from aliasing artifacts because it significantly undersamples the k-space data. In this work, we propose a compressed sensing (CS) framework for simultaneously estimating multiple tissue-related parameters based on the MRF method. It is more robust to low sampling ratio and is therefore more efficient in estimating MR parameters for all voxels of an object. Furthermore, the MRF method requires identifying the nearest atoms of the query fingerprints from the MR-signal-evolution dictionary with the L2 distance. However, we observed that the L2 distance is not always a proper metric to measure the similarities between MR Fingerprints. Adaptively learning a distance metric from the undersampled training data can significantly improve the matching accuracy of the query fingerprints. Numerical results on extensive simulated cases show that our method substantially outperforms stateof-the-art methods in terms of accuracy of parameter estimation.

eess.IV↗

arXiv2022

PPT: token-Pruned Pose Transformer for monocular and multi-view human pose estimation

Recently, the vision transformer and its variants have played an increasingly important role in both monocular and multi-view human pose estimation. Considering image patches as tokens, transformers can model the global dependencies within the entire image or across images from other views. However, global attention is computationally expensive. As a consequence, it is difficult to scale up these transformer-based methods to high-resolution features and many views. In this paper, we propose the token-Pruned Pose Transformer (PPT) for 2D human pose estimation, which can locate a rough human mask and performs self-attention only within selected tokens. Furthermore, we extend our PPT to multi-view human pose estimation. Built upon PPT, we propose a new cross-view fusion strategy, called human area fusion, which considers all human foreground pixels as corresponding candidates. Experimental results on COCO and MPII demonstrate that our PPT can match the accuracy of previous pose transformer methods while reducing the computation. Moreover, experiments on Human 3.6M and Ski-Pose demonstrate that our Multi-view PPT can efficiently fuse cues from multiple views and achieve new state-of-the-art results.

cs.CV↗

arXiv2022

Monolithically integrated active passive waveguide array fabricated on thin film lithium niobate using a single continuous photolithography process

We demonstrate a robust low-loss optical interface by tiling passive (i.e., without doping of active ions) thin film lithium niobate (TFLN) and active (i.e., doped with rare earth ions) TFLN substrates for monolithic integration of passive/active lithium niobate photonics. The tiled substrates composed of both active and passive areas allow to pattern the mask of the integrated active passive photonic device at once using a single continuous photolithography process. The interface loss of tiled substrate is measured as low as 0.26 dB. Thanks to the stability provided by this approach, a four-channel waveguide amplifier is realized in a straightforward manner, which shows a net gain of ~5 dB at 1550-nm wavelength and that of ~8 dB at 1530-nm wavelength for each channel. The robust low-loss optical interface for passive/active photonic integration will facilitate large-scale high performance photonic devices which require on-chip light sources and amplifiers.

physics.optics↗

arXiv2022

Joint Learning of Deep Texture and High-Frequency Features for Computer-Generated Image Detection

Distinguishing between computer-generated (CG) and natural photographic (PG) images is of great importance to verify the authenticity and originality of digital images. However, the recent cutting-edge generation methods enable high qualities of synthesis in CG images, which makes this challenging task even trickier. To address this issue, a joint learning strategy with deep texture and high-frequency features for CG image detection is proposed. We first formulate and deeply analyze the different acquisition processes of CG and PG images. Based on the finding that multiple different modules in image acquisition will lead to different sensitivity inconsistencies to the convolutional neural network (CNN)-based rendering in images, we propose a deep texture rendering module for texture difference enhancement and discriminative texture representation. Specifically, the semantic segmentation map is generated to guide the affine transformation operation, which is used to recover the texture in different regions of the input image. Then, the combination of the original image and the high-frequency components of the original and rendered images are fed into a multi-branch neural network equipped with attention mechanisms, which refines intermediate features and facilitates trace exploration in spatial and channel dimensions respectively. Extensive experiments on two public datasets and a newly constructed dataset with more realistic and diverse images show that the proposed approach outperforms existing methods in the field by a clear margin. Besides, results also demonstrate the detection robustness and generalization ability of the proposed approach to postprocessing operations and generative adversarial network (GAN) generated images.

cs.CV↗

arXiv2022

A Brief History of Recommender Systems

Soon after the invention of the Internet, the recommender system emerged and related technologies have been extensively studied and applied by both academia and industry. Currently, recommender system has become one of the most successful web applications, serving billions of people in each day through recommending different kinds of contents, including news feeds, videos, e-commerce products, music, movies, books, games, friends, jobs etc. These successful stories have proved that recommender system can transfer big data to high values. This article briefly reviews the history of web recommender systems, mainly from two aspects: (1) recommendation models, (2) architectures of typical recommender systems. We hope the brief review can help us to know the dots about the progress of web recommender systems, and the dots will somehow connect in the future, which inspires us to build more advanced recommendation services for changing the world better.

cs.IR↗

arXiv2022

Ellipticity control of terahertz high-harmonic generation in a Dirac semimetal

We report on terahertz high-harmonic generation in a Dirac semimetal as a function of the driving-pulse ellipticity and on a theoretical study of the field-driven intraband kinetics of massless Dirac fermions.Very efficient control of third-harmonic yield and polarization state is achieved in electron-doped Cd$_3$As$_2$ thin films at room temperature. The observed tunability is understood as resulting from terahertz-field driven intraband kinetics of the Dirac fermions. Our study paves the way for exploiting nonlinear optical properties of Dirac matter for applications in signal processing and optical communications.

cond-mat.mtrl-sci↗