arXiv subjects

Heng Wang

Publications and source records attributed to Heng Wang.

At least 163 records · Page 9Linked to original sources

arXiv2023

Why Is Prompt Tuning for Vision-Language Models Robust to Noisy Labels?

Vision-language models such as CLIP learn a generic text-image embedding from large-scale training data. A vision-language model can be adapted to a new classification task through few-shot prompt tuning. We find that such a prompt tuning process is highly robust to label noises. This intrigues us to study the key reasons contributing to the robustness of the prompt tuning paradigm. We conducted extensive experiments to explore this property and find the key factors are: 1) the fixed classname tokens provide a strong regularization to the optimization of the model, reducing gradients induced by the noisy samples; 2) the powerful pre-trained image-text embedding that is learned from diverse and generic web data provides strong prior knowledge for image classification. Further, we demonstrate that noisy zero-shot predictions from CLIP can be used to tune its own prompt, significantly enhancing prediction accuracy in the unsupervised setting. The code is available at https://github.com/CEWu/PTNL.

cs.CV↗

arXiv2023

Exploring the Role of Audio in Video Captioning

Recent focus in video captioning has been on designing architectures that can consume both video and text modalities, and using large-scale video datasets with text transcripts for pre-training, such as HowTo100M. Though these approaches have achieved significant improvement, the audio modality is often ignored in video captioning. In this work, we present an audio-visual framework, which aims to fully exploit the potential of the audio modality for captioning. Instead of relying on text transcripts extracted via automatic speech recognition (ASR), we argue that learning with raw audio signals can be more beneficial, as audio has additional information including acoustic events, speaker identity, etc. Our contributions are twofold. First, we observed that the model overspecializes to the audio modality when pre-training with both video and audio modality, since the ground truth (i.e., text transcripts) can be solely predicted using audio. We proposed a Modality Balanced Pre-training (MBP) loss to mitigate this issue and significantly improve the performance on downstream tasks. Second, we slice and dice different design choices of the cross-modal module, which may become an information bottleneck and generate inferior results. We proposed new local-global fusion mechanisms to improve information exchange across audio and video. We demonstrate significant improvements by leveraging the audio modality on four datasets, and even outperform the state of the art on some metrics without relying on the text modality as the input.

cs.CV↗

arXiv2023

BotMoE: Twitter Bot Detection with Community-Aware Mixtures of Modal-Specific Experts

Twitter bot detection has become a crucial task in efforts to combat online misinformation, mitigate election interference, and curb malicious propaganda. However, advanced Twitter bots often attempt to mimic the characteristics of genuine users through feature manipulation and disguise themselves to fit in diverse user communities, posing challenges for existing Twitter bot detection models. To this end, we propose BotMoE, a Twitter bot detection framework that jointly utilizes multiple user information modalities (metadata, textual content, network structure) to improve the detection of deceptive bots. Furthermore, BotMoE incorporates a community-aware Mixture-of-Experts (MoE) layer to improve domain generalization and adapt to different Twitter communities. Specifically, BotMoE constructs modal-specific encoders for metadata features, textual content, and graphical structure, which jointly model Twitter users from three modal-specific perspectives. We then employ a community-aware MoE layer to automatically assign users to different communities and leverage the corresponding expert networks. Finally, user representations from metadata, text, and graph perspectives are fused with an expert fusion layer, combining all three modalities while measuring the consistency of user information. Extensive experiments demonstrate that BotMoE significantly advances the state-of-the-art on three Twitter bot detection benchmarks. Studies also confirm that BotMoE captures advanced and evasive bots, alleviates the reliance on training data, and better generalizes to new and previously unseen user communities.

cs.SI↗

arXiv2023

$R^{2}$Former: Unified $R$etrieval and $R$eranking Transformer for Place Recognition

Visual Place Recognition (VPR) estimates the location of query images by matching them with images in a reference database. Conventional methods generally adopt aggregated CNN features for global retrieval and RANSAC-based geometric verification for reranking. However, RANSAC only employs geometric information but ignores other possible information that could be useful for reranking, e.g. local feature correlations, and attention values. In this paper, we propose a unified place recognition framework that handles both retrieval and reranking with a novel transformer model, named $R^{2}$Former. The proposed reranking module takes feature correlation, attention value, and xy coordinates into account, and learns to determine whether the image pair is from the same location. The whole pipeline is end-to-end trainable and the reranking module alone can also be adopted on other CNN or transformer backbones as a generic component. Remarkably, $R^{2}$Former significantly outperforms state-of-the-art methods on major VPR datasets with much less inference time and memory consumption. It also achieves the state-of-the-art on the hold-out MSLS challenge set and could serve as a simple yet strong solution for real-world large-scale applications. Experiments also show vision transformer tokens are comparable and sometimes better than CNN local features on local matching. The code is released at https://github.com/Jeff-Zilence/R2Former.

cs.CV↗

arXiv2023

PAniC-3D: Stylized Single-view 3D Reconstruction from Portraits of Anime Characters

We propose PAniC-3D, a system to reconstruct stylized 3D character heads directly from illustrated (p)ortraits of (ani)me (c)haracters. Our anime-style domain poses unique challenges to single-view reconstruction; compared to natural images of human heads, character portrait illustrations have hair and accessories with more complex and diverse geometry, and are shaded with non-photorealistic contour lines. In addition, there is a lack of both 3D model and portrait illustration data suitable to train and evaluate this ambiguous stylized reconstruction task. Facing these challenges, our proposed PAniC-3D architecture crosses the illustration-to-3D domain gap with a line-filling model, and represents sophisticated geometries with a volumetric radiance field. We train our system with two large new datasets (11.2k Vroid 3D models, 1k Vtuber portrait illustrations), and evaluate on a novel AnimeRecon benchmark of illustration-to-3D pairs. PAniC-3D significantly outperforms baseline methods, and provides data to establish the task of stylized reconstruction from portrait illustrations.

cs.CV↗

arXiv2023

Simple and Fast Polarization Tracking algorithm for Continuous-Variable Quantum Key Distribution System Using Orthogonal Pilot Tone

To reduce the influence of random channel polarization variation, especially fast polarization perturbation,for continuous-variable quantum key distribution (CV-QKD) systems, a simple and fast polarization tracking algorithm is proposed and experimentally demonstrated. The algorithm is implemented by an orthogonal pilot tone scheme, one of the pilot tones is used for estimating the polarization rotation angle, and the other one is used for compensating the polarization perturbation introduced phase noise. Meanwhile, residual effects are compensated precisely with the help of a real-valued FIR filter. In this case, the excess noise introduced by polarization perturbation is effectively suppressed. Experimental results show that the polarization scrambling rate 12.57 krad/s can be tracked by using the proposed algorithm, and a good estimated parameters performance is achieved. Compared to conventional polarization tracking algorithms such as the constant modulus algorithm (CMA), experimental results show that the polarization tracking apability of the proposed algorithm is significantly improved. Furthermore, much faster polarization tracking performance is evaluated by digital simulations, and the simulation results show that about 188.50 Mrad/s can be tracked by the proposed algorithm. Thus, our method provides effective technology for the practical application of fiber-based CV-QKD.

quant-ph↗

arXiv2023

Practical Continuous-variable Quantum Key Distribution with Feasible Optimization Parameters

Continuous-variable quantum key distribution (CV-QKD) offers an approach to achieve a potential high secret key rate (SKR) in metropolitan areas. There are several challenges in developing a practical CV-QKD system from the laboratory to the real world. One of the most significant points is that it is really hard to adapt different practical optical fiber conditions for CV-QKD systems with unified hardware. Thus, how to improve the performance of practical CV-QKD systems in the field without modification of the hardware is very important. Here, a systematic optimization method, combining the modulation variance and error correction matrix optimization, is proposed to improve the performance of a practical CV-QKD system with a restricted capacity of postprocessing. The effect of restricted postprocessing capacity on the SKR is modeled as a nonlinear programming problem with modulation variance as an optimization parameter, and the selection of an optimal error correction matrix is studied under the same scheme. The results show that the SKR of a CV-QKD system can be improved by 24% and 200% compared with previous frequently used optimization methods theoretically with a transmission distance of 50 km. Furthermore, the experimental results verify the feasibility and robustness of the proposed method, where the achieved optimal SKR achieved practically deviates <1.6% from the theoretical optimal value. Our results pave the way to deploy high-performance CV-QKD in the real world.

quant-ph↗

arXiv2023

TwiBot-22: Towards Graph-Based Twitter Bot Detection

Twitter bot detection has become an increasingly important task to combat misinformation, facilitate social media moderation, and preserve the integrity of the online discourse. State-of-the-art bot detection methods generally leverage the graph structure of the Twitter network, and they exhibit promising performance when confronting novel Twitter bots that traditional methods fail to detect. However, very few of the existing Twitter bot detection datasets are graph-based, and even these few graph-based datasets suffer from limited dataset scale, incomplete graph structure, as well as low annotation quality. In fact, the lack of a large-scale graph-based Twitter bot detection benchmark that addresses these issues has seriously hindered the development and evaluation of novel graph-based bot detection approaches. In this paper, we propose TwiBot-22, a comprehensive graph-based Twitter bot detection benchmark that presents the largest dataset to date, provides diversified entities and relations on the Twitter network, and has considerably better annotation quality than existing datasets. In addition, we re-implement 35 representative Twitter bot detection baselines and evaluate them on 9 datasets, including TwiBot-22, to promote a fair comparison of model performance and a holistic understanding of research progress. To facilitate further research, we consolidate all implemented codes and datasets into the TwiBot-22 evaluation framework, where researchers could consistently evaluate new models and datasets. The TwiBot-22 Twitter bot detection benchmark and evaluation framework are publicly available at https://twibot22.github.io/

cs.SI↗

arXiv2023

Temporal Perceiving Video-Language Pre-training

Video-Language Pre-training models have recently significantly improved various multi-modal downstream tasks. Previous dominant works mainly adopt contrastive learning to achieve global feature alignment across modalities. However, the local associations between videos and texts are not modeled, restricting the pre-training models' generality, especially for tasks requiring the temporal video boundary for certain query texts. This work introduces a novel text-video localization pre-text task to enable fine-grained temporal and semantic alignment such that the trained model can accurately perceive temporal boundaries in videos given the text description. Specifically, text-video localization consists of moment retrieval, which predicts start and end boundaries in videos given the text description, and text localization which matches the subset of texts with the video features. To produce temporal boundaries, frame features in several videos are manually merged into a long video sequence that interacts with a text sequence. With the localization task, our method connects the fine-grained frame representations with the word representations and implicitly distinguishes representations of different instances in the single modality. Notably, comprehensive experimental results show that our method significantly improves the state-of-the-art performance on various benchmarks, covering text-to-video retrieval, video question answering, video captioning, temporal action localization and temporal moment retrieval. The code will be released soon.

cs.CV↗

arXiv2023

Persistent Josephson tunneling between Bi$_2$Sr$_2$CaCu$_2$O$_{8+x}$ flakes twisted by 45$^\circ$ across the superconducting dome

There is a heated debate on the Josephson effect in twisted Bi$_2$Sr$_2$CaCu$_2$O$_{8+x}$ flakes. Recent experimental results suggest the presence of either anomalously isotropic pairing or exotic $d$+i$d$-wave pairing, in addition to the commonly believed $d$-wave one. Here, we address this controversy by fabricating ultraclean junctions with uncompromised crystalline quality and stoichiometry at the junction interfaces. In the optimally doped regime, we obtain prominent Josephson coupling (2-4 mV) in multiple junctions with the twist angle of 45$^\circ$, in sharp contrast to a recent report that shows two orders of magnitude suppression around 45$^\circ$ from the value at 0$^\circ$. We further extend this study to the previously unexplored overdoped regime and observe pronounced Josephson tunneling at 45$^\circ$ together with Josephson diode effect up to 50 K. Our work helps establish the persistent presence of an isotropic pairing component across the entire superconducting phase diagram.

cond-mat.supr-con↗

arXiv2023

One is All: Bridging the Gap Between Neural Radiance Fields Architectures with Progressive Volume Distillation

Neural Radiance Fields (NeRF) methods have proved effective as compact, high-quality and versatile representations for 3D scenes, and enable downstream tasks such as editing, retrieval, navigation, etc. Various neural architectures are vying for the core structure of NeRF, including the plain Multi-Layer Perceptron (MLP), sparse tensors, low-rank tensors, hashtables and their compositions. Each of these representations has its particular set of trade-offs. For example, the hashtable-based representations admit faster training and rendering but their lack of clear geometric meaning hampers downstream tasks like spatial-relation-aware editing. In this paper, we propose Progressive Volume Distillation (PVD), a systematic distillation method that allows any-to-any conversions between different architectures, including MLP, sparse or low-rank tensors, hashtables and their compositions. PVD consequently empowers downstream applications to optimally adapt the neural representations for the task at hand in a post hoc fashion. The conversions are fast, as distillation is progressively performed on different levels of volume representations, from shallower to deeper. We also employ special treatment of density to deal with its specific numerical instability problem. Empirical evidence is presented to validate our method on the NeRF-Synthetic, LLFF and TanksAndTemples datasets. For example, with PVD, an MLP-based NeRF model can be distilled from a hashtable-based Instant-NGP model at a 10X~20X faster speed than being trained the original NeRF from scratch, while achieving a superior level of synthesis quality. Code is available at https://github.com/megvii-research/AAAI2023-PVD.

cs.CV↗

arXiv2022

Performance analysis for OFDM-based multi-carrier continuous-variable quantum key distribution with arbitrary modulation protocol

Multi-carrier continuous-variable quantum key distribution (CV-QKD) is considered to be a promising way to boost the secret key rate (SKR) over the existing single-carrier CV-QKD scheme. However, the extra excess noise induced in the imperfect multi-carrier quantum state preparation process of N subcarriers will limit the performance of the system. Here, a systematic modulation noise model is proposed for the multi-carrier CV-QKD based on the orthogonal frequency division multiplexing (OFDM). Subsequently, the performance of multi-carrier CV-QKD with arbitrary modulation protocol (e.g. QPSK, 256QAM and Gaussian modulation protocol) can be quantitatively evaluated by combining the security analysis method of the single-carrier CV-QKD. Under practical system parameters, the simulation results show that the SKR of the multi-carrier CV-QKD can still be significantly improved by increasing the carrier number N even with imperfect practical modulations. Specifically, the total SKR of multi-carrier CV-QKD can be optimized by carefully choosing N. The proposed model provides a feasible theoretical framework for the future multi-carrier CV-QKD experimental implementation.

quant-ph↗

arXiv2022

Sub-Mbps key-rate continuous-variable quantum key distribution with local-local-oscillator over 100 km fiber

We experimentally demonstrated a sub-Mbps key rate Gaussian-modulated coherent-state continuous-variable quantum key distribution (CV-QKD) over 100 km transmission distance. To efficiently control the excess noise, the quantum signal and the pilot tone are co-transmitted in fiber channel based on wide-band frequency and polarization multiplexing methods. Furthermore, a high-accuracy data-assisted time domain equalization algorithm is carefully designed to compensate the phase noise and polarization variation in low signal-to-noise ratio. The asymptotic secure key rate (SKR) of the demonstrated CV-QKD is experimentally evaluated to be 10.36 Mbps, 2.59 Mbps, and 0.69 Mbps over transmission distance of 50 km, 75 km, and 100 km, respectively. The experimental demonstrated CV-QKD system significantly improves transmission distance and SKR compared to the state-of-art GMCS CV-QKD experimental results, and shows the potential for long-distance and high-speed secure quantum key distribution.

quant-ph↗

arXiv2022

PointNeuron: 3D Neuron Reconstruction via Geometry and Topology Learning of Point Clouds

Digital neuron reconstruction from 3D microscopy images is an essential technique for investigating brain connectomics and neuron morphology. Existing reconstruction frameworks use convolution-based segmentation networks to partition the neuron from noisy backgrounds before applying the tracing algorithm. The tracing results are sensitive to the raw image quality and segmentation accuracy. In this paper, we propose a novel framework for 3D neuron reconstruction. Our key idea is to use the geometric representation power of the point cloud to better explore the intrinsic structural information of neurons. Our proposed framework adopts one graph convolutional network to predict the neural skeleton points and another one to produce the connectivity of these points. We finally generate the target SWC file through the interpretation of the predicted point coordinates, radius, and connections. Evaluated on the Janelia-Fly dataset from the BigNeuron project, we show that our framework achieves competitive neuron reconstruction performance. Our geometry and topology learning of point clouds could further benefit 3D medical image analysis, such as cardiac surface reconstruction. Our code is available at https://github.com/RunkaiZhao/PointNeuron.

cs.CV↗

arXiv2022

High-throughput decoder of quasi-cyclic LDPC codes with limited precision for continuous-variable quantum key distribution systems

More than Mbps secret key rate was demonstrated for continuous-variable quantum key distribution (CV-QKD) systems, but real-time postprocessing is not allowed, which is restricted by the throughput of the error correction decoding in postprocessing. In this paper, a high-throughput FPGA-based quasi-cyclic LDPC decoder is proposed and implemented to support Mbps real-time secret key rate generation for CV-QKD for the first time. A residual bit error correction algorithm is used to solve the problem of high frame errors rate (FER) caused by the limited precision of the decoder. Specifically, real-time high-speed decoding for CV-QKD systems with typical code rates 0.2 and 0.1 is implemented on a commercial FPGA, and two throughputs of 360.92Mbps and 194.65Mbps are achieved, respectively, which can support 17.97 Mbps and 2.48 Mbps real-time generation of secret key rates under typical transmission distances of 25km and 50km, correspondingly. The proposed method paves the way for high-rate real-time CV-QKD deployment in secure metropolitan area network.

quant-ph↗

arXiv2022

Towards Generalisable Audio Representations for Audio-Visual Navigation

In audio-visual navigation (AVN), an intelligent agent needs to navigate to a constantly sound-making object in complex 3D environments based on its audio and visual perceptions. While existing methods attempt to improve the navigation performance with preciously designed path planning or intricate task settings, none has improved the model generalisation on unheard sounds with task settings unchanged. We thus propose a contrastive learning-based method to tackle this challenge by regularising the audio encoder, where the sound-agnostic goal-driven latent representations can be learnt from various audio signals of different classes. In addition, we consider two data augmentation strategies to enrich the training sounds. We demonstrate that our designs can be easily equipped to existing AVN frameworks to obtain an immediate performance gain (13.4%$\uparrow$ in SPL on Replica and 12.2%$\uparrow$ in SPL on MP3D). Our project is available at https://AV-GeN.github.io/.

cs.SD↗

arXiv2022

Direct observation of nodeless superconductivity and phonon modes in electron-doped copper oxide Sr$_{1-x}$Nd$_x$CuO$_2$

The microscopic understanding of high-temperature superconductivity in cuprates has been hindered by the apparent complexity of crystal structures in these materials. We used scanning tunneling microscopy and spectroscopy to study an electron-doped copper oxide compound Sr$_{1-x}$Nd$_x$CuO$_2$ that has only bare cations separating the CuO$_2$ planes and thus the simplest infinite-layer structure among all cuprate superconductors. Tunneling conductance spectra of the major CuO$_2$ planes in the superconducting state revealed direct evidence for a nodeless pairing gap, regardless of variation of its magnitude with the local doping of trivalent neodymium. Furthermore, three distinct bosonic modes are observed as multiple peak-dip-hump features outside the superconducting gaps and their respective energies depend little on the spatially varying gaps. Along with the bosonic modes with energies identical to those of the external, bending and stretching phonons of copper oxides, our findings indicate their origin from lattice vibrations rather than spin excitations.

cond-mat.supr-con↗

arXiv2022

Spatiality-guided Transformer for 3D Dense Captioning on Point Clouds

Dense captioning in 3D point clouds is an emerging vision-and-language task involving object-level 3D scene understanding. Apart from coarse semantic class prediction and bounding box regression as in traditional 3D object detection, 3D dense captioning aims at producing a further and finer instance-level label of natural language description on visual appearance and spatial relations for each scene object of interest. To detect and describe objects in a scene, following the spirit of neural machine translation, we propose a transformer-based encoder-decoder architecture, namely SpaCap3D, to transform objects into descriptions, where we especially investigate the relative spatiality of objects in 3D scenes and design a spatiality-guided encoder via a token-to-token spatial relation learning objective and an object-centric decoder for precise and spatiality-enhanced object caption generation. Evaluated on two benchmark datasets, ScanRefer and ReferIt3D, our proposed SpaCap3D outperforms the baseline method Scan2Cap by 4.94% and 9.61% in CIDEr@0.5IoU, respectively. Our project page with source code and supplementary files is available at https://SpaCap3D.github.io/ .

cs.CV↗