arXiv subjects

Yi Wang

Publications and source records attributed to Yi Wang.

At least 649 records · Page 36Linked to original sources

arXiv2023

VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking

Scale is the primary factor for building a powerful foundation model that could well generalize to a variety of downstream tasks. However, it is still challenging to train video foundation models with billions of parameters. This paper shows that video masked autoencoder (VideoMAE) is a scalable and general self-supervised pre-trainer for building video foundation models. We scale the VideoMAE in both model and data with a core design. Specifically, we present a dual masking strategy for efficient pre-training, with an encoder operating on a subset of video tokens and a decoder processing another subset of video tokens. Although VideoMAE is very efficient due to high masking ratio in encoder, masking decoder can still further reduce the overall computational cost. This enables the efficient pre-training of billion-level models in video. We also use a progressive training paradigm that involves an initial pre-training on a diverse multi-sourced unlabeled dataset, followed by a post-pre-training on a mixed labeled dataset. Finally, we successfully train a video ViT model with a billion parameters, which achieves a new state-of-the-art performance on the datasets of Kinetics (90.0% on K400 and 89.9% on K600) and Something-Something (68.7% on V1 and 77.0% on V2). In addition, we extensively verify the pre-trained video ViT models on a variety of downstream tasks, demonstrating its effectiveness as a general video representation learner. The code and model is available at \url{https://github.com/OpenGVLab/VideoMAEv2}.

cs.CV↗

arXiv2023

Bitstream-Corrupted JPEG Images are Restorable: Two-stage Compensation and Alignment Framework for Image Restoration

In this paper, we study a real-world JPEG image restoration problem with bit errors on the encrypted bitstream. The bit errors bring unpredictable color casts and block shifts on decoded image contents, which cannot be resolved by existing image restoration methods mainly relying on pre-defined degradation models in the pixel domain. To address these challenges, we propose a robust JPEG decoder, followed by a two-stage compensation and alignment framework to restore bitstream-corrupted JPEG images. Specifically, the robust JPEG decoder adopts an error-resilient mechanism to decode the corrupted JPEG bitstream. The two-stage framework is composed of the self-compensation and alignment (SCA) stage and the guided-compensation and alignment (GCA) stage. The SCA adaptively performs block-wise image color compensation and alignment based on the estimated color and block offsets via image content similarity. The GCA leverages the extracted low-resolution thumbnail from the JPEG header to guide full-resolution pixel-wise image restoration in a coarse-to-fine manner. It is achieved by a coarse-guided pix2pix network and a refine-guided bi-directional Laplacian pyramid fusion network. We conduct experiments on three benchmarks with varying degrees of bit error rates. Experimental results and ablation studies demonstrate the superiority of our proposed method. The code will be released at https://github.com/wenyang001/Two-ACIR.

eess.IV↗

arXiv2023

A Byte Sequence is Worth an Image: CNN for File Fragment Classification Using Bit Shift and n-Gram Embeddings

File fragment classification (FFC) on small chunks of memory is essential in memory forensics and Internet security. Existing methods mainly treat file fragments as 1d byte signals and utilize the captured inter-byte features for classification, while the bit information within bytes, i.e., intra-byte information, is seldom considered. This is inherently inapt for classifying variable-length coding files whose symbols are represented as the variable number of bits. Conversely, we propose Byte2Image, a novel data augmentation technique, to introduce the neglected intra-byte information into file fragments and re-treat them as 2d gray-scale images, which allows us to capture both inter-byte and intra-byte correlations simultaneously through powerful convolutional neural networks (CNNs). Specifically, to convert file fragments to 2d images, we employ a sliding byte window to expose the neglected intra-byte information and stack their n-gram features row by row. We further propose a byte sequence \& image fusion network as a classifier, which can jointly model the raw 1d byte sequence and the converted 2d image to perform FFC. Experiments on FFT-75 dataset validate that our proposed method can achieve notable accuracy improvements over state-of-the-art methods in nearly all scenarios. The code will be released at https://github.com/wenyang001/Byte2Image.

cs.CV↗

arXiv2023

Detecting Chinese Fake News on Twitter during the COVID-19 Pandemic

The outbreak of COVID-19 has led to a global surge of Sinophobia partly because of the spread of misinformation, disinformation, and fake news on China. In this paper, we report on the creation of a novel classifier that detects whether Chinese-language social media posts from Twitter are related to fake news about China. The classifier achieves an F1 score of 0.64 and an accuracy rate of 93%. We provide the final model and a new training dataset with 18,425 tweets for researchers to study fake news in the Chinese language during the COVID-19 pandemic. We also introduce a new dataset generated by our classifier that tracks the dynamics of fake news in the Chinese language during the early pandemic.

cs.CY↗

arXiv2023

mcLARO: Multi-Contrast Learned Acquisition and Reconstruction Optimization for simultaneous quantitative multi-parametric mapping

Purpose: To develop a method for rapid sub-millimeter T1, T2, T2* and QSM mapping in a single scan using multi-contrast Learned Acquisition and Reconstruction Optimization (mcLARO). Methods: A pulse sequence was developed by interleaving inversion recovery and T2 magnetization preparations and single-echo and multi-echo gradient echo acquisitions, which sensitized k-space data to T1, T2, T2* and magnetic susceptibility. The proposed mcLARO used a deep learning framework to optimize both the multi-contrast k-space under-sampling pattern and the image reconstruction based on image feature fusion. The proposed mcLARO method with R=8 under-sampling was validated in a retrospective ablation study using fully sampled data as reference and evaluated in a prospective study using separately acquired conventionally sampled quantitative maps as reference standard. Results: The retrospective ablation study showed improved image sharpness of mcLARO compared to the baseline network without multi-contrast sampling pattern optimization or image feature fusion, and negligible bias and narrow 95% limits of agreement on regional T1, T2, T2* and QSM values were obtained by the under-sampled reconstructions compared to the fully sampled reconstruction. The prospective study showed small or negligible bias and narrow 95% limits of agreement on regional T1, T2, T2* and QSM values by mcLARO (5:39 mins) compared to reference scans (40:03 mins in total). Conclusion: mcLARO enabled fast sub-millimeter T1, T2, T2* and QSM mapping in a single scan.

eess.IV↗

arXiv2023

The Neumann problem on the Clifford torus in $\mathbb{S}^3$

We discuss the solution of the Neumann problem associated with the CR Yamabe operator on a subset $Ω$ of the CR manifold $\mathbb{S}^3$ bounded by the Clifford torus $Σ$. We also discuss the Yamabe-type problem of finding a contact form on $Ω$ which has zero Tanaka--Webster scalar curvature and for which $Σ$ has constant $p$-mean curvature.

math.AP↗

arXiv2023

NeuralLift-360: Lifting An In-the-wild 2D Photo to A 3D Object with 360° Views

Virtual reality and augmented reality (XR) bring increasing demand for 3D content. However, creating high-quality 3D content requires tedious work that a human expert must do. In this work, we study the challenging task of lifting a single image to a 3D object and, for the first time, demonstrate the ability to generate a plausible 3D object with 360° views that correspond well with the given reference image. By conditioning on the reference image, our model can fulfill the everlasting curiosity for synthesizing novel views of objects from images. Our technique sheds light on a promising direction of easing the workflows for 3D artists and XR designers. We propose a novel framework, dubbed NeuralLift-360, that utilizes a depth-aware neural radiance representation (NeRF) and learns to craft the scene guided by denoising diffusion models. By introducing a ranking loss, our NeuralLift-360 can be guided with rough depth estimation in the wild. We also adopt a CLIP-guided sampling strategy for the diffusion prior to provide coherent guidance. Extensive experiments demonstrate that our NeuralLift-360 significantly outperforms existing state-of-the-art baselines. Project page: https://vita-group.github.io/NeuralLift-360/

cs.CV↗

arXiv2023

Decoupling between gravitationally bounded systems and the cosmic expansion

Recently, it was hypothesized that some supermassive black holes (SMBHs) may couple to the cosmic expansion. The mass of these SMBHs increase as the cubic power of the cosmic scale factor, leaving the energy density of the SMBHs unchanged when the universe expands. However, following general principles of general relativity, namely, locality or junction conditions, we show that the inner part of a gravitationally bounded system is unaware of the cosmic expansion, since the outer solution can be smoothly replaced by an asymptotically flat background. For the same reason, the direct reason that we do not expand with the expansion of the universe is not that we are bound states, but rather we are positioned in a greater gravitationally bounded system, namely the local group.

gr-qc↗

arXiv2023

Exploiting prompt learning with pre-trained language models for Alzheimer's Disease detection

Early diagnosis of Alzheimer's disease (AD) is crucial in facilitating preventive care and to delay further progression. Speech based automatic AD screening systems provide a non-intrusive and more scalable alternative to other clinical screening techniques. Textual embedding features produced by pre-trained language models (PLMs) such as BERT are widely used in such systems. However, PLM domain fine-tuning is commonly based on the masked word or sentence prediction costs that are inconsistent with the back-end AD detection task. To this end, this paper investigates the use of prompt-based fine-tuning of PLMs that consistently uses AD classification errors as the training objective function. Disfluency features based on hesitation or pause filler token frequencies are further incorporated into prompt phrases during PLM fine-tuning. The decision voting based combination among systems using different PLMs (BERT and RoBERTa) or systems with different fine-tuning paradigms (conventional masked-language modelling fine-tuning and prompt-based fine-tuning) is further applied. Mean, standard deviation and the maximum among accuracy scores over 15 experiment runs are adopted as performance measurements for the AD detection system. Mean detection accuracy of 84.20% (with std 2.09%, best 87.5%) and 82.64% (with std 4.0%, best 89.58%) were obtained using manual and ASR speech transcripts respectively on the ADReSS20 test set consisting of 48 elderly speakers.

cs.CL↗

arXiv2023

NeRFLiX: High-Quality Neural View Synthesis by Learning a Degradation-Driven Inter-viewpoint MiXer

Neural radiance fields (NeRF) show great success in novel view synthesis. However, in real-world scenes, recovering high-quality details from the source images is still challenging for the existing NeRF-based approaches, due to the potential imperfect calibration information and scene representation inaccuracy. Even with high-quality training frames, the synthetic novel views produced by NeRF models still suffer from notable rendering artifacts, such as noise, blur, etc. Towards to improve the synthesis quality of NeRF-based approaches, we propose NeRFLiX, a general NeRF-agnostic restorer paradigm by learning a degradation-driven inter-viewpoint mixer. Specially, we design a NeRF-style degradation modeling approach and construct large-scale training data, enabling the possibility of effectively removing NeRF-native rendering artifacts for existing deep neural networks. Moreover, beyond the degradation removal, we propose an inter-viewpoint aggregation framework that is able to fuse highly related high-quality training images, pushing the performance of cutting-edge NeRF models to entirely new levels and producing highly photo-realistic synthetic views.

cs.CV↗

arXiv2023

Uniqueness of composite wave of shock and rarefaction in the inviscid limit of Navier-Stokes equations

The uniqueness of entropy solution for the compressible Euler equations is a fundamental and challenging problem. In this paper, the uniqueness of a composite wave of shock and rarefaction of 1-d compressible Euler equations is proved in the inviscid limit of compressible Navier-Stokes equations. Moreover, the relative entropy around the original Riemann solution consisting of shock and rarefaction under the large perturbation is shown to be uniformly bounded by the framework developed in \cite{Kang-Vasseur-2021-Invent}. The proof contains two new ingredients: 1) a cut-off technique and the expanding property of rarefaction are used to overcome the errors generated by the viscosity related to inviscid rarefaction; 2) the error terms concerning the interactions between shock and rarefaction are controlled by the compressibility of shock, the decay of derivative of rarefaction and the separation of shock and rarefaction as time increases.

math.AP↗

arXiv2023

Efficiency-boosted semiconductor optical amplifiers via mode-division multiplexing

Semiconductor optical amplifiers (SOA) are a fundamental building block for many photonic systems. However, their power inefficiency has been setting back operational cost reduction, and the resulting thermal losses constrain miniaturization, and the realization of more complex photonic functions such as large-scale switches and optical phased arrays. In this work, we demonstrate significant gain and efficiency enhancement using an extra degree of freedom of light - the mode space. This is done without changing the SOA's material design, and therefore high versatility and compatibility can be obtained. Light is multiplexed in different guided modes and is reinjected into the same gain section twice without introducing resonance, doubling the interaction length in a broadband manner. Up to 87% higher gain and 300% higher wall-plug efficiency are obtained in a double-pass SOA compared to a conventional single-pass SOA, at the same operating current, in the wavelength range of 1560 - 1580 nm.

physics.optics↗

arXiv2023

VPU-EM: An Event-based Modeling Framework to Evaluate NPU Performance and Power Efficiency at Scale

State-of-art NPUs are typically architected as a self-contained sub-system with multiple heterogeneous hardware computing modules, and a dataflow-driven programming model. There lacks well-established methodology and tools in the industry to evaluate and compare the performance of NPUs from different architectures. We present an event-based performance modeling framework, VPU-EM, targeting scalable performance evaluation of modern NPUs across diversified AI workloads. The framework adopts high-level event-based system-simulation methodology to abstract away design details for speed, while maintaining hardware pipelining, concurrency and interaction with software task scheduling. It is natively developed in Python and built to interface directly with AI frameworks such as Tensorflow, PyTorch, ONNX and OpenVINO, linking various in-house NPU graph compilers to achieve optimized full model performance. Furthermore, VPU-EM also provides the capability to model power characteristics of NPU in Power-EM mode to enable joint performance/power analysis. Using VPU-EM, we conduct performance/power analysis of models from representative neural network architecture. We demonstrate that even though this framework is developed for Intel VPU, an Intel in-house NPU IP technology, the methodology can be generalized for analysis of modern NPUs.

cs.AR↗

arXiv2023

PointPatchMix: Point Cloud Mixing with Patch Scoring

Data augmentation is an effective regularization strategy for mitigating overfitting in deep neural networks, and it plays a crucial role in 3D vision tasks, where the point cloud data is relatively limited. While mixing-based augmentation has shown promise for point clouds, previous methods mix point clouds either on block level or point level, which has constrained their ability to strike a balance between generating diverse training samples and preserving the local characteristics of point clouds. Additionally, the varying importance of each part of the point clouds has not been fully considered, cause not all parts contribute equally to the classification task, and some parts may contain unimportant or redundant information. To overcome these challenges, we propose PointPatchMix, a novel approach that mixes point clouds at the patch level and integrates a patch scoring module to generate content-based targets for mixed point clouds. Our approach preserves local features at the patch level, while the patch scoring module assigns targets based on the content-based significance score from a pre-trained teacher model. We evaluate PointPatchMix on two benchmark datasets, ModelNet40 and ScanObjectNN, and demonstrate significant improvements over various baselines in both synthetic and real-world datasets, as well as few-shot settings. With Point-MAE as our baseline, our model surpasses previous methods by a significant margin, achieving 86.3% accuracy on ScanObjectNN and 94.1% accuracy on ModelNet40. Furthermore, our approach shows strong generalization across multiple architectures and enhances the robustness of the baseline model.

cs.CV↗

arXiv2023

Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervision

In this paper, we consider the problem of open-vocabulary semantic segmentation (OVS), which aims to segment objects of arbitrary classes instead of pre-defined, closed-set categories. The main contributions are as follows: First, we propose a transformer-based model for OVS, termed as OVSegmentor, which only exploits web-crawled image-text pairs for pre-training without using any mask annotations. OVSegmentor assembles the image pixels into a set of learnable group tokens via a slot-attention based binding module, and aligns the group tokens to the corresponding caption embedding. Second, we propose two proxy tasks for training, namely masked entity completion and cross-image mask consistency. The former aims to infer all masked entities in the caption given the group tokens, that enables the model to learn fine-grained alignment between visual groups and text entities. The latter enforces consistent mask predictions between images that contain shared entities, which encourages the model to learn visual invariance. Third, we construct CC4M dataset for pre-training by filtering CC12M with frequently appeared entities, which significantly improves training efficiency. Fourth, we perform zero-shot transfer on three benchmark datasets, PASCAL VOC 2012, PASCAL Context, and COCO Object. Our model achieves superior segmentation results over the state-of-the-art method by using only 3\% data (4M vs 134M) for pre-training. Code and pre-trained models will be released for future research.

cs.CV↗

arXiv2023

Analytic combinatorics of coordination numbers of cubic lattices

We investigate coordination numbers of the cubic lattices with emphases on their analytic behaviors, including the total positivity of the coordination matrices, the distribution of zeros of the coordination polynomials, the asymptotic normality of the coefficients of the coordination polynomials, the log-concavity and the log-convexity of the coordination numbers.

math.CO↗

arXiv2023

On extremal spectral radius of blow-up uniform hypergraphs

Let $G$ be an $r$-uniform hypergraph of order $t$ and $ρ(G)$ is the spectral radius of $\mathcal{A}(G)$, where $\mathcal{A}(G)$ is the adjacency tensor of $G$. A blow-up of $G$ respected to a positive integer vector $(n_{1}, n_{2},\ldots,n_{t})$, denoted by $G \circ (n_{1}, n_{2},\ldots,n_{t})$, is an $r$-uniform hypergraph obtained from $G$ by replacing each vertex $j$ of $G$ with a class of vertices $V_{j}$ of size $n_{j}\ge 1$ and if $\{j_{1},j_{2},\ldots,j_{r}\}\in E(G)$, then $\{v_{i_1},v_{i_2},\ldots,v_{i_r}\}\in E(H)$ for every $v_{i_{1}}\in V_{j_{1}}, v_{i_{2}}\in V_{j_{2}},\ldots, v_{i_{r}}\in V_{j_{r}}$. Let $\mathcal{B}_{n}(G)$ be the set of all the blow-ups of $G$ such that each $n_i\ge 1$ and $\sum_{i=1}^n n_i=n$. Let $K_{t}^{r}$ be the complete $r$-uniform hypergraph of order $t$, and let $SH(m,q,r)$ be the $r$-uniform sunflower hypergraph with $m$ petals and a kernel of size $r-q$ on $t$ vertices. For any $H\in \mathcal{B}_{n}(K_{t}^{r})$, we prove that $$ρ(K_{t}^{r}\circ(n-t+1,1,1,\ldots,1))\leqρ(H)\leq ρ(T_{t}^{r}(n)),$$ with the left equality holds if and only if $H\cong K_{t}^{r}\circ(n-t+1,1,1,\ldots,1)$, and the right equality holds if and only if $H\cong T_{t}^{r}(n)$, where $T_{t}^{r}(n)$ is the complete $t$-partite $r$-uniform hypergraph of order $n$, with parts of size $\lfloor n / k\rfloor$ or $\lceil n / k \rceil$. For any $H\in \mathcal{B}_{n}(H(m,q,r))$, we determine the exact value of the spectral radius of $H$ and characterize the hypergraphs with maximum spectral radius and minimum spectral radius in $\mathcal{B}_{n}(H(m,q,r))$, respectively.

math.CO↗

arXiv2023

PulseDL-II: A System-on-Chip Neural Network Accelerator for Timing and Energy Extraction of Nuclear Detector Signals

Front-end electronics equipped with high-speed digitizers are being used and proposed for future nuclear detectors. Recent literature reveals that deep learning models, especially one-dimensional convolutional neural networks, are promising when dealing with digital signals from nuclear detectors. Simulations and experiments demonstrate the satisfactory accuracy and additional benefits of neural networks in this area. However, specific hardware accelerating such models for online operations still needs to be studied. In this work, we introduce PulseDL-II, a system-on-chip (SoC) specially designed for applications of event feature (time, energy, etc.) extraction from pulses with deep learning. Based on the previous version, PulseDL-II incorporates a RISC CPU into the system structure for better functional flexibility and integrity. The neural network accelerator in the SoC adopts a three-level (arithmetic unit, processing element, neural network) hierarchical architecture and facilitates parameter optimization of the digital design. Furthermore, we devise a quantization scheme compatible with deep learning frameworks (e.g., TensorFlow) within a selected subset of layer types. We validate the correct operations of PulseDL-II on field programmable gate arrays (FPGA) alone and with an experimental setup comprising a direct digital synthesis (DDS) and analog-to-digital converters (ADC). The proposed system achieved 60 ps time resolution and 0.40% energy resolution at signal to noise ratio (SNR) of 47.4 dB.

physics.ins-det↗