arXiv subjects

Yue Zhao

Publications and source records attributed to Yue Zhao.

At least 307 records · Page 17Linked to original sources

arXiv2023

What Makes for Robust Multi-Modal Models in the Face of Missing Modalities?

With the growing success of multi-modal learning, research on the robustness of multi-modal models, especially when facing situations with missing modalities, is receiving increased attention. Nevertheless, previous studies in this domain exhibit certain limitations, as they often lack theoretical insights or their methodologies are tied to specific network architectures or modalities. We model the scenarios of multi-modal models encountering missing modalities from an information-theoretic perspective and illustrate that the performance ceiling in such scenarios can be approached by efficiently utilizing the information inherent in non-missing modalities. In practice, there are two key aspects: (1) The encoder should be able to extract sufficiently good features from the non-missing modality; (2) The extracted features should be robust enough not to be influenced by noise during the fusion process across modalities. To this end, we introduce Uni-Modal Ensemble with Missing Modality Adaptation (UME-MMA). UME-MMA employs uni-modal pre-trained weights for the multi-modal model to enhance feature extraction and utilizes missing modality data augmentation techniques to better adapt to situations with missing modalities. Apart from that, UME-MMA, built on a late-fusion learning framework, allows for the plug-and-play use of various encoders, making it suitable for a wide range of modalities and enabling seamless integration of large-scale pre-trained encoders to further enhance performance. And we demonstrate UME-MMA's effectiveness in audio-visual datasets~(e.g., AV-MNIST, Kinetics-Sound, AVE) and vision-language datasets~(e.g., MM-IMDB, UPMC Food101).

cs.AI↗

arXiv2023

Improving Discriminative Multi-Modal Learning with Large-Scale Pre-Trained Models

This paper investigates how to better leverage large-scale pre-trained uni-modal models to further enhance discriminative multi-modal learning. Even when fine-tuned with only uni-modal data, these models can outperform previous multi-modal models in certain tasks. It's clear that their incorporation into multi-modal learning would significantly improve performance. However, multi-modal learning with these models still suffers from insufficient learning of uni-modal features, which weakens the resulting multi-modal model's generalization ability. While fine-tuning uni-modal models separately and then aggregating their predictions is straightforward, it doesn't allow for adequate adaptation between modalities, also leading to sub-optimal results. To this end, we introduce Multi-Modal Low-Rank Adaptation learning (MMLoRA). By freezing the weights of uni-modal fine-tuned models, adding extra trainable rank decomposition matrices to them, and subsequently performing multi-modal joint training, our method enhances adaptation between modalities and boosts overall performance. We demonstrate the effectiveness of MMLoRA on three dataset categories: audio-visual (e.g., AVE, Kinetics-Sound, CREMA-D), vision-language (e.g., MM-IMDB, UPMC Food101), and RGB-Optical Flow (UCF101).

cs.CV↗

arXiv2023

LEAP: Liberate Sparse-view 3D Modeling from Camera Poses

Are camera poses necessary for multi-view 3D modeling? Existing approaches predominantly assume access to accurate camera poses. While this assumption might hold for dense views, accurately estimating camera poses for sparse views is often elusive. Our analysis reveals that noisy estimated poses lead to degraded performance for existing sparse-view 3D modeling methods. To address this issue, we present LEAP, a novel pose-free approach, therefore challenging the prevailing notion that camera poses are indispensable. LEAP discards pose-based operations and learns geometric knowledge from data. LEAP is equipped with a neural volume, which is shared across scenes and is parameterized to encode geometry and texture priors. For each incoming scene, we update the neural volume by aggregating 2D image features in a feature-similarity-driven manner. The updated neural volume is decoded into the radiance field, enabling novel view synthesis from any viewpoint. On both object-centric and scene-level datasets, we show that LEAP significantly outperforms prior methods when they employ predicted poses from state-of-the-art pose estimators. Notably, LEAP performs on par with prior approaches that use ground-truth poses while running $400\times$ faster than PixelNeRF. We show LEAP generalizes to novel object categories and scenes, and learns knowledge closely resembles epipolar geometry. Project page: https://hwjiang1510.github.io/LEAP/

cs.CV↗

arXiv2023

Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistants

Recent Multimodal Large Language Models (MLLMs) exhibit impressive abilities to perceive images and follow open-ended instructions. The capabilities of MLLMs depend on two crucial factors: the model architecture to facilitate the feature alignment of visual modules and large language models; the multimodal instruction tuning datasets for human instruction following. (i) For the model architecture, most existing models introduce an external bridge module to connect vision encoders with language models, which needs an additional feature-alignment pre-training. In this work, we discover that compact pre-trained vision language models can inherently serve as ``out-of-the-box'' bridges between vision and language. Based on this, we propose Muffin framework, which directly employs pre-trained vision-language models to act as providers of visual signals. (ii) For the multimodal instruction tuning datasets, existing methods omit the complementary relationship between different datasets and simply mix datasets from different tasks. Instead, we propose UniMM-Chat dataset which explores the complementarities of datasets to generate 1.1M high-quality and diverse multimodal instructions. We merge information describing the same image from diverse datasets and transforms it into more knowledge-intensive conversation data. Experimental results demonstrate the effectiveness of the Muffin framework and UniMM-Chat dataset. Muffin achieves state-of-the-art performance on a wide range of vision-language tasks, significantly surpassing state-of-the-art models like LLaVA and InstructBLIP. Our model and dataset are all accessible at https://github.com/thunlp/muffin.

cs.CV↗

arXiv2023

Training a Large Video Model on a Single Machine in a Day

Videos are big, complex to pre-process, and slow to train on. State-of-the-art large-scale video models are trained on clusters of 32 or more GPUs for several days. As a consequence, academia largely ceded the training of large video models to industry. In this paper, we show how to still train a state-of-the-art video model on a single machine with eight consumer-grade GPUs in a day. We identify three bottlenecks, IO, CPU, and GPU computation, and optimize each. The result is a highly efficient video training pipeline. For comparable architectures, our pipeline achieves higher accuracies with $\frac{1}{8}$ of the computation compared to prior work. Code is available at https://github.com/zhaoyue-zephyrus/AVION.

cs.CV↗

arXiv2023

Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack

Training text-to-image models with web scale image-text pairs enables the generation of a wide range of visual concepts from text. However, these pre-trained models often face challenges when it comes to generating highly aesthetic images. This creates the need for aesthetic alignment post pre-training. In this paper, we propose quality-tuning to effectively guide a pre-trained model to exclusively generate highly visually appealing images, while maintaining generality across visual concepts. Our key insight is that supervised fine-tuning with a set of surprisingly small but extremely visually appealing images can significantly improve the generation quality. We pre-train a latent diffusion model on $1.1$ billion image-text pairs and fine-tune it with only a few thousand carefully selected high-quality images. The resulting model, Emu, achieves a win rate of $82.9\%$ compared with its pre-trained only counterpart. Compared to the state-of-the-art SDXLv1.0, Emu is preferred $68.4\%$ and $71.3\%$ of the time on visual appeal on the standard PartiPrompts and our Open User Input benchmark based on the real-world usage of text-to-image models. In addition, we show that quality-tuning is a generic approach that is also effective for other architectures, including pixel diffusion and masked generative transformer models.

cs.CV↗

arXiv2023

Accurate 3D Prediction of Missing Teeth in Diverse Patterns for Precise Dental Implant Planning

In recent years, the demand for dental implants has surged, driven by their high success rates and esthetic advantages. However, accurate prediction of missing teeth for precise digital implant planning remains a challenge due to the intricate nature of dental structures and the variability in tooth loss patterns. This study presents a novel framework for accurate prediction of missing teeth in different patterns, facilitating digital implant planning. The proposed framework begins by estimating point-to-point correspondence among a dataset of dental mesh models reconstructed from CBCT images of healthy subjects. Subsequently, tooth dictionaries are constructed for each tooth type, encoding their position and shape information based on the established point-to-point correspondence. To predict missing teeth in a given dental mesh model, sparse coefficients are learned by sparsely representing adjacent teeth of the missing teeth using the corresponding tooth dictionaries. These coefficients are then applied to the dictionaries of the missing teeth to generate accurate predictions of their positions and shapes. The evaluation results on real subjects shows that our proposed framework achieves an average prediction error of 1.04mm for predictions of single missing tooth and an average prediction error of 1.33mm for the prediction of 14 missing teeth, which demonstrates its capability of accurately predicting missing teeth in various patterns. By accurately predicting missing teeth, dental professionals can improve the planning and placement of dental implants, leading to better esthetic and functional outcomes for patients undergoing dental implant procedures.

cs.CV↗

arXiv2023

Probing the pulsar explanation of the Galactic-Center GeV excess using continuous gravitational-wave searches

Over ten years ago, Fermi observed an excess of GeV gamma rays from the Galactic Center whose origin is still under debate. One explanation for this excess involves annihilating dark matter; another requires an unresolved population of millisecond pulsars concentrated at the Galactic Center. In this work, we use the results from LIGO/Virgo's most recent all-sky search for quasi-monochromatic, persistent gravitational-wave signals from isolated neutron stars, which is estimated to be about 20-50\% of the population, to determine whether unresolved millisecond pulsars could actually explain this excess. First, we choose a luminosity function that determines the number of millisecond pulsars required to explain the observed excess. Then, we consider two models for deformations on millisecond pulsars to determine their ellipticity distributions, which are directly related to their gravitational-wave radiation. Lastly, based on null results from the O3 Frequency-Hough all-sky search for continuous gravitational waves, we find that a large set of the parameter space in the pulsar luminosity function can be excluded. We also evaluate how these exclusion regions may change with respect to various model choices. Our results are the first of their kind and represent a bridge between gamma-ray astrophysics, gravitational-wave astronomy, and dark-matter physics.

astro-ph.HE↗

arXiv2023

DSV: An Alignment Validation Loss for Self-supervised Outlier Model Selection

Self-supervised learning (SSL) has proven effective in solving various problems by generating internal supervisory signals. Unsupervised anomaly detection, which faces the high cost of obtaining true labels, is an area that can greatly benefit from SSL. However, recent literature suggests that tuning the hyperparameters (HP) of data augmentation functions is crucial to the success of SSL-based anomaly detection (SSAD), yet a systematic method for doing so remains unknown. In this work, we propose DSV (Discordance and Separability Validation), an unsupervised validation loss to select high-performing detection models with effective augmentation HPs. DSV captures the alignment between an augmentation function and the anomaly-generating mechanism with surrogate losses, which approximate the discordance and separability of test data, respectively. As a result, the evaluation via DSV leads to selecting an effective SSAD model exhibiting better alignment, which results in high detection accuracy. We theoretically derive the degree of approximation conducted by the surrogate losses and empirically show that DSV outperforms a wide range of baselines on 21 real-world tasks.

cs.LG↗

arXiv2023

Earth Shielding and Daily Modulation from Electrophilic Boosted Dark Particles

Boosted dark particles of astrophysical origin can lead to nonstandard nuclear or electron recoil signals in direct detection experiments. %It has been shown that this interpretation successfully explains the excess of keV electron recoil events recently observed by the XENON1T experiment, and that a daily modulation of the signal in the detector is expected. We conduct an investigation of the daily modulation feature of a potential future signal of this type. In particular, we perform simulations of the dark particle interactions with electrons in atoms building up the Earth on its path to the detector, and provide in-depth predictions for the expected daily changes in the signal for various direct detection experiments, including XENONnT, PandaX, and LUX-ZEPLIN.

hep-ph↗

arXiv2023

Evidence for mass-dependent peculiar velocities in compact object binaries: Towards better constraints on natal kicks

We compile a catalogue of low-mass and high-mass X-ray binaries, some recently reported binaries that likely host a neutron star (NS) or a black hole (BH), and binary pulsars (a pulsar and a non-degenerated companion) that have measured systemic radial velocities ($γ$). Using Gaia and radio proper motions together with $γ$, we integrate their Galactic orbits and infer their post-supernova (post-SN) 3D peculiar velocities ($v_\mathrm{pec}^{z=0}$ at Galactic plane crossing); these velocities bear imprint of natal kicks that compact objects received at birth. With the sample totalling 85 objects, we model the overall distribution of $v_\mathrm{pec}^{z=0}$ and find a two-component Maxwellian distribution with a low- ($σ_v \approx 21\,\mathrm{km~s^{-1}}$) and a high-velocity ($σ_v \approx 107\,\mathrm{km~s^{-1}}$) component. A further comparison between distributions of binary subgroups suggests that binaries hosting high-mass donors/luminous companions mostly have $v_\mathrm{pec}^{z=0}\lesssim 100\,\mathrm{km~s^{-1}}$, while binaries with low-mass companions exhibit a broader distribution that extends up to $\sim 400\,\mathrm{km~s^{-1}}$. We also find significant anti-correlations of $v_\mathrm{pec}^{z=0}$ with binary total mass ($M_\mathrm{tot}$) and orbital period ($P_\mathrm{orb}$), at over 99% confidence. Specifically, our fit suggests $v_\mathrm{pec}^{z=0}\propto M_\mathrm{tot}^{-0.5}$ and $v_\mathrm{pec}^{z=0}\propto P_\mathrm{orb}^{-0.2}$. Discussions are presented on possible interpretation of the correlations in the context of kinematics and possible biases. The sample should enable a range of follow-up studies on compact object binary kinematics and evolution.

astro-ph.HE↗

arXiv2023

Mathematical and numerical study of an inverse source problem for the biharmonic wave equation

In this paper, we study the inverse source problem for the biharmonic wave equation. Mathematically, we characterize the radiating sources and non-radiating sources at a fixed wavenumber. We show that a general source can be decomposed into a radiating source and a non-radiating source. The radiating source can be uniquely determined by Dirichlet boundary measurements at a fixed wavenumber. Moreover, we derive a Lipschitz stability estimate for determining the radiating source. On the other hand, the non-radiating source does not produce any scattered fields outside the support of the source function. Numerically, we propose a novel source reconstruction method based on Fourier series expansion by multi-wavenumber boundary measurements. Numerical experiments are presented to verify the accuracy and efficiency of the proposed method.

math.NA↗

arXiv2023

High-speed scanless entire bandwidth mid-infrared chemical imaging

Mid-infrared spectroscopy probes molecular vibrations to identify chemical species and functional groups. Therefore, mid-infrared hyperspectral imaging is one of the most powerful and promising candidates for chemical imaging using optical methods. Yet high-speed and entire bandwidth mid-infrared hyperspectral imaging has not been realized. Here we report a mid-infrared hyperspectral chemical imaging technique that uses chirped pulse upconversion of sub-cycle pulses at the image plane. This technique offers a lateral resolution of 15 $μ$m, and the field of view is adjustable between 800 $μ$m $\times$ 600 $μ$m to 12 mm $\times$ 9 mm. The hyperspectral imaging produces a 640 $\times$ 480 pixel image in 8 s, which covers a spectral range of 640-3015 cm$^{-1}$, comprising 1069 wavelength points and offering a wavenumber resolution of 2.6-3.7 cm$^{-1}$. For discrete frequency mid-infrared imaging, the measurement speed reaches a frame rate of 5 kHz, the repetition rate of the laser. As a demonstration, we effectively identified and mapped different components in a microfluidic device, plant cell, and mouse embryo section. The great capacity and latent force of this technique in chemical imaging promise to be applied to many fields such as chemical analysis, biology, and medicine.

physics.optics↗

arXiv2023

Regularized Multivariate Functional Principal Component Analysis

Multivariate Functional Principal Component Analysis (MFPCA) is a valuable tool for exploring relationships and identifying shared patterns of variation in multivariate functional data. However, controlling the roughness of the extracted Principal Components (PCs) can be challenging. This paper introduces a novel approach called regularized MFPCA (ReMFPCA) to address this issue and enhance the smoothness and interpretability of the multivariate functional PCs. ReMFPCA incorporates a roughness penalty within a penalized framework, using a parameter vector to regulate the smoothness of each functional variable. The proposed method generates smoothed multivariate functional PCs, providing a concise and interpretable representation of the data. Extensive simulations and real data examples demonstrate the effectiveness of ReMFPCA and its superiority over alternative methods. The proposed approach opens new avenues for analyzing and uncovering relationships in complex multivariate functional datasets.

stat.ME↗

arXiv2023

Exotica in the Globular Cluster M4, Studied with Chandra, HST, and the VLA

Using the Hubble Ultraviolet Globular Cluster Survey (HUGS) and additional HST archival data, we have carried out a search for optical counterparts to the low-luminosity Chandra X-ray sources in the globular cluster M4 (NGC 6121). We have also searched for optical or X-ray counterparts to radio sources detected by the VLA. We find 24 new confident optical counterparts to Chandra sources for a total of 40, including the 16 previously identified. Of the 24 new identifications, 18 are stellar coronal X-ray sources (active binaries, ABs), the majority located along the binary sequence in a V-I colour-magnitude diagram and generally showing an H-alpha excess. In addition to confirming the previously detected cataclysmic variable (CV, CX4), we identify one confident new CV (CX76), and two candidates (CX81 and CX101). One MSP is known in M4 (CX12), and another strong candidate has been suggested (CX1); we identify some possible MSP candidates among optical and radio sources, such as VLA20, which appears to have a white dwarf counterpart. One X-ray source with a sub-subgiant optical counterpart and a flat radio spectrum (CX8, VLA31) is particularly mysterious. The radial distribution of X-ray sources suggests a relaxed population of average mass ~ 1.2 - 1.5 Msun. Comparing the numbers of ABs, MSPs, and CVs in M4 with other clusters indicates that AB numbers are proportional to cluster mass (primordial population), MSPs to stellar encounter rate (dynamically formed population), while CVs seem to be produced both primordially and dynamically.

astro-ph.HE↗

arXiv2023

Do Not Train It: A Linear Neural Architecture Search of Graph Neural Networks

Neural architecture search (NAS) for Graph neural networks (GNNs), called NAS-GNNs, has achieved significant performance over manually designed GNN architectures. However, these methods inherit issues from the conventional NAS methods, such as high computational cost and optimization difficulty. More importantly, previous NAS methods have ignored the uniqueness of GNNs, where GNNs possess expressive power without training. With the randomly-initialized weights, we can then seek the optimal architecture parameters via the sparse coding objective and derive a novel NAS-GNNs method, namely neural architecture coding (NAC). Consequently, our NAC holds a no-update scheme on GNNs and can efficiently compute in linear time. Empirical evaluations on multiple GNN benchmark datasets demonstrate that our approach leads to state-of-the-art performance, which is up to $200\times$ faster and $18.8\%$ more accurate than the strong baselines.

cs.LG↗

arXiv2023

Increasing stability estimates for the inverse potential scattering problems

This paper is mainly concerned with the inverse scattering problem of determining the unknown potential for the classical Schrödinger equation in two and three dimensions. We establish the increasing stability of the inverse scattering problem from either multi-frequency near-field data or multi-frequency far-field pattern. The stability estimate consists of the Lipschitz type data discrepancy and the logarithmic high frequency tail of the potential function, where the latter decreases as the upper bound of the frequency increases. A novel method is proposed for the proof, which is based on choosing appropriate incident plane waves and an application of the quantitative analytic continuation. A key ingredient in the analysis is employing scattering theory to obtain an analytic region and resolvent estimates in this region for the resolvent in two and three dimensions. We further apply this method to study the inverse scattering problem of determining both the magnetic potential and electric potential for the three-dimensional magnetic Schrödinger equation.

math.AP↗

arXiv2023

Phonon promoted charge density wave in topological kagome metal ScV$_{6}$Sn$_{6}$

Charge density wave (CDW) orders in vanadium-based kagome metals have recently received tremendous attention due to their unique properties and intricate interplay with exotic correlated phenomena, topological and symmetry-breaking states. However, the origin of the CDW order remains a topic of debate. The discovery of ScV$_{6}$Sn$_{6}$, a vanadium-based bilayer kagome metal exhibiting an in-plane $\sqrt{3}$ x $\sqrt{3} $ $\textit{R}$30$°$ CDW order with time-reversal symmetry breaking, provides a novel platform to explore the underlying mechanism behind the unconventional CDW. Here, we combine high-resolution angle-resolved photoemission spectroscopy, Raman scattering measurements and density functional theory to investigate the electronic structures and phonon modes of ScV$_{6}$Sn$_{6}$ and their evolution with temperature. We identify topologically nontrivial Dirac surface states and multiple van Hove singularities (VHSs) in the vicinity of the Fermi level, with one VHS near the K point exhibiting nesting wave vectors in proximity to the $\sqrt{3}$ x $\sqrt{3}$ $\textit{R}$30$°$ CDW wave vector. Additionally, Raman measurements indicate a strong intrinsic electron-phonon coupling in ScV$_{6}$Sn$_{6}$, as evidenced by the presence of a two-phonon mode and a large frequency amplitude mode. Our findings highlight the fundamental role of lattice degrees of freedom in promoting the CDW in ScV$_{6}$Sn$_{6}$ and provide important insights into the fascinating correlation phenomena observed in kagome metals.

cond-mat.str-el↗