arXiv subjects

Yuan Liu

Publications and source records attributed to Yuan Liu.

At least 307 records · Page 17Linked to original sources

arXiv2024

Explicitly Guided Information Interaction Network for Cross-modal Point Cloud Completion

In this paper, we explore a novel framework, EGIInet (Explicitly Guided Information Interaction Network), a model for View-guided Point cloud Completion (ViPC) task, which aims to restore a complete point cloud from a partial one with a single view image. In comparison with previous methods that relied on the global semantics of input images, EGIInet efficiently combines the information from two modalities by leveraging the geometric nature of the completion task. Specifically, we propose an explicitly guided information interaction strategy supported by modal alignment for point cloud completion. First, in contrast to previous methods which simply use 2D and 3D backbones to encode features respectively, we unified the encoding process to promote modal alignment. Second, we propose a novel explicitly guided information interaction strategy that could help the network identify critical information within images, thus achieving better guidance for completion. Extensive experiments demonstrate the effectiveness of our framework, and we achieved a new state-of-the-art (+16% CD over XMFnet) in benchmark datasets despite using fewer parameters than the previous methods. The pre-trained model and code and are available at https://github.com/WHU-USI3DV/EGIInet.

cs.CV↗

arXiv2024

TeethDreamer: 3D Teeth Reconstruction from Five Intra-oral Photographs

Orthodontic treatment usually requires regular face-to-face examinations to monitor dental conditions of the patients. When in-person diagnosis is not feasible, an alternative is to utilize five intra-oral photographs for remote dental monitoring. However, it lacks of 3D information, and how to reconstruct 3D dental models from such sparse view photographs is a challenging problem. In this study, we propose a 3D teeth reconstruction framework, named TeethDreamer, aiming to restore the shape and position of the upper and lower teeth. Given five intra-oral photographs, our approach first leverages a large diffusion model's prior knowledge to generate novel multi-view images with known poses to address sparse inputs and then reconstructs high-quality 3D teeth models by neural surface reconstruction. To ensure the 3D consistency across generated views, we integrate a 3D-aware feature attention mechanism in the reverse diffusion process. Moreover, a geometry-aware normal loss is incorporated into the teeth reconstruction process to enhance geometry accuracy. Extensive experiments demonstrate the superiority of our method over current state-of-the-arts, giving the potential to monitor orthodontic treatment remotely. Our code is available at https://github.com/ShanghaiTech-IMPACT/TeethDreamer

cs.CV↗

arXiv2024

X-ray Sources Classification Using Machine Learning: A Study with EP-WXT Pathfinder LEIA

X-ray observations play a crucial role in time-domain astronomy. The Einstein Probe (EP), a recently launched X-ray astronomical satellite, emerges as a forefront player in the field of time-domain astronomy and high-energy astrophysics. With a focus on systematic surveys in the soft X-ray band, EP aims to discover high-energy transients and monitor variable sources in the universe. To achieve these objectives, a quick and reliable classification of observed sources is essential. In this study, we developed a machine learning classifier for autonomous source classification using data from the EP-WXT Pathfinder Lobster Eye Imager for Astronomy (LEIA) and EP-WXT simulations. The proposed Random Forest classifier, built on selected features derived from light curves, energy spectra, and location information, achieves an accuracy of approximately 95% on EP simulation data and 98% on LEIA observational data. The classifier is integrated into the LEIA data processing pipeline, serving as a tool for manual validation and rapid classification during observations. This paper presents an efficient method for the classification of X-ray sources based on single observations, along with implications of most effective features for the task. This work facilitates rapid source classification for the EP mission and also provides valuable insights into feature selection and classification techniques for enhancing the efficiency and accuracy of X-ray source classification that can be adapted to other X-ray telescope data.

astro-ph.IM↗

arXiv2024

Single-shot Quantum Signal Processing Interferometry

Quantum systems of infinite dimension, such as bosonic oscillators, provide vast resources for quantum sensing. Yet, a general theory on how to manipulate such bosonic modes for sensing beyond parameter estimation is unknown. We present a general algorithmic framework, quantum signal processing interferometry (QSPI), for quantum sensing at the fundamental limits of quantum mechanics by generalizing Ramsey-type interferometry. Our QSPI sensing protocol relies on performing nonlinear polynomial transformations on the oscillator's quadrature operators by generalizing quantum signal processing (QSP) from qubits to hybrid qubit-oscillator systems. We use our QSPI sensing framework to make efficient binary decisions on a displacement channel in the single-shot limit. Theoretical analysis suggests the sensing accuracy, given a single-shot qubit measurement, scales inversely with the sensing time or circuit depth of the algorithm. We further concatenate a series of such binary decisions to perform parameter estimation in a bit-by-bit fashion. Numerical simulations are performed to support these statements. Our QSPI protocol offers a unified framework for quantum sensing using continuous-variable bosonic systems beyond parameter estimation and establishes a promising avenue toward efficient and scalable quantum control and quantum sensing schemes beyond the NISQ era.

quant-ph↗

arXiv2024

Technique Report of CVPR 2024 PBDL Challenges

The intersection of physics-based vision and deep learning presents an exciting frontier for advancing computer vision technologies. By leveraging the principles of physics to inform and enhance deep learning models, we can develop more robust and accurate vision systems. Physics-based vision aims to invert the processes to recover scene properties such as shape, reflectance, light distribution, and medium properties from images. In recent years, deep learning has shown promising improvements for various vision tasks, and when combined with physics-based vision, these approaches can enhance the robustness and accuracy of vision systems. This technical report summarizes the outcomes of the Physics-Based Vision Meets Deep Learning (PBDL) 2024 challenge, held in CVPR 2024 workshop. The challenge consisted of eight tracks, focusing on Low-Light Enhancement and Detection as well as High Dynamic Range (HDR) Imaging. This report details the objectives, methodologies, and results of each track, highlighting the top-performing solutions and their innovative approaches.

cs.CV↗

arXiv2024

Universal Hyperuniform Organization in Looped Leaf Vein Networks

Leaf vein network is a hierarchical vascular system that transports water and nutrients to the leaf cells. The thick primary veins form a branched network, while the secondary veins develop closed circuits forming a well-defined cellular structure. Through extensive analysis of a variety of distinct leaf species, we discover that the apparently disordered cellular structures of the secondary vein networks exhibit a universal hyperuniform organization and possess a hidden order on large scales. Disorder hyperuniform (DHU) systems lack conventional long-range order, yet they completely suppress normalized large-scale density fluctuations like crystals. Specifically, we find that the distributions of the geometric centers associated with the vein network loops possess a vanishing static structure factor in the zero-wavenumber limit, i.e., $S(k) \sim k^α$, where $α\approx 0.64$, providing an example of class III hyperuniformity in biology. This hyperuniform organization leads to superior efficiency of diffusive transport, as evidenced by the much faster convergence of the time-dependent spreadability $\mathcal{S}(t)$ to its long-time asymptotic limit, compared to that of other uncorrelated or correlated disordered but non-hyperuniform organizations. Our results also have implications for the discovery and design of novel disordered network materials with optimal transport properties.

cond-mat.soft↗

arXiv2024

Coverage Axis++: Efficient Inner Point Selection for 3D Shape Skeletonization

We introduce Coverage Axis++, a novel and efficient approach to 3D shape skeletonization. The current state-of-the-art approaches for this task often rely on the watertightness of the input or suffer from substantial computational costs, thereby limiting their practicality. To address this challenge, Coverage Axis++ proposes a heuristic algorithm to select skeletal points, offering a high-accuracy approximation of the Medial Axis Transform (MAT) while significantly mitigating computational intensity for various shape representations. We introduce a simple yet effective strategy that considers shape coverage, uniformity, and centrality to derive skeletal points. The selection procedure enforces consistency with the shape structure while favoring the dominant medial balls, which thus introduces a compact underlying shape representation in terms of MAT. As a result, Coverage Axis++ allows for skeletonization for various shape representations (e.g., water-tight meshes, triangle soups, point clouds), specification of the number of skeletal points, few hyperparameters, and highly efficient computation with improved reconstruction accuracy. Extensive experiments across a wide range of 3D shapes validate the efficiency and effectiveness of Coverage Axis++. Our codes are available at https://github.com/Frank-ZY-Dou/Coverage_Axis.

cs.CV↗

arXiv2024

Part123: Part-aware 3D Reconstruction from a Single-view Image

Recently, the emergence of diffusion models has opened up new opportunities for single-view reconstruction. However, all the existing methods represent the target object as a closed mesh devoid of any structural information, thus neglecting the part-based structure, which is crucial for many downstream applications, of the reconstructed shape. Moreover, the generated meshes usually suffer from large noises, unsmooth surfaces, and blurry textures, making it challenging to obtain satisfactory part segments using 3D segmentation techniques. In this paper, we present Part123, a novel framework for part-aware 3D reconstruction from a single-view image. We first use diffusion models to generate multiview-consistent images from a given image, and then leverage Segment Anything Model (SAM), which demonstrates powerful generalization ability on arbitrary objects, to generate multiview segmentation masks. To effectively incorporate 2D part-based information into 3D reconstruction and handle inconsistency, we introduce contrastive learning into a neural rendering framework to learn a part-aware feature space based on the multiview segmentation masks. A clustering-based algorithm is also developed to automatically derive 3D part segmentation results from the reconstructed models. Experiments show that our method can generate 3D models with high-quality segmented parts on various objects. Compared to existing unstructured reconstruction methods, the part-aware 3D models from our method benefit some important applications, including feature-preserving reconstruction, primitive fitting, and 3D shape editing.

cs.GR↗

arXiv2024

DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models

2D diffusion model, which often contains unwanted baked-in shading effects and results in unrealistic rendering effects in the downstream applications. Generating Physically Based Rendering (PBR) materials instead of just RGB textures would be a promising solution. However, directly distilling the PBR material parameters from 2D diffusion models still suffers from incorrect material decomposition, such as baked-in shading effects in albedo. We introduce DreamMat, an innovative approach to resolve the aforementioned problem, to generate high-quality PBR materials from text descriptions. We find out that the main reason for the incorrect material distillation is that large-scale 2D diffusion models are only trained to generate final shading colors, resulting in insufficient constraints on material decomposition during distillation. To tackle this problem, we first finetune a new light-aware 2D diffusion model to condition on a given lighting environment and generate the shading results on this specific lighting condition. Then, by applying the same environment lights in the material distillation, DreamMat can generate high-quality PBR materials that are not only consistent with the given geometry but also free from any baked-in shading effects in albedo. Extensive experiments demonstrate that the materials produced through our methods exhibit greater visual appeal to users and achieve significantly superior rendering quality compared to baseline methods, which are preferable for downstream tasks such as game and film production.

cs.GR↗

arXiv2024

Rethinking Overlooked Aspects in Vision-Language Models

Recent advancements in large vision-language models (LVLMs), such as GPT4-V and LLaVA, have been substantial. LLaVA's modular architecture, in particular, offers a blend of simplicity and efficiency. Recent works mainly focus on introducing more pre-training and instruction tuning data to improve model's performance. This paper delves into the often-neglected aspects of data efficiency during pre-training and the selection process for instruction tuning datasets. Our research indicates that merely increasing the size of pre-training data does not guarantee improved performance and may, in fact, lead to its degradation. Furthermore, we have established a pipeline to pinpoint the most efficient instruction tuning (SFT) dataset, implying that not all SFT data utilized in existing studies are necessary. The primary objective of this paper is not to introduce a state-of-the-art model, but rather to serve as a roadmap for future research, aiming to optimize data usage during pre-training and fine-tuning processes to enhance the performance of vision-language models.

cs.CV↗

arXiv2024

Application of a Dense Fusion Attention Network in Fault Diagnosis of Centrifugal Fan

Although the deep learning recognition model has been widely used in the condition monitoring of rotating machinery. However, it is still a challenge to understand the correspondence between the structure and function of the model and the diagnosis process. Therefore, this paper discusses embedding distributed attention modules into dense connections instead of traditional dense cascading operations. It not only decouples the influence of space and channel on fault feature adaptive recalibration feature weights, but also forms a fusion attention function. The proposed dense fusion focuses on the visualization of the network diagnosis process, which increases the interpretability of model diagnosis. How to continuously and effectively integrate different functions to enhance the ability to extract fault features and the ability to resist noise is answered. Centrifugal fan fault data is used to verify this network. Experimental results show that the network has stronger diagnostic performance than other advanced fault diagnostic models.

cs.LG↗

arXiv2024

The fast X-ray transient EP240315a: a z ~ 5 gamma-ray burst in a Lyman continuum leaking galaxy

The nature of the minute-to-hour long Fast X-ray Transients (FXTs) localised by telescopes such as Chandra, Swift, and XMM-Newton remains mysterious, with numerous models suggested for the events. Here, we report multi-wavelength observations of EP240315a, a 1600 s long transient detected by the Einstein Probe, showing it to have a redshift of z=4.859. We measure a low column density of neutral hydrogen, indicating that the event is embedded in a low-density environment, further supported by direct detection of leaking ionising Lyman-continuum. The observed properties are consistent with EP240315a being a long-duration gamma-ray burst, and these observations support an interpretation in which a significant fraction of the FXT population are lower-luminosity examples of similar events. Such transients are detectable at high redshifts by the Einstein Probe and, in the (near) future, out to even larger distances by SVOM, THESEUS, and Athena, providing samples of events into the epoch of reionisation.

astro-ph.HE↗

arXiv2024

IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination

This paper aims to recover object materials from posed images captured under an unknown static lighting condition. Recent methods solve this task by optimizing material parameters through differentiable physically based rendering. However, due to the coupling between object geometry, materials, and environment lighting, there is inherent ambiguity during the inverse rendering process, preventing previous methods from obtaining accurate results. To overcome this ill-posed problem, our key idea is to learn the material prior with a generative model for regularizing the optimization process. We observe that the general rendering equation can be split into diffuse and specular shading terms, and thus formulate the material prior as diffusion models of albedo and specular. Thanks to this design, our model can be trained using the existing abundant 3D object data, and naturally acts as a versatile tool to resolve the ambiguity when recovering material representations from RGB images. In addition, we develop a coarse-to-fine training strategy that leverages estimated materials to guide diffusion models to satisfy multi-view consistent constraints, leading to more stable and accurate results. Extensive experiments on real-world and synthetic datasets demonstrate that our approach achieves state-of-the-art performance on material recovery. The code will be available at https://zju3dv.github.io/IntrinsicAnything.

cs.CV↗

arXiv2024

SyncDreamer: Generating Multiview-consistent Images from a Single-view Image

In this paper, we present a novel diffusion model called that generates multiview-consistent images from a single-view image. Using pretrained large-scale 2D diffusion models, recent work Zero123 demonstrates the ability to generate plausible novel views from a single-view image of an object. However, maintaining consistency in geometry and colors for the generated images remains a challenge. To address this issue, we propose a synchronized multiview diffusion model that models the joint probability distribution of multiview images, enabling the generation of multiview-consistent images in a single reverse process. SyncDreamer synchronizes the intermediate states of all the generated images at every step of the reverse process through a 3D-aware feature attention mechanism that correlates the corresponding features across different views. Experiments show that SyncDreamer generates images with high consistency across different views, thus making it well-suited for various 3D generation tasks such as novel-view-synthesis, text-to-3D, and image-to-3D.

cs.CV↗

arXiv2024

FreeReg: Image-to-Point Cloud Registration Leveraging Pretrained Diffusion Models and Monocular Depth Estimators

Matching cross-modality features between images and point clouds is a fundamental problem for image-to-point cloud registration. However, due to the modality difference between images and points, it is difficult to learn robust and discriminative cross-modality features by existing metric learning methods for feature matching. Instead of applying metric learning on cross-modality data, we propose to unify the modality between images and point clouds by pretrained large-scale models first, and then establish robust correspondence within the same modality. We show that the intermediate features, called diffusion features, extracted by depth-to-image diffusion models are semantically consistent between images and point clouds, which enables the building of coarse but robust cross-modality correspondences. We further extract geometric features on depth maps produced by the monocular depth estimator. By matching such geometric features, we significantly improve the accuracy of the coarse correspondences produced by diffusion features. Extensive experiments demonstrate that without any task-specific training, direct utilization of both features produces accurate image-to-point cloud registration. On three public indoor and outdoor benchmarks, the proposed method averagely achieves a 20.6 percent improvement in Inlier Ratio, a three-fold higher Inlier Number, and a 48.6 percent improvement in Registration Recall than existing state-of-the-arts.

cs.CV↗

arXiv2024

An Evolutionary Network Architecture Search Framework with Adaptive Multimodal Fusion for Hand Gesture Recognition

Hand gesture recognition (HGR) based on multimodal data has attracted considerable attention owing to its great potential in applications. Various manually designed multimodal deep networks have performed well in multimodal HGR (MHGR), but most of existing algorithms require a lot of expert experience and time-consuming manual trials. To address these issues, we propose an evolutionary network architecture search framework with the adaptive multimodel fusion (AMF-ENAS). Specifically, we design an encoding space that simultaneously considers fusion positions and ratios of the multimodal data, allowing for the automatic construction of multimodal networks with different architectures through decoding. Additionally, we consider three input streams corresponding to intra-modal surface electromyography (sEMG), intra-modal accelerometer (ACC), and inter-modal sEMG-ACC. To automatically adapt to various datasets, the ENAS framework is designed to automatically search a MHGR network with appropriate fusion positions and ratios. To the best of our knowledge, this is the first time that ENAS has been utilized in MHGR to tackle issues related to the fusion position and ratio of multimodal data. Experimental results demonstrate that AMF-ENAS achieves state-of-the-art performance on the Ninapro DB2, DB3, and DB7 datasets.

cs.CV↗

arXiv2024

High-throughput measurement of elastic moduli of microfibers by rope coiling

There are many fields where it is of interest to measure the elastic moduli of tiny fragile fibers, such as filamentous bacteria, actin filaments, DNA, carbon nanotubes, and functional microfibers. The elastic modulus is typically deduced from a sophisticated tensile test under a microscope, but the throughput is low and limited by the time-consuming and skill-intensive sample loading/unloading. Here, we demonstrate a simple microfluidic method enabling the high-throughput measurement of the elastic moduli of microfibers by rope coiling using a localized compression, where sample loading/unloading are not needed between consecutive measurements. The rope coiling phenomenon occurs spontaneously when a microfiber flows from a small channel into a wide channel. The elastic modulus is determined by measuring either the buckling length or the coiling radius. The throughput of this method, currently 3,300 fibers per hour, is a thousand times higher than that of a tensile tester. We demonstrate the feasibility of the method by testing a nonuniform fiber with axially varying elastic modulus. We also demonstrate its capability for in situ inline measurement in a microfluidic production line. We envisage that high-throughput measurements may facilitate potential applications such as screening or sorting by mechanical properties and real-time control during production of microfibers.

cond-mat.soft↗

arXiv2024

Multiple Population Alternate Evolution Neural Architecture Search

The effectiveness of Evolutionary Neural Architecture Search (ENAS) is influenced by the design of the search space. Nevertheless, common methods including the global search space, scalable search space and hierarchical search space have certain limitations. Specifically, the global search space requires a significant amount of computational resources and time, the scalable search space sacrifices the diversity of network structures and the hierarchical search space increases the search cost in exchange for network diversity. To address above limitation, we propose a novel paradigm of searching neural network architectures and design the Multiple Population Alternate Evolution Neural Architecture Search (MPAE), which can achieve module diversity with a smaller search cost. MPAE converts the search space into L interconnected units and sequentially searches the units, then the above search of the entire network be cycled several times to reduce the impact of previous units on subsequent units. To accelerate the population evolution process, we also propose the the population migration mechanism establishes an excellent migration archive and transfers the excellent knowledge and experience in the migration archive to new populations. The proposed method requires only 0.3 GPU days to search a neural network on the CIFAR dataset and achieves the state-of-the-art results.

cs.NE↗