arXiv subjects

Hao Zheng

Publications and source records attributed to Hao Zheng.

At least 163 records · Page 9Linked to original sources

arXiv2023

Caption Anything: Interactive Image Description with Diverse Multimodal Controls

Controllable image captioning is an emerging multimodal topic that aims to describe the image with natural language following human purpose, $\textit{e.g.}$, looking at the specified regions or telling in a particular text style. State-of-the-art methods are trained on annotated pairs of input controls and output captions. However, the scarcity of such well-annotated multimodal data largely limits their usability and scalability for interactive AI systems. Leveraging unimodal instruction-following foundation models is a promising alternative that benefits from broader sources of data. In this paper, we present Caption AnyThing (CAT), a foundation model augmented image captioning framework supporting a wide range of multimodel controls: 1) visual controls, including points, boxes, and trajectories; 2) language controls, such as sentiment, length, language, and factuality. Powered by Segment Anything Model (SAM) and ChatGPT, we unify the visual and language prompts into a modularized framework, enabling the flexible combination between different controls. Extensive case studies demonstrate the user intention alignment capabilities of our framework, shedding light on effective user interaction modeling in vision-language applications. Our code is publicly available at https://github.com/ttengwang/Caption-Anything.

cs.CV↗

arXiv2023

Heterogeneous field response of hierarchical polar laminates in relaxor ferroelectrics

Relaxor ferroelectrics are a class of materials that are widely perceived as deriving their exotic properties from structural heterogeneities. Understanding the microscopic origin of the superior electromechanical response requires knowledge not only concerning the formation of polar nanodomains (PNDs) built from individual atoms but more importantly the spatial distribution of PNDs over longer distances. The mesoscale PND arrangement is shaped by the interactions between these domains and, in turn, dictates the electric-field driven PND response directly relevant to the macroscopic material properties. Here, we show the emergence of mesoscale lattice order that we name "polar laminates" in the canonical relaxor ferroelectric 0.68PbMg$_{1/3}$Nb$_{2/3}$O$_{3}$-0.32PbTiO$_{3}$ (PMN-0.32PT) using X-ray coherent nano-diffraction. These laminates are nematic with a size of ~350 nm and arise from the staggered arrangement of ~13 nm monoclinic PNDs along the <110> of the pseudocubic lattice. The spatial distribution of c-axis strain is directly correlated with the tilting of the PNDs and is most prominent between the laminates. Further operando nano-diffraction studies demonstrate heterogeneous electric-field-driven responses. The most active regions tend to reside inside the laminates while the spatial pinning centers are between the laminates. This observation reveals the hierarchical assembly of lattice order as a novel form of electron and lattice self-organization in heterogenous materials and establishes the role of such mesoscale spatial arrangement in connecting the nanoscale heterogeneity and macroscopic material properties. These findings provide a guiding principle for the design and optimization of future relaxors and may shed light on the existence of similar behavior in a wide range of quantum and functional materials.

cond-mat.mtrl-sci↗

arXiv2023

Multi-site, Multi-domain Airway Tree Modeling (ATM'22): A Public Benchmark for Pulmonary Airway Segmentation

Open international challenges are becoming the de facto standard for assessing computer vision and image analysis algorithms. In recent years, new methods have extended the reach of pulmonary airway segmentation that is closer to the limit of image resolution. Since EXACT'09 pulmonary airway segmentation, limited effort has been directed to quantitative comparison of newly emerged algorithms driven by the maturity of deep learning based approaches and clinical drive for resolving finer details of distal airways for early intervention of pulmonary diseases. Thus far, public annotated datasets are extremely limited, hindering the development of data-driven methods and detailed performance evaluation of new algorithms. To provide a benchmark for the medical imaging community, we organized the Multi-site, Multi-domain Airway Tree Modeling (ATM'22), which was held as an official challenge event during the MICCAI 2022 conference. ATM'22 provides large-scale CT scans with detailed pulmonary airway annotation, including 500 CT scans (300 for training, 50 for validation, and 150 for testing). The dataset was collected from different sites and it further included a portion of noisy COVID-19 CTs with ground-glass opacity and consolidation. Twenty-three teams participated in the entire phase of the challenge and the algorithms for the top ten teams are reviewed in this paper. Quantitative and qualitative results revealed that deep learning models embedded with the topological continuity enhancement achieved superior performance in general. ATM'22 challenge holds as an open-call design, the training data and the gold standard evaluation are available upon successful registration via its homepage.

eess.IV↗

arXiv2023

Few-Shot Learning with Visual Distribution Calibration and Cross-Modal Distribution Alignment

Pre-trained vision-language models have inspired much research on few-shot learning. However, with only a few training images, there exist two crucial problems: (1) the visual feature distributions are easily distracted by class-irrelevant information in images, and (2) the alignment between the visual and language feature distributions is difficult. To deal with the distraction problem, we propose a Selective Attack module, which consists of trainable adapters that generate spatial attention maps of images to guide the attacks on class-irrelevant image areas. By messing up these areas, the critical features are captured and the visual distributions of image features are calibrated. To better align the visual and language feature distributions that describe the same object class, we propose a cross-modal distribution alignment module, in which we introduce a vision-language prototype for each class to align the distributions, and adopt the Earth Mover's Distance (EMD) to optimize the prototypes. For efficient computation, the upper bound of EMD is derived. In addition, we propose an augmentation strategy to increase the diversity of the images and the text prompts, which can reduce overfitting to the few-shot training images. Extensive experiments on 11 datasets demonstrate that our method consistently outperforms prior arts in few-shot learning. The implementation code will be available at https://github.com/bhrqw/SADA.

cs.CV↗

arXiv2023

CageViT: Convolutional Activation Guided Efficient Vision Transformer

Recently, Transformers have emerged as the go-to architecture for both vision and language modeling tasks, but their computational efficiency is limited by the length of the input sequence. To address this, several efficient variants of Transformers have been proposed to accelerate computation or reduce memory consumption while preserving performance. This paper presents an efficient vision Transformer, called CageViT, that is guided by convolutional activation to reduce computation. Our CageViT, unlike current Transformers, utilizes a new encoder to handle the rearranged tokens, bringing several technical contributions: 1) Convolutional activation is used to pre-process the token after patchifying the image to select and rearrange the major tokens and minor tokens, which substantially reduces the computation cost through an additional fusion layer. 2) Instead of using the class activation map of the convolutional model directly, we design a new weighted class activation to lower the model requirements. 3) To facilitate communication between major tokens and fusion tokens, Gated Linear SRA is proposed to further integrate fusion tokens into the attention mechanism. We perform a comprehensive validation of CageViT on the image classification challenge. Experimental results demonstrate that the proposed CageViT outperforms the most recent state-of-the-art backbones by a large margin in terms of efficiency, while maintaining a comparable level of accuracy (e.g. a moderate-sized 43.35M model trained solely on 224 x 224 ImageNet-1K can achieve Top-1 accuracy of 83.4% accuracy).

cs.CV↗

arXiv2023

SurfNN: Joint Reconstruction of Multiple Cortical Surfaces from Magnetic Resonance Images

To achieve fast, robust, and accurate reconstruction of the human cortical surfaces from 3D magnetic resonance images (MRIs), we develop a novel deep learning-based framework, referred to as SurfNN, to reconstruct simultaneously both inner (between white matter and gray matter) and outer (pial) surfaces from MRIs. Different from existing deep learning-based cortical surface reconstruction methods that either reconstruct the cortical surfaces separately or neglect the interdependence between the inner and outer surfaces, SurfNN reconstructs both the inner and outer cortical surfaces jointly by training a single network to predict a midthickness surface that lies at the center of the inner and outer cortical surfaces. The input of SurfNN consists of a 3D MRI and an initialization of the midthickness surface that is represented both implicitly as a 3D distance map and explicitly as a triangular mesh with spherical topology, and its output includes both the inner and outer cortical surfaces, as well as the midthickness surface. The method has been evaluated on a large-scale MRI dataset and demonstrated competitive cortical surface reconstruction performance.

eess.IV↗

arXiv2023

Categorical computation

In quantum computing, the computation is achieved by linear operators in or between Hilbert spaces. In this work, we explore a new computation scheme, in which the linear operators in quantum computing are replaced by (higher) functors between two (higher) categories. If from Turing computing to quantum computing is the first quantization of computation, then this new scheme can be viewed as the second quantization of computation. The fundamental problem in realizing this idea is how to realize a (higher) functor physically. We provide a theoretical idea of realizing (higher) functors physically based on the physics of topological orders.

quant-ph↗

arXiv2023

TAPS: Topology-Aware Intra-Operator Parallelism Strategy Searching Algorithm for Deep Neural Networks

TAPS is a Topology-Aware intra-operator Parallelism strategy Searching algorithm that generates intra-operator parallelism strategies by considering both intra-node and inter-node bandwidth. Most of the existing auto-parallelism works use the communication volume as the communication cost directly when generating strategies, which we prove to be sub-optimal in multi-nodes cases. We design a topology-aware cost model for multi-node intra-operator parallelism strategy searching. Numerical experiments demonstrate that TAPS can generate strategies with up to 85% fewer communication costs, which outperform the latest baselines.

cs.DC↗

arXiv2022

Real-World Compositional Generalization with Disentangled Sequence-to-Sequence Learning

Compositional generalization is a basic mechanism in human language learning, which current neural networks struggle with. A recently proposed Disentangled sequence-to-sequence model (Dangle) shows promising generalization capability by learning specialized encodings for each decoding step. We introduce two key modifications to this model which encourage more disentangled representations and improve its compute and memory efficiency, allowing us to tackle compositional generalization in a more realistic setting. Specifically, instead of adaptively re-encoding source keys and values at each time step, we disentangle their representations and only re-encode keys periodically, at some interval. Our new architecture leads to better generalization performance across existing tasks and datasets, and a new machine translation benchmark which we create by detecting naturally occurring compositional patterns in relation to a training set. We show this methodology better emulates real-world requirements than artificial challenges.

cs.CL↗

arXiv2022

Keep Your Friends Close & Enemies Farther: Debiasing Contrastive Learning with Spatial Priors in 3D Radiology Images

Understanding of spatial attributes is central to effective 3D radiology image analysis where crop-based learning is the de facto standard. Given an image patch, its core spatial properties (e.g., position & orientation) provide helpful priors on expected object sizes, appearances, and structures through inherent anatomical consistencies. Spatial correspondences, in particular, can effectively gauge semantic similarities between inter-image regions, while their approximate extraction requires no annotations or overbearing computational costs. However, recent 3D contrastive learning approaches either neglect correspondences or fail to maximally capitalize on them. To this end, we propose an extensible 3D contrastive framework (Spade, for Spatial Debiasing) that leverages extracted correspondences to select more effective positive & negative samples for representation learning. Our method learns both globally invariant and locally equivariant representations with downstream segmentation in mind. We also propose separate selection strategies for global & local scopes that tailor to their respective representational requirements. Compared to recent state-of-the-art approaches, Spade shows notable improvements on three downstream segmentation tasks (CT Abdominal Organ, CT Heart, MR Heart).

cs.CV↗

arXiv2022

A Point in the Right Direction: Vector Prediction for Spatially-aware Self-supervised Volumetric Representation Learning

High annotation costs and limited labels for dense 3D medical imaging tasks have recently motivated an assortment of 3D self-supervised pretraining methods that improve transfer learning performance. However, these methods commonly lack spatial awareness despite its centrality in enabling effective 3D image analysis. More specifically, position, scale, and orientation are not only informative but also automatically available when generating image crops for training. Yet, to date, no work has proposed a pretext task that distills all key spatial features. To fulfill this need, we develop a new self-supervised method, VectorPOSE, which promotes better spatial understanding with two novel pretext tasks: Vector Prediction (VP) and Boundary-Focused Reconstruction (BFR). VP focuses on global spatial concepts (i.e., properties of 3D patches) while BFR addresses weaknesses of recent reconstruction methods to learn more effective local representations. We evaluate VectorPOSE on three 3D medical image segmentation tasks, showing that it often outperforms state-of-the-art methods, especially in limited annotation settings.

cs.CV↗

arXiv2022

Dance of SNN and ANN: Solving binding problem by combining spike timing and reconstructive attention

The binding problem is one of the fundamental challenges that prevent the artificial neural network (ANNs) from a compositional understanding of the world like human perception, because disentangled and distributed representations of generative factors can interfere and lead to ambiguity when complex data with multiple objects are presented. In this paper, we propose a brain-inspired hybrid neural network (HNN) that introduces temporal binding theory originated from neuroscience into ANNs by integrating spike timing dynamics (via spiking neural networks, SNNs) with reconstructive attention (by ANNs). Spike timing provides an additional dimension for grouping, while reconstructive feedback coordinates the spikes into temporal coherent states. Through iterative interaction of ANN and SNN, the model continuously binds multiple objects at alternative synchronous firing times in the SNN coding space. The effectiveness of the model is evaluated on synthetic datasets of binary images. By visualization and analysis, we demonstrate that the binding is explainable, soft, flexible, and hierarchical. Notably, the model is trained on single object datasets without explicit supervision on grouping, but successfully binds multiple objects on test datasets, showing its compositional generalization capability. Further results show its binding ability in dynamic situations.

cs.AI↗

arXiv2022

Quasiparticle characteristics of the weakly ferromagnetic Hund's metal MnSi

Hund's metals are multi-orbital systems with $3d$ or $4d$ electrons exhibiting both itinerant character and local moments, and they feature Kondo-like screenings of local orbital and spin moments, with suppressed coherence temperature driven by Hund's coupling $J_H$. They often exhibit magnetic order at low temperature, but how the interaction between the Kondo-like screening and long-range magnetic order is manifested in the quasiparticle spectrum remains an open question. Here we present spectroscopic signature of such interaction in a Hund's metal candidate MnSi exhibiting weak ferromagnetism. Our photoemission measurements reveal renormalized quasiparticle bands near the Fermi level with strong momentum dependence: the ferromagnetism manifests through possibly exchange-split bands (Q1) below $T_C$ , while the spin/orbital screenings lead to gradual development of quasiparticles (Q2) upon cooling. Our results demonstrate how the characteristic spin/orbital coherence in a Hund's metal could coexist and compete with the magnetic order to form a weak itinerant ferromagnet, via quasiparticle bands that are well separated in momentum space and exhibit distinct temperature dependence. Our results imply that the competition between the spin/orbital screening and the magnetic order in a Hund's metal bears intriguing similarity to the Kondo lattice systems.

cond-mat.str-el↗

arXiv2022

Revising the Hubble constant, spatial curvature and dark energy dynamics with the latest observations of quasars

In this paper, we use a newly compiled sample of ultra-compact structure in radio quasars and strong gravitational lensing systems with quasars acting as background sources to constrain six spatially flat and non-flat cosmological models ($Λ$CDM, PEDE and DGP). These two sets of quasar data (the time-delay measurements of six strong lensing systems and 120 intermediate-luminosity quasars calibrated as standard rulers) could break the degeneracy between cosmological parameters ($H_0$, $Ω_m$ and $Ω_k$) and therefore provide more stringent cosmological constraints for the six cosmological models we study. A joint analysis of the quasar sample provides model-independent estimations of the Hubble constant $H_0$, which is strongly consistent with that derived from the local distance ladder by SH0ES collaboration in the $Λ$CDM and PEDE model. However, in the framework of a DGP cosmology (especially for the flat universe), the measured Hubble constant is in good agreement with that derived from the the recent Planck 2018 results. In addition, our results show that zero spatial curvature is supported by the current lensed and unlensed quasar observations and there is no significant deviation from a flat universe. For most of cosmological model we study (the flat $Λ$CDM, non-flat $Λ$CDM, flat PEDE, and non-flat PEDE models), the derived matter density parameter is completely consistent with $Ω_m\sim 0.30$ in all the data sets, as expected by the latest cosmological observations. Finally, according to the the statistical criteria DIC, although the joint constraints provide substantial observational support to the flat PEDE model, they do not rule out dark energy being a cosmological constant and non-flat spatial hypersurfaces.

astro-ph.CO↗

arXiv2022

Mining SoC Message Flows with Attention Model

High-quality system-level message flow specifications are necessary for comprehensive validation of system-on-chip (SoC) designs. However, manual development and maintenance of such specifications are daunting tasks. We propose a disruptive method that utilizes deep sequence modeling with the attention mechanism to infer accurate flow specifications from SoC communication traces. The proposed method can overcome the inherent complexity of SoC traces induced by the concurrent executions of SoC designs that existing mining tools often find extremely challenging. We conduct experiments on five highly concurrent traces and find that the proposed approach outperforms several existing state-of-the-art trace mining tools.

cs.AI↗

arXiv2022

Observation of magnetism induced topological edge state in antiferromagnetic topological insulator MnBi4Te7

Breaking time reversal symmetry in a topological insulator may lead to quantum anomalous Hall effect and axion insulator phase. MnBi4Te7 is a recently discovered antiferromagnetic topological insulator with TN ~12.5 K, which is constituted of alternatively stacked magnetic layer (MnBi2Te4) and non-magnetic layer (Bi2Te3). By means of scanning tunneling spectroscopy, we clearly observe the electronic state present at a step edge of a magnetic MnBi2Te4 layer but absent at non-magnetic Bi2Te3 layers at 4.5 K. Furthermore, we find that as the temperature rises above TN, the edge state vanishes, while the point defect induced state persists upon temperature increasing. These results confirm the observation of magnetism induced edge states. Our analysis based on an axion insulator theory reveals that the nontrivial topological nature of the observed edge state.

cond-mat.mtrl-sci↗

arXiv2022

Quantum phase transition in magnetic nanographenes on a lead superconductor

Quantum spins, referred to the spin operator preserved by full SU(2) symmetry in the absence of the magnetic anistropy, have been proposed to host exotic interactions with superconductivity4. However, spin orbit coupling and crystal field splitting normally cause a significant magnetic anisotropy for d/f-shell spins on surfaces6,9, breaking SU(2) symmetry and fabricating the spins with Ising properties10. Recently, magnetic nanographenes have been proven to host intrinsic quantum magnetism due to their negligible spin orbital coupling and crystal field splitting. Here, we fabricate three atomically precise nanographenes with the same magnetic ground state of spin S=1/2 on Pb(111) through engineering sublattice imbalance in graphene honeycomb lattice. Scanning tunneling spectroscopy reveals the coexistence of magnetic bound states and Kondo screening in such hybridized system. Through engineering the magnetic exchange strength between the unpaired spin in nanographenes and cooper pairs, quantum phase transition from the singlet to the doublet state has been observed, in consistent with quantum models of spins on superconductors. Our work demonstrates delocalized graphene magnetism host highly tunable magnetic bound states with cooper pairs, which can be further developed to study the Majorana bound states and other rich quantum physics of low-dimensional quantum spins on superconductors.

cond-mat.mes-hall↗

arXiv2022

Counterexample Generation for Infinite-State Chemical Reaction Networks

Counterexample generation is an indispensable part of model checking process. In stochastic model checking, counterexample generation is a challenging problem as it is not enough to find a single trace that violates the given property. Instead, a potentially large set of traces with enough probability to violate the property needs to be found. This paper considers counterexample generation for chemical reaction network (CRN) models with potentially infinite state space. A method based on bounded model checking using SMT solving is developed for counterexample generation for CRNs. It intends to find a small set of property violating paths of a given model such that they collectively have a total probability that is above a given threshold. A unique challenge is due to the highly connected state space of CRNs where a counterexample is only a tiny subset of all property violating paths. To address such challenges, this paper presents a number of optimizations including a divide-and-conquer technique to scale up the counterexample generation method for large CRN models. This paper reports results from experiments on a number of infinite-state CRN models.

cs.LO↗