arXiv Science⌕ Search

arXiv subjects

Liang Zheng

Publications and source records attributed to Liang Zheng.

At least 91 records · Page 5Linked to original sources
arXiv2023

SmartMask: Context Aware High-Fidelity Mask Generation for Fine-grained Object Insertion and Layout Control

The field of generative image inpainting and object insertion has made significant progress with the recent advent of latent diffusion models. Utilizing a precise object mask can greatly enhance these applications. However, due to the challenges users encounter in creating high-fidelity masks, there is a tendency for these methods to rely on more coarse masks (e.g., bounding box) for these applications. This results in limited control and compromised background content preservation. To overcome these limitations, we introduce SmartMask, which allows any novice user to create detailed masks for precise object insertion. Combined with a ControlNet-Inpaint model, our experiments demonstrate that SmartMask achieves superior object insertion quality, preserving the background content more effectively than previous methods. Notably, unlike prior works the proposed approach can also be used even without user-mask guidance, which allows it to perform mask-free object insertion at diverse positions and scales. Furthermore, we find that when used iteratively with a novel instruction-tuning based planning model, SmartMask can be used to design detailed layouts from scratch. As compared with user-scribble based layout design, we observe that SmartMask allows for better quality outputs with layout-to-image generation methods. Project page is available at https://smartmask-gen.github.io

cs.CV↗
arXiv2023

Divide, Evaluate, and Refine: Evaluating and Improving Text-to-Image Alignment with Iterative VQA Feedback

The field of text-conditioned image generation has made unparalleled progress with the recent advent of latent diffusion models. While remarkable, as the complexity of given text input increases, the state-of-the-art diffusion models may still fail in generating images which accurately convey the semantics of the given prompt. Furthermore, it has been observed that such misalignments are often left undetected by pretrained multi-modal models such as CLIP. To address these problems, in this paper we explore a simple yet effective decompositional approach towards both evaluation and improvement of text-to-image alignment. In particular, we first introduce a Decompositional-Alignment-Score which given a complex prompt decomposes it into a set of disjoint assertions. The alignment of each assertion with generated images is then measured using a VQA model. Finally, alignment scores for different assertions are combined aposteriori to give the final text-to-image alignment score. Experimental analysis reveals that the proposed alignment metric shows significantly higher correlation with human ratings as opposed to traditional CLIP, BLIP scores. Furthermore, we also find that the assertion level alignment scores provide a useful feedback which can then be used in a simple iterative procedure to gradually increase the expression of different assertions in the final image outputs. Human user studies indicate that the proposed approach surpasses previous state-of-the-art by 8.7% in overall text-to-image alignment accuracy. Project page for our paper is available at https://1jsingh.github.io/divide-evaluate-and-refine

cs.CV↗
arXiv2023

Optimizing Camera Configurations for Multi-View Pedestrian Detection

Jointly considering multiple camera views (multi-view) is very effective for pedestrian detection under occlusion. For such multi-view systems, it is critical to have well-designed camera configurations, including camera locations, directions, and fields-of-view (FoVs). Usually, these configurations are crafted based on human experience or heuristics. In this work, we present a novel solution that features a transformer-based camera configuration generator. Using reinforcement learning, this generator autonomously explores vast combinations within the action space and searches for configurations that give the highest detection accuracy according to the training dataset. The generator learns advanced techniques like maximizing coverage, minimizing occlusion, and promoting collaboration. Across multiple simulation scenarios, the configurations generated by our transformer-based model consistently outperform random search, heuristic-based methods, and configurations designed by human experts, shedding light on future camera layout optimization.

cs.CV↗
arXiv2023

Attribute Descent: Simulating Object-Centric Datasets on the Content Level and Beyond

This article aims to use graphic engines to simulate a large number of training data that have free annotations and possibly strongly resemble to real-world data. Between synthetic and real, a two-level domain gap exists, involving content level and appearance level. While the latter is concerned with appearance style, the former problem arises from a different mechanism, i.e, content mismatch in attributes such as camera viewpoint, object placement and lighting conditions. In contrast to the widely-studied appearance-level gap, the content-level discrepancy has not been broadly studied. To address the content-level misalignment, we propose an attribute descent approach that automatically optimizes engine attributes to enable synthetic data to approximate real-world data. We verify our method on object-centric tasks, wherein an object takes up a major portion of an image. In these tasks, the search space is relatively small, and the optimization of each attribute yields sufficiently obvious supervision signals. We collect a new synthetic asset VehicleX, and reformat and reuse existing the synthetic assets ObjectX and PersonX. Extensive experiments on image classification and object re-identification confirm that adapted synthetic data can be effectively used in three scenarios: training with synthetic data only, training data augmentation and numerically understanding dataset content.

cs.CV↗
arXiv2023

Privacy Assessment on Reconstructed Images: Are Existing Evaluation Metrics Faithful to Human Perception?

Hand-crafted image quality metrics, such as PSNR and SSIM, are commonly used to evaluate model privacy risk under reconstruction attacks. Under these metrics, reconstructed images that are determined to resemble the original one generally indicate more privacy leakage. Images determined as overall dissimilar, on the other hand, indicate higher robustness against attack. However, there is no guarantee that these metrics well reflect human opinions, which, as a judgement for model privacy leakage, are more trustworthy. In this paper, we comprehensively study the faithfulness of these hand-crafted metrics to human perception of privacy information from the reconstructed images. On 5 datasets ranging from natural images, faces, to fine-grained classes, we use 4 existing attack methods to reconstruct images from many different classification models and, for each reconstructed image, we ask multiple human annotators to assess whether this image is recognizable. Our studies reveal that the hand-crafted metrics only have a weak correlation with the human evaluation of privacy leakage and that even these metrics themselves often contradict each other. These observations suggest risks of current metrics in the community. To address this potential risk, we propose a learning-based measure called SemSim to evaluate the Semantic Similarity between the original and reconstructed images. SemSim is trained with a standard triplet loss, using an original image as an anchor, one of its recognizable reconstructed images as a positive sample, and an unrecognizable one as a negative. By training on human annotations, SemSim exhibits a greater reflection of privacy leakage on the semantic level. We show that SemSim has a significantly higher correlation with human judgment compared with existing metrics. Moreover, this strong correlation generalizes to unseen datasets, models and attack methods.

cs.CV↗
arXiv2023

An introduction to the parton and hadron cascade model PACIAE 3.0

We introduce a parton and hadron cascade model PACIAE 3.0 based on PYTHIA 6.428 and the PACIAE 2.2 program series. The simulation framework of C-, B-, and A-loops are designed for the high energy ($\sqrt{s_{NN}}\geq 3$ GeV) and low energy ($\sqrt{s_{NN}}<3$ GeV) nuclear collisions, respectively, in PACIAE 3.0. In the C-loop simulation, the parton-parton inelastic scattering processes are added in the partonic rescattering process. The single string structure and multiple string interaction mechanism have been introduced investigating the strangeness enhancement in C- and B-loop. An improved mapping relation between the centrality percentage definition and the impact parameter definition is proposed responding the observation of $b_{max}\approx 20$ fm from ALICE, ATLAS, and CMS collaborations. We have extensively modified the phenomenological coalescence hadronization model. The PACIAE 3.0 model simulated results of particle yield, transverse momentum distribution, and rapidity distribution well reproduce, respectively, the experimental data measured at FOPI, E895, RHIC, and LHC energies.

hep-ph↗
arXiv2023

Resolving the $R_{\rm pA}$ and $v_2$ puzzle of $D^0$ mesons in $p-$Pb collisions

It has been difficult to reconcile the experimental data on the $D^0$ meson nuclear modification factor and elliptic flow in $p-$Pb collisions at LHC energies. Here we study these observables with the string melting version of a multi-phase transport model, which has been improved with the implementation of the Cronin effect (or transverse momentum broadening) and independent fragmentation for charm quarks. Using a strong Cronin effect allows us to provide the first simultaneous description of the $D^0$ meson $R_{\rm pA}$ and $v_2$ data at $p_{\rm T} \leq$ 8 GeV$/c$. The model also provides a reasonable description of the $D^0$ meson $p_{\rm T}$ spectra and the low-$p_{\rm T}$ (below $\sim$ 2 GeV$/c$) charged hadron spectra in $p+p$ and $p-$Pb collisions as well as $R_{\rm pA}$ and $v_2$ in $p-$Pb collisions. We find that both parton scatterings and the Cronin effect are important for the $D^0$ meson $R_{\rm pA}$, while parton scatterings are mostly responsible for the $D^0$ meson $v_2$. Our results indicate that it is crucial to include the Cronin effect for the simultaneous description of the $D^0$ meson $R_{\rm pA}$ and $v_2$. Since the Cronin effect is expected to grow with the system size, this work implies that the Cronin effect could also be important for heavy hadrons in large systems.

nucl-th↗
arXiv2023

CircNet: Meshing 3D Point Clouds with Circumcenter Detection

Reconstructing 3D point clouds into triangle meshes is a key problem in computational geometry and surface reconstruction. Point cloud triangulation solves this problem by providing edge information to the input points. Since no vertex interpolation is involved, it is beneficial to preserve sharp details on the surface. Taking advantage of learning-based techniques in triangulation, existing methods enumerate the complete combinations of candidate triangles, which is both complex and inefficient. In this paper, we leverage the duality between a triangle and its circumcenter, and introduce a deep neural network that detects the circumcenters to achieve point cloud triangulation. Specifically, we introduce multiple anchor priors to divide the neighborhood space of each point. The neural network then learns to predict the presences and locations of circumcenters under the guidance of those anchors. We extract the triangles dual to the detected circumcenters to form a primitive mesh, from which an edge-manifold mesh is produced via simple post-processing. Unlike existing learning-based triangulation methods, the proposed method bypasses an exhaustive enumeration of triangle combinations and local surface parameterization. We validate the efficiency, generalization, and robustness of our method on prominent datasets of both watertight and open surfaces. The code and trained models are provided at https://github.com/EnyaHermite/CircNet.

cs.CV↗
arXiv2023

Training with Product Digital Twins for AutoRetail Checkout

Automating the checkout process is important in smart retail, where users effortlessly pass products by hand through a camera, triggering automatic product detection, tracking, and counting. In this emerging area, due to the lack of annotated training data, we introduce a dataset comprised of product 3D models, which allows for fast, flexible, and large-scale training data generation through graphic engine rendering. Within this context, we discern an intriguing facet, because of the user "hands-on" approach, bias in user behavior leads to distinct patterns in the real checkout process. The existence of such patterns would compromise training effectiveness if training data fail to reflect the same. To address this user bias problem, we propose a training data optimization framework, i.e., training with digital twins (DtTrain). Specifically, we leverage the product 3D models and optimize their rendering viewpoint and illumination to generate "digital twins" that visually resemble representative user images. These digital twins, inherit product labels and, when augmented, form the Digital Twin training set (DT set). Because the digital twins individually mimic user bias, the resulting DT training set better reflects the characteristics of the target scenario and allows us to train more effective product detection and tracking models. In our experiment, we show that DT set outperforms training sets created by existing dataset synthesis methods in terms of counting accuracy. Moreover, by combining DT set with pseudo-labeled real checkout data, further improvement is observed. The code is available at https://github.com/yorkeyao/Automated-Retail-Checkout.

cs.CV↗
arXiv2023

Stark tuning of telecom single-photon emitters based on a single Er$^{3+}$

The implementation of scalable quantum networks requires photons at the telecom band and long-lived spin coherence. The single Er$^{3+}$ in solid-state hosts is an important candidate that fulfills these critical requirements simultaneously. However, to entangle distant Er$^{3+}$ ions through photonic connections, the emission frequency of individual Er$^{3+}$ in solid-state matrix must be the same, which is challenging because the emission frequency of Er$^{3+}$ depends on its local environment. Herein, we propose and experimentally demonstrate the Stark tuning of the emission frequency of a single Er$^{3+}$ in a Y$_2$SiO$_5$ crystal by employing electrodes interfaced with a silicon photonic crystal cavity. We obtain a Stark shift of 182.9 $\pm$ 0.8 MHz which is approximately 27 times of the optical emission linewidth, demonstrating the promising applications in tuning the emission frequency of independent Er$^{3+}$ into the same spectral channels. Our results provide a useful solution for construction of scalable quantum networks based on single Er$^{3+}$ and a universal tool for tuning emission of individual rare-earth ions.

quant-ph↗
arXiv2023

Resolving the $R_{\rm pA}$ and $v_2$ puzzle of $D^0$ mesons in $p-$Pb collisions at the LHC

It has been a challenge to understand the experimental data on both the nuclear modification factor and elliptic flow of $D^0$ mesons in $p-$Pb collisions at LHC energies. In this work, we study these collisions with an improved multi-phase transport model. By including the Cronin effect (i.e., transverse momentum broadening) and independent fragmentation for charm quarks, we provide the first simultaneous description of the $D^0$ meson $R_{\rm pA}$ and $v_2$ data at $p_{\rm T} \leq 8$ GeV$/c$. The model also reasonably describes the $D^0$ meson $p_{\rm T}$ spectra and the low-$p_{\rm T}$ charged hadron spectra, $R_{\rm pA}$ and $v_2$. Our results show that both parton interactions and the Cronin effect are important for the $D^0$ meson $R_{\rm pA}$, while parton interactions are mostly responsible for the $D^0$ meson $v_2$. It is thus essential to include the Cronin effect for the simultaneous description of the $D^0$ meson $R_{\rm pA}$ and $v_2$. This work implies that the Cronin effect could also be important for heavy hadrons in large systems.

nucl-th↗
arXiv2023

The universal scaling of kinetic freeze-out parameters across different collision systems at the LHC energy

In this paper, we perform the Tsallis Blast-Wave analysis on the transverse momentum spectra of identified hadrons produced in a wide range of collision systems at the Large Hadron Collider (LHC) including pp, pPb, XeXe and PbPb collisions. The kinetic freeze-out properties are investigated across these systems varying with the event multiplicity. We find that the extracted kinetic freeze-out temperature, radial flow velocity and the non-extensive parameter exhibit a universal scaling behavior for these systems with very different geometric size, especially when the independent baryon Tsallis non-extensive parameter is considered. This universality may indicate the existence of a unified partonic evolution stage in different collision systems at the LHC energies.

hep-ph↗
arXiv2023

The 7th AI City Challenge

The AI City Challenge's seventh edition emphasizes two domains at the intersection of computer vision and artificial intelligence - retail business and Intelligent Traffic Systems (ITS) - that have considerable untapped potential. The 2023 challenge had five tracks, which drew a record-breaking number of participation requests from 508 teams across 46 countries. Track 1 was a brand new track that focused on multi-target multi-camera (MTMC) people tracking, where teams trained and evaluated using both real and highly realistic synthetic data. Track 2 centered around natural-language-based vehicle track retrieval. Track 3 required teams to classify driver actions in naturalistic driving analysis. Track 4 aimed to develop an automated checkout system for retail stores using a single view camera. Track 5, another new addition, tasked teams with detecting violations of the helmet rule for motorcyclists. Two leader boards were released for submissions based on different methods: a public leader board for the contest where external private data wasn't allowed and a general leader board for all results submitted. The participating teams' top performances established strong baselines and even outperformed the state-of-the-art in the proposed challenge tracks.

cs.CV↗
arXiv2023

Large-scale Training Data Search for Object Re-identification

We consider a scenario where we have access to the target domain, but cannot afford on-the-fly training data annotation, and instead would like to construct an alternative training set from a large-scale data pool such that a competitive model can be obtained. We propose a search and pruning (SnP) solution to this training data search problem, tailored to object re-identification (re-ID), an application aiming to match the same object captured by different cameras. Specifically, the search stage identifies and merges clusters of source identities which exhibit similar distributions with the target domain. The second stage, subject to a budget, then selects identities and their images from the Stage I output, to control the size of the resulting training set for efficient training. The two steps provide us with training sets 80\% smaller than the source pool while achieving a similar or even higher re-ID accuracy. These training sets are also shown to be superior to a few existing search methods such as random sampling and greedy sampling under the same budget on training data size. If we release the budget, training sets resulting from the first stage alone allow even higher re-ID accuracy. We provide interesting discussions on the specificity of our method to the re-ID problem and particularly its role in bridging the re-ID domain gap. The code is available at https://github.com/yorkeyao/SnP.

cs.CV↗
arXiv2023

A Bag-of-Prototypes Representation for Dataset-Level Applications

This work investigates dataset vectorization for two dataset-level tasks: assessing training set suitability and test set difficulty. The former measures how suitable a training set is for a target domain, while the latter studies how challenging a test set is for a learned model. Central to the two tasks is measuring the underlying relationship between datasets. This needs a desirable dataset vectorization scheme, which should preserve as much discriminative dataset information as possible so that the distance between the resulting dataset vectors can reflect dataset-to-dataset similarity. To this end, we propose a bag-of-prototypes (BoP) dataset representation that extends the image-level bag consisting of patch descriptors to dataset-level bag consisting of semantic prototypes. Specifically, we develop a codebook consisting of K prototypes clustered from a reference dataset. Given a dataset to be encoded, we quantize each of its image features to a certain prototype in the codebook and obtain a K-dimensional histogram. Without assuming access to dataset labels, the BoP representation provides a rich characterization of the dataset semantic distribution. Furthermore, BoP representations cooperate well with Jensen-Shannon divergence for measuring dataset-to-dataset similarity. Although very simple, BoP consistently shows its advantage over existing representations on a series of benchmarks for two dataset-level tasks.

cs.CV↗
arXiv2023

Unsupervised Evaluation of Out-of-distribution Detection: A Data-centric Perspective

Out-of-distribution (OOD) detection methods assume that they have test ground truths, i.e., whether individual test samples are in-distribution (IND) or OOD. However, in the real world, we do not always have such ground truths, and thus do not know which sample is correctly detected and cannot compute the metric like AUROC to evaluate the performance of different OOD detection methods. In this paper, we are the first to introduce the unsupervised evaluation problem in OOD detection, which aims to evaluate OOD detection methods in real-world changing environments without OOD labels. We propose three methods to compute Gscore as an unsupervised indicator of OOD detection performance. We further introduce a new benchmark Gbench, which has 200 real-world OOD datasets of various label spaces to train and evaluate our method. Through experiments, we find a strong quantitative correlation betwwen Gscore and the OOD detection performance. Extensive experiments demonstrate that our Gscore achieves state-of-the-art performance. Gscore also generalizes well with different IND/OOD datasets, OOD detection methods, backbones and dataset sizes. We further provide interesting analyses of the effects of backbones and IND/OOD datasets on OOD detection performance. The data and code will be available.

cs.CV↗
arXiv2023

Learning to Select Camera Views: Efficient Multiview Understanding at Few Glances

Multiview camera setups have proven useful in many computer vision applications for reducing ambiguities, mitigating occlusions, and increasing field-of-view coverage. However, the high computational cost associated with multiple views poses a significant challenge for end devices with limited computational resources. To address this issue, we propose a view selection approach that analyzes the target object or scenario from given views and selects the next best view for processing. Our approach features a reinforcement learning based camera selection module, MVSelect, that not only selects views but also facilitates joint training with the task network. Experimental results on multiview classification and detection tasks show that our approach achieves promising performance while using only 2 or 3 out of N available views, significantly reducing computational costs. Furthermore, analysis on the selected views reveals that certain cameras can be shut off with minimal performance impact, shedding light on future camera layout optimization for multiview systems. Code is available at https://github.com/hou-yz/MVSelect.

cs.CV↗
arXiv2023

Adaptive Calibrator Ensemble for Model Calibration under Distribution Shift

Model calibration usually requires optimizing some parameters (e.g., temperature) w.r.t an objective function (e.g., negative log-likelihood). In this paper, we report a plain, important but often neglected fact that the objective function is influenced by calibration set difficulty, i.e., the ratio of the number of incorrectly classified samples to that of correctly classified samples. If a test set has a drastically different difficulty level from the calibration set, the optimal calibration parameters of the two datasets would be different. In other words, a calibrator optimal on the calibration set would be suboptimal on the OOD test set and thus has degraded performance. With this knowledge, we propose a simple and effective method named adaptive calibrator ensemble (ACE) to calibrate OOD datasets whose difficulty is usually higher than the calibration set. Specifically, two calibration functions are trained, one for in-distribution data (low difficulty), and the other for severely OOD data (high difficulty). To achieve desirable calibration on a new OOD dataset, ACE uses an adaptive weighting method that strikes a balance between the two extreme functions. When plugged in, ACE generally improves the performance of a few state-of-the-art calibration schemes on a series of OOD benchmarks. Importantly, such improvement does not come at the cost of the in-distribution calibration accuracy.

cs.LG↗