arXiv Science⌕ Search

arXiv subjects

Jiaming Liu

Publications and source records attributed to Jiaming Liu.

234 records · Page 13Linked to original sources
arXiv2019

EATEN: Entity-aware Attention for Single Shot Visual Text Extraction

Extracting entity from images is a crucial part of many OCR applications, such as entity recognition of cards, invoices, and receipts. Most of the existing works employ classical detection and recognition paradigm. This paper proposes an Entity-aware Attention Text Extraction Network called EATEN, which is an end-to-end trainable system to extract the entities without any post-processing. In the proposed framework, each entity is parsed by its corresponding entity-aware decoder, respectively. Moreover, we innovatively introduce a state transition mechanism which further improves the robustness of entity extraction. In consideration of the absence of public benchmarks, we construct a dataset of almost 0.6 million images in three real-world scenarios (train ticket, passport and business card), which is publicly available at https://github.com/beacandler/EATEN. To the best of our knowledge, EATEN is the first single shot method to extract entities from images. Extensive experiments on these benchmarks demonstrate the state-of-the-art performance of EATEN.

cs.CV↗
arXiv2019

Block Coordinate Regularization by Denoising

We consider the problem of estimating a vector from its noisy measurements using a prior specified only through a denoising function. Recent work on plug-and-play priors (PnP) and regularization-by-denoising (RED) has shown the state-of-the-art performance of estimators under such priors in a range of imaging tasks. In this work, we develop a new block coordinate RED algorithm that decomposes a large-scale estimation problem into a sequence of updates over a small subset of the unknown variables. We theoretically analyze the convergence of the algorithm and discuss its relationship to the traditional proximal optimization. Our analysis complements and extends recent theoretical results for RED-based estimation methods. We numerically validate our method using several denoiser priors, including those based on convolutional neural network (CNN) denoisers.

cs.CV↗
arXiv2019

Disentangled Image Matting

Most previous image matting methods require a roughly-specificed trimap as input, and estimate fractional alpha values for all pixels that are in the unknown region of the trimap. In this paper, we argue that directly estimating the alpha matte from a coarse trimap is a major limitation of previous methods, as this practice tries to address two difficult and inherently different problems at the same time: identifying true blending pixels inside the trimap region, and estimate accurate alpha values for them. We propose AdaMatting, a new end-to-end matting framework that disentangles this problem into two sub-tasks: trimap adaptation and alpha estimation. Trimap adaptation is a pixel-wise classification problem that infers the global structure of the input image by identifying definite foreground, background, and semi-transparent image regions. Alpha estimation is a regression problem that calculates the opacity value of each blended pixel. Our method separately handles these two sub-tasks within a single deep convolutional neural network (CNN). Extensive experiments show that AdaMatting has additional structure awareness and trimap fault-tolerance. Our method achieves the state-of-the-art performance on Adobe Composition-1k dataset both qualitatively and quantitatively. It is also the current best-performing method on the alphamatting.com online evaluation for all commonly-used metrics.

cs.CV↗
arXiv2019

Online Regularization by Denoising with Applications to Phase Retrieval

Regularization by denoising (RED) is a powerful framework for solving imaging inverse problems. Most RED algorithms are iterative batch procedures, which limits their applicability to very large datasets. In this paper, we address this limitation by introducing a novel online RED (On-RED) algorithm, which processes a small subset of the data at a time. We establish the theoretical convergence of On-RED in convex settings and empirically discuss its effectiveness in non-convex ones by illustrating its applicability to phase retrieval. Our results suggest that On-RED is an effective alternative to the traditional RED algorithms when dealing with large datasets.

eess.IV↗
arXiv2019

Editing Text in the Wild

In this paper, we are interested in editing text in natural images, which aims to replace or modify a word in the source image with another one while maintaining its realistic look. This task is challenging, as the styles of both background and text need to be preserved so that the edited image is visually indistinguishable from the source image. Specifically, we propose an end-to-end trainable style retention network (SRNet) that consists of three modules: text conversion module, background inpainting module and fusion module. The text conversion module changes the text content of the source image into the target text while keeping the original text style. The background inpainting module erases the original text, and fills the text region with appropriate texture. The fusion module combines the information from the two former modules, and generates the edited text images. To our knowledge, this work is the first attempt to edit text in natural images at the word level. Both visual effects and quantitative results on synthetic and real-world dataset (ICDAR 2013) fully confirm the importance and necessity of modular decomposition. We also conduct extensive experiments to validate the usefulness of our method in various real-world applications such as text image synthesis, augmented reality (AR) translation, information hiding, etc.

cs.CV↗
arXiv2019

Semi-supervised Skin Detection by Network with Mutual Guidance

In this paper we present a new data-driven method for robust skin detection from a single human portrait image. Unlike previous methods, we incorporate human body as a weak semantic guidance into this task, considering acquiring large-scale of human labeled skin data is commonly expensive and time-consuming. To be specific, we propose a dual-task neural network for joint detection of skin and body via a semi-supervised learning strategy. The dual-task network contains a shared encoder but two decoders for skin and body separately. For each decoder, its output also serves as a guidance for its counterpart, making both decoders mutually guided. Extensive experiments were conducted to demonstrate the effectiveness of our network with mutual guidance, and experimental results show our network outperforms the state-of-the-art in skin detection.

cs.CV↗
arXiv2019

Learning Raw Image Denoising with Bayer Pattern Unification and Bayer Preserving Augmentation

In this paper, we present new data pre-processing and augmentation techniques for DNN-based raw image denoising. Compared with traditional RGB image denoising, performing this task on direct camera sensor readings presents new challenges such as how to effectively handle various Bayer patterns from different data sources, and subsequently how to perform valid data augmentation with raw images. To address the first problem, we propose a Bayer pattern unification (BayerUnify) method to unify different Bayer patterns. This allows us to fully utilize a heterogeneous dataset to train a single denoising model instead of training one model for each pattern. Furthermore, while it is essential to augment the dataset to improve model generalization and performance, we discovered that it is error-prone to modify raw images by adapting augmentation methods designed for RGB images. Towards this end, we present a Bayer preserving augmentation (BayerAug) method as an effective approach for raw image augmentation. Combining these data processing technqiues with a modified U-Net, our method achieves a PSNR of 52.11 and a SSIM of 0.9969 in NTIRE 2019 Real Image Denoising Challenge, demonstrating the state-of-the-art performance. Our code is available at https://github.com/Jiaming-Liu/BayerUnifyAug.

cs.CV↗
arXiv2019

A New Recurrent Plug-and-Play Prior Based on the Multiple Self-Similarity Network

Recent work has shown the effectiveness of the plug-and-play priors (PnP) framework for regularized image reconstruction. However, the performance of PnP depends on the quality of the denoisers used as priors. In this letter, we design a novel PnP denoising prior, called multiple self-similarity net (MSSN), based on the recurrent neural network (RNN) with self-similarity matching using multi-head attention mechanism. Unlike traditional neural net denoisers, MSSN exploits different types of relationships among non-local and repeating features to remove the noise in the input image. We numerically evaluate the performance of MSSN as a module within PnP for solving magnetic resonance (MR) image reconstruction. Experimental results show the stable convergence and excellent performance of MSSN for reconstructing images from highly compressive Fourier measurements.

eess.IV↗
arXiv2019

Frame-Recurrent Video Inpainting by Robust Optical Flow Inference

In this paper, we present a new inpainting framework for recovering missing regions of video frames. Compared with image inpainting, performing this task on video presents new challenges such as how to preserving temporal consistency and spatial details, as well as how to handle arbitrary input video size and length fast and efficiently. Towards this end, we propose a novel deep learning architecture which incorporates ConvLSTM and optical flow for modeling the spatial-temporal consistency in videos. It also saves much computational resource such that our method can handle videos with larger frame size and arbitrary length streamingly in real-time. Furthermore, to generate an accurate optical flow from corrupted frames, we propose a robust flow generation module, where two sources of flows are fed and a flow blending network is trained to fuse them. We conduct extensive experiments to evaluate our method in various scenarios and different datasets, both qualitatively and quantitatively. The experimental results demonstrate the superior of our method compared with the state-of-the-art inpainting approaches.

cs.CV↗
arXiv2019

Detecting Text in the Wild with Deep Character Embedding Network

Most text detection methods hypothesize texts are horizontal or multi-oriented and thus define quadrangles as the basic detection unit. However, text in the wild is usually perspectively distorted or curved, which can not be easily tackled by existing approaches. In this paper, we propose a deep character embedding network (CENet) which simultaneously predicts the bounding boxes of characters and their embedding vectors, thus making text detection a simple clustering task in the character embedding space. The proposed method does not require strong assumptions of forming a straight line on general text detection, which provides flexibility on arbitrarily curved or perspectively distorted text. For character detection task, a dense prediction subnetwork is designed to obtain the confidence score and bounding boxes of characters. For character embedding task, a subnet is trained with contrastive loss to project detected characters into embedding space. The two tasks share a backbone CNN from which the multi-scale feature maps are extracted. The final text regions can be easily achieved by a thresholding process on character confidence and embedding distance of character pairs. We evaluated our method on ICDAR13, ICDAR15, MSRA-TD500, and Total-Text. The proposed method achieves state-of-the-art or comparable performance on all these datasets, and shows substantial improvement in the irregular-text datasets, i.e. Total-Text.

cs.CV↗
arXiv2018

TextNet: Irregular Text Reading from Images with an End-to-End Trainable Network

Reading text from images remains challenging due to multi-orientation, perspective distortion and especially the curved nature of irregular text. Most of existing approaches attempt to solve the problem in two or multiple stages, which is considered to be the bottleneck to optimize the overall performance. To address this issue, we propose an end-to-end trainable network architecture, named TextNet, which is able to simultaneously localize and recognize irregular text from images. Specifically, we develop a scale-aware attention mechanism to learn multi-scale image features as a backbone network, sharing fully convolutional features and computation for localization and recognition. In text detection branch, we directly generate text proposals in quadrangles, covering oriented, perspective and curved text regions. To preserve text features for recognition, we introduce a perspective RoI transform layer, which can align quadrangle proposals into small feature maps. Furthermore, in order to extract effective features for recognition, we propose to encode the aligned RoI features by RNN into context information, combining spatial attention mechanism to generate text sequences. This overall pipeline is capable of handling both regular and irregular cases. Finally, text localization and recognition tasks can be jointly trained in an end-to-end fashion with designed multi-task loss. Experiments on standard benchmarks show that the proposed TextNet can achieve state-of-the-art performance, and outperform existing approaches on irregular datasets by a large margin.

cs.CV↗
arXiv2018

Image Restoration using Total Variation Regularized Deep Image Prior

In the past decade, sparsity-driven regularization has led to significant improvements in image reconstruction. Traditional regularizers, such as total variation (TV), rely on analytical models of sparsity. However, increasingly the field is moving towards trainable models, inspired from deep learning. Deep image prior (DIP) is a recent regularization framework that uses a convolutional neural network (CNN) architecture without data-driven training. This paper extends the DIP framework by combining it with the traditional TV regularization. We show that the inclusion of TV leads to considerable performance gains when tested on several traditional restoration tasks such as image denoising and deblurring.

cs.CV↗
arXiv2017

On the Relation of Silicates and SiO Maser in Evolved Stars

The SiO molecule is one of the candidates for the seed of silicate dust in the circumstellar envelope of evolved stars, but this opinion is challenged. In this work we investigate the relation of the SiO maser emis- sion power and the silicate dust emission power. With both our own observation by using the PMO/Delingha 13.7-m telescope and archive data, a sample is assembled of 21 SiO v=1,J=2-1 sources and 28 SiO v=1,J=1- 0 sources that exhibit silicate emission features in the ISO/SWS spectrum as well. The analysis of their SiO maser and silicate emission power indicates a clear correlation, which is not against the hypothesis that the SiO molecules are the seed nuclei of silicate dust. On the other hand, no correlation is found between SiO maser and silicate crystallinity, which may imply that silicate crystallinity does not correlate with mass loss rate.

astro-ph.SR↗
arXiv2016

On the Silicate Crystallinities of Oxygen-Rich Evolved Stars and Their Mass Loss Rates

For decades ever since the early detection in the 1990s of the emission spectral features of crystalline silicates in oxygen-rich evolved stars, there is a long-standing debate on whether the crystallinity of the silicate dust correlates with the stellar mass loss rate. To investigate the relation between the silicate crystallinities and the mass loss rates of evolved stars, we carry out a detailed analysis of 28 nearby oxygen-rich stars. We derive the mass loss rates of these sources by modeling their spectral energy distributions from the optical to the far infrared. Unlike previous studies in which the silicate crystallinity was often measured in terms of the crystalline-to-amorphous silicate mass ratio, we characterize the silicate crystallinities of these sources with the flux ratios of the emission features of crystalline silicates to that of amorphous silicates. This does not require the knowledge of the silicate dust temperatures which are the major source of uncertainties in estimating the crystalline-to-amorphous silicate mass ratio. With a Pearson correlation coefficient of ~0.24, we find that the silicate crystallinities and the mass loss rates of these sources are not correlated. This supports the earlier findings that the dust shells of low mass-loss rate stars can contain a significant fraction of crystalline silicates without showing the characteristic features in their emission spectra.

astro-ph.SR↗
arXiv2016

A Precise Determination of the Mid-Infrared Interstellar Extinction Law Based on the APOGEE Spectroscopic Survey

A precise measure of the mid-infrared interstellar extinction law is crucial to the investigation of the properties of interstellar dust, especially of the grains in the large size end. Based on the stellar parameters derived from the SDSS-III/APOGEE spectroscopic survey, we select a large sample of G- and K-type giants as the tracers of the Galactic mid-infrared extinction. We calculate the intrinsic stellar color excesses from the stellar effective temperatures and use them to determine the mid-infrared extinction for a given line of sight. For the entire sky of the Milky Way surveyed by APOGEE, we derive the extinction (relative to the K$_{\rm S}$ band at wavelength $λ=2.16μ$m) for the four WISE bands at 3.4, 4.6, 12 and 22$μ$m, the four Spitzer/IRAC bands at 3.6, 4.5, 5.8 and 8$μ$m, the Spitzer/MIPS24 band at 23.7$μ$m and for the first time, the AKARI/S9W band at 8.23$μ$m. Our results agree with previous works in that the extinction curve is flat in the ~3--8$μ$m wavelength range and is generally consistent with the $R_V=5.5$ model curve except our determination exceeds the model prediction in the WISE/W4 band. Although some previous works found that the mid-IR extinction law appears to vary with the extinction depth $A_{\rm{K_S}}$, no noticeable variation has been found in this work. The uncertainties are analyzed in terms of the bootstrap resampling method and Monte-Carlo simulation and are found to be rather small.

astro-ph.GA↗
arXiv2016

A systematic search for the spectra with features of crystalline silicates in the Spitzer IRS Enhanced Products

The crystalline silicates features are mainly reflected in infrared bands. The Spitzer Infrared Spectrograph (IRS) collected numerous spectra of various objects and provided a big database to investigate crystalline silicates in a wide range of astronomical environments. We apply the manifold ranking algorithm to perform a systematic search for the spectra with crystalline silicates features in the Spitzer IRS Enhanced Products available. In total, 868 spectra of 790 sources are found to show the features of crystalline silicate. These objects are cross-matched with the SIMBAD database as well as with the Large Sky Area Multi-Object Fibre Spectroscopic Telescope (LAMOST)/DR2. The average spectrum of young stellar objects show a variety of features dominated either by forsterite or enstatite or neither, while the average spectrum of evolved objects consistently present dominant features of forsterite in AGB, OH/IR, post-AGB and planetary nebulae. They are identified optically as early-type stars, evolved stars, galaxies and so on. In addition, the strength of spectral features in typical silicate complexes is calculated. The results are available through CDS for the astronomical community to further study crystalline silicate.

astro-ph.GA↗
arXiv2013

Unidentified species in envelopes around carbon stars

The infrared (IR) spectra of many evolved carbon-rich stars exhibit two prominent dust emission features peaking around 21um and 30um, with the former exclusively seen in proto-planetary nebulae (PPNe), while the latter seen in a much wider range of objects, including AGB stars, PPNe and planetary nebulae (PNe). The 30um feature is seen in all the 21um sources, but no correlation is found between these two features. Over a dozen carrier candidates have been proposed for the 21um feature, but none of them has been widely accepted and the nature of the 21um feature remains a mystery. The carrier of the 30um feature also remains unidentied. MgS dust, once widely accepted as a valid carrier, was ruled out because of the sulfur budget problem. In this work we examine nano-sized FeO dust as a carrier for the 21um feature. We calculate the IR emission spectrum of FeO nanodust which undergoes single-photon heating in PPNe. It is found that the 21um feature emitted by FeO nanodust is too broad to explain the observed feature. For the 30um feature, we argue that graphite could be a viable carrier. Graphite, provided its d.c. conductivity sigma_dc exceeding 100/ohm/cm, exhibits a pronounced band at 30um.

astro-ph.SR↗