arXiv subjects

Jue Wang

Publications and source records attributed to Jue Wang.

At least 235 records · Page 13Linked to original sources

arXiv2018

End-to-End Audio Visual Scene-Aware Dialog using Multimodal Attention-Based Video Features

Dialog systems need to understand dynamic visual scenes in order to have conversations with users about the objects and events around them. Scene-aware dialog systems for real-world applications could be developed by integrating state-of-the-art technologies from multiple research areas, including: end-to-end dialog technologies, which generate system responses using models trained from dialog data; visual question answering (VQA) technologies, which answer questions about images using learned image features; and video description technologies, in which descriptions/captions are generated from videos using multimodal information. We introduce a new dataset of dialogs about videos of human behaviors. Each dialog is a typed conversation that consists of a sequence of 10 question-and-answer(QA) pairs between two Amazon Mechanical Turk (AMT) workers. In total, we collected dialogs on roughly 9,000 videos. Using this new dataset for Audio Visual Scene-aware dialog (AVSD), we trained an end-to-end conversation model that generates responses in a dialog about a video. Our experiments demonstrate that using multimodal features that were developed for multimodal attention-based video description enhances the quality of generated dialog about dynamic scenes (videos). Our dataset, model code and pretrained models will be publicly available for a new Video Scene-Aware Dialog challenge.

cs.CL↗

arXiv2018

Audio Visual Scene-Aware Dialog (AVSD) Challenge at DSTC7

Scene-aware dialog systems will be able to have conversations with users about the objects and events around them. Progress on such systems can be made by integrating state-of-the-art technologies from multiple research areas including end-to-end dialog systems visual dialog, and video description. We introduce the Audio Visual Scene Aware Dialog (AVSD) challenge and dataset. In this challenge, which is one track of the 7th Dialog System Technology Challenges (DSTC7) workshop1, the task is to build a system that generates responses in a dialog about an input video

cs.CL↗

arXiv2018

Video Representation Learning Using Discriminative Pooling

Popular deep models for action recognition in videos generate independent predictions for short clips, which are then pooled heuristically to assign an action label to the full video segment. As not all frames may characterize the underlying action---indeed, many are common across multiple actions---pooling schemes that impose equal importance on all frames might be unfavorable. In an attempt to tackle this problem, we propose discriminative pooling, based on the notion that among the deep features generated on all short clips, there is at least one that characterizes the action. To this end, we learn a (nonlinear) hyperplane that separates this unknown, yet discriminative, feature from the rest. Applying multiple instance learning in a large-margin setup, we use the parameters of this separating hyperplane as a descriptor for the full video segment. Since these parameters are directly related to the support vectors in a max-margin framework, they serve as robust representations for pooling of the features. We formulate a joint objective and an efficient solver that learns these hyperplanes per video and the corresponding action classifiers over the hyperplanes. Our pooling scheme is end-to-end trainable within a deep framework. We report results from experiments on three benchmark datasets spanning a variety of challenges and demonstrate state-of-the-art performance across these tasks.

cs.CV↗

arXiv2018

Scale-recurrent Network for Deep Image Deblurring

In single image deblurring, the "coarse-to-fine" scheme, i.e. gradually restoring the sharp image on different resolutions in a pyramid, is very successful in both traditional optimization-based methods and recent neural-network-based approaches. In this paper, we investigate this strategy and propose a Scale-recurrent Network (SRN-DeblurNet) for this deblurring task. Compared with the many recent learning-based approaches in [25], it has a simpler network structure, a smaller number of parameters and is easier to train. We evaluate our method on large-scale deblurring datasets with complex motion. Results show that our method can produce better quality results than state-of-the-arts, both quantitatively and qualitatively.

cs.CV↗

arXiv2017

RANSAC Algorithms for Subspace Recovery and Subspace Clustering

We consider the RANSAC algorithm in the context of subspace recovery and subspace clustering. We derive some theory and perform some numerical experiments. We also draw some correspondences with the methods of Hardt and Moitra (2013) and Chen and Lerman (2009b).

math.ST↗

arXiv2017

Virtual Astronaut for Scientific Visualization - A Prototype for Santa Maria Crater on Mars

To support scientific visualization of multiple-mission data from Mars, the Virtual Astronaut (VA) creates an interactive virtual 3D environment built on the Unity3D Game Engine. A prototype study was conducted based on orbital and Opportunity Rover data covering Santa Maria Crater in Meridiani Planum on Mars. The VA at Santa Maria provides dynamic visual representations of the imaging, compositional, and mineralogical information. The VA lets one navigate through the scene and provides geomorphic and geologic contexts for the rover operations. User interactions include in-situ observations visualization, feature measurement, and an animation control of rover drives. This paper covers our approach and implementation of the VA system. A brief summary of the prototype system functions and user feedback is also covered. Based on external review and comments by the science community, the prototype at Santa Maria has proven the VA to be an effective tool for virtual geovisual analysis.

astro-ph.IM↗

arXiv2017

Human Action Forecasting by Learning Task Grammars

For effective human-robot interaction, it is important that a robotic assistant can forecast the next action a human will consider in a given task. Unfortunately, real-world tasks are often very long, complex, and repetitive; as a result forecasting is not trivial. In this paper, we propose a novel deep recurrent architecture that takes as input features from a two-stream Residual action recognition framework, and learns to estimate the progress of human activities from video sequences -- this surrogate progress estimation task implicitly learns a temporal task grammar with respect to which activities can be localized and forecasted. To learn the task grammar, we propose a stacked LSTM based multi-granularity progress estimation framework that uses a novel cumulative Euclidean loss as objective. To demonstrate the effectiveness of our proposed architecture, we showcase experiments on two challenging robotic assistive tasks, namely (i) assembling an Ikea table from its constituents, and (ii) changing the tires of a car. Our results demonstrate that learning task grammars offers highly discriminative cues improving the forecasting accuracy by more than 9% over the baseline two-stream forecasting model, while also outperforming other competitive schemes.

cs.CV↗

arXiv2017

Three-Dimensional Anisotropic Magnetoresistance in the Dirac Node-Line Material ZrSiSe

The family of materials defined as ZrSiX (X = S, Se, Te) has been established as Dirac node-line semimetals, and subsequent study is urgent to exploit the promising application of unusual magnetoresistance property. In this work, we systematically investigated the anisotropic magnetoresistance in the newly-discovered Dirac node-line material ZrSiSe. By applying a magnetic field of 3 T by a vector field, the three-dimensional (3D) magnetoresistance (MR) shows strong anisotropy. The MR ratio of maximum and minimum directions can reach 7 at 3 T and keeps increasing at the higher magnetic field. The anisotropic MR forms a butterfly-shaped curve, which indicates the quasi-2D electronic structures. This is further confirmed by the angular-dependent Shubnikov-de Haas (SdH) oscillations. The first-principles calculations establish the quasi-2D tubular-shaped Fermi surface near the X point in the Brillouin zone. Our findings shed light on the 3D mapping of MR and the potential applications in magnetic sensors based on ZrSiSe Dirac materials.

cond-mat.mes-hall↗

arXiv2017

Dynamic Prediction for Multiple Repeated Measures and Event Time Data: An Application to Parkinson's Disease

In many clinical trials studying neurodegenerative diseases such as Parkinson's disease (PD), multiple longitudinal outcomes are collected to fully explore the multidimensional impairment caused by this disease. If the outcomes deteriorate rapidly, patients may reach a level of functional disability sufficient to initiate levodopa therapy for ameliorating disease symptoms. An accurate prediction of the time to functional disability is helpful for clinicians to monitor patients' disease progression and make informative medical decisions. In this article, we first propose a joint model that consists of a semiparametric multilevel latent trait model (MLLTM) for the multiple longitudinal outcomes, and a survival model for event time. The two submodels are linked together by an underlying latent variable. We develop a Bayesian approach for parameter estimation and a dynamic prediction framework for predicting target patients' future outcome trajectories and risk of a survival event, based on their multivariate longitudinal measurements. Our proposed model is evaluated by simulation studies and is applied to the DATATOP study, a motivating clinical trial assessing the effect of deprenyl among patients with early PD.

stat.AP↗

arXiv2017

Secure Transmission with Large Numbers of Antennas and Finite Alphabet Inputs

In this paper, we investigate secure transmission over the large-scale multiple-antenna wiretap channel with finite alphabet inputs. First, we investigate the case where instantaneous channel state information (CSI) of the eavesdropper is known at the transmitter. We show analytically that a generalized singular value decomposition (GSVD) based design, which is optimal for Gaussian inputs, may exhibit a severe performance loss for finite alphabet inputs in the high signal-to-noise ratio (SNR) regime. In light of this, we propose a novel Per-Group-GSVD (PG-GSVD) design which can effectively compensate the performance loss caused by the GSVD design. More importantly, the computational complexity of the PG-GSVD design is by orders of magnitude lower than that of the existing design for finite alphabet inputs in [1] while the resulting performance loss is minimal. Then, we extend the PG-GSVD design to the case where only statistical CSI of the eavesdropper is available at the transmitter. Numerical results indicate that the proposed PG-GSVD design can be efficiently implemented in large-scale multiple-antenna systems and achieves significant performance gains compared to the GSVD design.

cs.IT↗

arXiv2017

Zero-order Reverse Filtering

In this paper, we study an unconventional but practically meaningful reversibility problem of commonly used image filters. We broadly define filters as operations to smooth images or to produce layers via global or local algorithms. And we raise the intriguingly problem if they are reservable to the status before filtering. To answer it, we present a novel strategy to understand general filter via contraction mappings on a metric space. A very simple yet effective zero-order algorithm is proposed. It is able to practically reverse most filters with low computational cost. We present quite a few experiments in the paper and supplementary file to thoroughly verify its performance. This method can also be generalized to solve other inverse problems and enables new applications.

cs.CV↗

arXiv2017

Detail-revealing Deep Video Super-resolution

Previous CNN-based video super-resolution approaches need to align multiple frames to the reference. In this paper, we show that proper frame alignment and motion compensation is crucial for achieving high quality results. We accordingly propose a `sub-pixel motion compensation' (SPMC) layer in a CNN framework. Analysis and experiments show the suitability of this layer in video SR. The final end-to-end, scalable CNN framework effectively incorporates the SPMC layer and fuses multiple frames to reveal image details. Our implementation can generate visually and quantitatively high-quality results, superior to current state-of-the-arts, without the need of parameter tuning.

cs.CV↗

arXiv2017

Ordered Pooling of Optical Flow Sequences for Action Recognition

Training of Convolutional Neural Networks (CNNs) on long video sequences is computationally expensive due to the substantial memory requirements and the massive number of parameters that deep architectures demand. Early fusion of video frames is thus a standard technique, in which several consecutive frames are first agglomerated into a compact representation, and then fed into the CNN as an input sample. For this purpose, a summarization approach that represents a set of consecutive RGB frames by a single dynamic image to capture pixel dynamics is proposed recently. In this paper, we introduce a novel ordered representation of consecutive optical flow frames as an alternative and argue that this representation captures the action dynamics more effectively than RGB frames. We provide intuitions on why such a representation is better for action recognition. We validate our claims on standard benchmark datasets and demonstrate that using summaries of flow images lead to significant improvements over RGB frames while achieving accuracy comparable to the state-of-the-art on UCF101 and HMDB datasets.

cs.CV↗

arXiv2017

Action Representation Using Classifier Decision Boundaries

Most popular deep learning based models for action recognition are designed to generate separate predictions within their short temporal windows, which are often aggregated by heuristic means to assign an action label to the full video segment. Given that not all frames from a video characterize the underlying action, pooling schemes that impose equal importance to all frames might be unfavorable. In an attempt towards tackling this challenge, we propose a novel pooling scheme, dubbed SVM pooling, based on the notion that among the bag of features generated by a CNN on all temporal windows, there is at least one feature that characterizes the action. To this end, we learn a decision hyperplane that separates this unknown yet useful feature from the rest. Applying multiple instance learning in an SVM setup, we use the parameters of this separating hyperplane as a descriptor for the video. Since these parameters are directly related to the support vectors in a max-margin framework, they serve as robust representations for pooling of the CNN features. We devise a joint optimization objective and an efficient solver that learns these hyperplanes per video and the corresponding action classifiers over the hyperplanes. Showcased experiments on the standard HMDB and UCF101 datasets demonstrate state-of-the-art performance.

cs.CV↗

arXiv2017

Interfacial Charge Transfer Circumventing Momentum Mismatch at 2D van der Waals Heterojunctions

Interfacial charge separation and recombination at heterojunctions of monolayer transition metal dichalcogenides (TMDCs) are of interest to two dimensional optoelectronic technologies. These processes can involve large changes in parallel momentum vector due to the confinement of electrons and holes to the K-valleys in each layer. Since these high-momentum valleys are usually not aligned across the interface of two TMDC monolayers, how parallel momentum is conserved in the charge separation or recombination process becomes a key question. Here we probe this question using the model system of a type-II heterojunction formed by MoS2 and WSe2 monolayers and the experimental technical of femtosecond pump-probe spectroscopy. Upon photo-excitation specifically of WSe2 at the heterojunction, we observe ultrafast (<40 fs) electron transfer from WSe2 to MoS2, independent of the angular alignment and, thus, momentum mismatch between the two TMDCs. The resulting interlayer charge transfer exciton decays via nonradiatively recombination, with rates varying by up to three-orders of magnitude from sample to sample, but with no correlation with inter-layer angular alignment. We suggest that the initial interfacial charge separation and the subsequent interfacial charge recombination processes circumvent momentum mismatch via excess electronic energy and via defect-mediated recombination, respectively.

cond-mat.mtrl-sci↗

arXiv2017

Approaching the Intrinsic Photoluminescence Linewidth in Transition Metal Dichalcogenide Monolayers

Excitonic states in monolayer transition metal dichalcogenides (TMDCs) have been the subject of extensive recent interest. Their intrinsic properties can, however, be obscured due to the influence of inhomogeneity in the external environment. Here we report methods for fabricating high quality TMDC monolayers with narrow photoluminescence (PL) linewidth approaching the intrinsic limit. We find that encapsulation in hexagonal boron nitride (h-BN) sharply reduces the PL linewidth, and that passivation of the oxide substrate by an alkyl monolayer further decreases the linewidth and also minimizes the charged exciton (trion) peak. The combination of these sample preparation methods results in much reduced spatial variation in the PL emission, with a full-width-at-half-maximum as low as 1.7 meV. Analysis of the PL line shape yields a homogeneous width of 1.43$\pm$0.08 meV and inhomogeneous broadening of 1.1$\pm$0.3 meV.

cond-mat.mtrl-sci↗

arXiv2017

Resolution of Indecomposable Integral Flows on Signed Graphs

It is well known that each nonnegative integral flow on a graph can be decomposed into a sum of nonnegative graphic circuit flows, which cannot be further decomposed into nonnegative integral sub-flows. This is equivalent to saying that the indecomposable flows on graphs are those graphic circuit flows. Turning from graphs to signed graphs, the indecomposable flows are much richer than those of unsigned graphs. This paper gives a complete description of indecomposable flows on signed graphs from the viewpoint of resolution of singularities by means of double covering graphs.

math.CO↗

arXiv2016

Large-Scale MIMO Secure Transmission with Finite Alphabet Inputs

In this paper, we investigate secure transmission over the large-scale multiple-antenna wiretap channel with finite alphabet inputs. First, we show analytically that a generalized singular value decomposition (GSVD) based design, which is optimal for Gaussian inputs, may exhibit a severe performance loss for finite alphabet inputs in the high signal-to-noise ratio (SNR) regime. In light of this, we propose a novel Per-Group-GSVD (PG-GSVD) design which can effectively compensate the performance loss caused by the GSVD design. More importantly, the computational complexity of the PG-GSVD design is by orders of magnitude lower than that of the existing design for finite alphabet inputs in [1] while the resulting performance loss is minimal. Numerical results indicate that the proposed PG-GSVD design can be efficiently implemented in large-scale multiple-antenna systems and achieves significant performance gains compared to the GSVD design.

cs.IT↗