arXiv subjects

Cheng Sun

Publications and source records attributed to Cheng Sun.

54 records · Page 3Linked to original sources

arXiv2022

Self-supervised 360$^{\circ}$ Room Layout Estimation

We present the first self-supervised method to train panoramic room layout estimation models without any labeled data. Unlike per-pixel dense depth that provides abundant correspondence constraints, layout representation is sparse and topological, hindering the use of self-supervised reprojection consistency on images. To address this issue, we propose Differentiable Layout View Rendering, which can warp a source image to the target camera pose given the estimated layout from the target image. As each rendered pixel is differentiable with respect to the estimated layout, we can now train the layout estimation model by minimizing reprojection loss. Besides, we introduce regularization losses to encourage Manhattan alignment, ceiling-floor alignment, cycle consistency, and layout stretch consistency, which further improve our predictions. Finally, we present the first self-supervised results on ZilloIndoor and MatterportLayout datasets. Our approach also shows promising solutions in data-scarce scenarios and active learning, which would have an immediate value in the real estate virtual tour software. Code is available at https://github.com/joshua049/Stereo-360-Layout.

cs.CV↗

arXiv2022

The $\mathbf{uvu}$-avoiding $(a,b,c)$-Generalized Motzkin paths with vertical steps: bijections and statistic enumerations

A generalized Motzkin path, called G-Motzkin path for short, of length $n$ is a lattice path from $(0, 0)$ to $(n, 0)$ in the first quadrant of the XOY-plane that consists of up steps $\mathbf{u}=(1, 1)$, down steps $\mathbf{d}=(1, -1)$, horizontal steps $\mathbf{h}=(1, 0)$ and vertical steps $\mathbf{v}=(0, -1)$. An $(a,b,c)$-G-Motzkin path is a weighted G-Motzkin path such that the $\mathbf{u}$-steps, $\mathbf{h}$-steps, $\mathbf{v}$-steps and $\mathbf{d}$-steps are weighted respectively by $1, a, b$ and $c$. In this paper, we first give bijections between the set of $\mathbf{uvu}$-avoiding $(a,b,b^2)$-G-Motzkin paths of length $n$ and the set of $(a,b)$-Schröder paths as well as the set of $(a+b,b)$-Dyck paths of length $2n$, between the set of $\{\mathbf{uvu, uu}\}$-avoiding $(a,b,b^2)$-G-Motzkin paths of length $n$ and the set of $(a+b,ab)$-Motzkin paths of length $n$, between the set of $\{\mathbf{uvu,uu}\}$-avoiding $(a,b,b^2)$-G-Motzkin paths of length $n+1$ beginning with an $\mathbf{h}$-step weighted by $a$ and the set of $(a,b)$-Dyck paths of length $2n+2$. In the last section, we focus on the enumeration of statistics "number of $\mathbf{z}$-steps" for $\mathbf{z}\in \{\mathbf{u}, \mathbf{h}, \mathbf{v}, \mathbf{d}\}$ and "number of points" at given level in $\mathbf{uvu}$-avoiding G-Motzkin paths. These counting results are linked with Riordan arrays.

math.CO↗

arXiv2021

HoHoNet: 360 Indoor Holistic Understanding with Latent Horizontal Features

We present HoHoNet, a versatile and efficient framework for holistic understanding of an indoor 360-degree panorama using a Latent Horizontal Feature (LHFeat). The compact LHFeat flattens the features along the vertical direction and has shown success in modeling per-column modality for room layout reconstruction. HoHoNet advances in two important aspects. First, the deep architecture is redesigned to run faster with improved accuracy. Second, we propose a novel horizon-to-dense module, which relaxes the per-column output shape constraint, allowing per-pixel dense prediction from LHFeat. HoHoNet is fast: It runs at 52 FPS and 110 FPS with ResNet-50 and ResNet-34 backbones respectively, for modeling dense modalities from a high-resolution $512 \times 1024$ panorama. HoHoNet is also accurate. On the tasks of layout estimation and semantic segmentation, HoHoNet achieves results on par with current state-of-the-art. On dense depth estimation, HoHoNet outperforms all the prior arts by a large margin.

cs.CV↗

arXiv2021

Indoor Panorama Planar 3D Reconstruction via Divide and Conquer

Indoor panorama typically consists of human-made structures parallel or perpendicular to gravity. We leverage this phenomenon to approximate the scene in a 360-degree image with (H)orizontal-planes and (V)ertical-planes. To this end, we propose an effective divide-and-conquer strategy that divides pixels based on their plane orientation estimation; then, the succeeding instance segmentation module conquers the task of planes clustering more easily in each plane orientation group. Besides, parameters of V-planes depend on camera yaw rotation, but translation-invariant CNNs are less aware of the yaw change. We thus propose a yaw-invariant V-planar reparameterization for CNNs to learn. We create a benchmark for indoor panorama planar reconstruction by extending existing 360 depth datasets with ground truth H\&V-planes (referred to as PanoH&V dataset) and adopt state-of-the-art planar reconstruction methods to predict H\&V-planes as our baselines. Our method outperforms the baselines by a large margin on the proposed dataset.

cs.CV↗

arXiv2021

Specialize and Fuse: Pyramidal Output Representation for Semantic Segmentation

We present a novel pyramidal output representation to ensure parsimony with our "specialize and fuse" process for semantic segmentation. A pyramidal "output" representation consists of coarse-to-fine levels, where each level is "specialize" in a different class distribution (e.g., more stuff than things classes at coarser levels). Two types of pyramidal outputs (i.e., unity and semantic pyramid) are "fused" into the final semantic output, where the unity pyramid indicates unity-cells (i.e., all pixels in such cell share the same semantic label). The process ensures parsimony by predicting a relatively small number of labels for unity-cells (e.g., a large cell of grass) to build the final semantic output. In addition to the "output" representation, we design a coarse-to-fine contextual module to aggregate the "features" representation from different levels. We validate the effectiveness of each key module in our method through comprehensive ablation studies. Finally, our approach achieves state-of-the-art performance on three widely-used semantic segmentation datasets -- ADE20K, COCO-Stuff, and Pascal-Context.

cs.CV↗

arXiv2021

Moving in a 360 World: Synthesizing Panoramic Parallaxes from a Single Panorama

We present Omnidirectional Neural Radiance Fields (OmniNeRF), the first method to the application of parallax-enabled novel panoramic view synthesis. Recent works for novel view synthesis focus on perspective images with limited field-of-view and require sufficient pictures captured in a specific condition. Conversely, OmniNeRF can generate panorama images for unknown viewpoints given a single equirectangular image as training data. To this end, we propose to augment the single RGB-D panorama by projecting back and forth between a 3D world and different 2D panoramic coordinates at different virtual camera positions. By doing so, we are able to optimize an Omnidirectional Neural Radiance Field with visible pixels collecting from omnidirectional viewing angles at a fixed center for the estimation of new viewing angles from varying camera positions. As a result, the proposed OmniNeRF achieves convincing renderings of novel panoramic views that exhibit the parallax effect. We showcase the effectiveness of each of our proposals on both synthetic and real-world datasets.

cs.CV↗

arXiv2019

360-Indoor: Towards Learning Real-World Objects in 360° Indoor Equirectangular Images

While there are several widely used object detection datasets, current computer vision algorithms are still limited in conventional images. Such images narrow our vision in a restricted region. On the other hand, 360° images provide a thorough sight. In this paper, our goal is to provide a standard dataset to facilitate the vision and machine learning communities in 360° domain. To facilitate the research, we present a real-world 360° panoramic object detection dataset, 360-Indoor, which is a new benchmark for visual object detection and class recognition in 360° indoor images. It is achieved by gathering images of complex indoor scenes containing common objects and the intensive annotated bounding field-of-view. In addition, 360-Indoor has several distinct properties: (1) the largest category number (37 labels in total). (2) the most complete annotations on average (27 bounding boxes per image). The selected 37 objects are all common in indoor scene. With around 3k images and 90k labels in total, 360-Indoor achieves the largest dataset for detection in 360° images. In the end, extensive experiments on the state-of-the-art methods for both classification and detection are provided. We will release this dataset in the near future.

cs.CV↗

arXiv2019

Flat2Layout: Flat Representation for Estimating Layout of General Room Types

This paper proposes a new approach, Flat2Layout, for estimating general indoor room layout from a single-view RGB image whereas existing methods can only produce layout topologies captured from the box-shaped room. The proposed flat representation encodes the layout information into row vectors which are treated as the training target of the deep model. A dynamic programming based postprocessing is employed to decode the estimated flat output from the deep model into the final room layout. Flat2Layout achieves state-of-the-art performance on existing room layout benchmark. This paper also constructs a benchmark for validating the performance on general layout topologies, where Flat2Layout achieves good performance on general room types. Flat2Layout is applicable on more scenario for layout estimation and would have an impact on applications of Scene Modeling, Robotics, and Augmented Reality.

cs.CV↗

arXiv2019

HorizonNet: Learning Room Layout with 1D Representation and Pano Stretch Data Augmentation

We present a new approach to the problem of estimating the 3D room layout from a single panoramic image. We represent room layout as three 1D vectors that encode, at each image column, the boundary positions of floor-wall and ceiling-wall, and the existence of wall-wall boundary. The proposed network, HorizonNet, trained for predicting 1D layout, outperforms previous state-of-the-art approaches. The designed post-processing procedure for recovering 3D room layouts from 1D predictions can automatically infer the room shape with low computation cost - it takes less than 20ms for a panorama image while prior works might need dozens of seconds. We also propose Pano Stretch Data Augmentation, which can diversify panorama data and be applied to other panorama-related learning tasks. Due to the limited data available for non-cuboid layout, we relabel 65 general layout from the current dataset for finetuning. Our approach shows good performance on general layouts by qualitative results and cross-validation.

cs.CV↗

arXiv2018

A Spatial and Temporal Features Mixture Model with Body Parts for Video-based Person Re-Identification

The video-based person re-identification is to recognize a person under different cameras, which is a crucial task applied in visual surveillance system. Most previous methods mainly focused on the feature of full body in the frame. In this paper we propose a novel Spatial and Temporal Features Mixture Model (STFMM) based on convolutional neural network (CNN) and recurrent neural network (RNN), in which the human body is split into $N$ parts in horizontal direction so that we can obtain more specific features. The proposed method skillfully integrates features of each part to achieve more expressive representation of each person. We first split the video sequence into $N$ part sequences which include the information of head, waist, legs and so on. Then the features are extracted by STFMM whose $2N$ inputs are obtained from the developed Siamese network, and these features are combined into a discriminative representation for one person. Experiments are conducted on the iLIDS-VID and PRID-2011 datasets. The results demonstrate that our approach outperforms existing methods for video-based person re-identification. It achieves a rank-1 CMC accuracy of 74\% on the iLIDS-VID dataset, exceeding the the most recently developed method ASTPN by 12\%. For the cross-data testing, our method achieves a rank-1 CMC accuracy of 48\% exceeding the ASTPN method by 18\%, which shows that our model has significant stability.

cs.CV↗

arXiv2015

Ultrafast All-optical Modulation Exploiting the Vibrational Dynamic of Metallic Meta-atoms

Optical control over elementary molecular vibration establishes fundamental capabilities for exploiting the broad range of optical linear and nonlinear phenomena. However, experimental demonstration of the coherently driven molecular vibration remains a challenge task due to the weak optical force imposed on natural materials. Here we report the design of "meta-atom" that exhibits giant artificial optical nonlinearity. These "meta-atoms" support co-localized magnetic resonance at optical frequency and vibration resonance at GHz frequency with a deep-sub-diffraction-limit spatial confinement ($λ^2/100$). The coherent coupling of those two distinct resonances manifests a strong optical force, which is fundamentally different from the commonly studied form of radiation forces, the gradient forces, or photo-thermal induced deformation. It results in a giant third-order susceptibility $χ^{(3)}$ of $10^{-13}$ $m^2$/$V^2$, which is more than six orders of magnitude larger than that found in natural materials. The all-optical modulation at the frequency well above 1 GHz has thus been demonstrated experimentally.

physics.optics↗

arXiv2012

Nonreciprocal resonant transmission/reflection based on a one-dimensional photonic crystal adjacent to the magneto-optical metal film

We report the design of nonreciprocal resonant transmission/reflection using a one-dimensional photonic crystal (1DPC) adjacent to the magneto-optical (MO) metal film. The nonreciprocal surface modes are found at the interface between the PC and MO metal within the forbidden band of native PC structure. Breaking time-reversal symmetry using external magnetic field gives rise of such unique nonreciprocal properties. Quantitatively understanding of nonreciprocal resonant optical transmission/reflection behavior is performed using an effective admittance-matching theory. With excitation of the nonreciprocal surface modes, light can transmit and be reflected in one-way. Such design offers promising potential in realizing the optical diode.

physics.optics↗

arXiv2011

Construction of Chiral Metamaterial with a Helix Array

Here we report the designing of chiral metamaterial with metallic helix array. The effective electric and magnetic dipoles, which originate from the induced surface electric current upon illumination of incident light, are collinear at the resonant frequency. Consequently, for the circularly polarized incident light, negative refractive index is realized. Our design provides a unique approach to tune the optical properties by assembling helices, and demonstrates a different approach in exploring three- dimensional chiral metamaterial.

physics.optics↗

arXiv2011

Hiding a Realistic Object Using a Broadband Terahertz Invisibility Cloak

The invisibility cloak has been a long-standing dream for many researchers over the decades. The introduction of transformational optics has revitalized this field by providing a general method to design material distributions to hide the subject from detection. By transforming space and light propagation, a three-dimensional (3D) object is perceived as having a reduced number of dimensions, in the form of points, lines, and thin sheets, making it "undetectable" judging from the scattered field. Although a variety of cloaking devices have been reported at microwave and optical frequencies, the spectroscopically important Terahertz (THz) domain remains unexplored. Moreover, due to the difficulties in fabricating cloaking devices that are optically large in all three dimensions, hiding realistic 3D objects has yet to be demonstrated. Here, we report the first experimental demonstration of a 3D THz cloaking device fabricated using a scalable Projection Microstereolithography process. The cloak operates at a broad frequency range between 0.3 and 0.6 THz, and is placed over an α- lactose monohydrate absorber with rectangular shape. Characterized using angularresolved reflection THz time-domain spectroscopy (THz-TDS), the results indicate that the THz invisibility cloak has successfully concealed both the geometrical and spectroscopic signatures of the absorber, making it undetectable to the observer.

physics.optics↗

arXiv2011

Direct measurement of the correlated dynamics of the protein-backbone and proximal waters of hydration in mechanically strained elastin

We report on the direct measurement of the correlation times of the protein backbone carbons and proximal waters of hydration in mechanically strained elastin by nuclear magnetic resonance methods. The experimental data indicate a decrease in the correlation times of the carbonyl carbons as the strain on the biopolymer is increased. These observations are in good agreement with short 4ns molecular dynamics simulations of (VPGVG)3, a well studied mimetic peptide of elastin. The experimental results also indicate a reduction in the correlation time of proximal waters of hydration with increasing strain applied to the elastomer. A simple model is suggested that correlates the increase in the motion of proximal waters of hydration to the increase in frequency of libration of the protein backbone that develops with increasing strain. Together, the reduction in the protein entropy accompanied with the increase in entropy of the proximal waters of hydration with increasing strain, support the notion that the source of elasticity is driven by an entropic mechanism arising from the change in entropy of the protein backbone.

physics.bio-ph↗

arXiv2010

Three-Dimensional Cloaking Device Operates at Terahertz Frequencies

The invisibility cloak has been a long-standing dream for many researchers over the decades. By transforming space and light propagation, a three-dimensional (3D) object can be perceived as having reduced number of dimensions, in the form of points, lines, and thin sheets, making it "undetectable" judging from scattered field. Although a variety of cloaking devices have been reported at microwave and optical frequencies, the Terahertz (THz) domain remains unexplored. Moreover, it should be noted that all the previous experimental demonstrations are performed in a two-dimensional (2D) waveguide configuration. Although those works represent a critical step in validating the concept of the invisibility cloak, one would expect the cloaking device to be realized in 3D with the ability to cloak an object of realistic size. This requires the construction of an optically large cloaking device with features much smaller than the wavelength. Fabricating 3D structures with aspect ratio close to 100:1 is obviously a challenging task. Here, we report an experimental demonstration of a 3D THz ground plane cloak. Reflection terahertz time-domain spectroscopy (THz-TDS) was employed to characterize the cloaking samples. Two distinct reflection peaks can be clearly observed across a broad frequency range, which is caused by the reflection at the surface of the bump. The measured peak positions are consistent with the numerical simulation peak positions. By contrast, in the spectral map of the cloak sample, the wavefront is relatively smooth with a single peak.

physics.optics↗

arXiv2009

Construction of Chiral Metamaterial with U-Shaped Resonator Assembly

Chiral structure can be applied to construct metamaterial with negative refractive index (NRI). In an assembly of double-layered metallic U-shaped resonators with two resonant frequencies wH and wL, the effective induced electric and magnetic dipoles, which are contributed by the specific surface current distributions, are collinear at the same frequency. Consequently, for left circularly polarized light, NRI occurs at wH, whereas for right circularly polarized light it occurs at wL. Our design provides a new example to apply chiral structures to tune electromagnetic properties, and could be enlightening in exploring chiral metamaterials.

physics.optics↗

arXiv2008

Cloaking of Matter Waves

Invariant transformation for quantum mechanical systems is proposed. A cloaking of matter wave can be realized at given energy by designing the potential and effective mass of the matter waves in the cloaking region. The general conditions required for such a cloaking are determined and confirmed by both the wave and particle (classical) approaches. We show that it may be possible to construct such a cloaking system for cold atoms using optical lattices.

physics.optics↗