arXiv subjects

Haoran Xie

Publications and source records attributed to Haoran Xie.

At least 127 records · Page 7Linked to original sources

arXiv2022

CF-YOLO: Cross Fusion YOLO for Object Detection in Adverse Weather with a High-quality Real Snow Dataset

Snow is one of the toughest adverse weather conditions for object detection (OD). Currently, not only there is a lack of snowy OD datasets to train cutting-edge detectors, but also these detectors have difficulties learning latent information beneficial for detection in snow. To alleviate the two above problems, we first establish a real-world snowy OD dataset, named RSOD. Besides, we develop an unsupervised training strategy with a distinctive activation function, called $Peak \ Act$, to quantitatively evaluate the effect of snow on each object. Peak Act helps grading the images in RSOD into four-difficulty levels. To our knowledge, RSOD is the first quantitatively evaluated and graded snowy OD dataset. Then, we propose a novel Cross Fusion (CF) block to construct a lightweight OD network based on YOLOv5s (call CF-YOLO). CF is a plug-and-play feature aggregation module, which integrates the advantages of Feature Pyramid Network and Path Aggregation Network in a simpler yet more flexible form. Both RSOD and CF lead our CF-YOLO to possess an optimization ability for OD in real-world snow. That is, CF-YOLO can handle unfavorable detection problems of vagueness, distortion and covering of snow. Experiments show that our CF-YOLO achieves better detection results on RSOD, compared to SOTAs. The code and dataset are available at https://github.com/qqding77/CF-YOLO-and-RSOD.

cs.CV↗

arXiv2022

Find Your ASMR: A Perceptual Retrieval Interface for Autonomous Sensory Meridian Response Videos

Autonomous sensory meridian response (ASMR) is a type of video contents designed to help people relax and feel comfortable. Users usually retrieve ASMR contents from various video websites using only keywords. However, it is challenging to examine satisfactory contents to reflect users' needs for ASMR videos using keywords or content-based retrieval. To solve this issue, we propose a perceptual video retrieval system for ASMR videos and provide a novel retrieval user interface that allows users to retrieve content according to watching purpose and anticipated expectations, such as excitement, calmness, stress and sadness. An ASMR video perception dataset is constructed with annotations on affective responses after watching the videos. To verify the proposed video retrieval system, a user study is conducted showing that users can retrieve satisfactory ASMR contents easily and efficiently compared to conventional keywords-based retrieval systems.

cs.HC↗

arXiv2022

UCL-Dehaze: Towards Real-world Image Dehazing via Unsupervised Contrastive Learning

While the wisdom of training an image dehazing model on synthetic hazy data can alleviate the difficulty of collecting real-world hazy/clean image pairs, it brings the well-known domain shift problem. From a different yet new perspective, this paper explores contrastive learning with an adversarial training effort to leverage unpaired real-world hazy and clean images, thus bridging the gap between synthetic and real-world haze is avoided. We propose an effective unsupervised contrastive learning paradigm for image dehazing, dubbed UCL-Dehaze. Unpaired real-world clean and hazy images are easily captured, and will serve as the important positive and negative samples respectively when training our UCL-Dehaze network. To train the network more effectively, we formulate a new self-contrastive perceptual loss function, which encourages the restored images to approach the positive samples and keep away from the negative samples in the embedding space. Besides the overall network architecture of UCL-Dehaze, adversarial training is utilized to align the distributions between the positive samples and the dehazed images. Compared with recent image dehazing works, UCL-Dehaze does not require paired data during training and utilizes unpaired positive/negative data to better enhance the dehazing performance. We conduct comprehensive experiments to evaluate our UCL-Dehaze and demonstrate its superiority over the state-of-the-arts, even only 1,800 unpaired real-world images are used to train our network. Source code has been available at https://github.com/yz-wang/UCL-Dehaze.

cs.CV↗

arXiv2022

ILoveEye: Eyeliner Makeup Guidance System with Eye Shape Features

Drawing eyeliner is not an easy task for whom lacks experience in eye makeup. Everyone has a unique pair of eyes, so they need to draw eyeliner in a style that suits their eyes. We proposed ILoveEye, an interactive system that supports eye-makeup novices to draw natural and suitable eyeliner. The proposed system analyzes the shape of the user's eyes and classifies the eye types from camera frame. The system can recommend the eyeliner style to the user based on the designed recommendation rules. Then, the system can generate the original patterns corresponding to the eyeliner style, and the user can draw the eyeliner while observing the real-time makeup guidance. The user evaluation experiments are conducted to verify that the proposed ILoveEye system can help some users to draw reasonable eyeliner based on the specific eye shapes and improve their eye makeup skills.

cs.HC↗

arXiv2022

Refine-Net: Normal Refinement Neural Network for Noisy Point Clouds

Point normal, as an intrinsic geometric property of 3D objects, not only serves conventional geometric tasks such as surface consolidation and reconstruction, but also facilitates cutting-edge learning-based techniques for shape analysis and generation. In this paper, we propose a normal refinement network, called Refine-Net, to predict accurate normals for noisy point clouds. Traditional normal estimation wisdom heavily depends on priors such as surface shapes or noise distributions, while learning-based solutions settle for single types of hand-crafted features. Differently, our network is designed to refine the initial normal of each point by extracting additional information from multiple feature representations. To this end, several feature modules are developed and incorporated into Refine-Net by a novel connection module. Besides the overall network architecture of Refine-Net, we propose a new multi-scale fitting patch selection scheme for the initial normal estimation, by absorbing geometry domain knowledge. Also, Refine-Net is a generic normal estimation framework: 1) point normals obtained from other methods can be further refined, and 2) any feature module related to the surface geometric structures can be potentially integrated into the framework. Qualitative and quantitative evaluations demonstrate the clear superiority of Refine-Net over the state-of-the-arts on both synthetic and real-scanned datasets. Our code is available at https://github.com/hrzhou2/refinenet.

cs.CV↗

arXiv2022

When A Conventional Filter Meets Deep Learning: Basis Composition Learning on Image Filters

Image filters are fast, lightweight and effective, which make these conventional wisdoms preferable as basic tools in vision tasks. In practical scenarios, users have to tweak parameters multiple times to obtain satisfied results. This inconvenience heavily discounts the efficiency and user experience. We propose basis composition learning on single image filters to automatically determine their optimal formulas. The feasibility is based on a two-step strategy: first, we build a set of filtered basis (FB) consisting of approximations under selected parameter configurations; second, a dual-branch composition module is proposed to learn how the candidates in FB are combined to better approximate the target image. Our method is simple yet effective in practice; it renders filters to be user-friendly and benefits fundamental low-level vision problems including denoising, deraining and texture removal. Extensive experiments demonstrate that our method achieves an appropriate balance among the performance, time complexity and memory efficiency.

cs.CV↗

arXiv2022

Hierarchical Visual Interface for Lecture Video Retrieval and Summarization

With the emergence of large-scale open online courses and online academic conferences, it has become increasingly feasible and convenient to access online educational resources. However, it is time consuming and challenging to effectively retrieve and present numerous lecture videos for common users. In this work, we propose a hierarchical visual interface for retrieving and summarizing lecture videos. Users can utilize the proposed interface to effectively explore the required video information through the results of the video summary generation in different layers. We retrieve the input keywords with the corresponding video layer with timestamps, a frame layer with slides, and the poster layer with summarization of the lecture videos. We verified the proposed interface with our user study by comparing it with other conventional interfaces. The results from our user study confirmed that the proposed interface can achieve high retrieval accuracy and good user experience.see video here https://www.youtube.com/watch?v=zrnejwsOVpc .

cs.HC↗

arXiv2022

Interactive 3D Character Modeling from 2D Orthogonal Drawings with Annotations

We propose an interactive 3D character modeling approach from orthographic drawings (e.g., front and side views) based on 2D-space annotations. First, the system builds partial correspondences between the input drawings and generates a base mesh with sweeping splines according to edge information in 2D images. Next, users annotates the desired parts on the input drawings (e.g., the eyes and mouth) by using two type of strokes, called addition and erosion, and the system re-optimizes the shape of the base mesh. By repeating the 2D-space operations (i.e., revising and modifying the annotations), users can design a desired character model. To validate the efficiency and quality of our system, we verified the generated results with state-of-the-art methods.

cs.CV↗

arXiv2022

Sketch-based 3D Shape Modeling from Sparse Point Clouds

3D modeling based on point clouds is an efficient way to reconstruct and create detailed 3D content. However, the geometric procedure may lose accuracy due to high redundancy and the absence of an explicit structure. In this work, we propose a human-in-the-loop sketch-based point cloud reconstruction framework to leverage users cognitive abilities in geometry extraction. We present an interactive drawing interface for 3D model creation from point cloud data with the help of user sketches. We adopt an optimization method in which the user can continuously edit the contours extracted from the obtained 3D model and retrieve the model iteratively. Finally, we verify the proposed user interface for modeling from sparse point clouds. see video here https://www.youtube.com/watch?v=0H19NyXDRJE .

cs.GR↗

arXiv2021

xBalloon: Animated Objects with Balloon Plastic Actuator

Shape-changing interfaces are promising for users to change the physical properties of common objects. However, prevailing approaches of actuation devices require either professional equipment or materials that are not commonly accessible to non-professional users. In this work, we focus on the controllable soft actuators with inflatable structures because they are soft thus safe for human computer interaction. We propose a soft actuator design, called xBalloon, that is workable, inexpensive and easy-to-fabricate. It consists of daily materials including balloons and plastics and can realize bending actuation very effectively. For characterization, we fabricated xBalloon samples with different geometrical parameters and tested them regarding the bending performance and found the analytical model describing the relationship between the shape and the bending width. We then used xBalloons to animate a series of common objects and all can work satisfactorily. We further verified the user experience about the the fabrication and found that even those with no prior robotic knowledge can fabricate xBalloons with ease and confidence. Given all these advantages, we believe that xBalloon is an ideal platform for interaction design and entertainment applications.

cs.HC↗

arXiv2021

NaviChoker: Augmenting Pressure Sensation via Pneumatic Actuator

Many technologies have been developed in recent years to present audiovisual information in new ways, but developing an information presentation interface to convey tactile information is still a challenge. We propose a tactile device using wearable technology that is an all-around pressure presentation system using pneumatic actuators. Specifically, we develop a system in which a choker equipped with a pneumatic actuator is worn around the neck, that applies pressure in any direction to indicate to the user the direction in which to walk and also when to start and stop walking. In this paper, we describe the construction of the device, evaluation experiments, our assessment of the prototype, and future plans for the device.

cs.HC↗

arXiv2021

SketchMeHow: Interactive Projection Guided Task Instruction with User Sketches

In this work, we propose an interactive general instruction framework SketchMeHow to guidance the common users to complete the daily tasks in real-time. In contrast to the conventional augmented reality-based instruction systems, the proposed framework utilizes the user sketches as system inputs to acquire the users' production intentions from the drawing interfaces. Given the user sketches, the designated task instruction can be analyzed based on the sub-task division and spatial localization for each task. The projector-camera system is adopted in the projection guidance to the end-users with the spatial augmented reality technology. To verify the proposed framework, we conducted two case studies of domino arrangement and bento production. From our user studies, the proposed systems can help novice users complete the tasks efficiently with user satisfaction. We believe the proposed SketchMeHow can broaden the research topics in sketch-based real-world applications in human-computer interaction.

cs.HC↗

arXiv2021

SpatialViewer: A Remote Work Sharing Tool that Considers Intimacy Among Workers

Due to the influence of the new coronavirus disease (COVID-19), teleworking has been expanding rapidly. Although existing interactive remote working systems are convenient, they do not allow users to adjust their spatial distance to team members at will, %"Arbitrarily" is probably not the best word here. It means without apparent reason. A better expression might be "at will." and they ignore the discomfort caused by different levels of intimacy. To solve this issue, we propose a telework support system using spatial augmented reality technology. This system calibrates the space in which videos are projected with real space and adjusts the spatial distance between users by changing the position of projections. Users can switch the projection position of the video using hand-wave gestures. We also synchronize audio according to distance to further emphasize the sense of space within the remote interaction: the distance between projection position and user is inversely proportional to the audio volume. We conducted a telework experiment and a questionnaire survey to evaluate our system. The results show that the system enables users to adjust distance according to intimacy and thus improve the users' comfort.

cs.HC↗

arXiv2021

AugLimb: Compact Robotic Limb for Human Augmentation

This work proposes a compact robotic limb, AugLimb, that can augment our body functions and support the daily activities. AugLimb adopts the double-layer scissor unit for the extendable mechanism which can achieve 2.5 times longer than the forearm length. The proposed device can be mounted on the user's upper arm, and transform into compact state without obstruction to wearers. The proposed device is lightweight with low burden exerted on the wearer. We developed the prototype of AugLimb to demonstrate the proposed mechanisms. We believe that the design methodology of AugLimb can facilitate human augmentation research for practical use. see http://www.jaist.ac.jp/~xie/auglimb.html

cs.HC↗

arXiv2021

Stroke Correspondence by Labeling Closed Areas

Constructing stroke correspondences between keyframes is one of the most important processes in the production pipeline of hand-drawn inbetweening frames. This process requires time-consuming manual work imposing a tremendous burden on the animators. We propose a method to estimate stroke correspondences between raster character images (keyframes) without vectorization processes. First, the proposed system separates the closed areas in each keyframe and estimates the correspondences between closed areas by using the characteristics of shape, depth, and closed area connection. Second, the proposed system estimates stroke correspondences from the estimated closed area correspondences. We demonstrate the effectiveness of our method by performing a user study and comparing the proposed system with conventional approaches.

cs.GR↗

arXiv2021

CoreUI: Interactive Core Training System with 3D Human Shape

We present an interactive core training system for core training using a monocular camera image as input in this paper. It is commonly expensive to capture human pose using depth cameras or multiple cameras with conventional approaches. To solve this issue, we employ the skinned multi-person linear model of human shape to recover the 3D human pose from 2D images using pose estimation and human mesh recovery approaches. In order to support the user in maintaining the correct postures from target poses in the training, we adopt 3D human shape estimation for both the target image and input camera video. We propose CoreUI, a user interface for providing visual guidance showing the differences among the estimated targets and current human shapes in core training, which are visualized by markers at ten body parts with color changes. From our user studies, the proposed core training system is effective and convenient compared with the conventional guidance of 2D skeletons.

cs.HC↗

arXiv2021

Learning Perceptual Manifold of Fonts

Along the rapid development of deep learning techniques in generative models, it is becoming an urgent issue to combine machine intelligence with human intelligence to solve the practical applications. Motivated by this methodology, this work aims to adjust the machine generated character fonts with the effort of human workers in the perception study. Although numerous fonts are available online for public usage, it is difficult and challenging to generate and explore a font to meet the preferences for common users. To solve the specific issue, we propose the perceptual manifold of fonts to visualize the perceptual adjustment in the latent space of a generative model of fonts. In our framework, we adopt the variational autoencoder network for the font generation. Then, we conduct a perceptual study on the generated fonts from the multi-dimensional latent space of the generative model. After we obtained the distribution data of specific preferences, we utilize manifold learning approach to visualize the font distribution. In contrast to the conventional user interface in our user study, the proposed font-exploring user interface is efficient and helpful in the designated user preference.

cs.GR↗

arXiv2021

Direction-aware Feature-level Frequency Decomposition for Single Image Deraining

We present a novel direction-aware feature-level frequency decomposition network for single image deraining. Compared with existing solutions, the proposed network has three compelling characteristics. First, unlike previous algorithms, we propose to perform frequency decomposition at feature-level instead of image-level, allowing both low-frequency maps containing structures and high-frequency maps containing details to be continuously refined during the training procedure. Second, we further establish communication channels between low-frequency maps and high-frequency maps to interactively capture structures from high-frequency maps and add them back to low-frequency maps and, simultaneously, extract details from low-frequency maps and send them back to high-frequency maps, thereby removing rain streaks while preserving more delicate features in the input image. Third, different from existing algorithms using convolutional filters consistent in all directions, we propose a direction-aware filter to capture the direction of rain streaks in order to more effectively and thoroughly purge the input images of rain streaks. We extensively evaluate the proposed approach in three representative datasets and experimental results corroborate our approach consistently outperforms state-of-the-art deraining algorithms.

cs.CV↗