arXiv subjects

Hongsheng Lu

Publications and source records attributed to Hongsheng Lu.

16 recordsLinked to original sources

arXiv2024

OOSTraj: Out-of-Sight Trajectory Prediction With Vision-Positioning Denoising

Trajectory prediction is fundamental in computer vision and autonomous driving, particularly for understanding pedestrian behavior and enabling proactive decision-making. Existing approaches in this field often assume precise and complete observational data, neglecting the challenges associated with out-of-view objects and the noise inherent in sensor data due to limited camera range, physical obstructions, and the absence of ground truth for denoised sensor data. Such oversights are critical safety concerns, as they can result in missing essential, non-visible objects. To bridge this gap, we present a novel method for out-of-sight trajectory prediction that leverages a vision-positioning technique. Our approach denoises noisy sensor observations in an unsupervised manner and precisely maps sensor-based trajectories of out-of-sight objects into visual trajectories. This method has demonstrated state-of-the-art performance in out-of-sight noisy sensor trajectory denoising and prediction on the Vi-Fi and JRDB datasets. By enhancing trajectory prediction accuracy and addressing the challenges of out-of-sight objects, our work significantly contributes to improving the safety and reliability of autonomous driving in complex environments. Our work represents the first initiative towards Out-Of-Sight Trajectory prediction (OOSTraj), setting a new benchmark for future research. The code is available at \url{https://github.com/Hai-chao-Zhang/OOSTraj}.

cs.CV↗

arXiv2023

SiCP: Simultaneous Individual and Cooperative Perception for 3D Object Detection in Connected and Automated Vehicles

Cooperative perception for connected and automated vehicles is traditionally achieved through the fusion of feature maps from two or more vehicles. However, the absence of feature maps shared from other vehicles can lead to a significant decline in 3D object detection performance for cooperative perception models compared to standalone 3D detection models. This drawback impedes the adoption of cooperative perception as vehicle resources are often insufficient to concurrently employ two perception models. To tackle this issue, we present Simultaneous Individual and Cooperative Perception (SiCP), a generic framework that supports a wide range of the state-of-the-art standalone perception backbones and enhances them with a novel Dual-Perception Network (DP-Net) designed to facilitate both individual and cooperative perception. In addition to its lightweight nature with only 0.13M parameters, DP-Net is robust and retains crucial gradient information during feature map fusion. As demonstrated in a comprehensive evaluation on the V2V4Real and OPV2V datasets, thanks to DP-Net, SiCP surpasses state-of-the-art cooperative perception solutions while preserving the performance of standalone perception solutions.

cs.CV↗

arXiv2023

Layout Sequence Prediction From Noisy Mobile Modality

Trajectory prediction plays a vital role in understanding pedestrian movement for applications such as autonomous driving and robotics. Current trajectory prediction models depend on long, complete, and accurately observed sequences from visual modalities. Nevertheless, real-world situations often involve obstructed cameras, missed objects, or objects out of sight due to environmental factors, leading to incomplete or noisy trajectories. To overcome these limitations, we propose LTrajDiff, a novel approach that treats objects obstructed or out of sight as equally important as those with fully visible trajectories. LTrajDiff utilizes sensor data from mobile phones to surmount out-of-sight constraints, albeit introducing new challenges such as modality fusion, noisy data, and the absence of spatial layout and object size information. We employ a denoising diffusion model to predict precise layout sequences from noisy mobile data using a coarse-to-fine diffusion strategy, incorporating the RMS, Siamese Masked Encoding Module, and MFM. Our model predicts layout sequences by implicitly inferring object size and projection status from a single reference timestamp or significantly obstructed sequences. Achieving SOTA results in randomly obstructed experiments and extremely short input experiments, our model illustrates the effectiveness of leveraging noisy mobile data. In summary, our approach offers a promising solution to the challenges faced by layout sequence and trajectory prediction models in real-world settings, paving the way for utilizing sensor data from mobile phones to accurately predict pedestrian bounding box trajectories. To the best of our knowledge, this is the first work that addresses severely obstructed and extremely short layout sequences by combining vision with noisy mobile modality, making it the pioneering work in the field of layout sequence trajectory prediction.

cs.CV↗

arXiv2023

Controlling the Latent Space of GANs through Reinforcement Learning: A Case Study on Task-based Image-to-Image Translation

Generative Adversarial Networks (GAN) have emerged as a formidable AI tool to generate realistic outputs based on training datasets. However, the challenge of exerting control over the generation process of GANs remains a significant hurdle. In this paper, we propose a novel methodology to address this issue by integrating a reinforcement learning (RL) agent with a latent-space GAN (l-GAN), thereby facilitating the generation of desired outputs. More specifically, we have developed an actor-critic RL agent with a meticulously designed reward policy, enabling it to acquire proficiency in navigating the latent space of the l-GAN and generating outputs based on specified tasks. To substantiate the efficacy of our approach, we have conducted a series of experiments employing the MNIST dataset, including arithmetic addition as an illustrative task. The outcomes of these experiments serve to validate our methodology. Our pioneering integration of an RL agent with a GAN model represents a novel advancement, holding great potential for enhancing generative networks in the future.

cs.LG↗

arXiv2023

Learning to Localize with Attention: from sparse mmWave channel estimates from a single BS to high accuracy 3D location

One strategy to obtain user location information in a wireless network operating at millimeter wave (mmWave) is based on the exploitation of the geometric relationships between the channel parameters and the user position. These relationships can be built from the line-of-sight (LOS) path and first-order reflections, or purely first-order reflections, requiring high resolution channel estimates to ensure centimeter level accuracy. In this paper, we consider a mmWave multiple-input multiple-output (MIMO) system employing a hybrid architecture, and develop a low complexity two-stage multidimensional orthogonal matching pursuit (MOMP) algorithm suitable for accurate estimation of high dimensional channels. Then, a deep neural network (DNN) called PathNet is designed to classify the order of the estimated channel paths, so that only the LOS path and first-order reflections are selected for localization. Next, a 3D localization strategy exploiting the geometry of the environment is developed to operate in both LOS and non-line-of-sight (NLOS) conditions, while considering the unknown clock offset between the transmitter (TX) and the receiver (RX). Finally, a Transformer based network exploiting attention mechanisms called ChanFormer is proposed to refine the initial position estimate obtained from geometric localization. Simulation results obtained with realistic vehicular channels indicate that localization errors below 28 cm can be achieved for 80% of the users when the LOS path is present, while sub-meter accuracy can be achieved for 55% of the users in NLOS conditions.

eess.SP↗

arXiv2023

Deep Masked Graph Matching for Correspondence Identification in Collaborative Perception

Correspondence identification (CoID) is an essential component for collaborative perception in multi-robot systems, such as connected autonomous vehicles. The goal of CoID is to identify the correspondence of objects observed by multiple robots in their own field of view in order for robots to consistently refer to the same objects. CoID is challenging due to perceptual aliasing, object non-covisibility, and noisy sensing. In this paper, we introduce a novel deep masked graph matching approach to enable CoID and address the challenges. Our approach formulates CoID as a graph matching problem and we design a masked neural network to integrate the multimodal visual, spatial, and GPS information to perform CoID. In addition, we design a new technique to explicitly address object non-covisibility caused by occlusion and the vehicle's limited field of view. We evaluate our approach in a variety of street environments using a high-fidelity simulation that integrates the CARLA and SUMO simulators. The experimental results show that our approach outperforms the previous approaches and achieves state-of-the-art CoID performance in connected autonomous driving applications. Our work is available at: https://github.com/gaopeng5/DMGM.git.

cs.RO↗

arXiv2022

ViFi-Loc: Multi-modal Pedestrian Localization using GAN with Camera-Phone Correspondences

In Smart City and Vehicle-to-Everything (V2X) systems, acquiring pedestrians' accurate locations is crucial to traffic safety. Current systems adopt cameras and wireless sensors to detect and estimate people's locations via sensor fusion. Standard fusion algorithms, however, become inapplicable when multi-modal data is not associated. For example, pedestrians are out of the camera field of view, or data from camera modality is missing. To address this challenge and produce more accurate location estimations for pedestrians, we propose a Generative Adversarial Network (GAN) architecture. During training, it learns the underlying linkage between pedestrians' camera-phone data correspondences. During inference, it generates refined position estimations based only on pedestrians' phone data that consists of GPS, IMU and FTM. Results show that our GAN produces 3D coordinates at 1 to 2 meter localization error across 5 different outdoor scenes. We further show that the proposed model supports self-learning. The generated coordinates can be associated with pedestrian's bounding box coordinates to obtain additional camera-phone data correspondences. This allows automatic data collection during inference. After fine-tuning on the expanded dataset, localization accuracy is improved by up to 26%.

cs.CV↗

arXiv2022

Joint Initial Access and Localization in Millimeter Wave Vehicular Networks: a Hybrid Model/Data Driven Approach

High resolution compressive channel estimation provides information for vehicle localization when a hybrid mmWave MIMO system is considered. Complexity and memory requirements can, however, become a bottleneck when high accuracy localization is required. An additional challenge is the need of path order information to apply the appropriate geometric relationships between the channel path parameters and the vehicle, RSU and scatterers position. In this paper, we propose a low complexity channel estimation strategy of the angle of departure and time difference of arrival based on multidimensional orthogonal matching pursuit. We also design a deep neural network that predicts the order of the channel paths so only the LoS and first order reflections are used for localization. Simulation results obtained with realistic vehicular channels generated by ray tracing show that sub-meter accuracy can be achieved for 50% of the users, without resorting to perfect synchronization assumptions or unfeasible all-digital high resolution MIMO architectures.

eess.SP↗

arXiv2022

Adaptive Neural Network-based OFDM Receivers

We propose and examine the idea of continuously adapting state-of-the-art neural network (NN)-based orthogonal frequency division multiplex (OFDM) receivers to current channel conditions. This online adaptation via retraining is mainly motivated by two reasons: First, receiver design typically focuses on the universal optimal performance for a wide range of possible channel realizations. However, in actual applications and within short time intervals, only a subset of these channel parameters is likely to occur, as macro parameters, e.g., the maximum channel delay, can assumed to be static. Second, in-the-field alterations like temporal interferences or other conditions out of the originally intended specifications can occur on a practical (real-world) transmission. While conventional (filter-based) systems would require reconfiguration or additional signal processing to cope with these unforeseen conditions, NN-based receivers can learn to mitigate previously unseen effects even after their deployment. For this, we showcase on-the-fly adaption to current channel conditions and temporal alterations solely based on recovered labels from an outer forward error correction (FEC) code without any additional piloting overhead. To underline the flexibility of the proposed adaptive training, we showcase substantial gains for scenarios with static channel macro parameters, for out-of-specification usage and for interference compensation.

cs.IT↗

arXiv2021

The Transdimensional Poisson Process for Vehicular Network Analysis

A comprehensive vehicular network analysis requires modeling the street system and vehicle locations. Even when Poisson point processes (PPPs) are used to model the vehicle locations on each street, the analysis is barely tractable. That holds for even a simple average-based performance metric -- the success probability, which is a special case of the fine-grained metric, the meta distribution (MD) of the signal-to-interference ratio (SIR). To address this issue, we propose the transdimensional approach as an alternative. Here, the union of 1D PPPs on the streets is simplified to the transdimensional PPP (TPPP), a superposition of 1D and 2D PPPs. The TPPP includes the 1D PPPs on the streets passing through the receiving vehicle and models the remaining vehicles as a 2D PPP ignoring their street geometry. Through the SIR MD analysis, we show that the TPPP provides good approximations to the more cumbrous models with streets characterized by Poisson line/stick processes; and we prove that the accuracy of the TPPP further improves under shadowing. Lastly, we use the MD results to control network congestion by adjusting the transmit rate while maintaining a target fraction of reliable links. A key insight is that the success probability is an inadequate measure of congestion as it does not capture the reliabilities of the individual links.

cs.NI↗

arXiv2021

Asynchronous Collaborative Localization by Integrating Spatiotemporal Graph Learning with Model-Based Estimation

Collaborative localization is an essential capability for a team of robots such as connected vehicles to collaboratively estimate object locations from multiple perspectives with reliant cooperation. To enable collaborative localization, four key challenges must be addressed, including modeling complex relationships between observed objects, fusing observations from an arbitrary number of collaborating robots, quantifying localization uncertainty, and addressing latency of robot communications. In this paper, we introduce a novel approach that integrates uncertainty-aware spatiotemporal graph learning and model-based state estimation for a team of robots to collaboratively localize objects. Specifically, we introduce a new uncertainty-aware graph learning model that learns spatiotemporal graphs to represent historical motions of the objects observed by each robot over time and provides uncertainties in object localization. Moreover, we propose a novel method for integrated learning and model-based state estimation, which fuses asynchronous observations obtained from an arbitrary number of robots for collaborative localization. We evaluate our approach in two collaborative object localization scenarios in simulations and on real robots. Experimental results show that our approach outperforms previous methods and achieves state-of-the-art performance on asynchronous collaborative localization.

cs.RO↗

arXiv2021

Wiener Filter versus Recurrent Neural Network-based 2D-Channel Estimation for V2X Communications

We compare the potential of neural network (NN)-based channel estimation with classical linear minimum mean square error (LMMSE)-based estimators, also known as Wiener filtering. For this, we propose a low-complexity recurrent neural network (RNN)-based estimator that allows channel equalization of a sequence of channel observations based on independent time- and frequency-domain long short-term memory (LSTM) cells. Motivated by Vehicle-to-Everything (V2X) applications, we simulate time- and frequency-selective channels with orthogonal frequency division multiplex (OFDM) and extend our channel models in such a way that a continuous degradation from line-of-sight (LoS) to non-line-of-sight (NLoS) conditions can be emulated. It turns out that the NN-based system cannot just compete with the LMMSE equalizer, but it also can be trained w.r.t. resilience against system parameter mismatch. We thereby showcase the conceptual simplicity of such a data-driven system design, as this not only enables more robustness against, e.g., signal-to-noise-ratio (SNR) or Doppler spread estimation mismatches, but also allows to use the same equalizer over a wider range of input parameters without the need of re-building (or re-estimating) the filter coefficients. Particular attention has been paid to ensure compatibility with the existing IEEE 802.11p piloting scheme for V2X communications. Finally, feeding the payload data symbols as additional equalizer input unleashes further performance gains. We show significant gains over the conventional LMMSE equalization for highly dynamic channel conditions if such a data-augmented equalization scheme is used.

cs.IT↗

arXiv2020

Multi-view Sensor Fusion by Integrating Model-based Estimation and Graph Learning for Collaborative Object Localization

Collaborative object localization aims to collaboratively estimate locations of objects observed from multiple views or perspectives, which is a critical ability for multi-agent systems such as connected vehicles. To enable collaborative localization, several model-based state estimation and learning-based localization methods have been developed. Given their encouraging performance, model-based state estimation often lacks the ability to model the complex relationships among multiple objects, while learning-based methods are typically not able to fuse the observations from an arbitrary number of views and cannot well model uncertainty. In this paper, we introduce a novel spatiotemporal graph filter approach that integrates graph learning and model-based estimation to perform multi-view sensor fusion for collaborative object localization. Our approach models complex object relationships using a new spatiotemporal graph representation and fuses multi-view observations in a Bayesian fashion to improve location estimation under uncertainty. We evaluate our approach in the applications of connected autonomous driving and multiple pedestrian localization. Experimental results show that our approach outperforms previous techniques and achieves the state-of-the-art performance on collaboration localization.

cs.CV↗

arXiv2020

Feature Sharing and Integration for Cooperative Cognition and Perception with Volumetric Sensors

The recent advancement in computational and communication systems has led to the introduction of high-performing neural networks and high-speed wireless vehicular communication networks. As a result, new technologies such as cooperative perception and cognition have emerged, addressing the inherent limitations of sensory devices by providing solutions for the detection of partially occluded targets and expanding the sensing range. However, designing a reliable cooperative cognition or perception system requires addressing the challenges caused by limited network resources and discrepancies between the data shared by different sources. In this paper, we examine the requirements, limitations, and performance of different cooperative perception techniques, and present an in-depth analysis of the notion of Deep Feature Sharing (DFS). We explore different cooperative object detection designs and evaluate their performance in terms of average precision. We use the Volony dataset for our experimental study. The results confirm that the DFS methods are significantly less sensitive to the localization error caused by GPS noise. Furthermore, the results attest that detection gain of DFS methods caused by adding more cooperative participants in the scenes is comparable to raw information sharing technique while DFS enables flexibility in design toward satisfying communication requirements.

cs.CV↗

arXiv2020

Securing Vehicle-to-Everything (V2X) Communication Platforms

Modern vehicular wireless technology enables vehicles to exchange information at any time, from any place, to any network -- forms the vehicle-to-everything (V2X) communication platforms. Despite benefits, V2X applications also face great challenges to security and privacy -- a very valid concern since breaches are not uncommon in automotive communication networks and applications. In this survey, we provide an extensive overview of V2X ecosystem. We also review main security/privacy issues, current standardization activities and existing defense mechanisms proposed within the V2X domain. We then identified semantic gaps of existing security solutions and outline possible open issues.

cs.NI↗

arXiv2018

Performance and Scaling of Collaborative Sensing and Networking for Automated Driving Applications

A critical requirement for automated driving systems is enabling situational awareness in dynamically changing environments. To that end vehicles will be equipped with diverse sensors, e.g., LIDAR, cameras, mmWave radar, etc. Unfortunately the sensing `coverage' is limited by environmental obstructions, e.g., other vehicles, buildings, people, objects etc. A possible solution is to adopt collaborative sensing amongst vehicles possibly assisted by infrastructure. This paper introduces new models and performance analysis for vehicular collaborative sensing and networking. In particular, coverage gains are quantified, as are their dependence on the penetration of vehicles participating in collaborative sensing. We also evaluate the associated communication loads in terms of the Vehicle-to-Vehicle (V2V) and Vehicle-to-Infrastructure (V2I) capacity requirements and how these depend on penetration. We further explore how collaboration with sensing capable infrastructure improves sensing performance, as well as the benefits in utilizing spatio-temporal dynamics, e.g., collaborating with vehicles moving in the opposite direction. Collaborative sensing is shown to greatly improve sensing performance, e.g., improves coverage from 20% to 80% with a 20% penetration. In scenarios with limited penetration and high coverage requirements, infrastructure can be used to both help sense the environment and relay data. Once penetration is high enough, sensing vehicles provide good coverage and data traffic can be effectively `offloaded' to V2V connectivity, making V2I resources available to support other in-car services.

eess.SP↗