arXiv subjects

Lisha Yu

Publications and source records attributed to Lisha Yu.

3 recordsLinked to original sources

arXiv2025

UNeMo: Collaborative Visual-Language Reasoning and Navigation via a Multimodal World Model

Vision-and-Language Navigation (VLN) requires agents to autonomously navigate complex environments via visual images and natural language instructions--remains highly challenging. Recent research on enhancing language-guided navigation reasoning using pre-trained large language models (LLMs) has shown promising prospects. However, the reasoning of such methods is limited to the linguistic modality, lacking visual reasoning capabilities. Moreover, existing reasoning modules are optimized separately from navigation policies, leading to incompatibility and potential conflicts in optimization objectives.To tackle these challenges, we introduce UNeMo, a novel framework designed for the collaborative optimization of visual state reasoning and navigational decision-making. It introduces a Multimodal World Model (MWM) that takes visual features, language instructions, and navigational actions as inputs to jointly predict subsequent visual states, enabling cross-modal reasoning. Via a Hierarchical Prediction-Feedback (HPN) mechanism, MWM collaborates with navigation policies: the first layer generates actions using current vision-and-language features; MWM then infers post-action visual states to guide the second layer's fine-grained decisions. This forms a dynamic bidirectional promotion mechanism where MWM reasoning optimizes navigation policies, while policy decisions feedback to improve MWM's reasoning accuracy. Experiments on R2R and REVERIE datasets show UNeMo outperforms state-of-the-art methods by 2.1% and 0.7% in navigation accuracy for unseen scenes, validating its effectiveness.

cs.AI↗

arXiv2024

MC-GPT: Empowering Vision-and-Language Navigation with Memory Map and Reasoning Chains

In the Vision-and-Language Navigation (VLN) task, the agent is required to navigate to a destination following a natural language instruction. While learning-based approaches have been a major solution to the task, they suffer from high training costs and lack of interpretability. Recently, Large Language Models (LLMs) have emerged as a promising tool for VLN due to their strong generalization capabilities. However, existing LLM-based methods face limitations in memory construction and diversity of navigation strategies. To address these challenges, we propose a suite of techniques. Firstly, we introduce a method to maintain a topological map that stores navigation history, retaining information about viewpoints, objects, and their spatial relationships. This map also serves as a global action space. Additionally, we present a Navigation Chain of Thoughts module, leveraging human navigation examples to enrich navigation strategy diversity. Finally, we establish a pipeline that integrates navigational memory and strategies with perception and action prediction modules. Experimental results on the REVERIE and R2R datasets show that our method effectively enhances the navigation ability of the LLM and improves the interpretability of navigation reasoning.

cs.AI↗

arXiv2019

Monitoring dynamic networks: a simulation-based strategy for comparing monitoring methods and a comparative study

Recently there has been a lot of interest in monitoring and identifying changes in dynamic networks, which has led to the development of a variety of monitoring methods. Unfortunately, these methods have not been systematically compared; moreover, new methods are often designed for a specialized use case. In light of this, we propose the use of simulation to compare the performance of network monitoring methods over a variety of dynamic network changes. Using our family of simulated dynamic networks, we compare the performance of several state-of-the-art social network monitoring methods in the literature. We compare their performance over a variety of types of change; we consider both increases in communication levels, node propensity change as well as changes in community structure. We show that there does not exist one method that is uniformly superior to the others; the best method depends on the context and the type of change one wishes to detect. As such, we conclude that a variety of methods is needed for network monitoring and that it is important to understand in which scenarios a given method is appropriate.

stat.CO↗