ReLI: Cross-Lingual Language-to-Action Grounding for Human-Robot Interaction

Abstract

Adapting autonomous agents for real-world industrial, domestic, and other daily tasks is currently gaining momentum. However, in global or cross-lingual application contexts, the ability to instruct these agents in one's native language remains until today a formidable challenge. Existing language-conditioned human-robot interaction frameworks typically support only a handful of high-resource languages, e.g., English and Chinese, limiting accessibility for billions of potential end users. To address this gap, we propose ReLI, a cross-lingual framework that enables autonomous agents to converse naturally, reason semantically about their environment, and execute downstream tasks, regardless of the tasks' instruction linguistic origin or input modalities. We ground large-scale pre-trained foundation models and transform them into language-to-action models that can directly provide common-sense reasoning and high-level robot control through free-form conversational interactions. We then perform an implicit language-conditioned cross-lingual adaptation of the models to ensure that ReLI generalises effectively across diverse global languages. We conducted extensive empirical evaluation on a diverse set of short- and long-horizon tasks, including zero-shot and few-shot spatial navigation, scene information retrieval, and query-oriented tasks, and then benchmarked the performance across more than $70K+$ multi-turn conversations in over $140$ languages spanning high-resource, low-resource, and vulnerable/creole tiers. Across the benchmarked languages, ReLI achieved consistently high instruction-parsing accuracy, task success rate, and rapid response time. Further, we c..