arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2610.25006

Scalable, Intuitive, Mixed Reality-Based Semi-Autonomous Robotic Telemanipulation for the Execution of a Fine-Grained Bimanual Task

Shifei Duan·Zhe Wang·Francesco De Pace·Minas Liarokapis
View original source ↗Read source document ↗Export metadata ↓

Abstract

Nowadays, robotic telemanipulation enables humans to control robots so as to execute tasks in a manner similar to how they perform them, and serves as an essential approach for collecting on-robot data for robot learning by human demonstrations. The quality of collected demonstrations directly impacts the performance of the learned policy. However, existing telemanipulation systems rely heavily on human operation, particularly for large-scale data collection. Collecting dozens or hundreds of demonstrations is a demanding process, as operators tend to fatigue quickly, especially during the execution of dexterous tasks that require sustained concentration. Recently, semi-autonomous robotic telemanipulation has emerged as a promising direction that reduces operator workload while preserving human-in-the-loop control for task execution. We therefore introduce a scalable, intuitive, semi-autonomous robotic telemanipulation system that leverages a mixed reality (MR) headset for on-robot data collection. The system autonomously executes the majority of the task trajectories and only requires the operator to make fine adjustments at key stages. We validate the effectiveness of our system by collecting data and training imitation learning policies on the execution of a high-precision, fine-grained bimanual peg-in-hole task. With lower data collection effort, our system obtains high-quality demonstrations that can be used to train fully autonomous task execution policies.

Explore related subjects

cs.RO

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Shifei Duan, Zhe Wang, Francesco De Pace, Minas Liarokapis. 2026-10-18. Scalable, Intuitive, Mixed Reality-Based Semi-Autonomous Robotic Telemanipulation for the Execution of a Fine-Grained Bimanual Task. https://arxiv.org/abs/2610.25006

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

AnyNav: Visual Neuro-Symbolic Friction Learning for Off-road Navigation

Off-road navigation is critical for a wide range of field robotics applications from planetary exploration to disaster response. However, it remains a longstanding challenge due to unstructured environments and the inherently complex terrain-vehicle interactions. Traditional physics-based methods struggle to accurately capture the nonlinear dynamics underlying these interactions, while purely data-driven approaches often overfit to specific motion patterns, vehicle geometries, or platforms, limiting their generalization in diverse, real-world scenarios. To address these limitations, we introduce AnyNav, a vision-based friction estimation and navigation framework grounded in neuro-symbolic principles. Our approach integrates neural networks for visual perception with symbolic physical models for reasoning about terrain-vehicle dynamics. To enable self-supervised learning in real-world settings, we adopt the imperative learning paradigm, employing bilevel optimization to train the friction network through physics-based optimization. This explicit incorporation of physical reasoning substantially enhances generalization across terrains, vehicle types, and operational conditions. Leveraging the predicted friction coefficients, we further develop a physics-informed navigation system capable of generating physically feasible, time-efficient paths together with corresponding speed profiles. We demonstrate that AnyNav seamlessly transfers from simulation to real-world robotic platforms, exhibiting strong robustness across different four-wheeled vehicles and diverse off-road environments.

cs.RO↗
arXiv2026

LLM-Flock: LLM-Driven Decentralized Multi-Robot Flocking via Influence-Based Consensus with Collision Avoidance

Large Language Models (LLMs) have shown strong capabilities in reasoning and planning, prompting researchers to explore their use as decentralized decision-makers for multi-robot formation control. However, directly applying LLMs to such tasks often leads to unstable behavior, in which robots may collapse to the centroid of their positions or diverge entirely, due to hallucinated reasoning and limited coordination awareness. We propose LLM-Flock, a decentralized multi-robot flocking framework that addresses these challenges through two key mechanisms: (1) an influence-based consensus protocol, where robots iteratively refine individually generated formation plans by adopting those of more connected neighbors, driving the team toward a unified plan without centralized control, and (2) a two-layer collision avoidance strategy that pairs LLM-based urgency reasoning with a programmatic collision checker and re-prompting mechanism. We evaluate LLM-Flock across five state-of-the-art LLMs, including both closed-source (GPT-5, Claude 4.5) and open-source (Llama3.1-405b, Qwen-Max, DeepSeek-R1) models, demonstrating notable improvements in formation stability and convergence over prior LLM-based methods. Real-world experiments with Crazyflie drones further validate the framework's applicability to physical multi-robot systems.

cs.RO↗
arXiv2026

RoboPocket: Improve Robot Policies Instantly with Your Phone

Scaling robot learning is bottlenecked not only by data volume, but also by whether the data targets the policy's failures. Existing robot-free handheld interfaces enable portable in-the-wild data collection, yet remain open-loop: users record demonstrations without feedback from the current policy. Interactive methods such as DAgger provide such feedback by collecting corrective on-policy data, but require deploying imperfect policies on physical robots, making them costly, unsafe, and difficult to scale. We present RoboPocket, a smartphone-based system for robot-free policy iteration. By combining on-device sensing, display, and communication with cloud-side policy inference, RoboPocket provides real-time collection guidance and visualizes the policy's predicted trajectory in augmented reality as AR Visual Foresight. This allows users to identify policy weaknesses and collect targeted corrective data without deploying a physical robot. An asynchronous finetuning pipeline continuously incorporates new corrections and updates the policy within minutes, realizing a DAgger-inspired robot-free correction loop. Across diverse real-world manipulation tasks, we validate RoboPocket as a scalable data engine and show that online corrective data collection improves sample efficiency by up to $2\times$ over scaling offline demonstrations. Project page and videos: https://robo-pocket.github.io.

cs.RO↗