arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · cs/0505064

Multi-Modal Human-Machine Communication for Instructing Robot Grasping Tasks

P. C. McGuire·J. Fritsch·J. J. Steil·F. Roethling·G. A. Fink·S. Wachsmuth·G. Sagerer·H. Ritter
View original source ↗Read source document ↗Export metadata ↓

Abstract

A major challenge for the realization of intelligent robots is to supply them with cognitive abilities in order to allow ordinary users to program them easily and intuitively. One way of such programming is teaching work tasks by interactive demonstration. To make this effective and convenient for the user, the machine must be capable to establish a common focus of attention and be able to use and integrate spoken instructions, visual perceptions, and non-verbal clues like gestural commands. We report progress in building a hybrid architecture that combines statistical methods, neural networks, and finite state machines into an integrated system for instructing grasping tasks by man-machine interaction. The system combines the GRAVIS-robot for visual attention and gestural instruction with an intelligent interface for speech recognition and linguistic interpretation, and an modality fusion module to allow multi-modal task-oriented man-machine communication with respect to dextrous robot manipulation of objects.

Explore related subjects

cs.HCcs.AIcs.CVcs.LGcs.RO

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

P. C. McGuire, J. Fritsch, J. J. Steil, F. Roethling, G. A. Fink, S. Wachsmuth, G. Sagerer, H. Ritter. 2005-05-24. Multi-Modal Human-Machine Communication for Instructing Robot Grasping Tasks. https://doi.org/10.1109/irds.2002.1043875

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Smell with Genji: Rediscovering Sensory Experience with an AI Co-Smelling Partner

Olfaction contributes substantially to human perception, yet the olfactory-verbal gap often leaves people struggling to articulate what they smell. We present Smell with Genji, an AI-mediated interactive experience adapting Genji-kō, a traditional Japanese scent-matching game. This practice maps comparative judgments across a sequence of scents onto Genji-mon, providing visual patterns as shared reference points for communication. Within this framework, we introduce an AI co-smelling partner integrating multichannel olfactory sensing, a Transformer-based classification model, and a large language model (LLM)-driven conversational interface to translate sensor signals into descriptive dialogue. Rather than deploying machine olfaction purely for identification, the system positions the AI as a co-present partner sharing its sensory impressions while human judgment remains open. Placing the machine's perspective alongside human intuition creates a space for shared reflection, encouraging participants to deliberately evaluate and articulate their choices. This work explores how computational systems can support sensory interaction grounded in dialogue, fostering reflective articulation and preserving human agency in making sense of the olfactory world.

cs.HC↗
arXiv2026

AromaGen: Interactive Generation of Rich Olfactory Experiences with Multimodal Language Models

Smell's connections with food, memory, and social experience have motivated researchers to bring olfaction into interactive systems. Multimodal AI opens new possibilities for generating smells from natural language, yet it remains unclear whether pretrained models encode sufficient olfactory knowledge to translate language into perceptible compositions. We present AromaGen, an AI-powered 12-channel wearable olfactory system that maps free-form natural-language descriptions to 12 base odorants selected to cover a semantically derived olfactory space, while supporting iterative refinement through natural-language feedback. In a between-subjects study (N=60) across 50 real-world benchmark smells, participants distinguished AromaGen-generated target smells above chance in a three-alternative forced-choice task, with retrieval-augmented AI composition performing comparably to human-expert composition and zero-shot AI composition. Natural-language feedback significantly improved the perceived similarity of both AI- and human-expert compositions. Our findings demonstrate the feasibility of using pretrained multimodal AI, with human aroma composition data, to generate perceptible olfactory experiences from natural language.

cs.HC↗
arXiv2026

Stayin' Aligned Over Time: Towards Longitudinal Human-LLM Alignment via Contextual Reflection and Privacy-Preserving Behavioral Data

Current human-AI alignment and evaluation methods for large language models (LLMs) often rely on preference signals collected immediately after an interaction. This practice implicitly treats preference as static, even though many LLM-mediated decisions unfold over time and may be re-evaluated differently after real-world consequences and observed outcomes. Therefore, we argue for a methodological shift from single-moment preference elicitation to longitudinal, context-situated alignment measurement. We present a methodological framework for collecting temporally grounded alignment signals by combining (1) in-situ preference capture, (2) context-triggered follow-up preference reflection, and (3) privacy-preserving behavioral traces that help interpret preference change. As an instantiation of this methodology, we introduce BITE, a browser-based system that detects consequential LLM interactions, prompts reflection across later decision points, and supports progressive, user-controlled consent for sharing behavioral data. Through a two week longitudinal deployment study with 8 participants, our approach surfaced differences between immediate and later user preferences in accuracy, relevance and other dimensions of the LLM output. Our findings highlight the limitations of single-moment preference datasets and underscore the importance of longitudinal methods for alignment evaluation in everyday use.

cs.HC↗