arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2610.08943

"I'm Very Happy for It to Start Hallucinating a Little Bit": Using ClayFlect to Negotiate Multimodal AI Representations in Material Meaning-Making

Kellie Yu Hui Sim·Quoc-Nam Nguyen·Shuenn Yuen Han·Kenny Tsu Wei Choo
View original source ↗Read source document ↗Export metadata ↓

Abstract

As AI enters reflection and emotional support, understanding how it can participate in personal meaning-making while preserving users' authority over interpretation is increasingly important. We present ClayFlect, a novel MLLM-powered system integrating tactile clay-making with conversational and visual generative AI, and report a mixed-methods study with 50 participants. Reflection developed across material, conversational, and generated forms rather than through AI interaction alone. Participants treated AI representations as provisional: they compared, redirected, reinterpreted, selectively incorporated, or left them aside as their artefacts and meanings evolved. Clay provided a directly manipulable space in which participants could continue developing meaning independently of the AI, while generated representations externalised possibilities beyond what they could readily make or visualise. We show how generative AI can participate through representations that remain negotiable, and derive implications for supporting movement across representations, preserving parallel sites of control, and allowing AI support to recede or deepen as reflection unfolds.

Explore related subjects

cs.HC

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Kellie Yu Hui Sim, Quoc-Nam Nguyen, Shuenn Yuen Han, Kenny Tsu Wei Choo. 2026-10-06. "I'm Very Happy for It to Start Hallucinating a Little Bit": Using ClayFlect to Negotiate Multimodal AI Representations in Material Meaning-Making. https://arxiv.org/abs/2610.08943

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Before Bringing It Up: When and How AI Companions Should Use Memory

Memory can sustain AI companionship, yet even accurate recollection can be inappropriate to use. Two rounds of formative interviews with 14 users (n = 6 exploratory, n = 8 memory-focused) motivate asking what a companion should consider before using past information. Eight themes inform Reconsider, a single-call procedure with five checks and four handling modes, evaluated on 80 scenarios across five models over 400 blinded within-model pairs. Two LLM judges favored Reconsider by net margins of +15 and +23 percentage points, with bootstrap intervals excluding zero for three of five models but not for GPT or Claude. Evaluator analysis linked judge scoring differences to model family, and a preliminary matched-guidance control isolating memory-specific content gave positive margins. We contribute an interview-grounded design framework for memory use and an evaluation that scrutinizes its own evaluators.

cs.HC↗
arXiv2026

Intent Graph: Navigating the Analytical Reasoning Space for Exploratory Data Analysis

Exploratory data analysis (EDA) is rarely open-ended in practice: analysts work from high-level domain questions toward the concrete analyses that can answer them, prioritizing directions with domain knowledge and prior hypotheses. Large language models (LLMs) can supply such knowledge, but their responses are unstructured, leaving analysts no way to see what has been explored, what is missing, or why one direction was chosen over another. We present DAG-EDA, a system that lets analysts and an LLM co-navigate the space of possible analyses through two linked structures. An intent graph, governed by a grammar of analytical intent, decomposes an ambiguous natural-language question into progressively concrete analysis tasks, keeping alternative framings open and letting analysts branch, backtrack, and compare paths. A multi-layered knowledge graph externalizes the LLM's domain knowledge, linking domain concepts to the dataset variables that can measure them, so analysts can inspect and contest how their question is grounded in the data. Both graphs are constructed from only the dataset and the analyst's question, and the analyses the analyst reaches are rendered as interactive dashboards. We illustrate the system through a usage scenario and describe a user study design for examining whether the system scaffold analysts' reasoning and navigation.

cs.HC↗
arXiv2026

Magic Pen: Automatic Pen Mode Switching for Document Annotation

Traditional digital pen interfaces use menu buttons to change the pen mode, which results in time and cognitive load spent on round-trip interactions and mode errors from tapping small mode selection buttons. This work presents the Magic Pen, a technique which uses machine learning to automatically switch between digital pen modes without requiring explicit mode changes. Magic Pen is driven by an LSTM model trained on pen data collected from 27 participants across two studies and uses transfer learning to iteratively tune the model towards how a specific user annotates. Error mitigation techniques using a flick gesture or on-screen tap are incorporated to correct mode errors or remove a stroke quickly. We evaluated Magic Pen in a comparative study with 18 participants, followed by iterative improvements and a deployment study with 8 participants. Magic Pen was preferred compared to a conventional menu-based approach, and transfer learning allowed for greater model predictability and stability.

cs.HC↗