arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2610.25343

UnexpectedParrots: Iterating Behavioral Experiment Design with In-Silico Simulations

Sanchaita Hazra·Bodhisattwa Prasad Majumder·Pao Siangliulue·Peter Clark·Amy X. Zhang·Joseph Chang
View original source ↗Read source document ↗Export metadata ↓

Abstract

Running behavioral experiments is expensive, slow, and prone to design flaws, often detectable after collecting data. We introduce UnexpectedParrots to iteratively refine behavioral study designs before a researcher recruits costly human participants. UnexpectedParrots relies on reasoning traces in addition to responses from LLM participants to uncover diagnostic critiques of structural errors and missed opportunities in existing study designs. When evaluated against a human-pilot or a no-simulation design-only baseline, UnexpectedParrots adds most value, producing unique, high-quality critiques. In a moderated user study with 12 proficient researchers, we find UnexpectedParrots useful, time-saving, and thought-provoking. Critiques provided by UnexpectedParrots often converge on issues researchers identify only during independent human pilots or additional formative interviews, even when the fidelity of the simulated populations remains questionable. Our users deem UnexpectedParrots useful not only as a tool to conduct early pilot studies and identify errors but also as an ideation tool to expand one's research horizon.

Explore related subjects

cs.HC

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Sanchaita Hazra, Bodhisattwa Prasad Majumder, Pao Siangliulue, Peter Clark, Amy X. Zhang, Joseph Chang. 2026-10-18. UnexpectedParrots: Iterating Behavioral Experiment Design with In-Silico Simulations. https://arxiv.org/abs/2610.25343

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

FlexMind: Supporting Deeper Creative Thinking with LLMs

Expert ideation requires more than generating ideas. People must articulate trade-offs, develop mitigations, and compare parallel directions, yet this reasoning process is cognitively demanding. Existing ideation tools fall short by either focusing only on expanding or organizing the idea space, or by offering evaluations without supporting mitigation generation and continued reasoning over newly introduced trade-offs. We present FlexMind, an AI-augmented system that proactively scaffolds multi-threaded trade-off-mitigation chains within a structured workspace. FlexMind helps users surface concrete trade-offs, generate actionable mitigations, and navigate and expand multiple idea branches. In a controlled study (n=20), FlexMind supported deeper exploration without sacrificing breadth and produced higher-quality ideas, as judged by domain experts, compared to a baseline. A case study with six experts (n=6) further reveals how FlexMind supports idea analysis while helping uncover new solutions. These results inform implications for enhancing human-AI co-creation.

cs.HC↗
arXiv2026

ZORO: Active Controls to Support Rule Following in Vibe Coding

Rules files (e.g., AGENTS$.$md, CLAUDE$.$md) are the primary mechanism for human-agent alignment when developers vibe code. However, they remain passive: it is not immediately apparent when rules are being used or followed, or how to improve them. To transform rules from passive text into active controls, we introduce ZORO, an interactive interface that integrates directly with a coding agent and anchors rules to every step of the coding process. After an agent generates an initial plan, ZORO enriches the plan with rules, enforces the rules during implementation by requiring the agent prove that each rule was followed, and allows users to provide in-situ feedback when they are unsatisfied with a rule application to evolve the ruleset. A technical evaluation shows that coding agents follow rules more with ZORO than without. A user study demonstrates a change in people's behavior and cognitive strategies when rules are at the forefront of vibe coding. We discuss how making rules active in agentic systems unlocks broader opportunities for human-agent alignment in coding settings and beyond.

cs.HC↗
arXiv2026

Calibrating LLM Judges for Human and AI Conversations

Measuring how successful a conversation is remains difficult, even for humans judging spoken dialogue. We evaluate state-of-the-art LLMs as pointwise and pairwise judges of conversational success on CANDOR, finding pointwise scoring correlates moderately with human ratings, while pairwise comparison suffers from long transcripts and positional bias. Since this leaves judge scores incomparable across models, we propose a small anchor set and a calibration function that calibrates any judge onto a shared, interpretable scale. We further release the Voice Arena Goal Dataset (VA), 200 task-oriented human-AI and human-agent conversations with pairwise annotations, revealing a substantial gap between current judges and human-level discrimination. Using VA, we test whether CANDOR-fitted calibration transfers to human-AI conversations, finding it brings judges onto a shared scale despite never observing VA during fitting.

cs.HC↗