Human Thinking under Plural LLM Assistance: Mathematical Problem Solving and Open-Ended Writing
Large language models are changing not only the kind of assistance people receive, but also how that assistance is organized. Instead of working with a single general-purpose chatbot, people can now receive help from systems arranged as peers, specialists, or multiple agents with distinct roles. However, it remains unclear how these forms of plural LLM assistance affect human performance, confidence, and diversity of thought. We conducted two controlled experiments involving 562 participants to examine the effects of using multiple LLMs on mathematical problem-solving and writing. In a math task, participants worked with no LLM, an expert assistant, peer-like agents that surfaced common errors, or both an expert and a peer-like assistant. The expert-plus-peer condition produced the strongest unassisted post-task performance. In a writing task, participants wrote with no LLM, a single generalist assistant, or a pair of role-specialized assistants. LLM assistance improved essay quality, but the role-specialized pair preserved greater idea diversity than the single assistant. Together, these findings identify the arrangement of LLM assistance as a consequential design variable for human-AI collaboration.