VibeJam: A User Study Platform for Web Development with Agents

Nishant Balepur; Connor Baumler; Valerie Chen; Eunsol Choi; Rachel Rudinger; Jordan Boyd-Graber

Home/Archive/arXiv

arXiv · 2608.29889

VibeJam: A User Study Platform for Web Development with Agents

Nishant Balepur·Connor Baumler·Valerie Chen·Eunsol Choi·Rachel Rudinger·Jordan Boyd-Graber

View original source ↗Read source document ↗Export metadata ↓

Abstract

Programming with AI is increasingly agentic, users prompt LLMs to directly edit their code and review the changes, with adoption growing especially for web development tasks. Despite this growth, most NLP work uses offline evaluation and lacks support for online studies, losing insights into how programmers truly use coding agents. We release VibeJam, a browser-based user study platform for users to collaborate with AI agents to develop websites. VibeJam enables agent customization and uses the open-source Aider agent by default, and to mirror downstream use, we add diff review, chat and plan modes, and live website previews. In a pilot study with 55 released, game-based website creation tasks, five experienced AI programmers rate our system as fun, simple, and resembling commercial tools, while 13 junior students use VibeJam to make websites of higher quality than agents in the same task. We open-source VibeJam to spur extensions and support studies on how coding agents can help users.

Explore related subjects

cs.CL

Keep this discovery

BibTeX RIS

Nishant Balepur, Connor Baumler, Valerie Chen, Eunsol Choi, Rachel Rudinger, Jordan Boyd-Graber. 2026-08-30. VibeJam: A User Study Platform for Web Development with Agents. https://arxiv.org/abs/2608.29889

Cite the original work for its findings. Save a collection to share your selection of sources.

Discover connections

cs.CL subject

Connections use source metadata and explicit phrase matches, not verified experimental comparisons.

KEEP EXPLORING

Related papers

arXiv2024

Density Matrices for Metaphor Understanding

In physics, density matrices are used to represent mixed states, i.e. probabilistic mixtures of pure states. This concept has previously been used to model lexical ambiguity. In this paper, we consider metaphor as a type of lexical ambiguity, and examine whether metaphorical meaning can be effectively modelled using mixtures of word senses. We find that modelling metaphor is significantly more difficult than other kinds of lexical ambiguity, but that our best-performing density matrix method outperforms simple baselines as well as some neural language models.

cs.CL↗

arXiv2026

Informational Antilocality and the Locality Bias in LLMs

We consider the ability of transformer-based language models (LLMs) to learn what we call k-antilocal languages, i.e., languages that have no mutual information across any span of $k$ contiguous symbols. We construct such languages with increasing $k$, finding that LLMs trained on them achieve comparable cross-entropy loss regardless of antilocality, but converge more slowly on more antilocal languages. Our findings support the idea that non-local dependencies are more difficult to learn, but the evidence for this bias comes from learning speed rather than learning success.

cs.CL↗

arXiv2026

FLAME: A New Dataset on FLemish Accounts of Momentary Experiences

We introduce FLAME (FLemish Accounts of Momentary Experiences), a corpus of nearly 25,000 personal narratives in Belgian-Dutch (Flemish), collected through experience sampling to support Natural Language Processing (NLP) research on an underrepresented variety. Such everyday narratives are rich in culturally grounded themes, but their informal register and low-resource setting make thematic extraction hard. Comparing K-Means, LDA, and BERTopic, we find that human evaluation favors BERTopic, which produces the most coherent, culturally resonant topics. FLAME, thereby, offers a new resource for studying everyday language use in a low-resource variety.

cs.CL↗