arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2610.25244

It's Complicated: Factors Influencing the Effectiveness of LLM-Generated Feedback in a Live Programming Course

Tsvetomila Mihaylova·Arto Hellas·Bita Akram·Narges Norouzi·Peter Brusilovsky·Juho Leinonen
View original source ↗Read source document ↗Export metadata ↓

Abstract

We report on three iterations of an LLM-generated feedback system deployed in a live web development course, spanning three underlying models and three feedback formats. We ask how feedback presence and its characteristics (length, complexity, format) relate to student performance, whether prior feedback exposure is associated with final-project performance, and how performance changes after first exposure. Feedback was associated with more favorable immediate outcomes in several comparisons, but the pattern varied by outcome and period. In pooled models, longer feedback was associated with lower next-submission correctness, more remaining attempts, and longer resubmission time. Prior same-part feedback was associated with lower final-project solve probability. Student-level comparisons found differences on final-project steps begun before exposure, while within-student changes across steps were uncertain. These comparisons do not identify the causal effect of feedback. Changes on standard exercises were more favorable among exposed students, but the groups' different course-stage splits limit causal interpretation. Bounded message length and student choice of format are directions for experimental evaluation. Our final-project analysis captures follow-up performance rather than directly measuring learning or understanding.

Explore related subjects

cs.HC

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Tsvetomila Mihaylova, Arto Hellas, Bita Akram, Narges Norouzi, Peter Brusilovsky, Juho Leinonen. 2026-10-18. It's Complicated: Factors Influencing the Effectiveness of LLM-Generated Feedback in a Live Programming Course. https://arxiv.org/abs/2610.25244

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

FlexMind: Supporting Deeper Creative Thinking with LLMs

Expert ideation requires more than generating ideas. People must articulate trade-offs, develop mitigations, and compare parallel directions, yet this reasoning process is cognitively demanding. Existing ideation tools fall short by either focusing only on expanding or organizing the idea space, or by offering evaluations without supporting mitigation generation and continued reasoning over newly introduced trade-offs. We present FlexMind, an AI-augmented system that proactively scaffolds multi-threaded trade-off-mitigation chains within a structured workspace. FlexMind helps users surface concrete trade-offs, generate actionable mitigations, and navigate and expand multiple idea branches. In a controlled study (n=20), FlexMind supported deeper exploration without sacrificing breadth and produced higher-quality ideas, as judged by domain experts, compared to a baseline. A case study with six experts (n=6) further reveals how FlexMind supports idea analysis while helping uncover new solutions. These results inform implications for enhancing human-AI co-creation.

cs.HC↗
arXiv2026

ZORO: Active Controls to Support Rule Following in Vibe Coding

Rules files (e.g., AGENTS$.$md, CLAUDE$.$md) are the primary mechanism for human-agent alignment when developers vibe code. However, they remain passive: it is not immediately apparent when rules are being used or followed, or how to improve them. To transform rules from passive text into active controls, we introduce ZORO, an interactive interface that integrates directly with a coding agent and anchors rules to every step of the coding process. After an agent generates an initial plan, ZORO enriches the plan with rules, enforces the rules during implementation by requiring the agent prove that each rule was followed, and allows users to provide in-situ feedback when they are unsatisfied with a rule application to evolve the ruleset. A technical evaluation shows that coding agents follow rules more with ZORO than without. A user study demonstrates a change in people's behavior and cognitive strategies when rules are at the forefront of vibe coding. We discuss how making rules active in agentic systems unlocks broader opportunities for human-agent alignment in coding settings and beyond.

cs.HC↗
arXiv2026

Calibrating LLM Judges for Human and AI Conversations

Measuring how successful a conversation is remains difficult, even for humans judging spoken dialogue. We evaluate state-of-the-art LLMs as pointwise and pairwise judges of conversational success on CANDOR, finding pointwise scoring correlates moderately with human ratings, while pairwise comparison suffers from long transcripts and positional bias. Since this leaves judge scores incomparable across models, we propose a small anchor set and a calibration function that calibrates any judge onto a shared, interpretable scale. We further release the Voice Arena Goal Dataset (VA), 200 task-oriented human-AI and human-agent conversations with pairwise annotations, revealing a substantial gap between current judges and human-level discrimination. Using VA, we test whether CANDOR-fitted calibration transfers to human-AI conversations, finding it brings judges onto a shared scale despite never observing VA during fitting.

cs.HC↗