arXiv Science⌕ Search

arXiv · 2611.14333

AI Isn't Getting Dumber, We're Getting Pickier: Rising Standards in Human Evaluation of Large Language Models

Abstract

Users often complain that AI assistants such as ChatGPT are getting worse. Some versions do change, but human evaluation leaves a second possibility untested: the people judging may change. We separate the two on Chatbot Arena, a public platform where people compare answers from two anonymous large language models (LLMs) and pick the better one or say both are bad. We study 1,260,967 blind votes cast from May 2023 to August 2024 on pairs of models whose versions are fixed: dated vendor snapshots and models with fixed open weights. For the same pair of unchanged models, the share of votes saying "both are bad" rises by 1.71 percentage points a month on a mean of 17.1%, and by 1.91 points a month in 2024. In fourteen pairs observed often in both 2023 and 2024, it rose in every one, from 15.5% to 23.3%. The ranking does not trend: in 2024, the stronger model's share of decisive votes is flat. The rise holds within the same voter and on the same pair, and across arrival cohorts from their first recorded vote. It holds with prompt controls, in every major language, for dated vendor snapshots, open-weight models and models with stable answer length, and between interface changes. The drift is shared across models: month effects estimated from other models' pairs remove it from each model's own pairs. Platform-wide, the average model would be expected to score 78% against the 2023 average, yet the share of "both are bad" votes ended near where it began. The pattern fits a standard that follows the category's progress, so a product that stands still is judged as declining. Absolute verdicts on AI drifted while relative comparisons held steady, an evaluation drift that matters for LLM leaderboards and for the human feedback used to train models.

Explore related subjects

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Zihan Chen, Lei Nico Zheng, Di Zhu. 2026-11-10. AI Isn't Getting Dumber, We're Getting Pickier: Rising Standards in Human Evaluation of Large Language Models. https://arxiv.science/abs/2611.14333

Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite arXiv Science by Indiciae: BibTeX · RIS.

KEEP EXPLORING

Related papers

Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs

In this paper, we conduct a structured synthesis of existing theories and frameworks on human-human collaborative writing to assess how co-writing with AI reshapes writing in knowledge work settings, and draw seven insights along with design implications for human-AI collaborative writing tools. Our main argument is that, as we delegate more writing to AI, our cognitive process shifts from the traditional plan/write/review process to a plan/delegate/review process, reshaping the distribution of the cognitive labor between human writers and AI writing tools. To ensure that humans cognitively stay in the loop, we discuss a ``prototyping'' approach, where the writing tool scaffolds a healthier distribution of cognition through rapid, small-scale iterations that keep writers mentally in the loop even when generation is delegated to AI. We aim to bring theoretical grounding and practical guidance to the interaction designs of human-AI collaborative writing, with the goal of balancing automation with cognitive integrity.

cs.HC↗

How K-12 Teachers Adapt Generative AI to Local Classroom Demands: Cultural Distance Between GenAI Defaults and Everyday Teaching Practices

Generative AI (GenAI) is entering K-12 classrooms, but its default culture does not always fit local teaching. Existing accounts either treat this misfit as macro-level structural bias or examine it through single-case studies; neither captures how much work teachers must do to make outputs usable across tasks. This exploratory, theory-building qualitative study introduces cultural distance, the gap between GenAI's default cultural repertoire and the situated demands of teachers' practice. Through interviews with 30 K-12 teachers who frequently use GenAI, 10 each from South Africa, Taiwan, and the United States, we identified three levels defined by the effort teachers invested and whether it yielded a usable output: low (minor edits), mid (sustained prompting and revision), and high (structural gaps rooted in unsupported languages, undigitized knowledge, or policy filters that no teacher effort can close). We present cultural distance as a promising framework requiring further examination and draw implications for designers, policymakers, and educators.

cs.HC↗

AI Behavioral Science: A Framework and Agenda

We discuss the challenges and opportunities present in the rapidly emerging area of ``AI Behavioral Science.'' We frame it via three themes. First, as AI becomes ubiquitous and is increasingly proprietary and opaque, it becomes vital to develop models of AI and methods for assessing AI behavior. We outline how tools developed to assess people's behaviors by social scientists can be used to model, assess and infer AI's behaviors biases, tendencies, and heuristics. Second, we also discuss how AI can change the ways in which we learn about human behavior. Beyond its computational power, AI offers new techniques for simulating, inferring, predicting, and analyzing human behaviors. Third, as humans and AI are interacting in increasingly complex and intertwined systems, we need to analyze and model human-AI interactions including how human and AI behaviors depend on interactions at the individual level, how interacting systems of humans and AI behave, and ultimately how AI's integration into society affects economic and political outcomes. We discuss current research, questions, agendas, and goals in each of these three themes and how they depend upon each other.

cs.HC↗