arXiv Science⌕ Search

arXiv subjects

Thorben Jansen

Publications and source records attributed to Thorben Jansen.

5 recordsLinked to original sources

Detecting AI-Authored Text in Authentic Student Writing: Evidence from PISA 2025

AI-text detectors reach near-perfect scores in evaluations that contrast fully human-written texts with texts generated from researcher-designed prompts. In authentic student writing, however, AI use is voluntary and often partial. We study this setting with the PISA 2025 FLA dataset, in which 15-year-olds in Germany wrote English argumentative essays with and without access to a gpt-4o-mini-based chatbot. Keystroke, clipboard and chat logs let us trace every character of the final essay to its origin. From this provenance, we derive text- and sentence-level labels for human-written, AI-generated and mixed text, and for text written after chatbot use without traceable AI material. We evaluate a feature-based detector and a classifier on frozen Llama-3.1-8B embeddings, both adapted from top-ranked shared-task systems. Both separate AI from human text reliably (macro-F1 up to 0.96 for texts and 0.95 for sentences), independently of spelling errors. The feature-based detector, however, relies on lexical complexity, which a simple student-style rewriting prompt removes (AI recall 0.92 $\to$ 0.60 with gpt-4o-mini, 0.41 with Qwen3). The embedding-based detector stays robust for whole texts (AI recall $\geq$ 0.94) but not for single sentences (AI recall as low as 0.59). Mixed text falls between the two classes, and text without traceable AI material is classified as human-written in about 95\% of texts.

cs.CL↗

Agents' Last Exam

Recent AI systems have achieved strong results on a wide range of benchmarks, yet these gains have not translated into economically meaningful deployment across many professional domains. We argue that this gap is largely an evaluation problem: widely used benchmarks lack sustained performance measurement on real and economically valuable workflows. This paper introduces Agents' Last Exam (ALE), a benchmark designed to evaluate AI agents on long horizon, economically valuable, real world tasks with verifiable outcomes. Developed in collaboration with 250+ industry experts, ALE covers non-physical industries defined with reference to O*NET / SOC 2018 (the U.S. federal occupational taxonomy). It is organized around a task taxonomy with 55 sub fields grouped into 13 industry clusters covering 1K+ tasks. Current results show that the hardest tier remains far from saturated: across mainstream harness and backbone configurations, the average full pass rate is below 1%. ALE is designed as a living benchmark: its task pool grows continuously as new workflows and industries are onboarded. More broadly, ALE is intended not merely as another leaderboard, but as an instrument for closing the gap between benchmark success and GDP relevant impact.

cs.AI↗

AI, Expert or Peer? Provider Biases and Feedback Uptake Among Pre-Service Teachers

The EU AI Act places teachers in charge of using high-risk AI safely in their classes, which requires them to assess AI-generated outputs. Feedback is one of the most consequential of these outputs, yet little is known about pre-service teachers perceptions of AI-generated feedback. In a randomised experiment, 273 pre-service teachers each received one of 30 written feedback messages on a mathematics learning goal, produced under identical instructions by an expert, a peer, or a large language model (LLM). Without knowing the source, the participants judged who had written the message, rated six feedback perception subscales, and revised the learning goal. Source judgements were inaccurate (peer 46%, expert 40%, LLM 36%) and followed message length, not coded feedback quality. LLM feedback received more positive evaluations when ascribed to a human source. Ratings did not differ between feedback ascribed to experts and to peers. Relative uptake was highest for LLM feedback (52%). Coded feedback quality was the only variable significantly associated with uptake. Therefore, beliefs about who wrote a message shaped perceptions but not uptake. A feature-level analysis revealed that Valence was associated with perceptions only and an instructional composite with both perceptions and uptake. The implications for AI-related evaluative skills in teacher education are discussed.

cs.HC↗

The Future of Feedback: How Can AI Help Transform Feedback to Be More Engaging, Effective, and Scalable?

With digital learning environments becoming more prevalent, the ease with which generative AI enables the scalable production of real-time, automated feedback holds the potential to reshape learning and teaching experiences. This meeting report synthesizes the interdisciplinary perspectives of 50 scholars from educational psychology, computer science, science education, and the learning sciences on the use of generative AI for feedback and its promises and risks in educational practice. We highlight points of convergence in the scholarship, identify areas of debate and unresolved challenges, and outline open questions and future directions for research and educational practice that emerged from structured small-group activities designed to bridge disciplinary barriers.

cs.CY↗

Protecting and Promoting Human Agency in Education in the Age of Artificial Intelligence

Human agency is crucial in education and increasingly challenged by the use of generative AI. This meeting report synthesizes interdisciplinary insights and conceptualizes four aspects that delineate human agency: human oversight, AI-human complementarity, AI competencies, and relational emergence. We explore practical dilemmas for protecting and promoting agency, focusing on normative constraints, transparency, and cognitive offloading, and highlight key tensions and implications to inform ethical and effective AI integration in education.

cs.HC↗