arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2610.09460

Beyond Score Accuracy: Examining the Diagnostic Quality of LLM-Generated Structured Assessment in Higher Education

Xi Zhao·Xinyue Jiao·Zhen Xu
View original source ↗Read source document ↗Export metadata ↓

Abstract

As Large Language Models (LLMs) are increasingly adopted for automated grading and feedback in higher education, their structured outputs, including multi-dimensional rubric scores, detailed feedback comments, and improvement suggestions, create an appearance of thorough analytic evaluation. This study examines whether these outputs deliver what they appear to offer. Using the JorGPT dataset of 3,041 student responses to 50 open-ended computer science questions, scored by both human instructors and three commercial LLMs, we identify three systematic discrepancies between the apparent and actual quality of LLM-generated grading and feedback. The sub-dimension scores are highly correlated (r = 0.82-0.99, VIF up to 45), providing redundant rather than independent diagnostic information. The textual feedback rarely detects student misconceptions (5-7% vs. 15-31% for teachers), functioning as a coverage checklist rather than a diagnostic instrument. The feedback tone remains uniformly positive regardless of response quality, lacking the severity modulation observed in human feedback. Additionally, grading accuracy varies significantly by knowledge domain, with procedural topics most reliable. These findings provide empirically grounded guidance on which aspects of LLM-generated grading and feedback can be relied upon and which require continued human oversight.

Explore related subjects

cs.CY

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Xi Zhao, Xinyue Jiao, Zhen Xu. 2026-10-07. Beyond Score Accuracy: Examining the Diagnostic Quality of LLM-Generated Structured Assessment in Higher Education. https://arxiv.org/abs/2610.09460

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Bridging MOOCs, Smart Teaching, and AI-Assisted Learning: A Unified Quantitative Model

MOOCs, Smart Teaching (ST), and AI-assisted learning support different stages of higher education, yet a unified quantitative basis for analyzing their contributions, course design, and resource allocation remains limited. This study develops a mathematical model integrating MOOC-based pre-class learning, ST-based in-class adaptation, and AI-assisted post-class personalization. Learner mastery is represented as a bounded multidimensional state, with a common exponential learning-response function describing how instructional resources reduce remaining knowledge gaps. The stages differ in their allocation rules: predefined course-content emphasis for MOOCs, feedback-driven class-level adaptation for ST, and individualized gap-based allocation for AI-assisted learning. Simulations across 100 independently generated classes demonstrate stable cumulative progression and quantify stage-wise gains, while budget analysis reveals diminishing returns from additional AI support. For a fixed learner and learning mechanism, varying course-content emphasis shows a strong association between course-learner alignment and post-MOOC mastery. An optimal AI allocation is also derived under a fixed budget: supported components reach a common residual mastery gap, while components below this threshold receive no resources. A controlled comparison yields over 14% greater learning gain than proportional allocation. These analyses make the instructional process quantitatively analyzable and provide a basis for examining course-learner fit and coordinating limited learning resources. The model offers an analytical foundation for instructional decisions, with practical application requiring empirical estimation of learner states and calibration of learning-response parameters.

cs.CY↗
arXiv2026

Reasoning Enhances Robustness to Prompt Injection in LLM-Based Consensus

Large Language Models (LLMs) are gaining traction as a method to generate consensus statements and aggregate preferences in digital democracy experiments. Yet, participants can introduce critical vulnerabilities in LLM-based systems. Here, we examine the vulnerability and robustness of off-the-shelf consensus-generating LLMs to prompt-injection attacks, which consist of introducing strategically designed texts to amplify particular viewpoints, erase certain opinions, or divert consensus toward unrelated or irrelevant topics. Using data collected from a 2023 experiment conducted in the UK and predefining a majority-rule, we construct attack-free and adversarial variants of prompts containing public policy questions and opinion texts, classify opinion and consensus valences with a fine-tuned BERT model, and estimate LLM--human majority agreement rates. Overall, we find that default LLMs exhibit widespread vulnerability, especially when: (i) disagreement and agreement are finely balanced, (ii) under rational, instruction-like rhetorical strategies, and (iii) for attacks that shift consensus toward positions aligned with GB-unionist conservative manifestos relative to pro-independence left manifestos. A robustness pipeline combining GPT-OSS-SafeGuard injection detection, structured opinion representations, and GSPO-based reinforcement learning substantially reduces directional failures, outperforming state-of-the-art alternatives. While more effective detectors may enable more sophisticated attacks, these findings advance our understanding of both the vulnerabilities and the potential defenses of consensus-generating LLMs in digital democracy applications.

cs.CY↗
arXiv2026

Scaling Legal AI: Benchmarking Mamba and Transformers for Statutory Classification and Case Law Retrieval

Statutory corpora and judicial decisions are growing faster than legal professionals can read them, while individual judgments often exceed the context limits of standard encoder models. Transformer architectures dominate legal NLP benchmarks, but their quadratic attention complexity can require truncating or fragmenting documents that demand whole-document reasoning. Selective state-space models (SSMs), such as Mamba, offer linear-time sequence modeling and are a promising alternative for long legal documents, yet their performance on legal classification and retrieval remains underexplored. We present a preliminary benchmark comparing Mamba and SSD-Mamba with BERT, DeBERTa, and Longformer across four legal classification tasks (ECtHR, EUR-Lex, SCOTUS, and ILDC/ILC) and two case-retrieval tasks (ECtHR and ILDC), using a shared windowing and aggregation pipeline. The strongest SSM performs within approximately 1.3 percentage points of the strongest transformer across tasks and metrics. SSD-Mamba achieves the best results on most metrics for ECtHR classification, ILDC classification, and ECtHR retrieval, while processing approximately 3 times more tokens per second than DeBERTa and 4 times more than Longformer. DeBERTa remains strongest on SCOTUS and EUR-Lex F1. These results are preliminary because they do not include variance estimates across random seeds or statistical significance testing. Rather than presenting a definitive ranking, we use these findings to motivate further evaluation with repeated-seed experiments, statistical testing, and controls for model capacity and computational efficiency.

cs.CY↗