From IR to RecSys: Evaluating LLM-based Judges in Cranfield-style Recommendation Collections

Abstract

The Cranfield paradigm has long provided reliable, reproducible evaluation in ad hoc retrieval, and recent work has begun extending this framework to recommender systems. A recent development in IR is the use of Large Language Models (LLMs) as automatic relevance judges, showing promising agreement with human assessors. Whether this LLM-judge paradigm---studied predominantly on query--document pairs---transfers to the subjective, profile-driven nature of recommendation remains an open question. This paper bridges the IR and RecSys evaluation traditions by systematically investigating LLM-based judges within a Cranfield-style recommendation collection. Using the ML-32M-ext movie recommendation collection, we first demonstrate that traditional train--test splits yield substantially incomplete relevance labels and unreliable system rankings compared to Cranfield-style pooling. We then assess LLM-judge alignment with human labels, finding that richer item metadata and longer user histories improve agreement, although item-level agreement remains moderate overall. Rankings derived from LLM-judge labels achieve high agreement with human-based rankings (Kendall's tau up to 0.92 for nDCG@100 across 52 system configurations), comparable to values reported for TREC ad hoc retrieval collections. Crucially, LLM-judge recovers system rankings that are distorted under traditional evaluation---correctly identifying systems that are undervalued or overvalued by incomplete labels. An industrial case study in podcast recommendation further demonstrates the practical value of LLM-judge for model selection. Rather than positioning LLM-judges as a replacement for human or interaction-based evaluation, our results support their use as a promising complementary signal: item-level agreement with humans is moderate, yet system-level rankings---which aggregate judgments over many user--item pairs---remain stable.