Same Text, Different Filename: Are LLM Judges Biased?

Sergey V. Popov

Home/Archive/arXiv

arXiv · 2611.01829

Same Text, Different Filename: Are LLM Judges Biased?

Sergey V. Popov

View original source ↗Read source document ↗Export metadata ↓

Abstract

Yes. In 2026, leading economic journals started using Refine.ink in the technical quality assessment of submissions before publication. I study whether renaming a file changes the perception of the document's quality by LLM judges. I find that three frontier LLMs score the same document differently if you change the document's filename. A person-related label scores higher than no label; there is no consistent preference of own- or rival-based label, nor a penalty for an Asian name, but there is a context-specific bias in assessments whose direction and size vary with both document and label. With LLM-written documents, some self-premium can be detected, but a significant context-dependent label bias remains. Whether to combat it is a question of the costs of evaluation errors.

Explore related subjects

econ.GN q-fin.EC

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Sergey V. Popov. 2026-11-01. Same Text, Different Filename: Are LLM Judges Biased?. https://arxiv.org/abs/2611.01829

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

What Drives Energy Use? Prices, Policy, and Inefficiency

Why is U.S. energy use so high, even among rich economies? Using a 2006--2022 panel of the 50 states and DC, an LMDI decomposition documents a 12.8% national decline alongside substantial cross-state heterogeneity. To explain differences in energy-use levels across states, we estimate a microfounded stochastic-frontier model and adapt the LMG relative-importance method to the frontier and inefficiency components. Our approach assigns 56.3% of variation to the frontier, 42.2% to inefficiency, and 1.5% to noise. Prices and efficiency-policy environments are the two leading observed contributors, jointly accounting for 31%: prices operate mainly through the benchmark, whereas policy contributes through both components. Their leading ranks are stable across 21 specifications. Persistent low prices, uneven efficiency-policy environments, structural demand, and a sizeable composite inefficiency component help explain high U.S. energy use. Policy should therefore combine durable price signals, coherent efficiency programs, structural investment, targeted measures addressing avoidable energy use, and targeted household support.

econ.GN↗

arXiv2026

The Costs and Benefits of Leveraged ETFs

We study the costs and benefits of leveraged ETFs (LETFs), focusing on five determinants of performance: volatility drag, fees, financing costs, product launches, and market timing. Long broad-equity-index LETFs generated more than 100 billion US dollars in investor gains, including over 40 billion US dollars relative to counterfactual investments in the underlying assets, due to modest structural headwinds and rising markets. Single-stock LETFs, however, are launched on stocks near the top of the volatility and past-return distributions. This caters to retail demand for lottery-like payoffs, but it also creates larger volatility drag, higher financing costs, and greater exposure to losses from poor market timing.

econ.GN↗

arXiv2026

Prizes, Shared Hazard, and the Speed of Frontier AI

Frontier AI labs compete in a contest for commercial leadership while sharing the risk of catastrophe. Each lab gains from arriving earlier but bears only part of the hazard its speed adds, which this paper calls the hazard externality. In an illustrative calibrated takeoff model, three racing labs get through the takeoff period in 18 months, while a common rule on research shares would stretch it to 69 months. Neither force alone explains racing that fast. Removing the contest first closes over ninety percent of the gap in catastrophe probability between racing and a common rule when the catastrophe probability at full speed is 75 percent, and 69 percent when it is 25 percent; averaged over both orders of removal, the contest explains 65 and 44 percent. More labs make each lab race harder, when industry hazard at a common research share does not depend on the number of labs. Under open source in its ideal form, where every lab publishes its capability findings at once, labs slow to the coordinated pace, because no lab can get ahead and each bears all the hazard its research adds. One lab's restraint achieves much less than a common rule. The White House Accord on Super Intelligence of September 2026 audits safety but sets no limit on speed. In the model it does not slow the race, because each lab still gains as much from speeding up.

econ.GN↗