arXiv Science⌕ Search

arXiv · 2610.28666

Learning Checklists to Reduce Lottery Effects in Scientific Peer Review

Abstract

Reviewer lottery error describes error due to a small set of peer reviews attending to only some of the issues with a paper that a larger pool of eligible reviewers would raise. We show this error can be reduced by surfacing a checklist of criticisms learned from a historical review corpus. Our approach generates a tree of checklists representing coarsenings of the atomic review claim space, then selects the coarsening that maximizes a theoretically-motivated measure of correlated agreement among reviewers. We realize this approach with an LLM-assisted pipeline and test it on NeurIPS 2021 and ICLR 2022 review data. In simulated review with LLM reviewers, the learned checklist reduces the mean squared error (MSE) of a three reviewer committee by 58% on NeurIPS and 21% on ICLR, and on both venues the checklist with the highest correlated agreement has the lowest error for committees of three or more. In a human study with 20 participants, the learned NeurIPS checklist lowers the MSE of a paper ranking, measured against the NeurIPS 2021 order, from 0.48 to 0.22.

Explore related subjects

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Xuanyou Liu, Ziyang Guo, Jason Hartline, Jessica Hullman. 2026-10-14. Learning Checklists to Reduce Lottery Effects in Scientific Peer Review. https://arxiv.org/abs/2610.28666

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

Replication Certification at Publication: Checkable by Anyone, Anywhere, in Seconds, Forever

Authors increasingly deposit replication datasets---the data and code behind publications---in archives, and journals increasingly require them to, under progressively onerous rules. But a deposit alone proves little, so authors keep paying: months for the journal's reanalyses, years answering readers' questions, and rancor as failed replications become accusations because no record establishes what actually ran. We introduce a replication certificate: a cryptographically signed record of the deposited files and logged output. It replaces the journal's reanalyses with evidence anyone can check in seconds, which no one, including the archive, can backdate, and no one can fake without risking detection. Certification asks little of anyone and benefits everyone: The author deposits, then reruns the analysis once under a random seed the archive computes from the deposit's hash and a blockchain block; the archive signs, timestamps (in that blockchain), and stores the record---confidential data included---without rerunning anything; the journal merely checks a website for the certificate; and anyone can check or replay any step decades later.

cs.DL↗

Entering the Elite, But Remaining Outside Its Core? The Temporal Structure of Gender Inequality in the Global Citation

This study analyzed the gender dynamics of the global citation elite using the annual Top 2% Stanford/Elsevier lists for 2019-2024. While previous analyses have discussed women s representation (inclusion), I discuss what happens once male and female scientists are within the elite (position and persistence). The database used included 1.22 million person-year observations (N = 1,216,089) and 465,707 unique scientists and scholars from more than 150 countries. I studied entry, exit, and permanent membership in the elite. The share of women in the top 2% increased from 18.35% in 2019 to 20.85% in 2024. However, women were more strongly represented among first-observed entrants than among continuing members, and their share decreased from 21.58% among single-year members to 17.69% among scientists present in all six annual lists. Women were located closer to the lower boundary of the elite in terms of citation index deciles (the lowest decile included 26.88% of women). Men, in contrast, were closer to its upper boundary (the highest decile included only 15.23% of women). Logistic regression models estimated a lower probability of women s membership in the durable core of the elite (odds ratio estimate [OR] = 0.886). Consequently, increasing the inclusion of women in the elite does not mean their equal position within or equal persistence of membership in it. I draw conclusions about temporal gender dynamics within the global citation elite and gender inequalities in science more generally.

cs.DL↗

Light Fidelity in Academic Libraries: A Radiant Journey to Light the Future

Academic contexts have changed due to the fast expansion of communication technology, affecting information access, sharing, and dissemination. Light Fidelity (LiFi) is a new technology that may transform academic library connections in this changing environment. This article examines how LiFi can accelerate data transmission, secure data, and provide a more dynamic and collaborative learning environment to spur creativity. Academic libraries may accelerate information interchange using visible light communication with LiFi technology. LiFi is the solution for the increasing need of digital access it adress the situation by enabling faster research activities and enhancing collaboration among students, faculty, and researchers. Present paper discusses the importance and application of LiFi in libraries.

cs.DL↗