arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2610.34245

Sealed bets on emerging research topics: a protocol for judging early detection, applied to a detector and to eight research funders

Sébastien Gracia
View original source ↗Read source document ↗Export metadata ↓

Abstract

Tools that claim to detect emerging research topics are rarely judged on their own lists. Our main contribution is a protocol that makes such judgement possible. A bet freezes four things before its verdict can be known: the date of the photograph, the population of candidates, the verdict rule, and the comparison with chance drawn among the same candidates. The list has a fixed length, is read on two axes (champions and failures) against a computed ceiling, and is sealed by cryptographic fingerprint; sealing proves that a list existed at a date, not that it is right. A topic is a champion if its yearly volume of records triples within three years and reaches 100, a failure if it falls by 30%. We apply the protocol to one detector on PubMed/MEDLINE, in three separate blocks. On a retrospective benchmark (2017-2022), chance among the candidates yields 26.7 champions and 15.1 failures out of 120 slots; the detector yields 85 and 3, and under nine pre-registered variants of the verdict rule its champions stay between 2.5 and 5.0 times chance, with no more failures. The detector was developed with these verdicts known, and its specification is not published. Prospective validation rests on six sealed bets, judged in early 2029 and early 2030; only bets 5 and 6 test exactly the instrument of the benchmark. A secondary, exploratory study applies the same judge to eight research funders: under exact name matching, which may disadvantage grant texts, the detector's lists are more concentrated in future champions than the twenty topics named in the most grants of each funder (70.8% against 20.8% to 35.8%, chance 22.3%). We document data defects, negative results and the limits of each comparison.

Explore related subjects

cs.DL

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Sébastien Gracia. 2026-10-25. Sealed bets on emerging research topics: a protocol for judging early detection, applied to a detector and to eight research funders. https://arxiv.org/abs/2610.34245

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Interpreting LLM Scores in Peer Review: Alignment and Divergence Across Prompts, Domains, and Manuscript Features

Large language model (LLM) scores are being studied as information for scholarly evaluation. Interpreting these scores requires evidence about agreement with human judgments and its variation across evaluation conditions. Guided by an argument-based approach to validation, we compare four LLMs on 7,083 public ICLR 2024 submissions under a standard review condition and examine ten prompt conditions using DeepSeek-V3.2. We distinguish average score divergence, paper-level absolute error, and rank correlation, then examine domain differences and adjusted manuscript-feature associations. For DeepSeek-V3.2, Area Chair guidance reduces mean rating divergence from 1.460 to 0.199 points and mean absolute error from 1.577 to 0.942, while rank correlation changes from 0.377 to 0.387. Historical references combined with downward scoring instructions yield mean divergence of 0.052 and rank correlation of 0.321. Domain differences remain after global calibration. Longer titles have positive adjusted associations with rating divergence, and formula density has negative associations across all four configurations. These findings connect score interpretation to the scoring procedure, evaluation context, and manuscript information represented in the comparison. They show why aggregate alignment, manuscript ordering, and correspondence within manuscript groups require distinct evidence when evaluating LLM-generated scholarly assessment information.

cs.DL↗
arXiv2026

Is this Citation on Point?

In 2023, a New York judge sanctioned two attorneys in \emph{Mata v. Avianca} for filing a brief containing hallucinated citations generated by ChatGPT. These failures can be largely caught by database lookups. The harder problem is detecting citations that point to real cases but do not support the propositions for which they are offered, a failure mode that existing evaluations of LLMs for legal use cases largely overlook. In this paper, we study proposition-level citation support verification through controlled perturbations of real legal citations obtained from two legal corpora. We either replace the cited case or change only the pinpoint page within the same case. Our evaluation of fourteen model configurations on the resulting examples shows that models catch $93$--$100\%$ of wrong-case corruptions, but they only catch $37$--$61\%$ of wrong-pinpoint corruptions on court opinions and $52$--$83\%$ on legal briefs. When models fail to catch wrong-pinpoint corruptions, they accept the citation based on topical overlap rather than page-level support. Larger-scale models and extended reasoning narrow the gap but do not close it: GPT-5.4 with high reasoning effort still misses $40\%$ of pinpoint mismatches on court opinions and $18\%$ on briefs. Prompting the model to verify support at the cited page improves recall, but it also raises the false positive rate. These results show that citation misrepresentations remain difficult to detect automatically. Recognizing the right legal topic and verifying support for the cited proposition are distinct capabilities, and current models conflate them.

cs.DL↗
arXiv2026

Officially corrected errors persist in AI training data: preregistered measurements on EU law, Internet standards and retracted science

Official texts are corrected after publication: EU corrigenda, verified RFC errata, retractions. We measured whether these corrections are present in four public datasets used to train language models, and what models answer about the corrected passages. In EurlexResources, a legal training dataset, 1,989 of 2,082 testable single-language corrections to EU acts (95.5%, 94.6-96.3) remain in their erroneous form; in a random sample of 100, 46 change the meaning, a figure, a cross-reference, the competent authority or the scope (20 without scope). At least 124 corrected EU-law errors and at least 60 verified RFC errata (38 classified "Technical") are reproduced in their erroneous form in FineWeb-2 and in a sample of FineWeb. About 12,200 retracted papers (11,700-12,500) are present in the peS2o corpus, 79% with no mention of the retraction; separately, 53% of a random sample of retracted papers are still cited at least a year after retraction (OpenAlex). Asked 59 questions on serious corrected errors without documents, five models gave the erroneous version in 45% of the answers in which they committed to one version (31-60% allowing for clustering by question): 53% for scope corrections, 32% for the others. One answer in 295 said it did not know. Shown both versions side by side, the models chose the corrected one in 53% of cases (49-57), and naming the act made no difference; when each version carried its date and the corrigendum label, the four larger models chose it in 99-100% of cases. A comparison between a legal masked language model and its base model is null, against our bet: we do not show that model answers come from the datasets measured. All steps were preregistered and their fingerprints timestamped in the Bitcoin blockchain; six controls failed, several bets were lost, and every deviation is reported. Data and proofs: doi:10.5281/zenodo.23642239.

cs.DL↗