arXiv Science⌕ Search

arXiv subjects

Gabriel Sargent

Publications and source records attributed to Gabriel Sargent.

2 recordsLinked to original sources

Prediction-Powered LLM Ranking with Adaptively Collected Human Feedback

Ranking large language models (LLMs) through human evaluations is essential for downstream decision-making. Because human feedback is expensive, evaluation platforms often collect it adaptively, prioritizing informative model comparisons using previously observed feedback. To further improve precision with limited human feedback, evaluators can draw on abundant unrated responses and relatively inexpensive LLM judges; however, judge bias and adaptive dependence challenge standard i.i.d. inference frameworks. We propose Prediction-Powered Ranking (PP-Rank), a framework that combines LLM-judge predictions with human feedback to improve precision while providing valid inference under adaptive sampling. Data-driven prediction weights minimize the asymptotic covariance trace within the proposed class. In theory, we establish asymptotic normality for general evaluation targets and simultaneous ranking inference. A joint martingale analysis handles dependence between adaptive human feedback and auxiliary data. We further derive a rule for reducing exploration that prioritizes lower-cost comparisons while matching a conservative variance bound, and characterize allocation limits for regular $\sqrt{T}$-rate estimation of pairwise means. Semi-synthetic experiments on UltraFeedback-Extended show improvements over human-only baselines in score estimation, confidence interval width, and ranking certification. Synthetic experiments demonstrate robustness to judge quality and adaptive allocation, while showing that PP-Rank can substantially reduce evaluation costs without sacrificing precision.

cs.LG↗

Permutation Invariant Parking Assortments

We introduce parking assortments, a generalization of parking functions with cars of assorted lengths. In this setting, there are $n\in\mathbb{N}$ cars of lengths $\mathbf{y}=(y_1,y_2,\ldots,y_n)\in\mathbb{N}^n$ entering a one-way street with $m=\sum_{i=1}^ny_i$ parking spots. The cars have parking preferences $\mathbf{x}=(x_1,x_2,\ldots,x_n)\in[m]^n$, where $[m]:=\{1,2,\ldots,m\}$, and enter the street in order. Each car $i \in [n]$, with length $y_i$ and preference $x_i$, follows a natural extension of the classical parking rule: it begins looking for parking at its preferred spot $x_i$ and parks in the first $y_i$ contiguously available spots thereafter, if there are any. If all cars are able to park under the preference list $\mathbf{x}$, we say $\mathbf{x}$ is a parking assortment for $\mathbf{y}$. Parking assortments also generalize parking sequences, introduced by Ehrenborg and Happ, since each car seeks for the first contiguously available spots it fits in past its preference. Given a parking assortment $\mathbf{x}$ for $\mathbf{y}$, we say it is permutation invariant if all rearrangements of $\mathbf{x}$ are also parking assortments for $\mathbf{y}$. While all parking functions are permutation invariant, this is not the case for parking assortments in general, motivating the need for characterization of this property. Although obtaining a full characterization for arbitrary $n\in\mathbb{N}$ and $\mathbf{y}\in\mathbb{N}^n$ remains elusive, we do so for $n=2,3$. Given the technicality of these results, we introduce the notion of minimally invariant car lengths, for which the only invariant parking assortment is the all ones preference list. We provide a concise, oracle-based characterization of minimally invariant car lengths for any $n\in\mathbb{N}$. Our results around minimally invariant car lengths also hold for parking sequences.

math.CO↗