arXiv · 2611.11279
How Efficient Is Your Agent? Online Predictive Bounds under Budget Constraints
Abstract
We evaluate AI agents in an online setting where tasks arrive sequentially and, before running each task, we construct an upper predictive bound on its \emph{time-to-success}---the number of interactions until the task is completed successfully. Under a limited budget, some runs are stopped early, so their time-to-success is unobserved (right-censored), and the correctness of the bound may be unknown. We introduce \emph{\textbf{PR}edictive Bounds for \textbf{O}nline \textbf{B}udget \textbf{E}fficiency} (\texttt{PROBE}), a method for constructing calibrated upper predictive bounds while adaptively allocating a limited budget across agent tasks to reduce the anticipated coverage deficit. Our method calibrates the bounds online using inverse-probability-weighted feedback together with a correction designed to minimize coverage deficit while remaining unbiased. We prove that \texttt{PROBE} provides a coverage guarantee for any finite time horizon while satisfying the budget constraint. Experiments on software engineering, retrieval-augmented generation, and instruction following show that \texttt{PROBE} attains a coverage rate closer to the target level with tighter predictive bounds than a calibrated baseline.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Roy Maor Lotan, Shai Feldman, Yaniv Romano. 2026-11-08. How Efficient Is Your Agent? Online Predictive Bounds under Budget Constraints. https://arxiv.science/abs/2611.11279
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite arXiv Science by Indiciae: BibTeX · RIS.