Dr. Zero: Self-Evolving Search Agents without Training Data

Abstract

As high-quality data becomes increasingly difficult to obtain, self-evolution without curated training data has emerged as a promising paradigm. This approach allows large language models (LLMs) to autonomously generate and solve complex problems, thereby improving their reasoning capabilities. However, multi-turn search agents struggle in this setting due to limited question diversity and the substantial compute required for multi-step reasoning and tool use. In this work, we introduce Dr. Zero, a framework that enables search agents to effectively self-evolve without human-annotated training data, relying solely on an external search engine as their knowledge environment. In particular, we design a self-evolution feedback loop where a proposer generates structurally diverse questions to train a solver initialized from the same base model. As the solver evolves, it incentivizes the proposer to produce increasingly difficult yet solvable tasks, thus establishing an automated curriculum to refine both agents. To enhance training efficiency, we also introduce hop-grouped relative policy optimization (HRPO). This method clusters structurally similar questions to construct group-level baselines, effectively minimizing the sampling overhead in evaluating each query's individual difficulty and solvability. Consequently, HRPO significantly reduces the compute requirements for proposer training and reward estimation without compromising performance or stability. Extensive experimental results demonstrate that Dr. Zero matches or surpasses fully supervised search agents on several question answering benchmarks, showing that strong agentic search and evidence-grounded reasoning can emerge solely through self-evolution.