arXiv Science⌕ Search
Home/Archive/arXiv

arXiv · 2610.19572

Divergence-Routed On-Policy Distillation for Language Models

Tiantian Peng·Guannan Lv·Hongyang Li·Qiuhe Hong·Hang Cheng·Hongjian Dou·Xiangyu Wu·Yuyang Liu
View original source ↗Read source document ↗Export metadata ↓

Abstract

On-policy distillation (OPD) transfers knowledge from a teacher model to a student along student-generated trajectories, typically by minimizing the reverse Kullback-Leibler (RKL) divergence. However, RKL can provide weak corrective signals for teacher-supported tokens to which the student assigns very low probability. This raises two questions: which positions are most consequential for the behavioral gap between the student and teacher, and what additional supervision is needed at these positions? Through cross-sampling, we find that divergence-based routing more effectively pinpoints positions contributing to the teacher-student performance gap than entropy-based routing. Further analysis reveals teacher-mode under-coverage at high Jensen-Shannon (JS) divergence positions, where standard OPD increases the large teacher-student probability gaps. Motivated by these findings, we propose Divergence-Routed On-Policy Distillation (DrOPD). DrOPD retains RKL supervision at all positions and selectively adds forward Kullback-Leibler (FKL) divergence supervision at positions with high JS divergence. Experiments across six math benchmarks show that DrOPD improves over standard OPD and entropy-based routing. DrOPD also reduces severe under-coverage cases by 37.9 percent relative to OPD, supporting targeted FKL as an effective complement to OPD.

Explore related subjects

cs.AI

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Tiantian Peng, Guannan Lv, Hongyang Li, Qiuhe Hong, Hang Cheng, Hongjian Dou, Xiangyu Wu, Yuyang Liu. 2026-10-14. Divergence-Routed On-Policy Distillation for Language Models. https://arxiv.org/abs/2610.19572

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

SEDM: Selective Self-Evolving Distributed Memory for Agents

Agents in a society can reuse each other's experience, but indiscriminate memory use increases processing cost and can reduce task success. We present SEDM (Selective Self-Evolving Distributed Memory), a framework that selects both what agents share and when recipients use it. On the sender side, SEDM admits memories through paired execution tests, ranks them by relevance and measured utility, and abstracts them for cross-domain reuse. On the recipient side, Calibrated Memory Gating (CMG) predicts success with and without memory in a single forward pass, learning from paired outcomes including ties. It injects memory only when the calibrated success gain outweighs a context-cost penalty. SEDM exceeds the strongest evaluated baseline on LoCoMo open-domain questions by 15.1 F1 points and outperforms G-Memory on FEVER and HotpotQA with 16-32% fewer prompt tokens. In a controlled HotpotQA study, CMG achieves success comparable to always using memory with 18% fewer agent input tokens. Within the full SEDM pipeline on LoCoMo, CMG outperforms both always-use and no-memory policies for two SEDM memories while using 19-31% fewer input tokens than always-use. These results show that selective use can recover value from shared memories that hurt performance when injected indiscriminately.

cs.AI↗
arXiv2026

Toward Agents That Reason About Their Computation

While reinforcement learning agents can achieve superhuman performance in many complex tasks, they typically do not become more computationally efficient as they improve. In contrast, humans gradually require less cognitive effort as they become more proficient at a task. If agents could reason about their compute as they learn, could they similarly reduce their computation footprint? If they could, we could have more energy efficient agents or free up compute cycles for other processes like planning. In this paper, we experiment with showing agents the cost of their computation and giving them the ability to control when they use compute. We conduct our experiments on the Arcade Learning Environment, and our results demonstrate that with the same training compute budget, agents that reason about their compute perform better on 75% of games. Furthermore, these agents use three times less compute on average. We analyze individual games and show where agents gain these efficiencies. An interactive version of this paper is available at https://adrianorenstein.com/paper/toward-compute.

cs.AI↗
arXiv2026

LingLanMedBench: A Unified Benchmark for Evaluating Large Language Models in Traditional Chinese Medicine

Reliable evaluation of large language models in traditional Chinese medicine remains limited because existing benchmarks mainly focus on examination-style question answering or isolated generative tasks. We developed LingLanMedBench, a unified benchmark comprising 25,620 expert-verified items across 13 subtasks and five domains, covering knowledge, language understanding, information extraction, and diagnostic-therapeutic decision-making. LingLanMedBench standardizes task formats and evaluation metrics and further incorporates synonym-tolerant clinical-label matching, empirically derived hard subsets, and decision-recognition reformulations of open-ended clinical tasks. Fourteen open-source and proprietary models were evaluated under a standardized zero-shot protocol. Performance varied substantially across domains, model families, task formulations, and difficulty levels. The leading overall score decreased from 51.1 on the full benchmark to 31.9 on the hard subsets. Prescription performance also differed markedly between open-ended generation, with F1 scores of 32.7 and 20.3, and decision recognition, with accuracies of 90.6 and 56.8 on the full and hard subsets, respectively. These findings reveal a substantial gap between knowledge-oriented performance and clinically faithful reasoning. LingLanMedBench provides a standardized and reproducible framework for characterizing current capabilities, identifying systematic limitations, and tracking meaningful progress in large language models for traditional Chinese medicine.

cs.AI↗