arXiv · 2611.13249
Domain-Routed Delta Optimization for Multi-Teacher On-Policy Distillation
Abstract
Multi-teacher on-policy distillation can yield uneven domain capabilities. We propose Domain-Routed Delta Optimization (DRD-MOPD), composing a frozen base with expert-initialized trainable deltas. Domains update their deltas with independent optimizer states and clipping. After joint training, Post-separate exports the base plus one trained delta per domain, selected by known task domain without retraining. Using AdamW, the unweighted full student reaches 32.83% Macro across six benchmarks after 20 updates, versus 32.21% for a matched shared control. Post-separate raises Macro with Open-MOPD weighting.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Lin Yao. 2026-11-12. Domain-Routed Delta Optimization for Multi-Teacher On-Policy Distillation. https://arxiv.science/abs/2611.13249
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite arXiv Science by Indiciae: BibTeX · RIS.