Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation
In briefBalancing Multiple AI Teachers When Merging Skills into One Model
SummaryReinforcement learning can produce separate language models that each excel at one skill, such as mathematics or instruction following, but users want a single model with all of them. Multi-teacher on-policy distillation (MOPD) combines these specialists by having each one give feedback to a shared student model on prompts from its domain. The authors find that instruction-following feedback is much more variable than mathematics feedback and dominates training, so the student gains little from the math specialist. They propose Domain-Normalized MOPD (DN-MOPD), which rescales each domain's feedback by its measured spread, and report improved average scores over MOPD on six public benchmarks across three model sizes.