Self-Distillation with Verbal Regularization
Abstract
Self-distillation with verbal regularization improves reasoning by using system prompts to reduce shortcut bias from privileged information, preserving output diversity and boosting accuracy.
On-policy self-distillation (OPSD) trains a large language model using supervision from a teacher instantiated from the same model as the student. The teacher is conditioned on privileged information (PI), such as verified solutions. The student, which does not observe PI, learns from the teacher's next-token distributions along its own sampled responses. However, PI can bias supervision toward shortcut reasoning with limited verification or reconsideration, accompanied by a sharp drop in the entropy of the output distribution. To address this challenge, we introduce self-distillation with verbal regularization (VRSD), which uses system prompts to regulate how the teacher uses PI, thereby encouraging rethinking and exploration to counteract premature commitment. Theoretically, we model prompting as attenuating the influence of PI on the teacher distribution and derive conditions for entropy preservation. Across three models and four tasks, VRSD improves Pass@1 over OPSD in 11 of 12 model-task settings, by 3.9 percentage points on average, while generally maintaining higher token entropy.
Community
On-policy self-distillation (OPSD) builds a teacher from the model itself by giving it privileged information (PI) the student can't see. We find PI often pushes the teacher toward shortcut answers, suppressing re-checking tokens like "Wait" and "However" and collapsing the student's token entropy (near 0 in some runs). The model stops thinking, and downstream accuracy drops.
VRSD is a simple fix: we add a training-only system prompt to the teacher, asking it to verify intermediate steps, consider alternative reasoning paths, and revisit earlier judgments. The student never sees this prompt. We model the prompt as attenuating the PI-induced log-prob shift and derive conditions under which teacher entropy is preserved.
Across 3 models × 4 tasks (Grading, Chemistry, Text-to-SQL, Python):
- Pass@1 beats OPSD in 11/12 settings (+3.9 pp avg)
- Maj@16 never below OPSD; Worst@16 also improves
- Entropy recovers (e.g., SQL: 0.03 → 0.54)
- More faithful reasoning: misleading-reasoning rate 3.0% → 1.6%
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- OPSRD: On-Policy Self-Role Distillation (2026)
- Teach Yourself Where to Look: On-Policy Attention Self-Distillation for Reasoning (2026)
- Recursive Self-Improvement via On-Policy Distillation for Reasoning (2026)
- VISTA: Verifier-Informed Student-to-Teacher Adaptation for On-Policy Self-Distillation (2026)
- E$^2$-OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation (2026)
- DuoOPD: Learning from Joint Teacher-Student Outcomes for Multi-Task On-Policy Distillation (2026)
- Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2610.16354 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 1
Collections including this paper 0
No Collection including this paper