Why Self-Distillation Helps and Hurts: Denoising vs. Signal Forgetting

Abstract

Self-distillation, a special case of self-training, repeatedly trains a model on pseudo-labels produced by its previous iterate. Why can this iteration improve generalization, and why can it later hurt? We answer this question in overparameterized linear regression, starting from an estimator fit to noisy labels and then repeatedly refitting on fresh covariates labeled by the current estimator. In the proportional asymptotic regime, we derive deterministic recursions for the test risk and prove concentration around them. The recursions reveal a simple trade-off: iteration denoises by attenuating stochastic error inherited from the initial fit, but it also induces signal forgetting because each teacher--student transfer projects the estimate through a new data-dependent subspace. This competition produces a $U$-shaped risk curve and an optimal stopping time. The same dynamics act as a spectral filter: eigendirections carrying more of the true regression signal are retained longer, while weakly aligned or noise-like directions are suppressed. We further develop an iterated generalized cross-validation criterion with uniform consistency for selecting the stopping time. Synthetic simulations corroborate the recursions and data-driven stopping rule, while neural-network experiments exhibit the same denoising--forgetting pattern.