Publications

Why Diffusion Models Don’t Memorize: The Role of Implicit Dynamical Regularization in Training

Tony Bonnaire*, Raphaël Urfin*, Giulio Biroli, Marc Mézard.

NeurIPS 2025 Oral Best Paper

In this work, we investigate the role of the training dynamics in the transition from generalization to memorization in Diffusion Models. Through extensive experiments and theoretical analysis, we identify two distinct timescales: an early time $\tau_\mathrm{gen}$ at which models begin to generate high-quality samples, and a later time $\tau_\mathrm{mem}$ beyond which memorization emerges. Crucially, we find that $\tau_\mathrm{mem}$ increases linearly with the training set size $n$, while $\tau_\mathrm{gen}$ remains constant. This creates a growing window of training times with $n$ where models generalize effectively, despite showing strong memorization if training continues beyond it. It is only when $n$ becomes larger than a model-dependent threshold that overfitting disappears at infinite training times. These findings reveal a form of implicit dynamical regularization in the training dynamics, which allow to avoid memorization even in highly overparameterized settings.

Proceedings J. Stat. Mech. arXiv OpenReview Code Video (Voxel51 talk) Video (ICTP talk)

@inproceedings{NEURIPS2025_ceb7f3cc,
 author = {Bonnaire, Tony and Urfin, Rapha\"{e}l and Biroli, Giulio and Mezard, Marc},
 booktitle = {Advances in Neural Information Processing Systems},
 editor = {D. Belgrave and C. Zhang and H. Lin and R. Pascanu and P. Koniusz and M. Ghassemi and N. Chen},
 pages = {141266--141286},
 publisher = {Curran Associates, Inc.},
 title = {Why Diffusion Models Don't Memorize: The Role of Implicit Dynamical Regularization in Training},
 url = {https://proceedings.neurips.cc/paper_files/paper/2025/file/ceb7f3cc876a6dcb15130a645b5a4507-Paper-Conference.pdf},
 volume = {38},
 year = {2025}
}

First Learn, Then Memorize: The Spectral Bias of Diffusion Models

Raphaël Urfin, Tony Bonnaire, Giulio Biroli, Marc Mézard.

Preprint, 2026 Preprint

Diffusion models trained on a finite dataset first learn to generate novel, high-quality samples and only much later collapse onto their training set. We show that the training dynamics of the score are governed — exactly, and at any width — by the Gram matrix of the Neural Tangent Kernel evaluated on the noisy training data, and that using $m$ noise realizations per sample splits its spectrum into two bulks. The large eigenvalues carry the global features of the target distribution; the smallest ones, created by the repeated noising, align with sample-specific noise directions and set a memorization timescale parametrically larger in the dataset size $n$. We solve the spectrum analytically in the lazy high-dimensional limit, and find the same two-bulk structure in convolutional NTKs on CelebA and in finite-width U-Nets, where truncating the Gram matrix tunes the generalization–memorization transition.

arXiv

@misc{urfin2026learnmemorizespectralbias,
      title={First Learn, Then Memorize: The Spectral Bias of Diffusion Models},
      author={Raphaël Urfin and Tony Bonnaire and Giulio Biroli and Marc Mézard},
      year={2026},
      eprint={2609.35377},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2609.35377},
}

Weighting Schedules Govern What and When Score-Based Generative Models Learn from Multimodal Data

Jérémie Klinger, Raphaël Urfin, Giulio Biroli, Marylou Gabrié.

Preprint, 2026 Preprint

Score-based generative models are trained on a loss integrated over time $t$ with a weighting schedule $w(t)$. Decomposing that loss into its single-time contributions, we show that the signal-to-noise ratio $\Lambda(t)$ sets the rate at which each feature of a multimodal target — the mode directions and their relative weights — is acquired during training. At high $\Lambda(t)$ all mode directions are learned together, on a single timescale insensitive to their amplitudes, while the relative weights are not learned at all. Only near the speciation time, where $\Lambda(t)\sim 1$, does every feature become learnable, each on its own timescale. This clarifies how the choice of $w(t)$ shapes learning dynamics, and is confirmed on image and human genome haplotype generation.

arXiv

@misc{klinger2026weightingschedulesgovernscorebased,
      title={Weighting Schedules Govern What and When Score-Based Generative Models Learn from Multimodal Data},
      author={Jérémie Klinger and Raphaël Urfin and Giulio Biroli and Marylou Gabrié},
      year={2026},
      eprint={2609.35322},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2609.35322},
}

Double Descent and Malign Overfitting in Diffusion Models

Raphaël Urfin*, Tony Bonnaire*, Giulio Biroli, Marc Mézard.

Preprint, 2026 Preprint

Conventional wisdom in deep learning says overparameterization is benign: models with more parameters $p$ than training samples $n$ still generalize well, and the test error follows a double-descent curve. Diffusion models seem to do the opposite — overparameterization leads to memorization. Studying the empirical score-matching loss with a fixed number $m$ of noised realizations per sample, we show that the interpolation peak shifts from the classical $p \sim n$ to $p \sim nm$, and that after the peak the test loss settles on a plateau higher than its earlier minimum — what we call malign overfitting. Since $m \gg 1$ in practice, real diffusion models sit on the rising branch before the peak, where the test loss already grows and the model starts memorizing. Overparameterization nonetheless remains beneficial with optimal regularization.

arXiv

@misc{urfin2026doubledescentmalignoverfitting,
      title={Double Descent and Malign Overfitting in Diffusion Models},
      author={Raphaël Urfin and Tony Bonnaire and Giulio Biroli and Marc Mézard},
      year={2026},
      eprint={2609.26392},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2609.26392},
}