Improved Contrastive Divergence Training of Energy-Based Models - Research summary

Aug 5, 2026

Upload by Maeve

Contrastive divergence training for energy-based models is widely used but often suffers from unstable optimization dynamics. The work improves stability by analyzing an extra gradient term that is difficult to compute and usually omitted, showing it is numerically significant yet tractable to estimate. The approach supports better robustness and generation quality via data augmentation and multi-scale processing, and it evaluates stability across model architectures. Experiments demonstrate improved performance across benchmarks and tasks including image generation, out-of-distribution detection, and compositional generation.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages13

Reading Time33 min

AccessFree to download

Abstract

Introduction

Energy-Based Models and Contrastive Divergence

Motivation: Training Instabilities and Common Mitigations

The Neglected Gradient Term and Its Efficient Estimation

MIT Open Access Articles Improved Contrastive Divergence Training of Energy-Based Models The MIT Faculty has made this article openly available. Please share how this access benefits you. Your story matters. Citation: Du , Yilun , Li , Shuang , Tenenbaum , Joshua and Mordatch , Igor. 2021. "Improved Contrastive Divergence Training of Energy-Based Models." INTERNATIONAL CONFERENCE ON MACHINE LEARNING , VOL 139 , 139. Persistent URL: [https://hdl.handle. net/1721.1/150393](https://hdl.handle. net/1721.1/150393) Version: Final published version: final published article , as it appeared in a journal , conference proceedings , or other formally published context Terms of Use: Article is made available in accordance with the publisher 's policy and may be subject to US copyright law. Please refer to the publisher 's site for terms of use. Improved Contrastive Divergence Training of Energy-Based Models Yilun Du 1 Shuang Li 1 Joshua Tenenbaum 1 Igor Mordatch 2 Abstract Contrastive divergence is a popular method of training energy-based models, but is known to have difﬁculties with training stability. We propose an adaptation to improve contrastive divergence training by scrutinizing a gradient term that is difﬁcult to calculate and is often left out for convenience. We show that this gradient term is numerically signiﬁcant and in practice is important to avoid training instabilities, while being tractable to estimate. We further highlight how data augmentation and multi-scale processing can be used to improve model robustness and generation quality. Finally, we empirically evaluate stability of model architectures and show improved performance on a host of benchmarks and use cases,such as image generation, OOD detection, and compositional generation. 1 Introduction Energy-Based models (EBMs) have received an inﬂux of interest recently and have been applied to realistic image generation (Han et al., 2019 ; Du & Mordatch, 2019), 3D shapes synthesis (Xie et al., 2018b) , out of distribution and adversarial robustness (Lee et al., 2018 ; Du & Mordatch, 2019 ; Grathwohl et al., 2019), compositional generation (Hinton, 1999 ; Du et al., 2020a), memory modeling (Bartunov et al., 2019), text generation (Deng et al., 2020), video generation (Xie et al., 2017), reinforcement learning (Haarnoja et al., 2017 ; Du et al., 2019), continual learning (Li et al., 2020), protein design and folding (Ingraham et al.; Du et al., 2020b) and biologically-plausible training (Scellier & Bengio, 2017) . Contrastive divergence is a popular and elegant procedure for training EBMs proposed by (Hinton, 2002) which lowers the energy of the training data and raises the energy of the sampled confabulations generated by the model. The model confabulations are generated via an MCMC process (commonly Gibbs sampling or Langevin 1MIT CSAIL 2 Google Brain. Correspondence to: Yilun Du <[yilundu@mit.edu](yilundu@mit.edu) >. Proceedings of the 38 th International Conference on Machine Learning, PMLR 139, 2021 . Copyright 2021 by the author(s) . Figure 1: (Left) 128x128 samples on unconditional CelebA-HQ.(Right) 128x128 samples on unconditional LSUN Bedroom. dynamics), leveraging the extensive body of research on sampling and stochastic optimization. The appeal of contrastive divergence is its simplicity and extensibility. It does not require training additional auxiliary networks (Kim & Bengio, 2016 ; Dai et al., 2019) (which introduce additional tuning and balancing demands), and can be used to compose models zero-shot. Despite these advantages, training EBMs with contrastive divergence has been challenging due to training instabilities. Ensuring training stability required either combinations of spectral normalization and Langevin dynamics gradient clipping (Du & Mordatch, 2019), parameter tuning (Grathwohl et al., 2019), early stopping of MCMC chains (Nijkampet al., 2019b), or avoiding the use of modern deep learning components, such as self

FAQs

What problem does improved contrastive divergence training address?

It addresses training instabilities commonly observed when using contrastive divergence to train energy-based models.

What is the key change proposed by the work?

The method scrutinizes a gradient term introduced by changes to the energy function, estimating it efficiently instead of ignoring it.

How does the approach improve robustness and generation quality?

It leverages data augmentation and multi-scale processing to enhance mixing in MCMC transitions and improve sample diversity and quality.

Aug 5, 2026

Upload by Maeve