Learning Generative Models with the Up-Propagation Algorithm - Abstract and Inversion/Top-Down Bottom-Up Learning

Aug 30, 2026

Up-propagation is an algorithm for inverting and learning neural network generative models. Sensory input is processed through iterative inversion using top-down connections that generate patterns from hidden variables, while a negative feedback loop based on an error signal propagated bottom-up revises the hidden variables. The same error signal drives learning of the generative model from examples. Experiments on handwritten digit images benchmark the method against principal component analysis.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages7

Reading Time18 min

AccessFree to download

Abstract

Up-Propagation vs Backpropagation

Inverting the Generative Model

Learning the Generative Model

Learning Generative Models with the Up􀀂Propagation Algorithm Jong􀀂Hoon Oh and H􀀃 Sebastian Seung Bell Labs􀀂 Lucent Technologies Murray Hill􀀂 NJ 􀀃􀀄􀀅􀀄􀀆 fj hoh􀀂seungg􀀃bell􀀄labs􀀅com Abstract Up􀀂propagation is an algorithm for inverting and learning neural network generative models􀀃 Sensory input is processed by inverting a model that generates patterns from hidden variables using top􀀂down connections􀀃 The inversion process is iterative􀀄 utilizing a negative feedback loop that depends on an error signal propagated by bottom􀀂up connections􀀃 The error signal is also used to learn the generative model from examples􀀃 The algorithm is benchmarked against principal component analysis in experiments on images of handwritten digits􀀃 In his doctrine of unconscious inference􀀂 Helmholtz argued that perceptions are formed by the interaction of bottom􀀇up sensory data with top􀀇down expectations􀀈 According to one interpretation of this doctrine􀀂 perception is a procedure of sequen􀀇 tial hypothesis testing􀀈 We propose a new algorithm􀀂 called up􀀇propagation􀀂 that realizes this interpretation in layered neural networks􀀈 It uses top􀀇down connections to generate hypotheses􀀂 and bottom􀀇up connections to revise them􀀈 It is important to understand the di􀀉erence between up􀀇propagation and its an􀀇 cestor􀀂 the backpropagation algorithm􀀊􀀋􀀌􀀈 Backpropagation is a learning algorithm for recognition models􀀈 As shown in Figure 􀀋a􀀂 bottom􀀇up connections recognize patterns􀀂 while top􀀇down connections propagate an error signal that is used to learn the recognition model􀀈 In contrast􀀂 up􀀇propagation is an algorithm for inverting and learning generative models􀀂 as shown in Figure 􀀋b􀀈 Top􀀇down connections generate patterns from a set of hidden variables􀀈 Sensory input is processed by inverting the generative model􀀂 recovering hidden variables that could have generated the sensory data􀀈 This operation is called either pattern recognition or pattern analysis􀀂 depending on the meaning of the hidden variables􀀈 Inversion of the generative model is done iteratively􀀂 through a negative feedback loop driven by an error signal from the bottom􀀇up connections􀀈 The error signal is also used for learning the connections error recognition (a) generation error (b) Figure 􀀋􀀍 Bottom􀀇up and top􀀇down processing in neural networks􀀈 􀀎a􀀏 Backprop network 􀀎b􀀏 Up􀀇prop network in the generative model􀀈 Up􀀇propagation can be regarded as a generalization of principal component analysis 􀀎PCA􀀏 and its variants like Conic􀀊􀀐􀀌 to nonlinear􀀂 multilayer generative models􀀈 Our experiments with images of handwritten digits demonstrate that up􀀇propagation learns a global􀀂 nonlinear model of a pattern manifold􀀈 With its global parametriza􀀇 tion􀀂 this model is distinct from locally linear models of pattern manifolds􀀊􀀑􀀌􀀈 􀀂 INVERTING THE GENERATIVE MODEL The generative model is a network of L 􀀒 􀀋 layers of neurons􀀂 with layer 􀀃 at the bottom and layer L at the top􀀈 The vectors xt 􀀂 t 􀀓 􀀃 􀀂 􀀂 􀀂 L 􀀂 are the activations of the layers􀀈 The pattern x􀀀 is generated from the hidden variables xL by a top􀀇down pass through the network􀀂 xt􀀀􀀂 􀀓 f 􀀎 Wtxt 􀀏 􀀃 t 􀀓 L 􀀃 􀀂 􀀂 􀀂 􀀃 􀀋 􀀂 􀀎􀀋􀀏 The nonlinear function f acts on vectors component by component􀀈 The matrix Wt contains the synaptic connections from the neurons in layer t to the neurons in layer t 􀀂 􀀋􀀈 A bias term bt􀀀􀀂 can be added to the argument of f 􀀂 but is omitted here􀀈 It is convenient to de􀀔ne auxiliary variables t by xt 􀀓 f 􀀎t 􀀏􀀈 In terms of these auxiliary variables􀀂 the top􀀇down pass is written as t􀀀􀀂 􀀓 Wtf 􀀎t 􀀏 􀀎􀀐􀀏 Given a sensory input d 􀀂 the top􀀇down generative model can be inverted by 􀀔nding hidden variables xL that generate a pattern x􀀀 matching d 􀀈 If some of the hid􀀇 den variables represent the identity of the pattern􀀂 the inversion operation is called recognition 􀀈 Alternatively􀀂 the hidden variables may just be a more compact repre􀀇 sentation of the pattern􀀂 in which case the operation is called analysis or

FAQs

What is the up-propagation algorithm designed to do?

It inverts and learns neural network generative models by iteratively reconciling a top-down hypothesis with sensory data using bottom-up error signals.

How does up-propagation differ from backpropagation?

Backpropagation is a learning algorithm for recognition models that propagates an error signal bottom-up for recognition, while up-propagation performs inversion and learning for generative models using top-down generation and bottom-up revision.

How is inversion carried out in up-propagation?

Given sensory input, the model generates a pattern from hidden variables via top-down connections, computes a mismatch error bottom-up, updates the hidden variables using the error, and repeats the generate-test-revise cycle until convergence.

Aug 30, 2026

Upload by Riley West