[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-160242-en":3,"doc-seo-160242-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},160242,1099523885074,"Riley West","https://ap-avatar.wpscdn.com/davatar_9964176cb1d06d4a9deccf72a44ae3dc",8,"Research & Report","Learning Generative Models with the Up-Propagation Algorithm - Abstract and Inversion/Top-Down Bottom-Up Learning","Up-propagation is an algorithm for inverting and learning neural network generative models. Sensory input is processed through iterative inversion using top-down connections that generate patterns from hidden variables, while a negative feedback loop based on an error signal propagated bottom-up revises the hidden variables. The same error signal drives learning of the generative model from examples. Experiments on handwritten digit images benchmark the method against principal component analysis.","Learning Generative Models with the Up􀀂Propagation Algorithm  \nJong􀀂Hoon Oh and H􀀃 Sebastian Seung  \nBell Labs􀀂 Lucent Technologies  \nMurray Hill􀀂 NJ 􀀃􀀄􀀅􀀄􀀆 fj hoh􀀂seungg􀀃bell􀀄labs􀀅com  \nAbstract  \nUp􀀂propagation is an algorithm for inverting and learning neural network generative models􀀃 Sensory input is processed by inverting a model that generates patterns from hidden variables using top􀀂down connections􀀃 The inversion process is iterative􀀄 utilizing a negative feedback loop that depends on an error signal propagated by bottom􀀂up connections􀀃 The error signal is also used to learn the generative model from examples􀀃 The algorithm is benchmarked against principal component analysis in experiments on images of handwritten digits􀀃  \nIn his doctrine of unconscious inference􀀂 Helmholtz argued that perceptions are formed by the interaction of bottom􀀇up sensory data with top􀀇down expectations􀀈 According to one interpretation of this doctrine􀀂 perception is a procedure of sequen􀀇 tial hypothesis testing􀀈 We propose a new algorithm􀀂 called up􀀇propagation􀀂 that realizes this interpretation in layered neural networks􀀈 It uses top􀀇down connections to generate hypotheses􀀂 and bottom􀀇up connections to revise them􀀈  \nIt is important to understand the di􀀉erence between up􀀇propagation and its  \nan􀀇  \ncestor􀀂 the backpropagation algorithm􀀊􀀋􀀌􀀈 Backpropagation is a learning algorithm for recognition models􀀈 As shown in Figure 􀀋a􀀂 bottom􀀇up connections recognize patterns􀀂 while top􀀇down connections propagate an error signal that is used to learn the recognition model􀀈  \nIn contrast􀀂 up􀀇propagation is an algorithm for inverting and learning generative models􀀂 as shown in Figure 􀀋b􀀈 Top􀀇down connections generate patterns from a set of hidden variables􀀈 Sensory input is processed by inverting the generative model􀀂 recovering hidden variables that could have generated the sensory data􀀈 This operation is called either pattern recognition or pattern analysis􀀂 depending on the meaning of the hidden variables􀀈 Inversion of the generative model is done iteratively􀀂 through a negative feedback loop driven by an error signal from the bottom􀀇up connections􀀈 The error signal is also used for learning the connections  \nerror recognition  \n(a)  \ngeneration error  \n(b)  \nFigure 􀀋􀀍 Bottom􀀇up and top􀀇down processing in neural networks􀀈 􀀎a􀀏 Backprop network 􀀎b􀀏 Up􀀇prop network  \nin the generative model􀀈  \nUp􀀇propagation can be regarded as a generalization of principal component analysis  \n􀀎PCA􀀏 and its variants like Conic􀀊􀀐􀀌 to nonlinear􀀂 multilayer generative models􀀈 Our experiments with images of handwritten digits demonstrate that up􀀇propagation learns a global􀀂 nonlinear model of a pattern manifold􀀈 With its global parametriza􀀇 tion􀀂 this model is distinct from locally linear models of pattern manifolds􀀊􀀑􀀌􀀈  \n􀀂 INVERTING THE GENERATIVE MODEL  \nThe generative model is a network of L 􀀒 􀀋 layers of neurons􀀂 with layer 􀀃 at the bottom and layer L at the top􀀈 The vectors xt 􀀂 t 􀀓 􀀃 􀀂 􀀂 􀀂 L 􀀂 are the activations of the layers􀀈 The pattern x􀀀 is generated from the hidden variables xL by a top􀀇down pass through the network􀀂  \nxt􀀀􀀂 􀀓 f 􀀎 Wtxt 􀀏 􀀃 t 􀀓 L 􀀃 􀀂 􀀂 􀀂 􀀃 􀀋 􀀂 􀀎􀀋􀀏  \nThe nonlinear function f acts on vectors component by component􀀈 The matrix Wt contains the synaptic connections from the neurons in layer t to the neurons in layer t 􀀂 􀀋􀀈 A bias term bt􀀀􀀂 can be added to the argument of f 􀀂 but is omitted here􀀈 It is convenient to de􀀔ne auxiliary variables t by xt 􀀓 f 􀀎t 􀀏􀀈 In terms of these auxiliary variables􀀂 the top􀀇down pass is written as  \nt􀀀􀀂 􀀓 Wtf 􀀎t 􀀏 􀀎􀀐􀀏  \nGiven a sensory input d 􀀂 the top􀀇down generative model can be inverted by 􀀔nding hidden variables xL that generate a pattern x􀀀 matching d 􀀈 If some of the hid􀀇 den variables represent the identity of the pattern􀀂 the inversion operation is called recognition 􀀈 Alternatively􀀂 the hidden variables may just be a more compact repre􀀇 sentation of the pattern􀀂 in which case the operation is called analysis or","cbCailJidzmEdZzO","https://ap.wps.com/l/cbCailJidzmEdZzO","pdf",256632,1,7,"English","en",105,"# Abstract\n## Up-Propagation vs Backpropagation\n## Inverting the Generative Model\n## Learning the Generative Model","[{\"question\":\"What is the up-propagation algorithm designed to do?\",\"answer\":\"It inverts and learns neural network generative models by iteratively reconciling a top-down hypothesis with sensory data using bottom-up error signals.\"},{\"question\":\"How does up-propagation differ from backpropagation?\",\"answer\":\"Backpropagation is a learning algorithm for recognition models that propagates an error signal bottom-up for recognition, while up-propagation performs inversion and learning for generative models using top-down generation and bottom-up revision.\"},{\"question\":\"How is inversion carried out in up-propagation?\",\"answer\":\"Given sensory input, the model generates a pattern from hidden variables via top-down connections, computes a mismatch error bottom-up, updates the hidden variables using the error, and repeats the generate-test-revise cycle until convergence.\"}]","Learning Generative Models with the Up-Propagation Algorithm - Abstract and Inversion/Top-Down Bottom-Up Learning | PDF",1788052674,18,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"learning-generative-models-with-the-up-propagation-algorithm-abstract-and-inversiontop-down-bottom-up-learning","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/learning-generative-models-with-the-up-propagation-algorithm-abstract-and-inversiontop-down-bottom-up-learning/160242/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-30",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What is the up-propagation algorithm designed to do?","Question",{"text":75,"@type":76},"It inverts and learns neural network generative models by iteratively reconciling a top-down hypothesis with sensory data using bottom-up error signals.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does up-propagation differ from backpropagation?",{"text":80,"@type":76},"Backpropagation is a learning algorithm for recognition models that propagates an error signal bottom-up for recognition, while up-propagation performs inversion and learning for generative models using top-down generation and bottom-up revision.",{"name":82,"@type":73,"acceptedAnswer":83},"How is inversion carried out in up-propagation?",{"text":84,"@type":76},"Given sensory input, the model generates a pattern from hidden variables via top-down connections, computes a mismatch error bottom-up, updates the hidden variables using the error, and repeats the generate-test-revise cycle until convergence.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":21,"doc_module":4,"doc_module_name":46,"category_name":116,"show_sort_weight":117,"slug":118},"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]