[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-290791-105":59,"doc-detail-290791-en":131},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":124,"head_meta":126,"extra_data":128,"updated_unix":130},105,"en","how-deep-convolutional-neural-networks-lose-spatial-information-with-training","How deep convolutional neural networks lose spatial information with training","","The work studies why deep convolutional neural networks (CNNs) lose spatial information after training and how this relates to task performance and increased sensitivity to white noise. It tests a representation-learning hypothesis using analyses of layer-wise, trained CNN representations, separating spatial pooling effects from channel pooling. It further introduces scale-detection tasks to reproduce the observed behaviors. Results show that in deep ReLU networks, noise sensitivity rises because perturbing noise accumulates during spatial pooling and is rectified by ReLU units, and that odd activations like tanh reduce noise sensitivity.",{"@graph":69,"@context":123},[70,84,106],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/how-deep-convolutional-neural-networks-lose-spatial-information-with-training/290791/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/how-deep-convolutional-neural-networks-lose-spatial-information-with-training/290791.png","ImageObject",300,407,{"name":92,"@type":93},"River Wang","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-11-03","2026-09-17",true,{"@type":102,"interactionType":103,"userInteractionCount":105},"InteractionCounter",{"@type":104},"ViewAction",13,{"@type":107,"mainEntity":108},"FAQPage",[109,115,119],{"name":110,"@type":111,"acceptedAnswer":112},"What central idea explains how deep nets learn in high dimensions according to the paper?","Question",{"text":113,"@type":114},"The paper argues that deep networks learn representations that drop information irrelevant to the task, effectively reducing the problem’s effective dimensionality.","Answer",{"name":116,"@type":111,"acceptedAnswer":117},"How does training affect CNN sensitivity to image diffeomorphisms?",{"text":118,"@type":114},"After training, neural representations become progressively less sensitive to diffeomorphisms acting on images as information propagates through the network.",{"name":120,"@type":111,"acceptedAnswer":121},"Why does the paper say CNN sensitivity to noise increases with training?",{"text":122,"@type":114},"For deep ReLU networks, increased noise sensitivity is attributed to perturbing noise piling up during spatial pooling and then being rectified by ReLU units; using odd activations like tanh reduces this sensitivity.","https://schema.org",{"og:url":83,"og:type":125,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":127,"canonical":83},"index,follow",{"doc_id":129,"site_id":62},290791,1789642576,{"code":4,"msg":5,"data":132},{"doc_id":129,"user_id":133,"nickname":92,"user_avatar":134,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":105,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":56,"language":140,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":67,"update_tm":130,"read_time":144,"preview_page_count":4},1099514067438,"https://ap-avatar.wpscdn.com/avatar/100002539ee87300030?x-image-process=image/resize,m_fixed,w_180,h_180&k=1780474512215547542","Machine Learning: Science and  \nTechnology   \nPAPER • OPEN ACCESS  \nHow deep convolutional neural networks lose spatial information with training  \nTo cite this article: Umberto M Tomasini et al 2023 Mach. Learn. : Sci. Technol. 4 045026  \nView the article online for updates and enhancements.  \nYou may also like  \n-Enhancing unsteady heat transfer simulation in porous media through the application of convolutional neural networks  \nMohammad Saraireh  \n-MD-NDNet: a multi-dimensional convolutional neural network for falsepositive reduction in pulmonary nodule detection  \nZhan Wu, Rongjun Ge, Gonglei Shi et al.  \n-Predicting lung nodule malignancies by combining deep convolutional neural network and handcrafted features  \nShulong Li, Panpan Xu, Bin Li et al.  \nThis content was downloaded from IP address [128.178.116.78](128.178.116.78) on 14/03/2024 at 10:07  \n Mach. Learn.: Sci. Technol. 4 (2023) 045026 [https://doi.org/10.1088/2632-2153/ad092c](https://doi.org/10.1088/2632-2153/ad092c)  \nOPEN ACCESS  \nRECEIVED  \n14 June 2023  \nREVISED  \n9 October 2023  \nACCEPTED FOR PUBLICATION 2 November 2023  \nPUBLISHED  \n9 November 2023  \nOriginal Content from this work may be used under the terms of the  \nCreative Commons Attribution 4 .0 licence.  \nAny further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.  \nPAPER  \nHow deep convolutional neural networks lose spatial information with training  \nUmberto M Tomasini∗􀁂, Leonardo Petrini􀁂, Francesco Cagnetta and Matthieu Wyart Institute of Physics, ´Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland  \n∗ Author to whom any correspondence should be addressed.  \n[E-mail:](E-mail: umberto.tomasini@epfl.ch)[ umberto.tomasini@epfl.ch](E-mail: umberto.tomasini@epfl.ch)  \nKeywords: deep learning theory, convolutional neural networks, curse of dimensionality, representation learning, feature learning, learning invariants  \nAbstract  \nA central question of machine learning is how deep nets manage to learn tasks in high dimensions. An appealing hypothesis is that they achieve this feat by building a representation of the data where information irrelevant to the task is lost. For image datasets, this view is supported by the observation that after (and not before) training, the neural representation becomes less and less sensitive to diffeomorphisms acting on images as the signal propagates through the network. This loss of sensitivity correlates with performance and surprisingly correlates with a gain of sensitivity to white noise acquired during training. Which are the mechanisms learned by convolutional neural networks (CNNs) responsible for the these phenomena? In particular, why is the sensitivity to noise heightened with training? Our approach consists of two steps. (1) Analyzing the layer-wise representations of trained CNNs, we disentangle the role of spatial pooling in contrast to channel pooling in decreasing their sensitivity to image diffeomorphisms while increasing their sensitivity to noise. (2) We introduce model scale-detection tasks, which qualitatively reproduce the phenomena reported in our empirical analysis. In these models we can assess quantitatively how spatial pooling affects these sensitivities. We find that the increased sensitivity to noise observed in deep ReLU networks is a mechanistic consequence of the perturbing noise piling up during spatial pooling, after being rectified by ReLU units. Using odd activation functions like tanh drastically reduces the CNNs’ sensitivity to noise.  \n1. Introduction  \nDeep learning algorithms can be successfully trained to solve a large variety of tasks [1–5], often revolving around classifying data in high-dimensional spaces. If there was little structure in such tasks, the learning procedure would be cursed by the dimension d of these spaces. Indeed, generic tasks like regression of a continuous function [6] rely on sampling neighboring points in t","cbCaiod4UxwNU0ar","https://ap.wps.com/l/cbCaiod4UxwNU0ar","pdf",2387814,"English","# Abstract\n# Introduction","[{\"question\":\"What central idea explains how deep nets learn in high dimensions according to the paper?\",\"answer\":\"The paper argues that deep networks learn representations that drop information irrelevant to the task, effectively reducing the problem’s effective dimensionality.\"},{\"question\":\"How does training affect CNN sensitivity to image diffeomorphisms?\",\"answer\":\"After training, neural representations become progressively less sensitive to diffeomorphisms acting on images as information propagates through the network.\"},{\"question\":\"Why does the paper say CNN sensitivity to noise increases with training?\",\"answer\":\"For deep ReLU networks, increased noise sensitivity is attributed to perturbing noise piling up during spatial pooling and then being rectified by ReLU units; using odd activations like tanh reduces this sensitivity.\"}]","How deep convolutional neural networks lose spatial information with training | PDF",48]