[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-203639-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-203639-en":131},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","similarity-of-neural-network-representations-revisited","Similarity of Neural Network Representations Revisited","","Recent work seeks to understand neural network behavior by comparing representations across layers and across separately trained models. Methods based on canonical correlation analysis (CCA) can characterize multivariate similarity, but invariant statistics—including any invariant to invertible linear transforms—fail for higher-dimensional representations beyond the number of data points. A new similarity index is introduced to relate representational similarity matrices without this limitation. The index is equivalent to centered kernel alignment (CKA) and ties closely to CCA, while enabling reliable correspondences across different initializations.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/similarity-of-neural-network-representations-revisited/203639/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/similarity-of-neural-network-representations-revisited/203639.png","ImageObject",300,407,{"name":42,"@type":43},"Cart","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-11-04","2026-09-04",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",13,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"Why does CCA fail to measure meaningful similarity for higher-dimensional representations?","Question",{"text":63,"@type":64},"CCA belongs to statistics invariant to invertible linear transformations, but such invariances cannot measure meaningful similarities when representation dimension exceeds the number of data points.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"What similarity index is introduced, and what limitation does it avoid?",{"text":68,"@type":64},"The paper introduces a similarity index that measures the relationship between representational similarity matrices. It avoids the higher-dimension limitation affecting CCA and other invertible-transform-invariant statistics.",{"name":70,"@type":61,"acceptedAnswer":71},"How is the proposed similarity index related to CKA and CCA?",{"text":72,"@type":64},"The similarity index is equivalent to centered kernel alignment (CKA) and is closely connected to CCA, while also supporting reliable correspondences between representations trained from different initializations.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},203639,1790237189,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,111,115,120,123,127],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":107,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":112,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":113,"slug":114},8,30,"research-report",{"id":116,"doc_module":4,"doc_module_name":25,"category_name":117,"show_sort_weight":118,"slug":119},9,"Religion & Spirituality",20,"religion-spirituality",{"id":118,"doc_module":4,"doc_module_name":25,"category_name":121,"show_sort_weight":118,"slug":122},"World Cup","world-cup",{"id":124,"doc_module":4,"doc_module_name":25,"category_name":125,"show_sort_weight":124,"slug":126},10,"Lifestyle","lifestyle",{"id":128,"doc_module":4,"doc_module_name":25,"category_name":129,"show_sort_weight":97,"slug":130},19,"General","general",{"code":4,"msg":82,"data":132},{"doc_id":79,"user_id":133,"nickname":42,"user_avatar":134,"doc_module":4,"category_id":112,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":140,"language":141,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":142,"faqs":143,"seo_title":144,"seo_description":12,"update_tm":145,"read_time":146,"preview_page_count":4},18829141979164,"https://eur-avatar.wpscdn.com/davatar_6f874abed73319feea01a86fa6f0fab8","Similarity of Neural Network Representations Revisited  \nSimon Kornblith 1 Mohammad Norouzi 1 Honglak Lee 1 Geoffrey Hinton 1  \nAbstract  \nRecent work has sought to understand the behavior of neural networks by comparing representations between layers and between different trained models. We examine methods for comparing neural network representations based on canonical correlation analysis (CCA) . We show that CCA belongs to a family of statistics for measuring multivariate similarity, but that neither CCA nor anyother statistic that is invariant to invertible linear transformation can measure meaningful similarities between representations of higher dimension than the number of data points. We introduce a similarity index that measures the relationship between representational similarity matrices and does not suffer from this limitation. This similarity index is equivalent to centered kernel alignment (CKA) and is also closely connected to CCA.  \nUnlike CCA, CKA can reliably identify correspondences between representations in networks trained from different initializations.  \n1. Introduction  \nAcross a wide range of machine learning tasks, deep neural networks enable learning powerful feature representations automatically from data. Despite impressive empirical advances of deep neural networks in solving various tasks, the problem of understanding and characterizing the neural network representations learned from data remains relatively under-explored. Previous work (e.g. Advani & Saxe (2017); Amari et al. (2018); Saxe et al. (2014)) has made progress in understanding the theoretical dynamics of the neural network training process. These studies are insightful, but fundamentally limited, because they ignore the complex interaction between the training dynamics and structured data. A window into the network's representation can provide more information about the interaction between machine learning algorithms and data than the value of the loss function alone.  \n1 Google Brain. Correspondence to: Simon Kornblith \u003Cskorn[blith@google.com](blith@google.com) >.  \nProceedings of the 36 th International Conference on Machine Learning, Long Beach, California, PMLR 97, 2019 . Copyright 2019 by the author(s) .  \nThis paper investigates the problem of measuring similarities between deep neural network representations. An effective method for measuring representational similarity could help answer many interesting questions, including: (1) Do deep neural networks with the same architecture trained from different random initializations learn similar representations? (2) Can we establish correspondences between layers of different network architectures? (3) How similar are the representations learned using the same network architecture from different datasets?  \nWe build upon previous studies investigating similarity between the representations of neural networks (Laakso & Cottrell, 2000 ; Li et al., 2015 ; Raghu et al., 2017 ; Morcoset al., 2018 ; Wang et al., 2018) . We are also inspired by the extensive neuroscience literature that uses representational similarity analysis (Kriegeskorte et al., 2008a ; Edelman, 1998) to compare representations across brain areas (Haxbyet al., 2001 ; Freiwald & Tsao, 2010), individuals (Connolly et al., 2012), species (Kriegeskorte et al., 2008b), and behaviors (Elsayed et al., 2016), as well as between brainsand neural networks (Yamins et al., 2014 ; Khaligh-Razavi & Kriegeskorte, 2014 ; Sussillo et al., 2015) .  \nOur key contributions are summarized as follows:  \n􀀏 We discuss the invariance properties of similarity indexes and their implications for measuring similarity of neural network representations.  \n􀀏 We motivate and introduce centered kernel alignment (CKA) as a similarity index and analyze the relationship between CKA, linear regression, canonical correlation analysis (CCA), and related methods (Raghu et al., 2017 ; Morcos et al., 2018) .  \n􀀏 We show that CKA is able to determine the correspo","cbCaiaW74l38Yuwv","https://ap.wps.com/l/cbCaiaW74l38Yuwv","pdf",1479027,11,"English","# Abstract\n# Introduction\n# Problem Statement\n# What Should Similarity Be Invariant To?\n## Invariance to Invertible Linear Transformation","[{\"question\":\"Why does CCA fail to measure meaningful similarity for higher-dimensional representations?\",\"answer\":\"CCA belongs to statistics invariant to invertible linear transformations, but such invariances cannot measure meaningful similarities when representation dimension exceeds the number of data points.\"},{\"question\":\"What similarity index is introduced, and what limitation does it avoid?\",\"answer\":\"The paper introduces a similarity index that measures the relationship between representational similarity matrices. It avoids the higher-dimension limitation affecting CCA and other invertible-transform-invariant statistics.\"},{\"question\":\"How is the proposed similarity index related to CKA and CCA?\",\"answer\":\"The similarity index is equivalent to centered kernel alignment (CKA) and is closely connected to CCA, while also supporting reliable correspondences between representations trained from different initializations.\"}]","Similarity of Neural Network Representations Revisited | PDF",1788562349,28]