[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-86296-en":3,"doc-seo-86296-105":30,"detail-sidebar-cat-0-en-105":92},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},86296,13056703020460,"Valentina","https://ap-avatar.wpscdn.com/avatar/be000253dac470eee5d?_k=1778207105932848923",8,"Research & Report","How to Tame Grokking: Representation Geometry as a Control Signal","Grokking is a delayed-generalization phenomenon where neural networks first memorize training data and only later achieve strong test performance. Despite extensive research, the drivers and timing of grokking are not fully explained. This work studies how representation geometry relates to delayed generalization, showing that dimensionality collapse reliably precedes grokking. It introduces Geometric Dimensionality Regularization (GeomDR), a spectral regularizer that reshapes the effective dimensionality of hidden activations, controlling grokking onset and speed, sometimes accelerating generalization up to 52×.","How to Tame Grokking: Representation Geometry as a Control Signal  \nMaksim A. Kazanskii  \nIndependent Researcher  \n[mkazanskii@gmail.com](mkazanskii@gmail.com)  \narXiv :2607 . 1 1666v 1 [ cs .LG] 13 Jul 2026  \nAbstract  \nGrokking is a phenomenon in which neural networks initially memorize training data and only later exhibit strong generalization after prolonged optimization. Despite extensive recent study, the factors influencing the emergence and timing of grokking remain incompletely understood. We investigate the relationship between representation geometry and delayed generalization. We find that dimensionality collapse consistently precedes the onset of grokking in all evaluated settings. Motivated by these observations, we introduce Geometric Dimensionality Regularization (GeomDR), a simple spectral regularizer that modifies the effective dimensionality of hidden representations during training. Across modular addition, modular division, and permutation composition tasks, GeomDR consistently alters grokking dynamics and can substantially accelerate the onset of generalization depending on the intervention schedule and target dimensionality. In several settings, grokking is accelerated by up to 52 times relative to standard AdamW training. Similar qualitative effects are observed in both multilayer perceptrons and transformers. Together, these results suggest that representation geometry can serve as an effective control signal for grokking and provide evidence that geometric interventions offer a practical approach for studying and influencing delayed generalization in neural networks.  \nIntroduction  \nGrokking is a delayed-generalization phenomenon in which neural networks first memorize training data and only much later achieve strong test performance after prolonged optimization [19] . This behavior differs from conventional learning dynamics, where training and test performance typically improve together, and has become a useful setting for studying generalization in overparameterized neural networks [19, 13, 17, 24] .  \nPrior work has linked grokking to weight decay, feature compression, circuit formation, and representation learning dynamics [19, 17, 13] . A common theme in these explanations is that learned representations often become progressively compressed into lower-dimensional structures. More broadly, representation geometry and dimensionality have been shown to play important roles in optimization and generalization [2, 12] .  \nRecent work has shown that grokking can be accelerated or altered through optimization dynamics, weight-  \nnorm control, sparse subnetworks, or embedding transfer [14, 16, 11, 26] . However, comparatively less attention has been paid to controlling grokking through direct interventions on the geometry of hidden representations. If delayed generalization is closely connected to representation geometry, then explicitly modifying this geometry may influence the onset and speed of grokking.  \nWe investigate this hypothesis by introducing Geometric Dimensionality Regularization (GeomDR), a representationlevel spectral regularizer that suppresses variance outside a target subspace and thereby controls the effective dimensionality of hidden representations during training. GeomDR directly modifies the covariance spectrum of hidden activations.  \nWe perform a systematic study across grokking tasks, architectures, intervention schedules, target dimensionalities, and random seeds. Our results show that geometric interventions can substantially alter delayed generalization dynamics, accelerating grokking by up to 52 times in some settings and, under stronger interventions, delaying or suppressing generalization. We further find that changes ineffective dimensionality consistently precede the transition from memorization to generalization, suggesting that representation dimensionality is not merely a diagnostic statistic but a controllable variable associated with grokking dynamics.  \nOur cont","cbCaicYbJS3KPWtE","https://ap.wps.com/l/cbCaicYbJS3KPWtE","pdf",7439505,7,1,22,"English","en",105,"# Introduction\n# Related Work","[{\"question\":\"What is grokking, and why is it important to study its timing?\",\"answer\":\"Grokking refers to a delayed-generalization behavior where networks memorize training data first and only later reach strong test performance after prolonged optimization. Studying its timing helps clarify what drives the transition from memorization to generalization in overparameterized networks.\"},{\"question\":\"What key relationship does the paper find between representation geometry and grokking?\",\"answer\":\"Dimensionality collapse consistently precedes the onset of grokking across all evaluated settings. The results suggest representation dimensionality is closely tied to the dynamics that trigger generalization.\"},{\"question\":\"How does Geometric Dimensionality Regularization (GeomDR) work, and what effect does it have?\",\"answer\":\"GeomDR is a representation-level spectral regularizer that suppresses variance outside a target subspace, directly modifying the covariance spectrum of hidden activations. It alters grokking dynamics—often accelerating generalization substantially, and under stronger interventions it can delay or suppress generalization.\"}]",1784210248,55,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":87,"head_meta":89,"extra_data":91,"updated_unix":28},"how-to-tame-grokking-representation-geometry-as-a-control-signal","",{"@graph":36,"@context":86},[37,54,69],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/how-to-tame-grokking-representation-geometry-as-a-control-signal/86296/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":24,"description":14,"dateModified":62,"datePublished":63,"encodingFormat":61,"isAccessibleForFree":64,"interactionStatistic":65},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-07-27","2026-07-16",true,{"@type":66,"interactionType":67,"userInteractionCount":20},"InteractionCounter",{"@type":68},"ViewAction",{"@type":70,"mainEntity":71},"FAQPage",[72,78,82],{"name":73,"@type":74,"acceptedAnswer":75},"What is grokking, and why is it important to study its timing?","Question",{"text":76,"@type":77},"Grokking refers to a delayed-generalization behavior where networks memorize training data first and only later reach strong test performance after prolonged optimization. Studying its timing helps clarify what drives the transition from memorization to generalization in overparameterized networks.","Answer",{"name":79,"@type":74,"acceptedAnswer":80},"What key relationship does the paper find between representation geometry and grokking?",{"text":81,"@type":77},"Dimensionality collapse consistently precedes the onset of grokking across all evaluated settings. The results suggest representation dimensionality is closely tied to the dynamics that trigger generalization.",{"name":83,"@type":74,"acceptedAnswer":84},"How does Geometric Dimensionality Regularization (GeomDR) work, and what effect does it have?",{"text":85,"@type":77},"GeomDR is a representation-level spectral regularizer that suppresses variance outside a target subspace, directly modifying the covariance spectrum of hidden activations. It alters grokking dynamics—often accelerating generalization substantially, and under stronger interventions it can delay or suppress generalization.","https://schema.org",{"og:url":52,"og:type":88,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":90,"canonical":52},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":93},[94,98,102,106,111,116,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":95,"show_sort_weight":96,"slug":97},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":99,"show_sort_weight":100,"slug":101},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":103,"show_sort_weight":104,"slug":105},"Exam",70,"exam",{"id":107,"doc_module":4,"doc_module_name":46,"category_name":108,"show_sort_weight":109,"slug":110},5,"Comic",60,"comic",{"id":112,"doc_module":4,"doc_module_name":46,"category_name":113,"show_sort_weight":114,"slug":115},6,"Technology",50,"technology",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":107,"slug":138},19,"General","general"]