[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84962-en":3,"doc-seo-84962-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},84962,7971461740886,"Theodore","https://ap-avatar.wpscdn.com/davatar_3d24733baf745e90a7e4bdd5f77d97b2",8,"Research & Report","Multiplication Beyond Groups: Stratified Fourier Mechanisms in Transformer Circuits","Transformers can learn algorithmic reasoning, yet mechanistic studies usually analyze globally invertible operations like cyclic addition and group composition. This work studies how small transformers learn modular integer multiplication over composite moduli, where the operation is non-invertible due to zero-divisors. A monoid extension of Group Composition via Representation (GCR) is proposed: computation localizes to disjoint algebraic strata, enabling Fourier mechanisms. Experiments on square-free modular multiplication show region-organized embeddings, class-sensitive routing, low-rank write directions, and local character features driving logits.","Multiplication Beyond Groups: Stratified Fourier Mechanisms in Transformer Circuits  \nZitong Andrew Chen 1 * Junaid Hasan 1 * Akhil Srinivasan 1 * Hemkesh Bandi 1 Jarod Alper 1  \narXiv :2607 .07066v 1 [ cs .LG] 8 Jul 2026  \nAbstract  \nTransformers have demonstrated a remarkable ability to learn algorithmic reasoning, yet mechanistic analyses have mostly focused on globally invertible operations such as cyclic addition and group composition. In this work, we investigate how small transformers learn modular integer multiplication over composite moduli, a fundamentally non-invertible operation due to the presence of zero-divisors. We propose the monoid extension: a localized generalization of Group Composition via Representation (GCR) that suggests the learned computation does not rely on a single global representation space. Instead, the model partitions the input space into local hierarchical algebraic regions, where group-like structure survives and Fourier mechanisms can be applied. In transformers trained on square-free modular multiplication, we find that embeddings organize around these regions, attention exhibits class-sensitive routing and low-rank write directions, and local character features explain a large fraction of the model’s output logits. Our results suggest that representation-theoretic mechanisms previously identified for group operations can extend beyond groups to more general structures.  \n1. Introduction  \nNeural networks can learn to perform structured mathematical tasks, often generalizing far beyond the examples seen during training. A central goal of mechanistic interpretability is to reverse-engineer these learned algorithms to be able to deeply understand the internal computations that produce the correct answer.  \nEarly work on modular addition and related arithmetic tasks showed that small transformers often learn rigid mathe-  \n*Equal contribution 1UW Math AI Lab, University of Washington. Correspondence to: Zitong A. Chen \u003C[zchen66@uw.edu](zchen66@uw.edu) >.  \nMechanistic Interpretability Workshop at the 43 rd International Conference on Machine Learning, Seoul, South Korea, 2026 . Copyright 2026 by the author(s) .  \nmatical structure rather than relying on rote memorization (Power et al., 2022 ; Nanda et al., 2023 ; Gromov, 2023 ; Zhong et al., 2023) . These analytical works often involved concepts from the mathematical fields of group theory, Fourier analysis, and representation and character theory, which bridge the gap between the theoretical abstract operations and the physical linear algebra as computed by the networks.  \nIn particular, prior work on learning group operations has found that models represent the group elements using Fourier features and compute group operations through representation-theoretic mechanisms. These results are generalized by Group Composition via Representation (GCR) as introduced by Chughtai et al. (2023), which explainshow models can compute group operations by embedding elements into representation space, compose the representations internally, and use the unembedding to score candidates via character-like identity tests.  \nHowever, this line of work largely assumes that the underlying algebraic structure is a group, which ensures global invertibility for all elements. An extension to arbitrary associative multiplication tables breaks this assumption, where the underlying structures now involve non-invertible elements. This raises a natural question: what happens to GCR-style mechanisms when global invertibility is no longer available?  \nThis is exactly the question posed by Chughtai et al. (2023) . In this work, we address this question by exploring the noninvertible algebraic setting through analyzing how models learn modular multiplication over composite moduli. We show that, in a transformer trained on modular multiplication over Zn for select values of n, GCR-style computation appears to localize to disjoint algebraic strata that partitions the input sp","cbCaib6qPsyoxXmy","https://ap.wps.com/l/cbCaib6qPsyoxXmy","pdf",10840401,2,1,29,"English","en",105,"# Introduction\n# Related Work","[{\"question\":\"Why does modular integer multiplication over composite moduli make mechanistic analysis harder than group operations?\",\"answer\":\"Multiplication over composite moduli is non-invertible because zero-divisors exist. This breaks the global invertibility assumption that many group-based GCR explanations rely on.\"},{\"question\":\"What is the monoid extension proposed to adapt GCR to the non-invertible setting?\",\"answer\":\"The monoid extension localizes the computation by generalizing Group Composition via Representation through partitioning the input space into local hierarchical algebraic regions. Group-like structure survives locally, allowing Fourier mechanisms to apply.\"},{\"question\":\"What evidence is reported for transformer behavior on square-free modular multiplication?\",\"answer\":\"Embeddings organize around the algebraic regions, attention shows class-sensitive routing, low-rank write directions appear in OV circuits, and local Fourier characters plus local inverses explain a large fraction of output logits.\"}]",1784199738,73,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"multiplication-beyond-groups-stratified-fourier-mechanisms-in-transformer-circuits","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,47,50],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":20},"https://docshare.wps.com/document/","Document",{"item":48,"name":12,"@type":43,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":43,"position":52},"https://docshare.wps.com/document/multiplication-beyond-groups-stratified-fourier-mechanisms-in-transformer-circuits/84962/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-24","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why does modular integer multiplication over composite moduli make mechanistic analysis harder than group operations?","Question",{"text":75,"@type":76},"Multiplication over composite moduli is non-invertible because zero-divisors exist. This breaks the global invertibility assumption that many group-based GCR explanations rely on.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What is the monoid extension proposed to adapt GCR to the non-invertible setting?",{"text":80,"@type":76},"The monoid extension localizes the computation by generalizing Group Composition via Representation through partitioning the input space into local hierarchical algebraic regions. Group-like structure survives locally, allowing Fourier mechanisms to apply.",{"name":82,"@type":73,"acceptedAnswer":83},"What evidence is reported for transformer behavior on square-free modular multiplication?",{"text":84,"@type":76},"Embeddings organize around the algebraic regions, attention shows class-sensitive routing, low-rank write directions appear in OV circuits, and local Fourier characters plus local inverses explain a large fraction of output logits.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]