[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84392-en":3,"doc-seo-84392-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84392,7971461741311,"Ophelia","https://ap-avatar.wpscdn.com/avatar/74000253aff267980c6?x-image-process=image/resize,m_fixed,w_180,h_180&k=1779345379180704826",8,"Research & Report","When Structured Sparse Autoencoders Learn Consistent Concepts Across Modalities","Sparse autoencoders (SAEs) learn sparse latent features as candidate monosemantic concepts, but in vision-language models they often fail to produce modality-consistent concepts, yielding fragmented coverage in the visual space. This work introduces Structured Sparse AutoEncoder (S2AE), which enforces concept consistency using semantic and spatial structure. It groups image patches via Transformer attention similarity and spatial proximity, then applies exclusive inter-group sparsity and group sparsity within groups. On Qwen2.5-VL-7B-Instruct, S2AE improves semantic alignment and representational efficiency while retaining near-perfect reconstruction and increased monosemanticity.","arXiv :2607 .08605v 1 [ cs .CV] 9 Jul 2026  \nWhen Structured Sparse Autoencoders Learn Consistent Concepts Across Modalities  \n1 2 1  \nWeiduo Liao , Yunqiao Yang , and Ying Wei  \n1 2  \nZhejiang University, Nanyang Technological University  \nSparse autoencoders (SAEs) have emerged as a promising technique for mechanistic interpretability by learning a set of sparse latent features in large models, each of which encodes a distinct concept. However, in vision-language models (VLMs), vanilla SAEs struggle to learn modality-consistent concepts, with concepts often exhibiting fragmented coverage (i.e., disjoint regions) in the visual modality. To address this challenge, we propose a Structured Sparse AutoEncoder (S2AE) that enforces concept consistency from both semantic and spatial perspectives in the visual modality. Specifically, we group image patches based on Transformer attention similarity and spatial proximity, and introduce a structured sparsity regularization when training the vanilla SAE. The regularization consists of exclusive sparsity for inter-group concept disentanglement and group sparsity for intra-group concept consistency, which drives the latent neurons by SAEs to specialize indistinct, semantically grounded concepts. Evaluated on the Qwen2 .5-VL-7B-Instruct model, the method achieves 6.06% average improvement in semantic alignment (mIoU) and 60.81 in representational efficiency (lower l 0 norm) while maintaining near-perfect reconstruction fidelity with an Explained Variance above 99% . Cross-modal analysis further demonstrates that S2AE enhances neuronal monosemanticity by this visual structural prior, achieving a 3.08% average gain in semantic consistency and a 2.37% average gain in monosemanticity scores for both modalities of multimodal features, thereby fostering more coherent and disentangled representations.  \n Code Repository: [github.com/liaoweiduo/s2ae](github.com/liaoweiduo/s2ae).  \n SAE visualization space: [huggingface.co/spaces/liaoweiduo/SAE-explorer](huggingface.co/spaces/liaoweiduo/SAE-explorer).  \n1. Introduction  \nRecent large vision-language models (VLMs) have achieved strong performance across diverse vision-language tasks, including image understanding (Hu et al., 2022, Jung et al., 2025), medical image diagnosis (Xianget al., 2025, Ding et al., 2025), and increasingly complex multi-modal reasoning (Zhao et al., 2025, Liu et al., 2025b, Wang et al., 2026) . However, their increasing deployment has raised concerns regarding the reliability (Guan et al., 2024, Yang et al., 2025a), generalization (Li et al., 2025, Yang et al., 2025b), and internal mechanisms (Neo et al., 2025, Jiang et al., 2025a) of these models. Mechanistic interpretability aims to address these challenges by uncovering the internal representations and computational mechanisms underlying model predictions (Wang et al., 2023, Bricken et al., 2023a, Dreyer et al., 2025), offering a principled pathway toward understanding, diagnosing, and ultimately improving VLMs in turn (Yao et al., 2024, Cywiński and Deja, 2025, Li et al., 2026) .  \nPrior neuron-level interpretation of hidden representations (Zhou et al., 2018, Oikarinen and Weng, 2023, Ahn et al., 2024) faces a key challenge: individual neurons are often polysemantic, activating in response to multiple unrelated concepts (Olah et al., 2020, Elhage et al., 2022, Gandelsman et al., 2025) . Motivated by the linear representation hypothesis (Park et al., 2024), which posits that model representations can be expressed as linear combinations of monosemantic features, sparse autoencoders (SAEs) (Lee et al., 2006)  \nVanilla Ours  \n0.51 0.58  \nVanilla Ours  \n0.46 0.66  \nVanilla Ours  \n0.68 0.90  \nFigure 1: Visualization of SAE activation masks. SAE is trained on Qwen2.5-VL-7B-Instruct’s layer 5 . Three concept cases are shown in green masks on the original images to compare the SAE activation mask performance. Takeaway: Our method achieves a significantly closer visual match with t","cbCaisy0wSbD4937","https://ap.wps.com/l/cbCaisy0wSbD4937","pdf",6916649,1,34,"English","en",105,"# Introduction\n## Mechanistic interpretability and limitations of vanilla SAEs\n## Structured grouping with attention similarity and spatial proximity\n## Structured sparsity regularization and evaluation highlights","[{\"question\":\"What problem do vanilla sparse autoencoders face in vision-language models?\",\"answer\":\"Vanilla SAEs struggle to learn modality-consistent concepts, often showing fragmented or disjoint coverage across the visual modality. They also exhibit residual polysemanticity, where one feature responds to multiple unrelated concepts.\"},{\"question\":\"How does S2AE enforce concept consistency in the visual modality?\",\"answer\":\"S2AE partitions image patches using Transformer attention similarity combined with spatial proximity, then applies structured sparsity during training. Exclusive sparsity separates inter-group concepts, while group sparsity encourages intra-group consistency.\"},{\"question\":\"What improvements does S2AE achieve on the Qwen2.5-VL-7B-Instruct model?\",\"answer\":\"The method reports a 6.06% average improvement in semantic alignment (mIoU) and 60.81 in representational efficiency (lower l0 norm) while keeping reconstruction fidelity with explained variance above 99%. Cross-modal analysis also shows gains in semantic consistency and monosemanticity.\"}]",1784195268,86,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"when-structured-sparse-autoencoders-learn-consistent-concepts-across-modalities","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/when-structured-sparse-autoencoders-learn-consistent-concepts-across-modalities/84392/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem do vanilla sparse autoencoders face in vision-language models?","Question",{"text":75,"@type":76},"Vanilla SAEs struggle to learn modality-consistent concepts, often showing fragmented or disjoint coverage across the visual modality. They also exhibit residual polysemanticity, where one feature responds to multiple unrelated concepts.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does S2AE enforce concept consistency in the visual modality?",{"text":80,"@type":76},"S2AE partitions image patches using Transformer attention similarity combined with spatial proximity, then applies structured sparsity during training. Exclusive sparsity separates inter-group concepts, while group sparsity encourages intra-group consistency.",{"name":82,"@type":73,"acceptedAnswer":83},"What improvements does S2AE achieve on the Qwen2.5-VL-7B-Instruct model?",{"text":84,"@type":76},"The method reports a 6.06% average improvement in semantic alignment (mIoU) and 60.81 in representational efficiency (lower l0 norm) while keeping reconstruction fidelity with explained variance above 99%. Cross-modal analysis also shows gains in semantic consistency and monosemanticity.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]