[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84185-en":3,"doc-seo-84185-105":30,"detail-sidebar-cat-0-en-105":83},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},84185,1374391974468,"Eden","https://ap-avatar.wpscdn.com/davatar_29158cc5080c5b710cf443261637dec0",8,"Research & Report","Distributed Sparse Interventions in Language Models","Language models can infer and compose tasks from context, yet the mechanisms behind task representations and composition remain unclear. This work studies causal effects of neuron-level interventions on task behavior and shows that neuron-specific nonlinearities and cross-neuron interactions challenge prior global, vector-based steering assumptions. It introduces Distributed Sparse Interventions (DSI), identifying sparse neuron sets for task-relevant computations and enabling fine-grained, localized control using only about 0.01% of neurons across multiple instruction-tuned models.","arXiv :2607 .07 128v 1 [ cs .LG] 8 Jul 2026  \nDistributed Sparse Interventions in Language Models  \nMaximilian S. Ernst 1 ,2 ,3 ∗ Lorenz Linhardt3 ,4 Aaron Peikert2 Oliver Eberle3 ,4  \n1Max Planck School of Cognition, Leipzig, Germany  \n2 Center for Lifespan Psychology, Max Planck Institute for Human Development, Berlin, Germany  \n3Machine Learning Group, Technische Universität Berlin, Berlin, Germany  \n4Berlin Institute for the Foundations of Learning and Data (BIFOLD), Berlin, Germany  \nAbstract  \nLanguage models perform a wide range of tasks at varying levels of abstraction with the capacity to flexibly infer tasks from context, execute multiple tasks simultaneously, and select among competing tasks. To study the role of model components in task behaviour, their causal influence can be investigated through interventions.  \nPrior work on model steering has largely focused on interventions along global directions in activation space, modeling task representations as approximately linear and additive. By studying interventions at the neuron level, we find substantial, neuron-specific nonlinear effects on model outputs that are not captured by current steering approaches. We introduce Distributed Sparse Interventions (DSI), an intervention approach that considers nonlinearities and interactions between neurons across layers to identify sparse sets of neurons that elicit task-relevant computations. Across a range of tasks, we demonstrate that DSI can activate task behaviour in instruction-tuned language models by localising and intervening on as few as 0.01% of neurons, highlighting the effectiveness of sparse, distributed interventions in the neuron basis. Additionally, adopting a set-based perspective enables computations over the identified neuron sets, offering insights into the roles of individual neurons by analysing their effects across tasks. Through sparse interventions, DSI enables fine-grained control over model behaviour, localisation of task-relevant neuron sets, and furthers our understanding of task composition.  \n1 Introduction  \nModern language models demonstrate flexible task-solving behaviours, including the adaptive combination and execution of multiple tasks through in-context learning (ICL) [10, 26, 53] . However, the mechanisms by which these models represent and compose tasks remain largely elusive, particularly in scenarios requiring the combination, isolation, or selective activation of behaviours. In the context of model steering, prior research has primarily focused on interventions along directions in activation space [43, 49, 18, 8, 32], leveraging the observation that many task-relevant behaviours exhibit approximately linear structure in representation space. Such vector-based steering approaches typically rely on a number of key assumptions: (1) effective interventions require modifying all neuron activations in a specific layer or attention head,(2) tasks are represented as linear directions in activation space that can be combined additively, and (3) local first-order effect approximationsin activation space are sufficient to identify interventions that induce approximately proportional changes in behaviour.  \nWe demonstrate that these assumptions can be violated in practice. Task-relevant computations are distributed across model components, and interventions often exhibit nonlinear interaction effects across neurons that violate the assumptions of uniform first-order approximations. This motivates the development of fine-grained intervention strategies that operate directly on neurons rather than on global directions in activation space. Such approaches could enable less invasive and more  \n∗ Correspondence to [ernst@mpib-berlin.mpg.de](ernst@mpib-berlin.mpg.de). Code is available at GitHub.  \nPreprint.  \ninterpretable interventions. However, they are currently constrained by the difficulty of reliably localising task-relevant neurons without relying on the strong assumptions inherent in firs","cbCain9qa381SAV7","https://ap.wps.com/l/cbCain9qa381SAV7","pdf",941611,3,1,29,"English","en",105,"# Introduction\n## Model steering assumptions\n## Neuron-level rationale for DSI\n# Related work\n## Model steering","[{\"question\":\"How many neurons does DSI typically need to activate tasks?\",\"answer\":\"Across 12 tasks on several models, DSI often activates task behavior by intervening on as few as 0.01% of neurons (about 8–64 neuron units), frequently matching or surpassing 10-shot ICL performance.\"}]",1784193735,73,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":78,"head_meta":80,"extra_data":82,"updated_unix":28},"distributed-sparse-interventions-in-language-models","",{"@graph":36,"@context":77},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,50],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":20},"https://docshare.wps.com/document/research-report/",{"item":51,"name":13,"@type":43,"position":52},"https://docshare.wps.com/document/distributed-sparse-interventions-in-language-models/84185/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-28","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71],{"name":72,"@type":73,"acceptedAnswer":74},"How many neurons does DSI typically need to activate tasks?","Question",{"text":75,"@type":76},"Across 12 tasks on several models, DSI often activates task behavior by intervening on as few as 0.01% of neurons (about 8–64 neuron units), frequently matching or surpassing 10-shot ICL performance.","Answer","https://schema.org",{"og:url":51,"og:type":79,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":81,"canonical":51},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":84},[85,89,93,97,102,107,112,115,120,123,127],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":86,"show_sort_weight":87,"slug":88},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":90,"show_sort_weight":91,"slug":92},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Exam",70,"exam",{"id":98,"doc_module":4,"doc_module_name":46,"category_name":99,"show_sort_weight":100,"slug":101},5,"Comic",60,"comic",{"id":103,"doc_module":4,"doc_module_name":46,"category_name":104,"show_sort_weight":105,"slug":106},6,"Technology",50,"technology",{"id":108,"doc_module":4,"doc_module_name":46,"category_name":109,"show_sort_weight":110,"slug":111},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":113,"slug":114},30,"research-report",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},9,"Religion & Spirituality",20,"religion-spirituality",{"id":118,"doc_module":4,"doc_module_name":46,"category_name":121,"show_sort_weight":118,"slug":122},"World Cup","world-cup",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":124,"slug":126},10,"Lifestyle","lifestyle",{"id":128,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":98,"slug":130},19,"General","general"]