[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-273808-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-273808-en":130},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","sparse-token-routing-in-efficient-transformers-sewn-sparse-research-paper","Sparse Token Routing in Efficient Transformers - SEWN-sparse - Research paper","","Efficient-transformer research often assumes that tokens can be routed with adaptive computation so only a subset receives full processing. This work tests that premise end to end using SEWN, a two-stream Transformer with a learned gate selecting lightweight versus full-capacity routing. Routing yields negligible accuracy loss versus parameter-matched baselines, while faithfulness strongly depends on gate learning. Static lexicon priors fail on counterfactual tests (BoolQ), whereas contextual gating produces highly significant separation without changing accuracy.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/sparse-token-routing-in-efficient-transformers-sewn-sparse-research-paper/273808/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/sparse-token-routing-in-efficient-transformers-sewn-sparse-research-paper/273808.png","ImageObject",300,407,{"name":42,"@type":43},"Clementine","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-10-29","2026-09-15",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",10,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"What is the main premise tested by this paper in efficient transformers?","Question",{"text":63,"@type":64},"The paper tests the idea that not all tokens need equal computation: function words and redundant context can be processed cheaply, while content-bearing tokens should receive full processing.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"How does SEWN implement token routing?",{"text":68,"@type":64},"SEWN is a two-stream Transformer where tokens are routed through either a lightweight or full-capacity stream via a learned gating mechanism that makes the routing decision explicit.",{"name":70,"@type":61,"acceptedAnswer":71},"Why do static lexicon-seeded priors fail, while contextual gates succeed?",{"text":72,"@type":64},"A static lexicon-seeded prior fails the counterfactual faithfulness test on BoolQ, indicating poor faithfulness. A fully contextual gate achieves highly significant separation on both tasks under the same style of counterfactual evaluation without reducing task accuracy.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},273808,1789472846,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,111,115,120,123,126],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":107,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":112,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":113,"slug":114},8,30,"research-report",{"id":116,"doc_module":4,"doc_module_name":25,"category_name":117,"show_sort_weight":118,"slug":119},9,"Religion & Spirituality",20,"religion-spirituality",{"id":118,"doc_module":4,"doc_module_name":25,"category_name":121,"show_sort_weight":118,"slug":122},"World Cup","world-cup",{"id":55,"doc_module":4,"doc_module_name":25,"category_name":124,"show_sort_weight":55,"slug":125},"Lifestyle","lifestyle",{"id":127,"doc_module":4,"doc_module_name":25,"category_name":128,"show_sort_weight":97,"slug":129},19,"General","general",{"code":4,"msg":82,"data":131},{"doc_id":79,"user_id":132,"nickname":42,"user_avatar":133,"doc_module":4,"category_id":112,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":134,"file_id":135,"file_url":136,"file_type":137,"file_size":138,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":55,"language":139,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":140,"faqs":141,"seo_title":142,"seo_description":12,"update_tm":80,"read_time":143,"preview_page_count":4},1374391974564,"https://ap-avatar.wpscdn.com/avatar/14000253aa45c000a9e?x-image-process=image/resize,m_fixed,w_180,h_180&k=1779874745381141002","Sparse Token Routing in Efficient Transformers  \nSai Krishna Arthanari, JaeHyeong Chang, Chengzhe Sun, Siwei Lyu  \nInstitute for Artificial Intelligence and Data Science (IAD), University at Buffalo  \nBuffalo, NY, USA  \n[arthanarisaikrishna@gmail.com](arthanarisaikrishna@gmail.com), [jchang46@buffalo.edu](jchang46@buffalo.edu), [csun22@buffalo.edu](csun22@buffalo.edu), [siweilyu@buffalo.edu](siweilyu@buffalo.edu)  \narXiv : 2608 .20632v 1 [cs .CL] 2 1 Aug 2026  \nAbstract—Efficient-transformer research often motivates token pruning and adaptive computation with the claim that not all tokens require equal computational effort. We test this claim end to end using SEWN, a two-stream Transformer that routes tokens through either lightweight or full-capacity processing using a learned gate. Across our experiments, routing introduces negligible accuracy change relative to parameter-matched baselines, while the gate’s token-importance signal depends critically on how it is learned. A static lexicon-seeded prior fails a counterfactual faithfulness test on BoolQ, whereas a fully contextual gate achieves highly significant separation (p \u003C 10 −10) on both evaluated tasks without changing task accuracy.  \nWe then convert the routing signal into an explicit compute mechanism. SEWN-sparse performs hard top-k selection before the expensive stream and achieves 5.2–8.7× throughput over BERT-base and 2.6–4.4× over DistilBERT across the evaluated datasets, with a measured accuracy trade-off that remains modest at larger scale and longer context on the 27k-example PubMedQA benchmark. Under an identical counterfactual masking protocol, the gate is also compared with raw attention, attention rollout, and integrated gradients. Attention-based methods provide meaningful signals at low cost, whereas integrated gradients incurs 46× the inference cost; among the evaluated methods, SEWN-sparse provides the strongest faithfulness–efficiency trade-off.  \nA random-selection ablation with the same architecture and bottleneck reduces accuracy by only 2.6 points but eliminates significant faithfulness (p = 0 .45), isolating learned token ranking rather than hard sparsity as the source of the effect. A direct reimplementation of Mixture-of-Depths also produces a meaningful routing signal, but fails our strict ordering-based faithfulness criterion. We further find that the static-prior versus contextual-gate result does not cleanly replicate on RoBERTa, and a systematic prior size, coverage, and content sweep rules out the tested confounds, indicating backbone dependence. Finally, across five task categories, the efficiency–accuracy trade-off is robust on binary QA and sentiment classification but breaks down for long-passage multiple-choice comprehension, where both SEWN and DistilBERT exhibit substantial capacity limitations. All experiments use a fixed three-seed protocol, including negative results.  \nIndex Terms—efficient transformers, token pruning, sparse attention, interpretability, faithfulness, knowledge distillation, big data  \nI. INTRODUCTION  \nThe premise behind adaptive-computation and tokenpruning methods for transformers is that not every input token requires the same amount of processing: function words, boilerplate, and redundant context can often be handled cheaply, while content-bearing tokens need the full model. This premise underlies a substantial line of work on token  \npruning (PoWER-BERT [5], Learned Token Pruning [7], TRBERT [8]), token merging (ToMe [9]), and depth-adaptive computation (Mixture-of-Depths [10]) .  \nTwo distinct claims are usually bundled together under this premise:  \n1) An efficiency claim: skipping or cheaply processing“unimportant” tokens saves compute without much accuracy cost.  \n2) An interpretability claim: whatever signal decides which tokens are “important” doubles as a humanlegible explanation of the model’s behavior.  \nThis paper is an attempt to test both claims rigorously, separately, and honestly, using a c","cbCaicJEmm4Vdu4g","https://ap.wps.com/l/cbCaicJEmm4Vdu4g","pdf",375402,"English","# Abstract\n# Introduction\n## Adaptive computation and token pruning premise\n## Testing efficiency and interpretability claims","[{\"question\":\"What is the main premise tested by this paper in efficient transformers?\",\"answer\":\"The paper tests the idea that not all tokens need equal computation: function words and redundant context can be processed cheaply, while content-bearing tokens should receive full processing.\"},{\"question\":\"How does SEWN implement token routing?\",\"answer\":\"SEWN is a two-stream Transformer where tokens are routed through either a lightweight or full-capacity stream via a learned gating mechanism that makes the routing decision explicit.\"},{\"question\":\"Why do static lexicon-seeded priors fail, while contextual gates succeed?\",\"answer\":\"A static lexicon-seeded prior fails the counterfactual faithfulness test on BoolQ, indicating poor faithfulness. A fully contextual gate achieves highly significant separation on both tasks under the same style of counterfactual evaluation without reducing task accuracy.\"}]","Sparse Token Routing in Efficient Transformers - SEWN-sparse - Research paper | PDF",25]