[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84758-en":3,"doc-seo-84758-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84758,4398048950312,"Violet","https://ap-avatar.wpscdn.com/avatar/400002538284de19e3c?_k=1778320343897328908",8,"Research & Report","Elastic Gang: Per-Token Membership Change for a Hard-Barriered LLM Inference Gang Co-Scheduled with OS","On-device LLM decoding is a hard-barriered CPUSIMD workload that needs nearly all CPU cores for milliseconds per token, while the OS concurrently needs those same cores for general processes. Static partitioning and naive preemption break hard barriers: a departed core can deadlock a barrier, and an unlatched arrival can silently corrupt logits. Elastic gang in Anima OS is a bare-metal x86_64 Rust kernel where inference membership changes per token without named-core waits, using an ACK-latched epoch protocol plus a generation-tagged participant latch. Hardware measurements on Zen 5 show bit-exact outputs under verified per-token membership change and improved throughput.","arXiv :2607 .04668v 1 [ cs .OS] 6 Jul 2026  \nElastic Gang: Per-Token Membership Change for a Hard-Barriered LLM Inference Gang Co-Scheduled with OS  \nProcesses  \nDaeyeon Son  \nIndependent Researcher  \nRepublic of Korea  \n[sdy1350@gmail. com](sdy1350@gmail. com)  \nJuly 2026—draft  \nAbstract  \nOn-device LLM decoding is a hard-barriered CPUSIMD computation that wants every core for milliseconds per token—while the rest of the OS wants those same cores continuously. A barriered gang cannot simply be thrown into a preemptive scheduler: an unannounced departure deadlocks a barrier, and an unannounced arrival silently corrupts logits. I present the elastic gang of Anima OS, a bare-metal x86 _64 Rust kernel in which the inference gang is a first-class schedulable entity whose core membership may change between any two tokens. The core mechanism is an ACKlatched epoch protocol that never waits on a named core—a seqlock-style generation-tagged participant latch composed with RCU/epoch-style membership consent: each token’s participant set is the intersection of the cores the gang requested and the cores that acked the current epoch, snapshotted into a single tagged latch word; an un-acked core is simply outside this token and joins at most one token later, while work-stealing absorbs its rows. Displaced general processes migrate and keep running; cores return to them the moment a generation ends. On a real AMD Zen 5 machine (8 C/16 T), inference output is bit-exact under verified per-token membership change on both a 135Mand a 7B model — the property that makes elasticity safe in a kernel whose safety gate reads logits.  \nAgainst fair static core partitions (93–100% of both solo references; the static baseline is a restriction of the same shipped binary, making the sweep an ablation of policy alone), elastic membership Pareto-dominates: at intermediate inference duty cycles it delivers 1.75 ×(25%), 1.52 × (50%), and 1.28 × (75%) the general throughput of a static 8-core split at equal or better inference throughput, recovers all eight stranded cores when inference is idle, and converges to the split at saturation—tracking a frontier no fixed partition tracks atmore than one point. Returning a lent core costs 0 .22µs  \n(p50); acquiring a busy, tenant-occupied core costs one scheduling quantum (≈16 ms) by design — a running tenant is never preempted mid-slice — amortized over the generation, while the per-token participation latch is a single tagged atomic read. Decode throughput saturates at gang width 8 in two different memory regimes (DRAM-bound 7B at 12.7 tok/s; cache-resident 135M, which loses throughput past the knee), so ceding cores past the knee is nearly free—elasticity auto-sizes the gang to the knee online.  \n1 Introduction  \nAn operating system that hosts LLM-driven agents runs two populations of work with opposite shapes. General processes—shells, services, agent runtimes, I/O—are preemptible, independent threads: the textbook workload of a weighted round-robin or virtual-time scheduler. LLM inference is the opposite: a single-programmultiple-data computation in which every participating core executes the same operator stream (∼226 barriers per token in my engine), synchronizing at hard barriers between matrix-vector waves. During a token, the gang wants every core it can get; between generations it wants none.  \nCommitting cores to either population strands the other’s cycles. A static “inference partition” of K coresidles whenever no generation is in flight—and on-device agent inference is bursty by construction (my agents’generations are 32-token syscall-bracketed bursts; interactive ask-style jobs arrive at human cadence) . A static general partition throttles every generation to 16 − K cores. And a barriered gang cannot simply be thrown into the preemptive scheduler: preempting one member mid-token stalls all members at the next barrier, and naive membership changes either deadlock a barrier on a departed core or silent","cbCaieW6dNSLqkau","https://ap.wps.com/l/cbCaieW6dNSLqkau","pdf",754844,1,14,"English","en",105,"# Abstract\n# 1 Introduction\n# Elastic Gang Protocol\n## ACK-latched epoch protocol\n## Generation-tagged participant latch","[{\"question\":\"Why is static core partitioning insufficient for on-device LLM decoding on a multitasking OS?\",\"answer\":\"Static inference partitions idle whenever no generation is in flight, while general processes throttle active generations; bursty agent workloads make both directions inefficient. Hard-barriered gang execution also cannot be safely preempted mid-token.\"},{\"question\":\"How does elastic gang prevent deadlocks when a core departs during hard-barrier inference?\",\"answer\":\"The protocol uses an ACK-latched epoch so token participation depends on acknowledgments for the current epoch at latch time. If a core departs, the generation start/end epoch bumps invalidate outstanding participation, and no barrier waits on a named core.\"},{\"question\":\"How does elastic gang ensure correctness when core membership changes between tokens?\",\"answer\":\"A generation-tagged participant latch snapshots the requested ∩ acked core set once per token into a single tagged word. All barrier wait sites and worker self-exclusion checks read the same snapshot, so membership changes only take effect at token boundaries.\"}]",1784198075,35,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"elastic-gang-per-token-membership-change-for-a-hard-barriered-llm-inference-gang-co-scheduled-with-os","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/elastic-gang-per-token-membership-change-for-a-hard-barriered-llm-inference-gang-co-scheduled-with-os/84758/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"Why is static core partitioning insufficient for on-device LLM decoding on a multitasking OS?","Question",{"text":74,"@type":75},"Static inference partitions idle whenever no generation is in flight, while general processes throttle active generations; bursty agent workloads make both directions inefficient. Hard-barriered gang execution also cannot be safely preempted mid-token.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"How does elastic gang prevent deadlocks when a core departs during hard-barrier inference?",{"text":79,"@type":75},"The protocol uses an ACK-latched epoch so token participation depends on acknowledgments for the current epoch at latch time. If a core departs, the generation start/end epoch bumps invalidate outstanding participation, and no barrier waits on a named core.",{"name":81,"@type":72,"acceptedAnswer":82},"How does elastic gang ensure correctness when core membership changes between tokens?",{"text":83,"@type":75},"A generation-tagged participant latch snapshots the requested ∩ acked core set once per token into a single tagged word. All barrier wait sites and worker self-exclusion checks read the same snapshot, so membership changes only take effect at token boundaries.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":105,"slug":137},19,"General","general"]