[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85673-en":3,"doc-seo-85673-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85673,549758252649,"Ivy","https://ap-avatar.wpscdn.com/avatar/8000253669c5317157?_k=1778319167496531819",8,"Research & Report","MawForge Memory-Bounded Expert Materialization for Local Mixture-of-Experts Inference","Sparse Mixture-of-Experts (MoE) models reduce active computation per token, yet local inference can still fail because full model residency, KV caches, runtime buffers, and OS memory headroom are required on unified-memory machines. MawForge makes local MoE serving practical by keeping common tensors resident, storing the full model on disk, and materializing routed expert tensors into a bounded execution cache on demand. The paper details the split-pack architecture, a native GGUF serving path, and 600-row validation on a MacBook Pro M5 Pro under an 18 GiB target.","MawForge: Memory-Bounded Expert Materialization for Local  \nMixture-of-Experts Inference  \nCraig Opie, Holocron Security, Inc.  \narXiv :2607 .09686v1 [ cs .LG] 17 Jun 2026  \nAbstract—Sparse Mixture-of-Experts (MoE) language models separate total parameter count from per-token active computation, but local inference systems often still require the full model, key-value cache, runtime buffers, and operatingsystem headroom to fit in fast memory. MawForge tests a different systems hypothesis: local MoE serving can be made practical on constrained unified-memory machines by storing the full model on disk, keeping common tensors resident, and materializing routed expert tensors into a bounded execution cache on demand.  \nThis article presents MawForge's split-pack architecture, budget model, native GGUF serving path, and completed local validation on a MacBook Pro M5 Pro with 24 GB unified memory under an 18 GiB explicit serving target. The validation matrix contains 600 non-speculative MawForge rows across three model profiles, five cache settings, two context lengths, four prompt classes, and five repetitions per cell. Static planning accepted 27 of 30 unique model/context/cache cells and rejected 3 cells before execution; all 540 statically feasible generation rows returned valid 96-token completions without triggering the memory guard. The results show that MawForge can serve large GGUF MoE models within a bounded local memory envelope, including a 34 GB Qwen3.6 35B A3B Q8_ 0 profile and a 25 GB Gemma 4 26B A4B Q8_ 0 profile. They also show that expert-cache size is non-monotonic: larger caches consistently improve hit rate and reduce materialized bytes, but they can reduce throughput by increasing host memory pressure. Qwen Q8 favored the smallest tested cache, 15%, at both 4K and 32K contexts; Gemma Q8 favored 35% among feasible points; Qwen Q4 shifted with context and prompt class. A focused speculative-decoding addendum found that Gemma MTP speculation accepted more draft tokens but reduced throughput and increased expert materialization, indicating that dense-model speculation assumptions do not transfer directly to split-pack MoE serving.  \nThe central finding is that MawForge is effective as abounded execution mechanism and measurement substrate for local MoE inference, but not as a cache-maximization policy. Performance depends on balancing expert reuse against resident footprint, KV-cache size, quantization, route locality, and macOS memory pressure.  \nIndex Terms—Mixture-of-Experts, local inference, memory hierarchy, expert caching, GGUF, Apple Silicon, model serving, systems evaluation  \nI. INTRODUCTION  \nLarge language model deployment is usually governed by a hard residency constraint: model weights must fit in fast memory with additional headroom for key-value (KV) cache, runtime allocations, accelerator buffers, and operating-system pressure. This constraint is especially sharp on consumer and workstation systems with unified memory because the CPU, GPU, kernel, display stack, filesystem cache, and inference runtime draw from the same physical pool.  \nSparse MoE models relax the compute side of this constraint. A token does not activate every expert; it activates a routed subset. However, most local inference paths still treat model residency as a whole-file loading problem. The full GGUF is mapped or loaded, and failure occurs when the aggregate model, KV cache, and runtime memory exceed what the machine can absorb. Sparse activation does not automatically become sparse residency.  \nMawForge reframes the problem as a memory hierarchy. The full model remains a durable disk object. Common tensors remain resident. Routed expert tensors are split into per-layer, per-expert payloads and copied into bounded inmemory slots only when the router demands them. The resulting question is not simply \"does the full model fit?\" but\"how much expert cache is enough to capture route locality without crossing the host memory-pressu","cbCaimE7PzcOt9Gy","https://ap.wps.com/l/cbCaimE7PzcOt9Gy","pdf",115996,1,7,"English","en",105,"# Introduction\n# Research Questions\n## Feasibility\n## Cache Behavior\n## Quantization and Context\n## Speculation","[{\"question\":\"What problem does MawForge address in local Mixture-of-Experts inference?\",\"answer\":\"Local serving can exceed fast-memory limits due to model residency plus KV cache, runtime buffers, and OS headroom. Sparse activation does not automatically make residency requirements sparse, so failures can occur despite reduced per-token compute.\"},{\"question\":\"How does MawForge manage memory during inference?\",\"answer\":\"MawForge stores the full model on disk, keeps common tensors resident, and materializes routed expert tensors into bounded in-memory cache slots only when the router demands them.\"},{\"question\":\"What were the key findings from the validation on constrained Apple Silicon hardware?\",\"answer\":\"Under an 18 GiB serving target, all statically feasible generation rows produced valid 96-token completions without triggering memory guards across multiple cache settings and context lengths. Expert-cache size showed non-monotonic performance: larger caches improved hit rate and reduced materialized bytes, but could reduce throughput due to host memory pressure.\"}]",1784205514,18,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"mawforge-memory-bounded-expert-materialization-for-local-mixture-of-experts-inference","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/mawforge-memory-bounded-expert-materialization-for-local-mixture-of-experts-inference/85673/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does MawForge address in local Mixture-of-Experts inference?","Question",{"text":75,"@type":76},"Local serving can exceed fast-memory limits due to model residency plus KV cache, runtime buffers, and OS headroom. Sparse activation does not automatically make residency requirements sparse, so failures can occur despite reduced per-token compute.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does MawForge manage memory during inference?",{"text":80,"@type":76},"MawForge stores the full model on disk, keeps common tensors resident, and materializes routed expert tensors into bounded in-memory cache slots only when the router demands them.",{"name":82,"@type":73,"acceptedAnswer":83},"What were the key findings from the validation on constrained Apple Silicon hardware?",{"text":84,"@type":76},"Under an 18 GiB serving target, all statically feasible generation rows produced valid 96-token completions without triggering memory guards across multiple cache settings and context lengths. Expert-cache size showed non-monotonic performance: larger caches improved hit rate and reduced materialized bytes, but could reduce throughput due to host memory pressure.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":21,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]