[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82519-en":3,"doc-seo-82519-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},82519,687197100911,"Himbo","https://ap-avatar.wpscdn.com/avatar/a000239b6f1da00475?x-image-process=image/resize,m_fixed,w_180,h_180&k=1782698725881665579",8,"Research & Report","BaseRT: Apple Silicon Native Metal 上的最佳级 LLM 推理运行时","BaseRT is a native Metal inference runtime for large language models on Apple Silicon, reporting the highest inference throughput on this hardware to date. It addresses overhead in existing runtimes by using chip-specific kernel fusion, unified-memory-aware optimization, and custom dispatch logic aligned with Metal execution. BaseRT supports many model families across eight quantization formats (Q2–FP16) on all Apple M-series devices and evaluates Qwen3, Llama 3.2, and Gemma 4 at Q4/Q8 on M3 and M4 Pro.","BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal  \nPrabod Rathnayaka∗ Fabian Waschkowski Lukas Wesemann  \narXiv :2607 .0050 1v 1 [ cs .CL] 1 Jul 2026  \nBase Compute, Melbourne, Australia  \nAbstract  \nWe present BaseRT, a native Metal inference runtime for large language models (LLMs) on Apple Silicon, and report the highest inference throughput on this hardware to date. Existing runtimes, including llama.cpp and MLX-based frameworks, incur overhead from abstractions not designed for Metal’s execution model or Apple Silicon’s unified memory topology. By building natively on Metal with chip-specific kernel fusion, unified memory-aware optimisation, and custom dispatch logic, BaseRT recovers performance that framework-based approaches leave on the table. BaseRT supports a wide range of model families across eight quantisation formats (Q2 to FP16) on all Apple M-series devices. In this paper, we evaluate the Qwen3, Llama 3.2, and Gemma 4 families at Q4 and Q8 quantisation on M3 and M4 Pro devices. BaseRT achieves up to 1 .56 × higher decode throughput than llama.cpp and up to 1 .35 × higher than MLX, with substantially larger margins on prefill for mixture-of-experts models, delivering consistent best-in-class throughput from sub-1B to 30B parameter models. These results establish Apple Silicon asa more capable inference platform than previously reported, with direct implications for the emerging edge inference paradigm: as privacy requirements, latency constraints, and cloud cost pressures drive inference toward on-device deployment, performance-optimised local runtimes are a critical enabling layer for this transition. BaseRT is publicly available at [https://github.com/basecompute/baseRT](https://github.com/basecompute/baseRT).  \n1 Introduction  \nAI inference demand is growing at a rate that challenges the scalability of centralised cloud infrastructure. Token usage across major providers has grown by roughly an order of magnitude year-on-year as evidenced by token usage via Google Cloud API and a broader mix of providers on Open Router [1, 2] . However, the majority of enterprise adoption is still nascent and this trend is likely to continue [3] . Global inference compute is projected to overtake training compute for the first in 2027, with AI inference expected to makeup 40% of global data center compute by 2030 [4] . This trajectory places increasing pressure on the economic and architectural assumptions underlying cloud-only inference, and makes a compelling case for a structural shift toward edge inference: running LLMs locally on consumer and professional hardware. The case for this shift rests on four independent drivers.  \nPrivacy and Data Residency: Prompts processed in the cloud are subject to multi-tenancy, third-party infrastructure risk, and complex legal jurisdictions [5], risks that Gartner projects will cause over 40% of AI-related data breaches by 2027 through insufficient oversight of cross-border prompt routing [6] . Local inference keeps all data within controlled environments, enabling compliance with air-gapped and zero-trust requirements.  \nLatency: Cloud inference adds request handling, queueing, and GPU scheduling overhead that can push time-to-first-token (TTFT) into the hundreds of milliseconds, especially under load [7] . On-device inference  \n∗ Correspondence: prabod@basecompute.co  \nDecode throughput (tok/s)  \n400  \n200  \n0  \nQwen3-0.6B Llama-3 .2-1B Llama-3 .2-3B  \nModel  \n|  | \u003Cbr> |  |\n| --- | --- | --- |\n\n128 256 512 1024 2048 Prompt length (tokens)  \n BaseRT  MLX  llama.cpp  \nFigure 1: BaseRT delivers best-in-class throughput on Apple Silicon. Decode throughput (left) is improved by 1 .15–1.56 × over llama.cpp and up to 1 .35 × over MLX across the Q4 models tested on Apple M4 Pro. For prefill throughput (right), BaseRT leads across all prompt lengths for the Qwen3-30B-A3B model in 4-bit quantisation with up to 1 .78 × improvement over MLX.  \navoids network and queueing del","cbCaitO0V654kUut","https://ap.wps.com/l/cbCaitO0V654kUut","pdf",351315,1,11,"English","en",105,"# Introduction\n## Privacy and Data Residency\n## Latency\n## Connectivity\n## Cost\n## Hardware Context","[{\"question\":\"BaseRT 相比现有运行时（如 llama.cpp、MLX）主要改进在哪里？\",\"answer\":\"BaseRT 通过原生构建于 Metal，并采用芯片定制的 kernel fusion、面向统一内存拓扑的优化以及自定义 dispatch 逻辑，减少了不匹配执行抽象带来的开销。\"},{\"question\":\"BaseRT 支持哪些量化格式与模型类型？\",\"answer\":\"BaseRT 覆盖从 Q2 到 FP16 的多种量化格式，并支持多种模型家族；文中在 Q4 与 Q8 上评估了 Qwen3、Llama 3.2 和 Gemma 4。\"},{\"question\":\"论文如何论证边缘推理相对于云端推理的价值？\",\"answer\":\"文章从隐私/数据驻留、时延（TTFT 与排队调度开销）、网络可用性依赖以及成本结构差异四个驱动因素说明，本地推理能更好满足落地需求，尤其是需要低时延与离线确定性的场景。\"}]",1784181128,28,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"basert-best-in-class-llm-inference-on-apple-silicon-via-native-metal","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/basert-best-in-class-llm-inference-on-apple-silicon-via-native-metal/82519/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"BaseRT 相比现有运行时（如 llama.cpp、MLX）主要改进在哪里？","Question",{"text":75,"@type":76},"BaseRT 通过原生构建于 Metal，并采用芯片定制的 kernel fusion、面向统一内存拓扑的优化以及自定义 dispatch 逻辑，减少了不匹配执行抽象带来的开销。","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"BaseRT 支持哪些量化格式与模型类型？",{"text":80,"@type":76},"BaseRT 覆盖从 Q2 到 FP16 的多种量化格式，并支持多种模型家族；文中在 Q4 与 Q8 上评估了 Qwen3、Llama 3.2 和 Gemma 4。",{"name":82,"@type":73,"acceptedAnswer":83},"论文如何论证边缘推理相对于云端推理的价值？",{"text":84,"@type":76},"文章从隐私/数据驻留、时延（TTFT 与排队调度开销）、网络可用性依赖以及成本结构差异四个驱动因素说明，本地推理能更好满足落地需求，尤其是需要低时延与离线确定性的场景。","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]