[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85838-en":3,"doc-seo-85838-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85838,8796095461610,"Oliver","https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c",6,"Technology","FlashAccel: Leveraging High-Bandwidth Flash for High-Throughput LLM Inference","Large language model inference is increasingly constrained by GPU high-bandwidth memory (HBM) capacity as both model weights and KV cache grow rapidly. High-Bandwidth Flash (HBF) offers substantially higher capacity than HBM while maintaining comparable bandwidth, but its high access latency, poor bandwidth utilization under naive layouts, and limited support for heterogeneous memory management make GPU integration difficult. FlashAccel is a co-designed system that adds architectural, data-layout, and HBF-aware storage management support for efficient HBF-based inference.","FlashAccel: Leveraging High-Bandwidth Flash for High-Throughput LLM Inference  \nXinyu Wang  \nInstitute of Computing Technology, Chinese Academy of Sciences Beijing, China  \nUniversity of Chinese Academy of Sciences Beijing, China [wangxinyu22s@ict.ac.cn](wangxinyu22s@ict.ac.cn)  \nYalong Xue  \nInstitute of Computing Technology, Chinese Academy of Sciences Beijing, China  \nSchool of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences Beijing, China [xueyalong22@mails.ucas.ac.cn](xueyalong22@mails.ucas.ac.cn)  \nXiaotian Sun  \nInstitute of Computing Technology, Chinese Academy of Sciences Beijing, China  \nUniversity of Chinese Academy of Sciences Beijing, China [sunxiaotian21s@ict.ac.cn](sunxiaotian21s@ict.ac.cn)  \narXiv :2607 . 10186v1 [ cs .AR] 11 Jul 2026  \nXiaoyu Zhang  \nInstitute of Computing Technology, Chinese Academy of Sciences Beijing, China [zhangxiaoyu@ict.ac.cn](zhangxiaoyu@ict.ac.cn)  \nChunmeng Dou  \nInstitute of Microelectronics, Chinese Academy of Sciences Beijing, China [douchunmeng@ime.ac.cn](douchunmeng@ime.ac.cn)  \nXueqi Li  \nInstitute of Computing Technology, Chinese Academy of Sciences Beijing, China [lixueqi@ict.ac.cn](lixueqi@ict.ac.cn)  \nXiaoming Chen∗ Institute of Computing Technology, Chinese Academy of Sciences Beijing, China [chenxiaoming@ict.ac.cn](chenxiaoming@ict.ac.cn)  \nAbstract  \nLarge language model (LLM) inference is increasingly limited by the capacity of High-Bandwidth Memory (HBM) in GPUs, as model weights and KV cache grow rapidly. HighBandwidth Flash (HBF) provides higher capacity than HBM while retaining comparable bandwidth, making it a promising substrate for capacity-constrained LLM inference. However, its inherently high access latency, low bandwidth utilization, and lack of support for heterogeneous resource management make it difficult to integrate HBF into GPUs forLLM inference. We present FlashAccel, a co-designed system that enables efficient LLM inference using HBF. FlashAccel integrates HBF into HBM-based GPUs, providing architectural support to mitigate access latency. It improves bandwidth utilization through specialized data layouts for both model weights and KV cache, and introduces an HBF-aware storage management layer together with a programming model to organize persistent data in HBF and coordinate heterogeneous memory resources at the system level. Experimental results demonstrate that integrating six HBF stacks into the GPU enables FlashAccel to deliver an average improvement of 2.54× and 1.93× in throughput per GPU and energy efficiency over the HBM-only GPU under 100ms latency constraint, respectively.  \nKeywords: High Bandwidth Flash, LLM Inference  \n∗ Corresponding author.  \n1 Introduction  \nThe rapid advancement of large language models (LLMs) is driving inference workloads toward multi-turn interactions [17, 66, 70], agents [43, 56, 64], and long-context processing [7, 8, 36] . Two key trends emerge in this evolution. First, model sizes continue to grow following the scaling law [29], at the cost of drastically increased parameter scale. Second, new workloads increasingly rely on long context lengths, leading to substantial growth in KV cache footprints during inference [9] . These trends fundamentally increase the memory capacity requirements of inference systems. However, the capacity of High-Bandwidth Memory (HBM) in GPUs has not kept pace with the rapid growth of model size. Figure 1 shows that, although GPU memory capacity has increased steadily over time, the memory footprint of popular models has grown much faster. Today, models can exceed terabytes in size, far surpassing the capacity of a single GPU, which is at most a few hundred gigabytes. Meanwhile, as the context length has grown by 4× on average over the past two years [5], the storage overhead of KV cache has increased proportionally by 4× .  \nThe limited HBM capacity introduces several critical challenges in LLM inference. First, during the memory-bound decode phase, in","cbCailgHYJ4jjx3F","https://ap.wps.com/l/cbCailgHYJ4jjx3F","pdf",2941496,1,16,"English","en",105,"# Introduction\n## Memory capacity trends in LLM inference\n## Challenges of HBM-limited inference\n## Prior flash-based approaches and their limitations\n## Proposed direction with HBF","[{\"question\":\"Why does LLM inference become limited by GPU memory capacity?\",\"answer\":\"As model sizes scale up and long-context workloads increase KV cache footprints, the HBM capacity cannot keep pace, restricting batch size and reducing throughput.\"},{\"question\":\"What advantages does High-Bandwidth Flash (HBF) provide over HBM?\",\"answer\":\"HBF offers much higher capacity while delivering bandwidth comparable to HBM, enabling potential increases in decode throughput when integrated effectively.\"},{\"question\":\"What prevents straightforward use of HBF for high-throughput inference?\",\"answer\":\"Flash access latency is inherently high, and achieving HBF peak bandwidth requires strong parallelism, which can be hindered by inefficient data layouts and the lack of system-level heterogeneous memory management.\"}]",1784206628,40,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"flashaccel-leveraging-high-bandwidth-flash-for-high-throughput-llm-inference","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/technology/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/flashaccel-leveraging-high-bandwidth-flash-for-high-throughput-llm-inference/85838/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why does LLM inference become limited by GPU memory capacity?","Question",{"text":75,"@type":76},"As model sizes scale up and long-context workloads increase KV cache footprints, the HBM capacity cannot keep pace, restricting batch size and reducing throughput.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What advantages does High-Bandwidth Flash (HBF) provide over HBM?",{"text":80,"@type":76},"HBF offers much higher capacity while delivering bandwidth comparable to HBM, enabling potential increases in decode throughput when integrated effectively.",{"name":82,"@type":73,"acceptedAnswer":83},"What prevents straightforward use of HBF for high-throughput inference?",{"text":84,"@type":76},"Flash access latency is inherently high, and achieving HBF peak bandwidth requires strong parallelism, which can be hindered by inefficient data layouts and the lack of system-level heterogeneous memory management.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,113,117,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":111,"slug":112},50,"technology",{"id":114,"doc_module":4,"doc_module_name":45,"category_name":115,"show_sort_weight":28,"slug":116},7,"Healthcare","healthcare",{"id":118,"doc_module":4,"doc_module_name":45,"category_name":119,"show_sort_weight":120,"slug":121},8,"Research & Report",30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]