[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-81896-en":3,"doc-seo-81896-105":31,"detail-sidebar-cat-0-en-105":93},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":28,"seo_description":14,"update_tm":29,"read_time":30},81896,8796095462418,"Noah","https://ap-avatar.wpscdn.com/avatar/80000253c1241d02b47?x-image-process=image/resize,m_fixed,w_180,h_180&k=1778826106357471780",8,"Research & Report","TileLens: Efficiently Using Large-Granularity Memory Systems with Transparent Two-Dimensional Memory Layout","Large Language Model (LLM) inference is limited by GPU High-Bandwidth Memory (HBM) capacity and bandwidth. High-Bandwidth Flash (HBF) and RoMe increase capacity or bandwidth, but impose kilobyte-scale access granularity. This work shows Large-Granularity Memory Systems can degrade tiled matrix multiplication in LLM inference by up to 10× due to read amplification: 1-D memory layouts mismatch 2-D compute tiles, causing each request to fetch data far beyond tile boundaries. TileLens introduces a tile-major layout and GPU software/hardware extensions to align granularity with tiles, reducing slowdown to within 1% of an HBM-only baseline.","TileLens: Efficiently Using Large-Granularity Memory Systems with Transparent Two-Dimensional Memory Layout  \n* *  \nJae Hyung Ju , Euijun Chung , Hritvik Taneja, Anish Saxena,  \nShinnung Jeong, Hyesoon Kim, Moinuddin K. Qureshi  \nGeorgia Institute of Technology  \nAtlanta, Georgia, USA  \n{jhju,euijun,htaneja3,asaxena317,sjeong306,[moin}@gatech.edu](moin}@gatech.edu), [hyesoon@cc.gatech.edu](hyesoon@cc.gatech.edu)  \narXiv :2607 .0403 1v 1 [ cs .AR] 4 Jul 2026  \nAbstract  \nLarge Language Model (LLM) inference is bottlenecked by the capacity and bandwidth of GPU High-Bandwidth Memory (HBM). Recent proposals, such as High-Bandwidth Flash (HBF) and RoMe, offer higher capacity or bandwidth than HBM, but require a minimum access granularity of kilobytes (e.g., 4 KB) . We show that these Large-Granularity Memory Systems (LGMS) can degrade the performance of tiled matrix-multiplication, which is the dominant operation in LLM inference, by up to an order of magnitude. The root cause of the slowdown is read amplification, where memory requests fetch far more data than the tile actually needs. This waste stems from a fundamental mismatch between the two-dimensional nature of compute tiles and the one-dimensional memory layout, leading to each request spilling well beyond the tile boundaries.  \nTo mitigate read amplification, we propose to use tile-major layout for LGMS. Rather than storing data as an one-dimensional strip, tile-major layout reshapes each contiguous memory block into a two-dimensional rectangle, aligning memory granularity with tile boundaries. To ease the adoption of tile-major layout on GPUs, we propose TileLens, lightweight software and hardware extensions that collectively cover major classes of GPU kernels. TileLens-SW extends GPU DSLs so that DSL-based kernels (e.g., CUTLASS, FlashAttention) can adopt tile-major in global memory by changing only the layout descriptor. TileLens-HW extends the Tensor Memory Accelerator (TMA) for transparent tile-major support in TMA-based kernels (e.g., cuBLAS, DeepGEMM) without code changes. We evaluate TileLens on a cycle-level simulator using matrix-multiplication kernels from Qwen-3 30B and Llama-3.1 70B. Combining a tile-major layout with an adaptive hardware prefetcher, TileLens achieves near-HBM performance on HBF-augmented GPUs with a 5 􀁠s HBF NAND read latency, reducing the geomean slowdown from 1.61–6.49× with conventional layouts to within 1% of an HBM-only baseline.  \nKeywords  \nGPU, HBM, HBF, Hybrid memory system, LLM  \n1 Introduction  \nThe rapid adoption of Large Language Models (LLMs) is pushing the limits of current GPU memory systems. Model sizes have grown to require up to terabytes of GPU memory [1, 9, 22, 41] . Moreover, emerging capabilities such as reasoning, multimodality, and agentic behavior require larger per-user state. LLM inference is dominated by matrix multiplications that read model weights and user state for each generated token with little data reuse. As a result, GPU  \n*  \nEqual contribution.  \nmemory capacity and bandwidth are the main bottlenecks for LLM inference performance [24] .  \nNew GPU memory technologies have been proposed to address these bottlenecks by providing higher capacity or bandwidth than the current HBM (High-Bandwidth Memory). High-Bandwidth Flash (HBF) [11, 12, 24, 37] is an emerging technology that stacks NAND flash memory dies in a manner similar to HBM, offering 8–16× the capacity with comparable bandwidth. RoMe [25] reclaims the bandwidth consumed by the address and command buses for data transfers by increasing the HBM access granularity to 4 KB. Both proposals require a minimum access granularity of kilobytes, much larger than the 32 B granularity of HBM. However, the impact of employing such Large-Granularity Memory Systems (LGMS) on GPUs for LLM inference has not been well studied.  \nWe show that using LGMS on GPUs results in read amplification during matrix multiplication (matmul), where each memory access fetches more da","cbCaiuX28ZQkk9bu","https://ap.wps.com/l/cbCaiuX28ZQkk9bu","pdf",1036503,6,1,13,"English","en",105,"# Abstract\n# Keywords\n# Introduction\n## Motivation: GPU memory bottlenecks in LLM inference\n## Large-Granularity Memory Systems and their access granularity constraints\n## Read amplification in tiled matrix multiplication\n## Root cause and design goal\n## Proposed approach: tile-major layout and TileLens","[{\"question\":\"为什么 Large-Granularity Memory Systems 会降低 LLM 推理中的性能？\",\"answer\":\"由于读取放大（read amplification），矩阵乘法的每次内存访问会获取远超计算 tile 所需的数据量，导致带宽浪费，并使性能最多下降一个数量级。\"},{\"question\":\"读取放大的根本原因是什么？\",\"answer\":\"GPU 当前的一维（1-D）全局内存布局与矩阵乘法的二维（2-D）分块计算方式不匹配，导致请求溢出到 tile 边界之外。\"},{\"question\":\"TileLens 如何缓解读取放大并接近 HBM 性能？\",\"answer\":\"TileLens 采用 tile-major 布局把连续内存块重塑为二维矩形，从而让内存粒度与 tile 边界对齐；同时通过 TileLens-SW 和 TileLens-HW 让常见 GPU kernel（如 CUTLASS、FlashAttention、cuBLAS、DeepGEMM）在全局内存中透明使用该布局，并结合自适应硬件预取器将性能降低控制在接近 HBM-only 基线（约 1% 内）。\"}]","TileLens: Efficiently Using Large-Granularity Memory Systems with Transparent Two-Dimensional Memory Layout | PDF",1784176916,33,{"code":4,"msg":32,"data":33},"ok",{"site_id":25,"language":24,"slug":34,"title":13,"keywords":35,"description":14,"schema_data":36,"social_meta":88,"head_meta":90,"extra_data":92,"updated_unix":29},"tilelens-efficiently-using-large-granularity-memory-systems-with-transparent-two-dimensional-memory-layout","",{"@graph":37,"@context":87},[38,55,70],{"@type":39,"itemListElement":40},"BreadcrumbList",[41,45,49,52],{"item":42,"name":43,"@type":44,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":46,"name":47,"@type":44,"position":48},"https://docshare.wps.com/document/","Document",2,{"item":50,"name":12,"@type":44,"position":51},"https://docshare.wps.com/document/research-report/",3,{"item":53,"name":13,"@type":44,"position":54},"https://docshare.wps.com/document/tilelens-efficiently-using-large-granularity-memory-systems-with-transparent-two-dimensional-memory-layout/81896/",4,{"url":53,"name":13,"@type":56,"author":57,"headline":13,"publisher":59,"fileFormat":62,"inLanguage":24,"description":14,"dateModified":63,"datePublished":64,"encodingFormat":62,"isAccessibleForFree":65,"interactionStatistic":66},"DigitalDocument",{"name":9,"@type":58},"Person",{"url":42,"name":60,"@type":61},"DocShare","Organization","application/pdf","2026-07-29","2026-07-16",true,{"@type":67,"interactionType":68,"userInteractionCount":20},"InteractionCounter",{"@type":69},"ViewAction",{"@type":71,"mainEntity":72},"FAQPage",[73,79,83],{"name":74,"@type":75,"acceptedAnswer":76},"为什么 Large-Granularity Memory Systems 会降低 LLM 推理中的性能？","Question",{"text":77,"@type":78},"由于读取放大（read amplification），矩阵乘法的每次内存访问会获取远超计算 tile 所需的数据量，导致带宽浪费，并使性能最多下降一个数量级。","Answer",{"name":80,"@type":75,"acceptedAnswer":81},"读取放大的根本原因是什么？",{"text":82,"@type":78},"GPU 当前的一维（1-D）全局内存布局与矩阵乘法的二维（2-D）分块计算方式不匹配，导致请求溢出到 tile 边界之外。",{"name":84,"@type":75,"acceptedAnswer":85},"TileLens 如何缓解读取放大并接近 HBM 性能？",{"text":86,"@type":78},"TileLens 采用 tile-major 布局把连续内存块重塑为二维矩形，从而让内存粒度与 tile 边界对齐；同时通过 TileLens-SW 和 TileLens-HW 让常见 GPU kernel（如 CUTLASS、FlashAttention、cuBLAS、DeepGEMM）在全局内存中透明使用该布局，并结合自适应硬件预取器将性能降低控制在接近 HBM-only 基线（约 1% 内）。","https://schema.org",{"og:url":53,"og:type":89,"og:title":13,"og:site_name":60,"og:description":14},"article",{"robots":91,"canonical":53},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":94},[95,99,103,107,112,116,121,124,129,132,136],{"id":21,"doc_module":4,"doc_module_name":47,"category_name":96,"show_sort_weight":97,"slug":98},"Story & Novel",90,"story-novel",{"id":48,"doc_module":4,"doc_module_name":47,"category_name":100,"show_sort_weight":101,"slug":102},"Literature",80,"literature",{"id":54,"doc_module":4,"doc_module_name":47,"category_name":104,"show_sort_weight":105,"slug":106},"Exam",70,"exam",{"id":108,"doc_module":4,"doc_module_name":47,"category_name":109,"show_sort_weight":110,"slug":111},5,"Comic",60,"comic",{"id":20,"doc_module":4,"doc_module_name":47,"category_name":113,"show_sort_weight":114,"slug":115},"Technology",50,"technology",{"id":117,"doc_module":4,"doc_module_name":47,"category_name":118,"show_sort_weight":119,"slug":120},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":47,"category_name":12,"show_sort_weight":122,"slug":123},30,"research-report",{"id":125,"doc_module":4,"doc_module_name":47,"category_name":126,"show_sort_weight":127,"slug":128},9,"Religion & Spirituality",20,"religion-spirituality",{"id":127,"doc_module":4,"doc_module_name":47,"category_name":130,"show_sort_weight":127,"slug":131},"World Cup","world-cup",{"id":133,"doc_module":4,"doc_module_name":47,"category_name":134,"show_sort_weight":133,"slug":135},10,"Lifestyle","lifestyle",{"id":137,"doc_module":4,"doc_module_name":47,"category_name":138,"show_sort_weight":108,"slug":139},19,"General","general"]