[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82135-en":3,"doc-seo-82135-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},82135,1099514067438,"River Wang","https://ap-avatar.wpscdn.com/avatar/100002539ee87300030?x-image-process=image/resize,m_fixed,w_180,h_180&k=1780474512215547542",8,"Research & Report","StreamDQ Near-Memory Weight DeQuantization in Custom HBM for Scalable AI Inference Acceleration","Large language models scaling increases both memory bandwidth demand and inference computation, making weight-only quantization common for reducing model size with limited accuracy loss. On current GPUs, CUDA-core-based dequantization adds instruction overhead, extra on-chip traffic, and pipeline stalls, becoming a key throughput bottleneck for cloud-scale serving. StreamDQ introduces lightweight near-memory dequantization using compact DeQuantization Blocks in custom HBM, performing inline dequantization during memory loads via selectable sideband tags. Results show up to 7.08× speedup and 90.23% lower energy, with end-to-end latency down to 54.68% and 2.20× decode throughput improvement.","StreamDQ: Near-Memory Weight DeQuantization in Custom HBM for Scalable AI Inference Acceleration  \nMinki Jeong  \nSK Hynix Icheon, South Korea [minki.jeong@sk.com](minki.jeong@sk.com)  \nSeungyong Lee  \nSK Hynix Icheon, South Korea [seungyong.lee@sk.com](seungyong.lee@sk.com)  \nIeryung Park  \nSK Hynix Icheon, South Korea [ieryung.park@sk.com](ieryung.park@sk.com)  \nDaegun Yoon  \nSK Hynix Icheon, South Korea [daegun.yoon@sk.com](daegun.yoon@sk.com)  \nNameun Kang  \nSK Hynix Icheon, South Korea [nameun.kang@sk.com](nameun.kang@sk.com)  \nJoonseop Sim  \nSK Hynix Icheon, South Korea [joonseop.sim@sk.com](joonseop.sim@sk.com)  \nSoohong Ahn  \nSK Hynix Icheon, South Korea [soohong.ahn@sk.com](soohong.ahn@sk.com)  \nHyeonseok Ju  \nSK Hynix Icheon, South Korea [hyeonseok.ju@sk.com](hyeonseok.ju@sk.com)  \nYoungpyo Joo  \nSK Hynix Icheon, South Korea [youngpyo.joo@sk.com](youngpyo.joo@sk.com)  \narXiv :2607 .08993v 1 [ cs .AR] 9 Jul 2026  \nHoshik Kim  \nSK Hynix Icheon, South Korea [hoshik.kim@sk.com](hoshik.kim@sk.com)  \nAbstract  \nAs large language models (LLMs) scale, their memory and computation demands have grown substantially, making weight-only quantization a widely adopted technique for reducing model size with minimal accuracy loss. However, on current GPUs, CUDA-core-based dequantization introduces substantial instruction overhead, on-chip traffic, and pipeline stalls, making it a major bottleneck for high-throughput, cloud-scale LLM serving. To address these limitations, we propose StreamDQ, a lightweight architectural enhancement that enables on-the-fly dequantization in the memory subsystem for high-throughput, large-batch LLM inference. StreamDQ integrates compact DeQuantization Blocks (DQBs) into the base die of high-bandwidth memory (HBM) and performs inline dequantization on standard memory loads. A lightweight sideband tag on each memory read request selects the dequantization mode while preserving conventional load semantics. By relocating dequantization  \nPermission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific [permission and/or a fee. Request permissions from permissions@acm.org](permission and/or a fee. Request permissions from permissions@acm.org). Conference acronym ’XX, Woodstock, NY  \n© 2018 Copyright held by the owner/author(s) . Publication rights licensed to ACM.  \nACM ISBN 978-1-4503-XXXX-X/2018/06  \n[https://doi.org/XXXXXXX.XXXXXXX](https://doi.org/XXXXXXX.XXXXXXX)  \nto the memory side, StreamDQ eliminates GPU-side CUDAcore-based dequantization, thereby reducing on-chip traffic on the GPU and avoiding extra HBM write-back andreload of dequantized weights at large batch sizes. Our evaluation shows that StreamDQ achieves up to 7.08× speedup and 90.23% lower energy for mixed-precision GEMM, with only 0 . 127 mm2 area and 0 .355 W power overhead per DQBin a 12 nm CMOS process. For end-to-end LLM inference, StreamDQ reduces latency by up to 54.68% and improves decode throughput by up to 2.20× .  \nKeywords: LLM, Quantization, Dequantization, Inference, Custom HBM  \nACM Reference Format:  \nMinki Jeong, Daegun Yoon, Soohong Ahn, Seungyong Lee, Nameun Kang, Hyeonseok Ju, Ieryung Park, Joonseop Sim, Youngpyo Joo, and Hoshik Kim. 2018. StreamDQ: Near-Memory Weight DeQuantization in Custom HBM for Scalable AI Inference Acceleration. In Proceedings of Make sure to enter the correct conference title from your rights confirmation email (Conference acronym ’XX) . ACM, New York, NY, USA, 15 pages. [https://doi.org/XXXXXXX.XXXXXXX](https://doi.org/XXXXXXX.XXXXXXX)  \n1 Introduction  ","cbCainesf8XtBr4k","https://ap.wps.com/l/cbCainesf8XtBr4k","pdf",3185574,1,15,"English","en",105,"# Abstract\n# Introduction\n## LLM Inference Bottlenecks\n## Quantization and Dequantization Background","[{\"question\":\"Why does dequantization become a bottleneck in high-throughput LLM serving on current GPUs?\",\"answer\":\"CUDA-core-based dequantization introduces substantial instruction overhead, extra on-chip traffic, and pipeline stalls. These effects slow down end-to-end throughput for large-batch cloud inference.\"},{\"question\":\"What is StreamDQ and where does it perform dequantization?\",\"answer\":\"StreamDQ integrates compact DeQuantization Blocks into the base die of custom HBM. It performs inline dequantization on standard memory loads rather than on the GPU CUDA cores.\"},{\"question\":\"How does StreamDQ select the dequantization behavior for each memory read request?\",\"answer\":\"A lightweight sideband tag is attached to each memory read request to choose the dequantization mode. This preserves conventional load semantics while enabling on-the-fly dequantization.\"}]",1784178391,38,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"streamdq-near-memory-weight-dequantization-in-custom-hbm-for-scalable-ai-inference-acceleration","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/streamdq-near-memory-weight-dequantization-in-custom-hbm-for-scalable-ai-inference-acceleration/82135/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"Why does dequantization become a bottleneck in high-throughput LLM serving on current GPUs?","Question",{"text":74,"@type":75},"CUDA-core-based dequantization introduces substantial instruction overhead, extra on-chip traffic, and pipeline stalls. These effects slow down end-to-end throughput for large-batch cloud inference.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"What is StreamDQ and where does it perform dequantization?",{"text":79,"@type":75},"StreamDQ integrates compact DeQuantization Blocks into the base die of custom HBM. It performs inline dequantization on standard memory loads rather than on the GPU CUDA cores.",{"name":81,"@type":72,"acceptedAnswer":82},"How does StreamDQ select the dequantization behavior for each memory read request?",{"text":83,"@type":75},"A lightweight sideband tag is attached to each memory read request to choose the dequantization mode. This preserves conventional load semantics while enabling on-the-fly dequantization.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":105,"slug":137},19,"General","general"]