[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85824-en":3,"doc-seo-85824-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85824,8796095461564,"Liam","https://ap-avatar.wpscdn.com/davatar_155a257f0dc6eb9ab79c44ca47cae57d",8,"Research & Report","RDQ: Residual Distribution Quantization for Large Language Models","Post-training quantization (PTQ) of large language models sharply degrades below 4-bit precision due to residual stream distributional drift: layer-wise quantization noise accumulates in the shared residual representation, driving KL divergence from FP16 upward super-linearly with transformer depth. Empirical analysis shows 84% of LLaMA-3-8B layers have non-Gaussian residuals and residual variance grows 6,548× across depth. RDQ introduces Cascaded Error Compensation (CEC), a sequential calibration capturing drifted activations, fitting per-channel AWQ-style scales folded into preceding RMSNorm weights, achieving state-of-the-art perplexity with zero inference overhead.","arXiv :2607 . 10 137v 1 [ cs .LG] 11 Jul 2026  \nRDQ: Residual Distribution Quantization for Large Language Models  \nPrateek Singh  \nAbstract  \nPost-training quantization (PTQ) of large language models degrades sharply below 4-bit precision. We identify the root cause as residual stream distributional drift: quantization noise injected at each transformer layer accumulates in the shared residual representation, causing KL divergence from the FP16 baseline to grow super-linearly with depth (Pearson r=0 .999 with log-perplexity, p\u003C0 .001, confirmed across all tested methods and bit-widths) . We discover that 84% of LLaMA-3-8B layers exhibit non-Gaussian residual distributions (KS test, p≤0 .05), and that per-layer residual stream variance grows 6 ,548 × across depth. We propose RDQ (Residual Distribution Quantization), a PTQ framework whose central contribution is Cascaded Error Compensation (CEC): a sequential calibration procedure that captures the actual drifted activations each layer receives (computed by running calibration data through already-quantized upstream layers) and fits per-channel AWQ-style scales against those drifted inputs, with scales folded into preceding RMSNorm weights for exact mathematical equivalence at zero inference overhead. RDQ achieves state-of-the-art results on all three tested architectures:  \nLLaMA-3-8B: 7 .55 / 5 .62 PPL (W3/W4); Qwen-2.5-7B: 7 .46 / 6 .38 PPL; Mistral-  \n7B: 6.88 / 5.73 PPL. RDQ beats the best published baseline (LeanQuant/SpinQuant) at every model and bit-width combination, with gains up to −46 .4% vs. RTN at W3A16 on LLaMA-3-8B. All output is standard group-128 asymmetric quantization, deployable on Qualcomm AIMET, GGUF, and any standard inference stack at zero runtime overhead.  \n1 Introduction  \nDeploying large language models (LLMs) on edge hardware demands 3–4-bit weight quantization Linet al. [2024], Frantar et al. [2023] . While 4-bit methods approach FP16 quality, performance degrades sharply at 3 bits: on LLaMA-3-8B, RTN-128 goes from 5.83 PPL at W4 to 14.09 at W3—a 2.42 × ratio for only 1.33 × bit reduction.  \nOur diagnosis: residual stream drift. The transformer residual stream hl is shared across all layers—every layer reads from it and writes to it Elhage et al. [2021] . When layer l is quantized, its error εl persists into hl+1, affecting every subsequent layer. We measure empirically (Figure 5) that total KL divergence from the FP16 baseline correlates with log-PPL at Pearson r = 0 .999 (p \u003C 0.001) across all methods and bit-widths. This single relationship is the paper’s central empirical finding: quantization quality is predictable from drift, and drift is controllable.  \nTwo complementary findings sharpen this diagnosis. First, 84% of LLaMA-3-8B layers are nonGaussian by KS test (Figure 7), invalidating Gaussian distributional assumptions implicit in most PTQ analysis. Second, per-layer variance grows 6 ,548 × across depth (Figure 5), showing that later layers are categorically more sensitive to quantization error.  \nOur solution: CEC. Unlike prior work that optimizes each layer independently against clean FP16 inputs, we show that cross-layer drift is the dominant failure mode and introduce a calibration  \nPreprint.  \nprocedure that directly corrects for it. Specifically: standard PTQ methods (GPTQ Frantar et al.[2023], AWQ Lin et al. [2024], LeanQuant Zhang and Shrivastava [2025]) assume each layer receives clean FP16 input during calibration. After quantization, layer L actually receives ˆhL = hL +Pj\u003CL εj , so its calibration is systematically mis-adapted to a distribution that no longer exists at inference. CEC fixes this by processing layers sequentially: after quantizing layers 0..l−1, fresh calibration activations are captured from the real (drifted) distribution at layer l and used to fit optimal per-channel scales. The method works with any per-layer quantizer; we build on AWQ-style scaling, which is why the critical ablation is drift-aware calibrat","cbCaieJVx38Nd6kq","https://ap.wps.com/l/cbCaieJVx38Nd6kq","pdf",856046,1,12,"English","en",105,"# Introduction\n## Residual stream drift diagnosis\n## Solution: Cascaded Error Compensation (CEC)\n# Related Work\n## PTQ methods for LLMs\n## Key distinction","[{\"question\":\"Why does PTQ quality degrade sharply below 4-bit precision in large language models?\",\"answer\":\"Because quantization noise introduced at each transformer layer accumulates in the shared residual representation, causing residual distributional drift and super-linear growth in KL divergence from the FP16 baseline with depth.\"},{\"question\":\"What empirical evidence supports the residual drift diagnosis?\",\"answer\":\"The paper reports a near-perfect correlation between KL drift and log-perplexity (Pearson r=0.999, p\\u003c0.001) across methods and bit-widths, finds 84% of LLaMA-3-8B layers are non-Gaussian, and shows per-layer residual variance grows 6,548× across depth.\"},{\"question\":\"How does RDQ (CEC) address the drift problem during calibration?\",\"answer\":\"CEC performs sequential, drift-aware calibration: after quantizing upstream layers, it runs calibration data through already-quantized layers to capture the actual drifted activations at each layer, then fits per-channel scales (AWQ-style) and folds them into preceding RMSNorm weights to maintain mathematical equivalence with zero runtime overhead.\"}]",1784206486,30,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"rdq-residual-distribution-quantization-for-large-language-models","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/rdq-residual-distribution-quantization-for-large-language-models/85824/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why does PTQ quality degrade sharply below 4-bit precision in large language models?","Question",{"text":75,"@type":76},"Because quantization noise introduced at each transformer layer accumulates in the shared residual representation, causing residual distributional drift and super-linear growth in KL divergence from the FP16 baseline with depth.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What empirical evidence supports the residual drift diagnosis?",{"text":80,"@type":76},"The paper reports a near-perfect correlation between KL drift and log-perplexity (Pearson r=0.999, p\u003C0.001) across methods and bit-widths, finds 84% of LLaMA-3-8B layers are non-Gaussian, and shows per-layer residual variance grows 6,548× across depth.",{"name":82,"@type":73,"acceptedAnswer":83},"How does RDQ (CEC) address the drift problem during calibration?",{"text":84,"@type":76},"CEC performs sequential, drift-aware calibration: after quantizing upstream layers, it runs calibration data through already-quantized layers to capture the actual drifted activations at each layer, then fits per-channel scales (AWQ-style) and folds them into preceding RMSNorm weights to maintain mathematical equivalence with zero runtime overhead.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":28,"slug":121},"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]