3DLS: 3D Logic-Stacked Architecture for Disaggregated LLM Serving

Jul 16, 2026

Upload by Violet

Large language model (LLM) serving increasingly uses prefill-decode (PD) disaggregation with tensor parallelism (TP) to handle large models and long contexts. In conventional 2D/2.5D chiplet designs, layer-wise KV-cache transfers and decode-side TP collectives contend for the same lateral die-to-die (D2D) interconnect, increasing communication latency and stretching token generation intervals. 3DLS proposes logic-on-logic 3D-stacked routing that isolates KV-cache traffic via vertical interconnects while keeping TP collectives on the lateral fabric, improving throughput up to 1.49× and reducing end-to-end latency by up to 60.2%.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages4

Reading Time10 min

AccessFree to download

Introduction

Traffic Contention in Conventional 2D/2.5D Chiplets

Proposed 3DLS Architecture

Communication Operations and Conflict Analysis

3DLS: A 3D Logic-Stacked Architecture for Disaggregated LLM Serving Jaehun Lee, In-Jun Jung, and Joo-Young Kim, Senior Member, IEEE arXiv :2607 .0 16 17v 1 [ cs .AR] 2 Jul 2026 Abstract—Large language model (LLM) serving increasingly combines prefill-decode (PD) disaggregation with tensor parallelism (TP) to support large models and long contexts. In conventional 2D/2.5D chiplet architectures, layer-wise prefill-todecode KV-cache transfer decode-side TP collectives share the same lateral die-to-die (D2D) interconnect, creating mixed-traffic contention on the decode critical path. This contention increases communication latency, prolongs token generation intervals, and degrades end-to-end serving performance. We propose 3DLS, a logic-on-logic 3D-stacked chiplet architecture that separates traffic classes by routing KV-cache transfers through vertical interconnects while preserving decode-side TP collectives on the lateral D2D fabric. 3DLS achieves up to 1.49× throughput and 60.2% lower end-to-end (E2E) latency over the sharedfabric planar baseline, and still achieves up to 1.17× throughput and 31.4% lower E2E latency over a workload-aware prioritymanaged planar baseline. These results highlight that physical isolation is an important design principle for future chiplet-based PD-disaggregated LLM serving systems. Index Terms—Large Language Model, LLM Serving, Disaggregated Serving, Chiplet, KV cache, 3D integration. I. INTRODUCTION LARGE language model (LLM) serving is shifting toward larger models and longer context windows, increasing compute and memory resource demands. At the same time, satisfying these demands with monolithic large-die accelerators has become increasingly difficult due to the rising cost of advanced process nodes, yield challenges, and reticle-size limits. These trends make chiplet-based integration a promising solution for scaling LLM inference hardware beyond the practical limits of monolithic designs [1], making on-package communication a major concern for serving performance. Modern LLM inference consists of two phases with distinct characteristics: a compute-intensive prefill phase and a memory-bound decode phase. Because these phases differ in resource demands, recent serving systems increasingly adopt prefill-decode (PD) disaggregation [2], [3], assigning each phase to a dedicated resource pool to improve efficiency and utilization. Meanwhile, serving large models continues to rely on tensor parallelism (TP) [4] to meet memory-capacity and throughput requirements. However, TP introduces frequent © 2026 IEEE. This is the author’s accepted manuscript of an article accepted for publication in IEEE Computer Architecture Letters. DOI: 10.1109/LCA.2026.3709108 . Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses. Jaehun Lee is with the Graduate School of System Architect, KAIST, Daejeon 34141, South Korea (e-mail: [jaehunlee@kaist.ac.kr](jaehunlee@kaist.ac.kr)). In-Jun Jung and Joo-Young Kim are with the School of Electrical Engineering, KAIST, Daejeon 34141, South Korea (e-mail: [injun@kaist.ac.kr](injun@kaist.ac.kr); [jooyoung1203@kaist.ac.kr](jooyoung1203@kaist.ac.kr)). (a) TP=4 case, 2 hop (b) TP=16 case, 4 hop :Prefill die Collective Communication :Decode die : KV cache transfer Fig. 1. Traffic contention in conventional 2D/2.5D chiplet-based PDdisaggregated serving. (a) TP=4 case with 2-hop KV cache transfer; (b) TP=16 case with 4-hop KV cache transfer. collective operations whose overhead grows with model scale and TP degree. As a result, a TP-enabled PD-disaggregated serving system must simultaneously support two heterogeneous traffic classes: KV-cache transfer from prefill to decode, and latency-critical decode-side collective communication. Fig. 1 shows that when PD-disaggregated serving with TP is mapped onto a conventional 2D/2.5D chiplet architecture [5], KV-cache transfer and decode-side TP collectives are fo

FAQs

What problem does 3DLS address in PD-disaggregated LLM serving with tensor parallelism?

It addresses contention on the lateral die-to-die interconnect between KV-cache transfers (prefill to decode) and decode-side TP collective communication, which increases latency and degrades end-to-end performance.

How does 3DLS physically isolate the two traffic classes?

3DLS routes KV-cache transfers through dedicated vertical 3D interconnects, while keeping decode-side TP collectives on the lateral D2D fabric.

What performance improvements does the document report for 3DLS?

It reports up to 1.49× throughput and 60.2% lower end-to-end latency versus a shared-fabric planar baseline, and up to 1.17× throughput with 31.4% lower end-to-end latency versus a workload-aware priority-managed planar baseline.

Jul 16, 2026

Upload by Violet