HCRMap: Pressure-Aware Hot-Expert Residency Mapping for 3.5D MoE Chiplet Inference

Jul 16, 2026

Upload by Aurelia

Mixture-of-Experts (MoE) LLM inference routes tokens to only a small subset of experts, yet token-dependent routing creates persistent hotness skew where a few hot experts dominate traffic while others remain lightly used. In 3.5D multi-chiplet systems, this skew simultaneously causes compute imbalance and escalates pressure on communication, memory bandwidth, I/O, and execution queues. HCRMap addresses this by dynamically managing expert replica residency across memory tiers. It promotes, retains, demotes, or evicts experts and maps routed token groups to resident replicas to mitigate bottlenecks.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages15

Reading Time38 min

Introduction

Hotness skew in MoE inference

Limitations of existing systems

Residency placement as a cross-resource tradeoff

arXiv :2607 . 1 1586v 1 [ cs .AI] 13 Jul 2026 HCRMap: Pressure-Aware Hot-Expert Residency Mapping for 3.5D MoE Chiplet Inference Yongqin Zhanga,∗ a Nanjing Vocational College of Information Technology, 99 Wenlan Road, Xianlin College Community, Nanjing, Jiangsu, 210023, P.R.C. ARTICLE INFO Keywords: Mixture-of-Experts 3.5D chiplets Runtime mapping Hot expert residency Hierarchical memory AB STRACT Mixture-of-Experts (MoE) large language models (LLM) activate only a small number of experts during inference, but token routing introduces persistent expert hotness skew: a small set of hot experts continuously receives most tokens, while the remaining experts are lightly loaded. On 3.5D multi-chiplet systems, this skew not only causes compute imbalance but also amplifies pressure on communication, memory bandwidth, I/O, and execution queues. Therefore, the core problem is not simply to reduce token movement, but to dynamically place and reuse hot expert replicas across different memory tiers. This paper proposes HCRMap, a hot expert residency mapping framework for pressure-aware expert replica management in 3.5D MoE inference. Based on expert hotness, weight loading cost, migration overhead, and runtime resource pressure, HCRMap dynamically determines which experts should be promoted, retained, demoted, or evicted. It then maps routed token groups to suitable resident replicas, thereby jointly mitigating communication, memory, and queue bottlenecks. Experimental results show that HCRMap reduces end-to-end latency by 43.6% and 43.0% over Hydra in the prefill and decode stages, respectively; by 34.5% and 33.1% over MoEntwine; and by 46.7% and 46 .0% over PIMoE. 1. Introduction Mixture-of-Experts large language models (MoELLMs) scale capacity by activating only a few experts per token through Top-􀁫 gating [1–4], but token-dependent routing produces persistent expert-popularity skew: a small set of hot experts repeatedly receives most tokens across serving windows while others stay lightly used [5–8] . As shown in Figure 1, this skew is not only a compute-load problem. It reshapes token dispatch and gather, expertweight access, memory-bank pressure, and shared-I/O pressure. The routed expert-FFN bottleneck is therefore a cross-resource imbalance over compute, memory, and interconnect, affecting full-model latency and energy in both prefill and decode execution. On 3.5D multi-chiplet systems with hierarchical memory, this skew surfaces as repeated expert-weight streaming [9–11] . Expert weights are too large to keep all active experts in the nearest stacked-SRAM tier; in our evaluation, each benchmark uses the routed-expert footprint computed from its own hidden and intermediate dimensions, while the near-tier budget of each chiplet group is only 64 MB. Therefore, finite near-tier capacity can hold only a small number of hot experts, while routing profiles may require more hot or warm resident experts in the same layer. Warm experts that overflow the near tier must repeatedly stream from groupshared DRAM through shared I/O and D2D/NoP links. This transfer recurs across windows and continuously occupies shared interfaces and package links. Because 3.5D integration co-packages stacked SRAM, local HBM, and groupshared DRAM into a finer-grained hierarchy, expert weights should not be classified simply as resident or non-resident; ∗Corresponding author. Email address: [zyq1350335@126.com](zyq1350335@126.com) (Y. Zhang) Figure 1: MoE FFN serving pipeline for selected experts. they can be placed at different residency levels according to runtime hotness, streaming cost, and resource pressure. Existing MoE systems optimize expert execution along separate axes: placement and routing that use popularity, affinity, or co-activation to cut dispatch latency; replication or shadow replicas that relieve hot-expert queueing; and heterogeneous execution or memory-aware offloading driven by compute and memory behavior

FAQs

What problem does HCRMap address in 3.5D MoE chiplet inference?

HCRMap targets persistent hot-expert skew that creates cross-resource imbalance across compute, memory, communication, I/O, and execution queues in 3.5D multi-chiplet systems.

How does HCRMap decide which expert replicas to keep or move?

HCRMap dynamically promotes, retains, demotes, or evicts experts using hotness, weight loading cost, migration overhead, and runtime resource pressure.

What performance improvements does the paper report for HCRMap?

Experiments show HCRMap reduces end-to-end latency by 43.6% and 43.0% over Hydra in prefill and decode, respectively, and also improves versus MoEntwine and PIMoE.

Jul 16, 2026

Upload by Aurelia