Nemotron-Labs-3-Puzzle-75B-A9B Compressing Hybrid MoE LLMs

Jul 16, 2026

Upload by Maeve

Nemotron-Labs-3-Puzzle-75B-A9B presents a compressed, deployment-optimized variant of Nemotron-3-Super designed for interactive serving. The model maximizes server throughput under constrained user load, delivering about 2× higher server throughput on a single 8×B200 node at matched user throughput, and increasing 1M-token concurrency on a single H100 from 1 to 8 requests. A multi-stage pipeline combines Iterative Puzzle with knowledge distillation, reinforcement learning, quantization, and a Multi-Token Prediction head, jointly optimizing MoE pruning, active parameter budgets, and Mamba pruning while preserving quality.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages25

Reading Time1 h 3 min

Abstract

Introduction

arXiv :2607 .0437 1v2 [ cs .AI ] 7 Jul 2026 Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs Akhiad Bercovich, Talor Abramovich, Daniel Afrimi, Shay Aharon, Nir Ailon, Vladimir Anisimov, Omer Ullman Argov, Maor Ashkenazi, Tomer Asida, Nave Assaf, Tomer Bar Natan, Alexander Bukharin, Grzegorz Chlebus, Marcin Chochowski, Eric Chung, Mohammad Dabbah, Carlo del Mundo, Ewa Dobrowolska, Ido Galil, Yaniv Galron, Amnon Geifman, Yonatan Geifman, Izik Golan, Alex Gronskiy, Tomasz Grzegorzek, Netanel Haber, Lior Kadoch, Grzegorz Karch, Tomer Keren, Abhinav Khattar, Amir Klein, Tugrul Konuk, Roi Koren, Daniel Korzekwa, Shaun Kotek, Konstantinos Krommydas, Itay Levy , Ofri Masad, Yoav Miron, Pavlo Molchanov, Shahar Mor, Zach Moshe, Saurav Muralidharan, Najeeb Nabwani, Besmira Nushi, Mostofa Patwary, Omri Puny, Johannes Rausch, Tomer Ronen, Sepehr Sameni, Itamar Schen, Elad Segal, Daniel Serebrenik, Ido Shahaf, Soumye Singhal, Daniil Sorokin, Sharath Turuvekere Sreenivas, Marta StepniewskaDziubinska, Ali Taghibakhshi, Nima Tajbakhsh, Oren Tropp, Dor Tzur, Anna Warno, Yi-Fu Wu, Michal Zawalski, Jiaqi Zeng, Yian Zhang, Ran Zilberstein, Amit Zuker, Ran El-Yaniv Abstract We present Nemotron-Labs-3-Puzzle-75B-A9B, a compressed variant of Nemotron-3-Super optimized for interactive deployment. We designed the model to maximize server throughput under high user throughput constraints. In interactive serving workloads on a single 8 ×B200 node, Puzzle-75B-A9B achieves approximately 2 × higher server throughput than Nemotron-3-Super at matched user throughput constraints. In ultra-long-context deployment on a single H100 GPU, the compressed model increases 1M-token concurrency from 1 request to 8 requests. Puzzle-75B-A9B is constructed using a multi-stage pipeline that combines the Iterative Puzzle compression framework with knowledge distillation, reinforcement learning, quantization, and Multi-Token Prediction head. The compression process jointly optimizes heterogeneous MoE pruning, active parameter budget, and Mamba pruning to improve inference efficiency while preserving model quality. We evaluate Puzzle-75B-A9B on a broad suite of reasoning, coding, multilingual, long-context, and agentic benchmarks. Despite substantial compression, the model retains strong downstream accuracy relative to the parent model across a wide range of tasks. These results demonstrate that large hybrid MoE models can be substantially optimized for deployment efficiency while maintaining strong downstream capability. Our model is publicly available on Hugging Face 1 . 1 Introduction Recent large language models (LLMs) have achieved substantial gains in reasoning, coding, long-context understanding, and agentic behavior by scaling both parameter count and architectural complexity. However, these improvements come at a significant inference cost, high memory consumption, elevated serving latency, and reduced deployment flexibility. These limitations become particularly acute in production inference settings, where systems must simultaneously satisfy strict throughput, latency, and context-length requirements. Consequently, improving inference efficiency without substantially degrading model quality has emerged as a central challenge for deploying next-generation LLMs at scale. A common approach to this problem is post-training compression, where a pretrained model is transformed into a smaller and more efficient model. Existing compression techniques, including pruning, quantization, and knowledge distillation, often involve a difficult tradeoff between efficiency and downstream capability preservation. This challenge is especially pronounced for hybrid MoE architectures, where performance 1The models are available at: NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-BF16, NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9BNVFP4, and NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-FP8 . Average accuracy (%) 74 72 70 68 66 64 62 60 58 Intelligence vs.

FAQs

What is Nemotron-Labs-3-Puzzle-75B-A9B and what problem does it target?

Nemotron-Labs-3-Puzzle-75B-A9B is a compressed, deployment-optimized variant of Nemotron-3-Super. It targets improving inference efficiency for production settings that must meet throughput, latency, and context-length constraints without substantially degrading model quality.

How much throughput and concurrency improvement does the compressed model achieve?

On a single 8×B200 node for interactive serving, it achieves approximately 2× higher server throughput than Nemotron-3-Super under matched user throughput constraints. For ultra-long-context deployment on a single H100 GPU, it increases 1M-token concurrency from 1 request to 8 requests.

What techniques are combined in the model compression pipeline?

The construction uses a multi-stage pipeline combining Iterative Puzzle with knowledge distillation, reinforcement learning, quantization, and a Multi-Token Prediction head. It jointly optimizes heterogeneous MoE pruning, active parameter budget, and Mamba pruning to improve inference efficiency while preserving quality.

Jul 16, 2026

Upload by Maeve