DeepSeek-V3 Technical Report - arXiv 2412.19437v2

Oct 22, 2026

DeepSeek-V3 is presented as a large Mixture-of-Experts (MoE) language model with 671B total parameters and 37B activated per token. Efficient inference and cost-effective training are enabled through Multi-head Latent Attention (MLA) and DeepSeekMoE, with an auxiliary-loss-free load balancing strategy and a multi-token prediction training objective. The model is pre-trained on 14.8T diverse tokens, then trained via Supervised Fine-Tuning and Reinforcement Learning. Evaluations show strong open-source performance, comparable to leading closed-source systems, with stable training and only 2.788M H800 GPU hours.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages53

Reading Time2 h 14 min

AccessFree to download

1 Introduction

2 Architecture

2.1 Basic Architecture

2.2 Multi-Token Prediction

3 Infrastructures

3.1 Compute Clusters

3.5 Suggestions on Hardware Design

4 Pre-Training

4.1 Data Construction

4.5 Discussion

5 Post-Training

5.1 Supervised Fine-Tuning

5.4 Discussion

6 Conclusion, Limitations, and Future Directions

arXiv :2412 . 19437v2 [ cs .CL] 18 Feb 2025 DeepSeek-V3 Technical Report DeepSeek-AI [research@deepseek.com](research@deepseek.com) Abstract We present DeepSeek-V3, a strong Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token. To achieve efficient inference and cost-effective training, DeepSeek-V3 adopts Multi-head Latent Attention (MLA) and DeepSeekMoE architectures, which were thoroughly validated in DeepSeek-V2 . Furthermore, DeepSeek-V3 pioneersan auxiliary-loss-free strategy for load balancing and sets a multi-token prediction training objective for stronger performance. We pre-train DeepSeek-V3 on 14.8 trillion diverse and high-quality tokens, followed by Supervised Fine-Tuning and Reinforcement Learning stages to fully harness its capabilities. Comprehensive evaluations reveal that DeepSeek-V3 outperforms other open-source models and achieves performance comparable to leading closed-source models. Despite its excellent performance, DeepSeek-V3 requires only 2.788M H800 GPU hours for its full training. In addition, its training process is remarkably stable. Throughout the entire training process, we did not experience any irrecoverable loss spikes or perform any rollbacks. The model checkpoints are available at [https://github. com/deepseek-ai/DeepSeek-V3](https://github. com/deepseek-ai/DeepSeek-V3) . | | DeepSeek-V3 | DeepSeek-V2.5 | Qwen2.5-72B-Inst | Llama-3 . 1-405B-Inst | GPT-4o-0513 | Claude-3.5-Sonnet-1022 | | --- | --- | --- | --- | --- | --- | --- | Figure 1 | Benchmark performance of DeepSeek-V3 and its counterparts. Contents 1 Introduction 4 2 Architecture 6 2.1 Basic Architecture ..................................... 6 2.1.1 Multi-Head Latent Attention .......................... 7 2.1.2 DeepSeekMoE with Auxiliary-Loss-Free Load Balancing .......... 8 2.2 Multi-Token Prediction .................................. 10 3 Infrastructures 11 3.1 Compute Clusters ..................................... 11 3.2 Training Framework ................................... 12 3.2.1 DualPipe and Computation-Communication Overlap ............ 12 3.2.2 Efficient Implementation of Cross-Node All-to-All Communication .... 13 3.2.3 Extremely Memory Saving with Minimal Overhead ............. 14 3.3 FP8 Training ........................................ 14 3.3.1 Mixed Precision Framework .......................... 15 3.3.2 Improved Precision from Quantization and Multiplication ......... 16 3.3.3 Low-Precision Storage and Communication ................. 18 3.4 Inference and Deployment ................................ 18 3.4.1 Prefilling ...................................... 19 3.4.2 Decoding ..................................... 19 3.5 Suggestions on Hardware Design ............................ 20 3.5.1 Communication Hardware ........................... 20 3.5.2 Compute Hardware ............................... 20 4 Pre-Training 21 4.1 Data Construction ..................................... 21 4.2 Hyper-Parameters ..................................... 22 4.3 Long Context Extension ................................. 23 4.4 Evaluations ........................................ 24 4.4.1 Evaluation Benchmarks ............................. 24 4.4.2 Evaluation Results ................................ 24 4.5 Discussion ......................................... 26 4.5.1 Ablation Studies for Multi-Token Prediction ................. 26 4.5.2 Ablation Studies for the Auxiliary-Loss-Free Balancing Strategy ...... 26 4.5.3 Batch-Wise Load Balance VS. Sequence-Wise Load Balance ......... 27 5 Post-Training 28 5.1 Supervised Fine-Tuning ................................. 28 5.2 Reinforcement Learning ................................. 29 5.2.1 Reward Model .................................. 29 5.2.2 Group Relative Policy Optimization ...................... 30 5.3 Evaluations ...................................

FAQs

What makes DeepSeek-V3 efficient to train and run?

DeepSeek-V3 uses Multi-head Latent Attention (MLA) and DeepSeekMoE, together with an auxiliary-loss-free load balancing strategy and a multi-token prediction training objective.

How is DeepSeek-V3 trained after pre-training?

After pre-training on 14.8T tokens, it undergoes Supervised Fine-Tuning followed by Reinforcement Learning to fully leverage its capabilities.

What do evaluations say about DeepSeek-V3 performance and training stability?

Comprehensive evaluations indicate DeepSeek-V3 outperforms other open-source models and reaches performance comparable to leading closed-source models. Training is reported as highly stable, with no irrecoverable loss spikes or rollbacks, and a total compute of 2.788M H800 GPU hours.

Oct 22, 2026

Uploaded by Genevieve