FRED - A Wafer-scale Fabric for 3D Parallel DNN Training

Aug 20, 2026

Wafer-scale systems tightly integrate high-end accelerator chiplets with high-speed wafer-scale interconnects, offering low-latency, high-bandwidth connectivity for deep neural network training. Existing network-on-wafer designs such as 2D Meshes lack the flexibility to match diverse parallelization patterns. This work introduces Fred, a wafer-scale fabric using tiny microswitches to form a distributed nonblocking on-wafer topology with in-switch collective support for arbitrary accelerator groups. Benchmarks show sizable end-to-end training-time reductions for ResNet-152, Transformer-17B, GPT-3, and Transformer-1T versus a wafer-scale Mesh baseline.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages15

Reading Time38 min

AccessFree to download

Introduction

DNN scaling and distributed training overhead

Wafer-scale substrate and fabric topologies

Parallelism strategies for DNN training

FRED: A Wafer-scale Fabric for 3D Parallel DNN Training Saeed Rashidi∗ [saeed.rashidi@gatech.edu](saeed.rashidi@gatech.edu)[ ](saeed.rashidi@gatech.edu)Georgia Institute of Technology Atlanta, Georgia, USA William Won [william.won@gatech.edu](william.won@gatech.edu)[ ](william.won@gatech.edu)Georgia Institute of Technology Atlanta, Georgia, USA Sudarshan Srinivasan [sudarshan.srinivasan@intel.com](sudarshan.srinivasan@intel.com)[ ](sudarshan.srinivasan@intel.com)Intel Bangalore, Karnataka, India Puneet Gupta [puneetg@ucla.edu](puneetg@ucla.edu)[ ](puneetg@ucla.edu)UCLA Los Angeles, California, USA Tushar Krishna [tushar@ece.gatech.edu](tushar@ece.gatech.edu)[ ](tushar@ece.gatech.edu)Georgia Institute of Technology Atlanta, Georgia, USA arXiv :2406 . 19580v2 [ cs .AR] 9 Jun 2025 Abstract Wafer-scale systems are an emerging technology that tightly integrates high-end accelerator chiplets with high-speed wafer-scale interconnects, enabling low-latency and high-bandwidth connectivity. This makes them a promising platform for deep neural network (DNN) training. However, current network-on-wafer topologies, such as 2D Meshes, lack the flexibility needed to support various parallelization strategies effectively. In this paper, we propose Fred, a wafer-scale fabric architecture tailored to the unique communication needs of DNN training. Fred creates a distributed on-wafer topology with tiny microswitches, providing nonblocking connectivity for collective communications between arbitrary groups of accelerators and enabling in-switch collective support. Our results show that for sample parallelization strategies, Fred can improve the average end-to-end training time of ResNet-152, Transformer-17B, GPT-3, and Transformer-1T by 1.76×, 1.87×, 1.34×, and 1.4×, respectively, compared to a baseline wafer-scale Mesh. CCS Concepts • Hardware → Emerging technologies; • Networks → Network architectures; • Computer systems organization → Architectures. Keywords distributed training, wafer-scale platforms ACM Reference Format: Saeed Rashidi, William Won, Sudarshan Srinivasan, Puneet Gupta, and Tushar Krishna. 2025. FRED: A Wafer-scale Fabric for 3D Parallel DNN Training. In Proceedings of The 52nd IEEE/ACM International Symposium on Computer Architecture (’ISCA 2025’) . ACM, New York, NY, USA, 15 pages. [https://doi.org/3695053.3731055](https://doi.org/3695053.3731055) 1 Introduction DNN models are on an exponential growth curve. A recent study shows that in less than two years, the compute and memory requirements for DNN training have increased by 1,800× and 1,500×, respectively [58] . Distributing the training across multiple accelerators or neural processing units (NPUs) is a common practice today to reduce the training time. However, one critical side effect ∗The author is now at Meta: [rashidi1saeed@meta.com](rashidi1saeed@meta.com) ’ISCA 2025’, June 21–25, 2025, Tokyo, Japan 2025. ACM ISBN 978-1-4503-XXXX-X/18/06 [https://doi.org/3695053.3731055](https://doi.org/3695053.3731055) of distributed training is the communication overhead between NPUs to synchronize model gradients and/or activations, depending on the parallelization strategy. As the number of NPUs scales, communication overhead increases, up to a point where it becomes the dominant factor in distributed training latency [26, 27, 54, 55] . There are fundamental limits to the bandwidth that can be provided even by high-speed rack-scale fabrics (such as NVLink [33]), and thus there has been a growing interest in platforms that integrate multiple NPUs together in the same package. Cerebras [10] demonstrated one extreme incarnation of this idea in the form of a monolithic wafer with NPUs connected to one another. More costand yield-effective approaches include silicon/organic interposerbased approaches [24, 59] or using Silicon Interconnect Fabric (SiIF), which bonds chiplets directly onto a full thickness silicon wafer without needing a

FAQs

Why does communication become a bottleneck in distributed DNN training?

As the number of NPUs increases, the communication overhead required to synchronize gradients and/or activations grows, eventually becoming the dominant contributor to distributed training latency.

What limitation of 2D Mesh fabrics motivates the Fred architecture?

2D Mesh topologies are inherently blocking, making them inefficient for DNN training communication patterns that require flexible, strategy-dependent connectivity.

How does Fred support collective communication during training?

Fred builds a distributed on-wafer topology with tiny microswitches to provide nonblocking connectivity and enabling in-switch collective support for arbitrary accelerator groups.

Aug 20, 2026

Upload by William