NEST - Network-and Memory-Aware Device Placement for Distributed Deep Learning - Core performance results

Sep 11, 2026

Uploaded by Patrick

Distributed deep learning suffers from inefficient device placement and hybrid parallelism selection because many frameworks use simplified, topology-agnostic searches and address memory constraints only after placement. NEST introduces a network-, compute-, and memory-aware framework that incrementally models communication and memory in a structured dynamic program, jointly optimizing hybrid parallelism and device placement. The approach captures realistic datacenter hierarchies, enables provable optimality, detects bottlenecks early, and achieves up to 2.4× higher throughput while scaling to clusters with 1,000+ GPUs.

LanguageEnglish

CategoryPresentations

FormatPDF

Pages1

AccessFree to download

Problem

Bottlenecks in distributed training

NEST Framework

Network- , compute- , and memory-aware modeling

Incremental optimization via structured dynamic programming

Key Insights

Topology-adaptive partitioning

Explicit memory modeling for early bottleneck detection

Template-based parallelism for Transformer sharding

Results

Throughput improvement over baselines

Scalability on large GPU clusters

NEST: Network-and Memory-Aware Device Placement for Distributed Deep Learning Irene Wang, Vishnu Venkata, Arvind Krishnamurthy, Divya Mahajan Problem: Efficient distributed training is bottlenecked by frameworks that rely on simplified, topology-agnostic search and handle memory constraints post-hoc, leading to excessive synchronization, over-sharding, and poor scalability on realistic, hierarchical datacenter networks. Our Approach: We introduce NEST, a network- , compute- , and memory-aware framework that incrementally models communication and memory within a structured dynamic program to jointly optimize hybrid parallelism and device placement with provable optimality on realistic datacenter topologies. Findings: NEST demonstrates that explicitly modeling network hierarchy and memory constraints enables superior placement strategies, achieving up to 2.4× higher throughput than state-of-the-art baselines and sustaining performance scalability on clusters with over 1,000 GPUs. Paper Link How to Train DNN Efficiently? Today’s AI models are trained at massive scales. Communication can become the bottleneck. Optimal partitioning and placement requires being aware of the underlying network. Network- & Memory-Awareness Prior works fail to consider Network and Memory realities atthe sametime and lead to suboptimal placement plans Oversimplify Networks Post-Placement Memory Check The NEST Framework Key Insights Topology-adaptive partitioning balances latencies across uneven network links Explicit memory modeling enables early bottleneck detection and scalable training Template-based parallelism matches finegrained sharding on repetitive Transformer architectures Validated to exceed profiling-based techniques on real-world clusters Results NEST’s distribution strategy achieves 2.4x higher throughput than state-of-the-art baseline Alpa Manual MCMC Alpa-E Phaze NEST Throughput Improvement 24 18 12 6 0 | Llam | | | a3- | 70B | | | | | | | | | | | | | | | | | | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | X
| | | | | | | | | | | | | | | | | | | | | | X | | | | | | | | | | | | | | | | | X | | | | 64 128 256 512 1024 12 10 8 6 4 2 0 | GPT | 3- | 5B | | | | | | | | | | | | | | | | | | | | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | X X
| | | | | | | | | | | | | | | | | | X | | | | 17 64 128 256 512 1024 16 12 8 4 0 | Mixt | | | | | | | | | | | | | | | | | | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | | | | ral 7B | x 8 | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | X | | | | | | | | | | | | | X | | | | 64 128 256 512 1024 Number of Devices

FAQs

What problem does NEST address in distributed deep learning?

Efficient distributed training is bottlenecked by placement frameworks that ignore network topology and handle memory constraints only after placement, causing excessive synchronization, over-sharding, and poor scalability.

How does NEST optimize device placement and parallelism?

NEST incrementally models communication and memory within a structured dynamic program to jointly optimize hybrid parallelism and device placement on realistic datacenter topologies.

What performance gains does NEST report?

NEST achieves up to 2.4× higher throughput than state-of-the-art baselines (e.g., Alpa) and sustains scalability on clusters with over 1,000 GPUs.

Sep 11, 2026

Uploaded by Patrick