MAST - Global Scheduling of ML Training across Geo-Distributed Datacenters at Hyperscale

Sep 17, 2026

Uploaded by McQueen

Public clouds require users to manually choose a datacenter region for both ML training data upload and training execution so that data and computation are colocated. Independent user choices can cause demand–supply mismatches across regions, reducing hardware utilization and hurting provider profitability. In Meta’s hyperscale private cloud, MAST provides a global scheduling abstraction that places both training data and workloads across regions. Three design principles—temporal decoupling, scope decoupling, and exhaustive search—enable global-scale scheduling and balance load, reducing GPU demand-to-supply ratio from 2.63 to 0.98 for high-priority workloads.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages19

Reading Time48 min

AccessFree to download

Abstract

Introduction

Contributions

Data-GPU colocation

Scalability

Design principles

MAST: Global Scheduling of ML Training across Geo-Distributed Datacenters at Hyperscale Arnab Choudhury, Meta Platforms; Yang Wang, Meta Platforms and The Ohio State University; Tuomas Pelkonen, Meta Platforms; Kutta Srinivasan, LinkedIn; Abha Jain, Shenghao Lin, Delia David, Siavash Soleimanifard, Michael Chen, Abhishek Yadav, Ritesh Tijoriwala, Denis Samoylov, and Chunqiang Tang, Meta Platforms [https://www.usenix.org/conference/osdi24/presentation/choudhury](https://www.usenix.org/conference/osdi24/presentation/choudhury) This paper is included in the Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation . July 10–12, 2024 • Santa Clara, CA, USA 978-1-939133-40-3 Open access to the Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation is sponsored by MAST: Global Scheduling of ML Training across Geo-Distributed Datacenters at Hyperscale Arnab Choudhury 1 , Yang Wang 1 †, Tuomas Pelkonen 1 , Kutta Srinivasan‡, Abha Jain 1 , Shenghao Lin 1 , Delia David 1 , Siavash Soleimanifard 1 , Michael Chen 1 , Abhishek Yadav 1 , Ritesh Tijoriwala 1 , Denis Samoylov 1 , and Chunqiang Tang 1 1 Meta Platforms † The Ohio State University ‡ LinkedIn (work done while at Meta) Abstract In public clouds, users must manually select a datacenter region to upload their ML training data and launch ML training workloads in the same region to ensure data and computation colocation. Unfortunately, isolated decisions by individual users can lead to a mismatch between workload demand and hardware supply across regions, hurting the cloud provider’s hardware utilization and profitability. To address this problem in Meta’s hyperscale private cloud, we provide a global-scheduling abstraction to all ML training workloads. Users simply submit their training workloads to MAST, our global scheduler, and rely on it to intelligently place both data and training workloads to different regions. We describe three design principles that enable MAST to schedule complex ML training workloads at a global scale: temporal decoupling, scope decoupling, and exhaustive search. MAST successfully balances the load across global regions. Before MAST, the most overloaded region had a GPU demand-to-supply ratio of 2.63 for high-priority workloads. With MAST, this ratio has been reduced to 0.98, effectively eliminating the overload. 1 Introduction The success of ML applications [8, 46] has resulted in ML training becoming the fastest-growing datacenter workload. Public cloud providers run ML training workloads in multiple geo-distributed datacenter regions [3,4, 15] to ensure sufficient capacity. Accordingly, users need to manually select a region to upload their ML training data and then launch training workloads in the same region to ensure colocation of data and computation. Unfortunately, such manual selection can lead to a regional mismatch between workload demand and hardware supply. For instance, one region may exhaust its capacity, accumulating a long queue of pending jobs, while another region has surplus capacity remaining idle. Contributions: Chunqiang, Kutta, and Tuomas initiated the MAST project in 2020 . In terms of paper writing, Yang wrote the majority of the paper, followed by Chunqiang. In terms of coding, Arnab, Kutta, and Tuomas led the project’s development at different times. All other authors also made major contributions to the project’s development. Meta’s private cloud used to experience this load imbalance. It comprised tens of datacenter regions, millions of machines, and tens of thousands of GPUs. Similar to public clouds, users initially had to manually select regions to store training data and launch workloads. Users’ suboptimal decisions previously led to an imbalance in the GPU demand-to-supply ratio, reaching as high as 2.63 in certain regions for highpriority workloads, which was later reduced to 0.98 through optimizations described in this p

FAQs

What problem does MAST address in geo-distributed ML training?

MAST addresses regional mismatch between workload demand and hardware supply caused by users manually selecting datacenter regions, which can lead to queue buildup in one region and idle capacity in another.

How does MAST help users submit training workloads?

Users submit training workloads to MAST, which intelligently places both training data and the training workloads into different regions to achieve data–GPU colocation.

Which design principles enable MAST to schedule workloads at global scale?

MAST uses temporal decoupling, scope decoupling, and exhaustive search to manage scheduling complexity across many regions and machines.

Sep 17, 2026

Uploaded by McQueen