Optimizing Performance on Trinity - Utilizing Machine Learning, Proxy Applications and Scheduling Priorities

Aug 4, 2026

Upload by Oliver

Supercomputers like Trinity scale to thousands of nodes, while real workloads progress at the rate of the slowest components. The paper addresses the ongoing need to detect underperforming nodes, assess their impact, and reduce downtime during performance-critical runs. It presents fast proxy tests based on MPI and OpenMP to replace long runtimes, then applies machine learning and clustering to isolate outliers. The resulting ordered node lists support scheduling policies that minimize slow-node effects and improve overall cluster efficiency.

LanguageEnglish

CategoryTechnology

FormatPDF

Pages7

Reading Time18 min

AccessFree to download

Introduction

Node bottlenecks and performance categories

Limitations of full HPL measurements

Methodology: Proxy applications

Outline of proxy test approach

Results and mapping to HPL

High Performance Linpack comparison

Mitigation and scheduling strategies

Conclusion

arXiv :2404 . 10617v1 [ cs .DC] 16 Mar 2024 Optimizing Performance on Trinity Utilizing Machine Learning, Proxy Applications and Scheduling Priorities Phil Romero High Performance Computing Division Los Alamos National Laboratory Email: [prr@lanl.gov](prr@lanl.gov) Abstract—Abstract—The sheer number of nodes continues to increase in today’s supercomputers, the first half of Trinity alone contains more than 9400 compute nodes. Since the speed of today’s clusters are limited by the slowest nodes, it more important than ever to identify slow nodes, improve their performance if it can be done, and assure minimal usage of slower nodes during performance critical runs. This is an ongoing maintenance task that occurs on a regular basis and, therefore, it is important to minimize the impact upon its users by assessing and addressing slow performing nodes and mitigating their consequences while minimizing down time. These issues can be solved, in large part, through a systematic application of fast running hardware assessment tests, the application of Machine Learning, and making use of performance data to increase efficiency of large clusters. Proxy applications utilizing both MPI and OpenMP were developed to produce data as a substitute for long runtime applications to evaluate node performance. Machine learning is applied to identify underperforming nodes, and policies are being discussed to both minimize the impact of underperforming nodes and increase the efficiency of the system. In this paper, I will describe the process used to produce quickly performing proxy tests, consider various methods to isolate the outliers, and produce ordered lists for use in scheduling to accomplish this task. Keywords–Machine Learning, High Performance Computing, Benchmarking, Artificial Intelligence. I. INTRODUCTION Today’s supercomputers consist of ever larger and growing numbers of nodes, since computational tasks running on these computers are limited to progressing at rates that are limited by the slowest performing components, it is more important than ever to identify bottlenecks and mitigate their damaging effects in a time efficient manner. There are several factors that contribute to producing the slow progression of computational tasks including: 1) Slow performing CPU’s 2) Slow performing Random Access Memory 3) Slow performing and/or less than optimally buffered input/output systems 4) Slow performing node interconnects. Factory tests allowed Los Alamos extensive time to test many different applications on a two thousand node subset of Trinity. Four different tests were conducted targeting each performance category of cpu speed, memory speed, and interconnect bandwidth. The results were then mined for clusters utilizing the kMeans clustering algorithm[1] . The output of the Figure 1 . This figure shows a map plot that attempts to preserve distances between the features of all items under consideration. Note the cluster labeled”3”, it shows a markedly different shape from the clusters labeled ”1” with large outliers. The x and y axes represent distances in arbitrary space and are orthogonal. clustering process proved interesting in that cpu speed tests produced the largest variations in performance. This can be seen in Figure 1, the clusters labeled 2 and 3 show widespread outliers that produce irregular shape clusters as compared to the cluster labeled 1 . This plot attempts to preserve distances in two dimensions that enable a better assessment of the variability in items than would be found through a Principal Components Analysis[4] that is sensitive to a given projection of data from many dimensions into two dimensions. Similar plots produced for memory speed and interconnect bandwidth do not show widespread outliers, they produce relatively compact and regular shapes.This performance imbalance indicates that extra effort should be expended in reducing the differences in performance amongst cpu relat

FAQs

Why is identifying slow nodes critical on Trinity-like supercomputers?

Cluster-wide task progress is limited by the slowest nodes, so detecting bottlenecks early helps avoid damaging performance drops and reduces user-impacting downtime.

What problem do proxy applications solve compared with standard HPL runs?

HPL can take more than four hours per tuned assessment and requires multiple measurements for confidence, so fast proxy tests provide quicker performance information.

How does machine learning contribute to improving scheduling decisions?

Machine learning is used to identify underperforming nodes, isolate outliers, and generate ordered node lists that scheduling policies can exploit to reduce the impact of slow nodes.

Aug 4, 2026

Upload by Oliver