GPUALERT: A ZERO-INSTRUMENTATION PROCESS-BOUNDARY MONITOR FOR DIAGNOSING GPU TRAINING-JOB FAILURES

Jul 16, 2026

GPU training jobs fail frequently on large production clusters, yet operators often learn hours later with minimal evidence: experiment trackers need script edits and cloud connectivity, while scheduler hooks provide only a single status line without logs or causes. GPUAlert is a command-line wrapper that monitors any training command at the process boundary without modifying the workload, then emails structured results including a classified failure cause, durable logs, and bounded output artifacts. The design uses three reliability primitives—pre-launch log guarantees, notifier isolation, and an artifact budget—validated by an evaluation on 474 logs across 15 failure classes.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

Abstract

Introduction

Problem of delayed failure feedback

Need for zero-instrumentation tools

GPUAlert overview and process-boundary operation

Reliability primitives and contributions

arXiv :2607 .0 1409v 1 [ cs . SE] 1 Jul 2026 GPUALERT: A ZERO-INSTRUMENTATION PROCESS-BOUNDARY MONITOR FOR DIAGNOSING GPU TRAINING-JOB FAILURES Parv Agarwal 1 Asif Ekbal 1 † ABSTRACT GPU training jobs fail often: roughly two in five on large production clusters-yet the operator typically learns of a failure only by reconnecting hours later. Experiment trackers require editing the training script and maintaining a cloud connection; the scheduler’s mail hook delivers a single status line with no cause and no logs. GPUAlert is a command-line wrapper that monitors any training command at the process boundary, and with no change to that command, it emails a structured notification on completion carrying a classified failure cause, durable logs, and output artifacts. The tool is organized around three reliability primitives: a pre-launch log guarantee that establishes the durable destination before the child process can crash, notifier isolation that makes the wrapper’s exit code a pure function of the child’s status regardless of whether the email succeeds and a non-silent artifact budget that bounds attachment size without ever dropping output silently. We release a labelled corpus of 474 GPU training logs across 15 failure classes, three of which are synthetic, as noted in Section 4, and a reproducible evaluation harness. On the twelve hardware-reproduced classes, the ordered-rule classifier reaches 0.997 macro-F1, against 0.830 for unordered keyword matching and 0.133 for exit-code inspection. Wrapper overhead is a constant ∼3 ms per job; the pre-launch guarantee preserves a log where a shell redirect yields nothing; and across all 15 failure modes the wrapper returns the child’s exit code unchanged even when the SMTP relay is unreachable. 1 INTRODUCTION The failure of a GPU training job is not a tail event; it is the common case. Large production clusters shed roughly 40% of their jobs before completion (Hu et al., 2024 ; Kokolis et al., 2025 ; Jeon et al., 2019 ; Weng et al., 2022), and infrastructure faults: CUDA errors, ECC events, NVLink and NCCL failures account for the majority of the GPUhours those failures waste. At thousand-GPU scale, the mean time to the first failure of a run is measured in singledigit hours (Kokolis et al., 2025) . These numbers describe a workload in which something breaks constantly and in which knowing what broke, quickly translates directly toreclaimed GPU-hours. The feedback loop available to the person running the job has barely changed. Submit a job, close the laptop, andreconnect hours later to a terminal that shows either a result or a dead process, usually with no context about why it died. 1AI-NLP-ML Research Laboratory, Department of Computer Science and Engineering, Indian Institute of Technology Patna, Patna, Bihar, India. Correspondence to: Parv Agarwal <[parvagarwal9759@gmail.com](parvagarwal9759@gmail.com) >, Asif Ekbal <[asif@iitp.ac.in](asif@iitp.ac.in) >. Preprint. Not peer reviewed. Copyright © 2026 held by the authors. The cost is not the failure itself but the latency to noticing it. A job that hits a CUDA out-of-memory error in its third epoch can sit dead for the rest of a twelve-hour reservation while the operator assumes it is still training. Tools exist, but they do not fit the setting where the gap is widest. Modern experiment trackers are powerful, yet they require importing an SDK into the training script and, in the common configuration, an outbound connection to a hosted backend. Neither assumption holds for an inherited script, a quick one-off run, a shared cluster with restrictive egress, or an air-gapped HPC partition. At the other extreme, the scheduler’s notification hook requires no code change but carries almost no information: one line saying“the job ended“, with no logs, no cause, and no artifacts. Between “instrument everything and phone a cloud” and“a single status line” there is a wide unserved middle, and it is where most day-to

FAQs

How effective is GPUAlert at classifying different failure modes?

On twelve hardware-reproduced classes, the ordered-rule classifier achieves 0.997 macro-F1, compared with 0.830 for unordered keyword matching and 0.133 for exit-code inspection. Wrapper overhead is about 3 ms per job and the child exit code is returned unchanged across failure modes.

Jul 16, 2026

Upload by Lucas Martin