Prior-Matched Evaluation of Operational Earth-Observation Classifiers: A Three-Number Reporting Method Demonstrated on Sentinel-1

Jul 16, 2026

Upload by Eden

Internal Waves Service screens the Sentinel-1 Wave-mode archive for internal solitary waves and routes detections to experts, where expert attention is the dominant cost of error and thus precision must lead. Classifiers trained and reported on balanced data can look good while failing operationally because balanced-test metrics hide the true positive rate (~1/20). A precision-first evaluation is required: a prior-matched three-number reporting method compares balanced-test, an operational-prior frozen test, and real post-deployment adjudication to measure the honest precision gap.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages24

Reading Time1 h 0 min

Introduction

Operational setting and precision-first objective

Why balanced-test reporting fails under operational priors

Three-number prior-matched reporting method

arXiv :2607 .07 146v 1 [ cs .LG] 8 Jul 2026 Prior-matched evaluation of operational Earth-observation classifiers: a three-number reporting method demonstrated on Sentinel-1 internal-wave detection ∗ João Pinelo, João Gonçalves, Arun Shukla, Adriana Santos-Ferreira† 8 July 2026 Abstract The Internal Waves Service screens the Sentinel-1 Wave-mode archive for internal solitary waves, routing detections to experts whose adjudication time is the resource the effort exists to conserve. Because attention is the cost of error, precision leads. Its classifier was trained and reported at a one-to-one class balance, fixed before the operational rate could be known. That rate has since emerged at roughly one scene in twenty, anda balanced-test score badly overstates the precision a validator meets. A model that scores 0.794 balanced-test precision scores 0.192 in real operation: the gap is a systematic artefact of reporting at the wrong prior, invisible to the metric most work quotes. We show the mismatch to be an evaluation problem in the costume of a training one at a fixed recall, prior correction and calibration cannot move precision, and answer it with a prior-matched reporting method based on three numbers: balanced-test, operational-prior, and real post-deployment, whose contrast is the honest measure. A precision-first, leakage-controlled development cycle then improves the classifier lever by lever, each promoted only against a preregistered margin; added capacity not clearing it, calibration inert, feature aggregation the one real lift, so the honest negatives are as much a result asthe gain. Holding recall at a floor of 0.80 and certifying against a sealed, single-read lockbox, the promoted model reports 0.927 precision at the operational prior; an out-of-time check confirms discrimination transfers to unseen periods while a fixed operating point does not. Prior-matched reporting, begin balanced, then move to the prior as the stream reveals it, transfers to any operational Earth-observation service bootstrapping a rare-event detector under a prior it has yet to discover. ∗ This manuscript is being submitted to Geoscientific Model Development. †Atlantic International Research Centre (AIR Centre), Azores, Portugal. ORCID—João Pinelo: 0000-0002-4890-0775; João Gonçalves: 0009-0001-4547-1696; Arun Shukla: 0009- 0001-5812-1660; Adriana Santos-Ferreira: 0000-0002-5704-6021 . Correspondence: João Pinelo ([joao.pinelo@aircentre.org](joao.pinelo@aircentre.org)) 1. Introduction Internal solitary waves are among the most energetic features of the ocean, and their surface signature — alternating bands of increased and reduced roughness as the wave-induced currents strain the short surface waves that syntheticaperture radar (SAR) responds to—is recorded in SAR imagery, which makes a satellite SAR archive a natural instrument for mapping where and how often they occur (e.g., Barintag et al. 2023) . The Internal Waves Service builds that map operationally from Sentinel-1 Wave-mode data, running a convolutional classifier over the incoming vignettes and passing its detections to domain experts who confirm or reject each one. The classifier’s purpose is not to be the final word but to spend the experts’ time well: the archive is far larger than any team could inspect exhaustively (Sect. 2), so the model exists to concentrate scarce expert attention on the scenes most likely to carry a wave. That framing sets the cost of error. A false positive lands a wave-free scene on a validator’s desk and consumes the attention the service is built to conserve; a missed wave is largely recoverable, because Wave-mode acquisitions recur over the same locations on the orbit repeat and the archive is reprocessed, so the same site returns for another look. Precision therefore leads and recall is held to a floor rather than maximised—the operating discipline the whole effort is organised around. The diﬀiculty is not the model so much as h

FAQs

Why can a classifier with good balanced-test precision perform poorly in real Sentinel-1 operations?

Because the classifier is evaluated on class-balanced data, while the operational stream has a much lower positive rate (~0.05). Balanced-test metrics do not expose the negatives-to-positives ratio seen in production, hiding the operational precision gap.

What is the core evaluation problem addressed by the paper?

The mismatch between the prior assumed by reported metrics and the prior encountered in deployment. The paper argues this is an evaluation issue presented as a training issue, requiring honest reporting rather than simple retraining or calibration.

How does the proposed three-number prior-matched reporting method work?

It characterizes an operational classifier with three figures: balanced-test performance, performance on a frozen test set drawn at the operational prior, and real post-deployment performance from expert adjudication. The contrast among these measures is treated as the true measure of operational precision.

Jul 16, 2026

Upload by Eden