On the use of adversarial validation for quantifying dissimilarity in geospatial machine learning prediction

Aug 4, 2026

Upload by Noah

Recent geospatial machine learning research shows that cross-validation (CV) outcomes depend strongly on how dissimilar the sample data are from the prediction locations. This paper introduces a feature-space method to quantify that dissimilarity on a 0–100% scale using adversarial validation via a binary classifier. Experiments on synthetic and real datasets with increasing dissimilarities demonstrate full-range quantification and clarify CV behavior under varying mismatch levels.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages18

Reading Time45 min

AccessFree to download

Introduction

Motivation for geospatial ML evaluation and CV

Random k-fold cross-validation and its limitations

Feature-space dissimilarity and spatial prediction settings

arXiv :2404 . 12575v1 [ cs .LG] 19 Apr 2024 On the use of adversarial validation for quantifying dissimilarity in geospatial machine learning prediction Yanwen Wanga,* , Mahdi Khodadadzadeha , and Ra´ul Zurita-Millaa a Faculty of Geo-Information Science and Earth Observation (ITC), University of Twente, 7522NH Enschede, the Netherlands ARTICLE HISTORY Compiled April 22, 2024 ABSTRACT Recent geospatial machine learning studies have shown that the results of model evaluation via cross-validation (CV) are strongly affected by the dissimilarity be tween the sample data and the prediction locations. In this paper, we propose a method to quantify such a dissimilarity in the interval 0 to 100%, and from the perspective of the data feature space. The proposed method is based on adversarial validation, which is an approach that can check whether sample data and prediction locations can be separated with a binary classifier. To study the effectiveness and generality of our method, we tested it on a series of experiments based on both syn thetic and real datasets and with gradually increasing dissimilarities. Results show that the proposed method can successfully quantify dissimilarity across the entire range of values. Next to this, we studied how dissimilarity affects CV evaluations by comparing the results of random CV and of two spatial CV methods, namely block and spatial+ CV. Our results showed that CV evaluations follow similar pat terns in all datasets and predictions: when dissimilarity is low (usually lower than 30%), random CV provides the most accurate evaluation results. As dissimilarity increases, spatial CV methods, especially spatial+ CV, become more and more ac curate and even outperforming random CV. When dissimilarity is high (>=90%), no CV method provides accurate evaluations. These results show the importance of considering feature space dissimilarity when working with geospatial machine learn ing predictions, and can help researchers and practitioners to select more suitable CV methods for evaluating their predictions. KEYWORDS Machine learning; geospatial regression; model evaluation; cross-validation; feature space. 1. Introduction Machine learning (ML) is widely used in geospatial prediction to estimate unknown values at specific prediction locations (Hengl et al. 2018; Aguilar et al. 2018; Usmanet al. 2023) . These predictions are often done to create spatially continuous products, for example, mineral (Khodadadzadeh and Gloaguen 2019), health risk (Garcia-Marti et al. 2018), or phenological (Zurita-Milla, Laurent, and van Gijsel 2015) maps. In these and many other applications, predictions come from ML regression models trained on limited sample data, where the number of samples is typically much smaller than the number of prediction locations. This imbalance between samples and prediction locations, is mostly due to practical limitations such as accessibility (Lamichhane, Kumar, and Wilson 2019) or sampling costs (Hengl et al. 2015) . For similar reasons, collecting additional data for an independent evaluation of geospatial ML prediction is rarely feasible (Valavi et al. 2019) . To address these operational constraints, the evaluation of geospatial ML models is mainly conducted by splitting the available sample data into training and validation subsets (de Bruin et al. 2022; Wang, Khodadadzadeh, and Zurita-Milla 2023) . Random k-fold cross-validation (RDM-CV) stands out as the most popular evaluation method (Chen et al. 2018; Nesha et al. 2020; Guo et al. 2022) . Asthe name indicates, RMD-CV randomly splits the sample data into k equal-size folds, and then, it iteratively uses one of them as a validation subset and the remaining ones as a training subset. When sample data are collected by a probability sampling strategy, such as simple random sampling (Brus, Kempen, and Heuvelink 2011; Wanget al. 2012) and regular sampling (Lagacherie et al. 20

FAQs

What problem does the paper address in geospatial machine learning evaluation?

It addresses that CV results for geospatial predictions are strongly affected by the dissimilarity between training sample data and the prediction locations.

How does the proposed method quantify dissimilarity?

It uses adversarial validation, training a binary classifier to test whether sample data and prediction locations can be separated in the data feature space, producing a 0–100% dissimilarity measure.

How does dissimilarity level affect which CV method works best?

When dissimilarity is low (typically below 30%), random CV is most accurate; as dissimilarity increases, spatial CV methods—especially spatial+ CV—become more accurate; at very high dissimilarity (>=90%), no CV method yields accurate evaluations.

Aug 4, 2026

Upload by Noah