Categorising the World into Local Climate Zones - Towards Quantifying Labelling Uncertainty for Machine Learning Models

Aug 5, 2026

Upload by Anda

Image classification in remote sensing is highly sensitive to labelling uncertainty, since training images depend on evaluations by human experts. Ambiguities in expert judgments can propagate into downstream models, motivating a probabilistic treatment of label uncertainty. The work models expert evaluations using a multinomial mixture framework, assuming class ambiguity is negligible while expert opinions vary. Parameters are estimated via a stochastic EM algorithm, enabling analysis of label-uncertainty sources such as general class ambiguity, expert heterogeneity, and image origin city, with broad relevance to human-in-the-loop machine learning.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages31

Reading Time1 h 18 min

AccessFree to download

Introduction

Local Climate Zones (LCZ) and satellite image classification

Label acquisition, crowdsourcing, and annotation challenges

Label noise and label ambiguity in machine learning

CATEGORISING THE WORLD INTO LOCAL CLIMATE ZONESTOWARDS QUANTIFYING LABELLING UNCERTAINTY FOR MACHINE LEARNING MODELS arXiv :2309 .01440v1 [ stat .AP] 4 Sep 2023 Katharina Hechinger Department of Statistics Ludwig-Maximilians-University Munich, 80539 [katharina.hechinger@stat.uni-muenchen.de](katharina.hechinger@stat.uni-muenchen.de) Xiao Xiang Zhu Chair of Data Science in Earth Observation Technical University of Munich Munich, 80333 Göran Kauermann Department of Statistics Ludwig-Maximilians-University Munich, 80539 ABSTRACT Image classification is often prone to labelling uncertainty. To generate suitable training data, images are labelled according to evaluations of human experts. This can result in ambiguities, which will affect subsequent models. In this work, we aim to model the labelling uncertainty in the context of remote sensing and the classification of satellite images. We construct a multinomial mixture model given the evaluations of multiple experts. This is based on the assumption that there is no ambiguity of the image class, but apparently in the experts’ opinion about it. The model parameters can be estimated by a stochastic EM algorithm. Analysing the estimates gives insights into sources of label uncertainty. Here, we focus on the general class ambiguity, the heterogeneity of experts, and the origin city of the images. The results are relevant for all machine learning applications where image classification is pursued and labelling is subject to humans. Keywords Expert Evaluations · Labelling Uncertainty · Mixture Models · Multiple Labellers · Stochastic Expectation Maximisation 1 Introduction Machine Learning has achieved impressive standards in recent years. In particular, in image analysis and classification, deep learning has completely changed the way to approach image data. Today, machine learning is increasingly used for the classification of images, with applications for instance in medical image analysis, face recognition, machine vision and many more. In this paper, we focus on satellite images and their use to classify the world into so-called Local Climate Zones (LCZ) as a categorization of the surface. The concept of LCZ, as proposed in Stewart (2011), has achieved a general standard in remote sensing and is based on the assumption that the structure of the landscape influences the local climate. The LCZ scheme categorizes the surface of the world into 17 classes that are supposed to influence local climate behaviour. The classes differ in surface structure (e.g. related to the density or height of trees and buildings) and surface cover (unsealed or sealed) . A schematic description and exemplary satellite images are shown in Figure 1 . This categorisation serves as an international standard for the mapping and analysis of urban areas and massive effort has been spent in developing algorithms that transform satellite images into an LCZ map. For this purpose, deep learning offers promising solutions to achieve high-quality maps and has already proven its utility in this regard, see e.g. Qiu et al. (2019) or Qiu et al. (2018) . Zhu et al. (2022) combine earth observation data with deep learning and reveal detailed morphology of urban agglomerations across the globe. For an extensive overview of challenges, advances and resources of deep learning in the field of remote sensing, we refer to Zhu et al. (2017) . Figure 1: Example of image scenes of the 17 LCZ classes (upper image: Sentinel-1, middle image: Sentinel-2, lower image: high-resolution aerial image from Google) . Machine learning is thereby based on labelled data, that is we are in the context of supervised learning, see e.g. Friedmanet al. (2001) . In this context, the problem of acquiring labels is very common and often solved by crowdsourcing, as introduced by Estellés-Arolas and González-Ladrón-de Guevara (2012) . A lot of effort has already been spent in analysing the quality of such labels, e.g.

FAQs

Why is labelling uncertainty important for satellite image classification into LCZs?

LCZ mapping relies on detailed expert instructions and hand-labelled images. Different experts can reach different conclusions, creating ambiguities that can affect training and model performance.

How does the paper model labelling uncertainty from multiple experts?

It constructs a multinomial mixture model based on evaluations from multiple experts, assuming there is no inherent ambiguity of the image class but expert opinions differ.

How are the model parameters estimated and what insights are produced?

Parameters are estimated using a stochastic EM algorithm. The estimates reveal sources of label uncertainty, focusing on general class ambiguity, expert heterogeneity, and the images’ origin city.

Aug 5, 2026

Upload by Anda