An Interactive Human-Machine Learning Interface for Collecting and Learning from Complex Annotations - Paper

Aug 3, 2026

Upload by Maeve

Human-Computer Interaction can enhance machine learning by improving model performance, accelerating learning, and strengthening user confidence. This work removes the assumption that annotators must conform to rigid constraints of traditional labels by collecting more flexible supervision signals. The proposed human-machine learning interface supports binary classification and lets annotators use counterfactual examples alongside standard labels as dataset annotations. The paper also reviews challenges for extending the approach.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages4

Reading Time10 min

AccessFree to download

Abstract

1 Introduction

Human-in-the-loop and related approaches

Active learning and machine teaching

Counterfactual annotation motivation

2 Methodology

Learning task

Interface and learning process

An Interactive Human-Machine Learning Interface for Collecting and Learning from Complex Annotations Jonathan Erskine 1 , Matt Clifford 1 , Alexander Hepburn 1 , Ral Santos-Rodrguez 1 1University of Bristol {jonathan.erskine, matt.clifford, alex.hepburn, [enrsr](enrsr}@brisol.ac.uk)[}](enrsr}@brisol.ac.uk)[@brisol.ac.uk](enrsr}@brisol.ac.uk) arXiv :2403 . 19339v1 [ cs .LG] 28 Mar 2024 Abstract Human-Computer Interaction has been shown to lead to improvements in machine learning systems by boosting model performance, accelerating learning and building user confidence. In this work, we aim to alleviate the expectation that human annotators adapt to the constraints imposed by traditional labels by allowing for extra flexibility in the form that supervision information is collected. For this, we propose a human-machine learning interface for binary classification tasks which enables human annotators to utilise counterfactual examples to complement standard binary labels as annotations for adataset. Finally we discuss the challenges in future extensions of this work. 1 Introduction While the field of Artificial intelligence (AI) research has undoubtedly made remarkable progress in recent years, it is important to recognize that this expansion does not necessarily imply a proportionate step towards a form of AI that has ability to understand, learn, and apply knowledge across various domains. Instead, we are witnessing the proliferation of specialized tools; each designed to tackle specific tasks with remarkable proficiency, often with a high associated cost. For example, Large Language Models (LLMs) excel in their designated domains [Wei et al., 2022], but their exceptional performance is often reliant on the availability of large datasets [Devlin et al., 2019] and their capabilities are often limited when confronted with unfamiliar or unforeseen challenges [Collins et al., 2022] . In this paper we ask if the reliance on large amounts of data and poor model generalisation can be alleviated through human-machine interaction, and propose an interface that enables human annotators to evaluate model behaviour and increase the complexity of individual annotations where required. Many recent approaches in human-in-the-loop (HITL) machine learning have focused on providing post-hoc explanations for the decisions generated by complex machine learning models [Wang et al., 2021; Kurzendorfer et al., 2017] . By integrating these explanations into the pipeline, researchers aim to empower humans to detect the presence of spurious correlations. Such correlations often imply biases present in the training data, which in turn presents a need for more intelligent data selection. Active learning typically engages humans in the annotation process to strategically choose samples that will most effectively improve the model’s performance [Ren et al., 2022], usually selected to reduce a form of uncertainty [Prince, 2004], mitigating the need for large datasets. However, solely relying on labels for supervision information may be too inflexible for some tasks and may under-utilise the skills of the annotators. The field of Machine Teaching provides flexibility by allowing humans to select groups of samples that effectively describe concepts and patterns [Zhu, 2015] . Recent work combines active learning and machine teaching to automatically generate the minimum viable dataset to learn a concept, and then apply an active learning approach to refine this initial prototype [MosqueiraRey et al., 2021] . Human involvement in the learning pipeline is not only restricted to data selection. Humans can provide feedback on the explanations produced by models e.g. by adjusting decision boundaries for a 3D segmentation task [Kurzendorfer et al., 2017] according to human input, or to augment training examples. Kaushik et al. create a new dataset of counterfactual pairs [Kaushik et al., 2020]; a counterfactual is usually defined as a point be

FAQs

What problem does the proposed interface aim to solve for human annotators?

It reduces the expectation that annotators must adapt to rigid constraints from traditional labels by allowing additional flexibility in how supervision information is collected.

How does the interface incorporate counterfactuals in binary classification?

It enables human annotators to use counterfactual examples to complement standard binary labels as annotations for the dataset.

What kind of learning task is used to demonstrate the approach?

The paper demonstrates the method on a simple synthetic, two-dimensional binary classification setup with separable clusters.

Aug 3, 2026

Upload by Maeve