Human-in-the-Loop Mixup

Aug 3, 2026

Upload by Finn

Aligning model representations to humans improves robustness and generalization, yet existing approaches often rely on standard observational data and assume synthetic labels are perceptually aligned. This work examines mixup synthetic data, using its controlled mixing mechanism as a testbed for perceptual–label alignment. The authors design the HILL MixE Suite interfaces, collect 159 participants’ perceptual judgments with uncertainty, and find inconsistent alignment with traditional synthetic labels. These results support reliability gains for downstream models, especially when incorporating human uncertainty, and release judgments via the H-Mix data hub.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages11

Reading Time28 min

AccessFree to download

Abstract

Introduction

Motivation: synthetic data and alignment

Why focus on mixup

Goal and elicitation approach

Human-in-the-Loop Mixup Katherine M. Collins* 1 Umang Bhatt 1,2 Weiyang Liu 1,3 Vihari Piratla 1 Ilia Sucholutsky4 Bradley Love2,5 Adrian Weller 1,2 1University of Cambridge 2The Alan Turing Institute 3Max Planck Institute for Intelligent Systems 4Princeton University 5University College London Abstract Aligning model representations to humans has been found to improve robustness and generalization. However, such methods often focus on standard observational data. Synthetic data is proliferating and powering many advances in machine learning; yet, it is not always clear whether synthetic labels are perceptually aligned to humans – rendering it likely model representations are not human aligned. We focus on the synthetic data used in mixup: a powerful regularizer shown to improve model robustness, generalization, and calibration. We design a comprehensive series of elicitation interfaces, which we release as HILL MixE Suite, and recruit 159 participants to provide perceptual judgments along with their uncertainties, over mixup examples. We find that human perceptions do not consistently align with the labels traditionally used for synthetic points, and begin to demonstrate the applicability of these findings to potentially increase the reliability of downstream models, particularly when incorporating human uncertainty. We release all elicited judgmentsin a new data hub we call H-Mix. 1 INTRODUCTION Synthetic data is proliferating, fueled by increasingly powerful generative models, e.g. [Goodfellow et al., 2014a, Dhariwal and Nichol, 2021] . These data are not only consumed directly by people – but, as training predictive models on synthetic data has been found to unlock tremendous advances in machine learning (ML) [Silver et al., 2016, de Melo et al., 2022, Emam et al., 2021, Jordon et al., 2022], synthetic data is increasingly employed to train algorithms serving as engines of many applications humans may in- * [Correspondence to: kmc61@cam.ac.uk](Correspondence to: kmc61@cam.ac.uk) mixup Data {0.1,0.9} λf = 0.1 {0.5,0.5} λf = 0.5 {0.7,0.3} λf = 0.7 Original Data {1,0} {0,1} (a) How mixup data is constructed Answer Pool construct Which mixup data best matches the label {0.5,0.5}? (b) Elicitation Setting I: Endorse a synthetic image to match a label construct What is the label that best matches this mixup data? |

Annotator | infer | λf
Mixing Coefficient
Confidence | | --- | --- | --- | (c) Elicitation Setting II: Infer the label for a synthetic image Figure 1: Framework overview. A) Synthetic data generating process used in mixup; B) and C) depict elicitation settings: B) participants endorse a synthetic image to match a label, C) participants infer the label for a synthetic image and provide their uncertainty in the corresponding inference. teract with. However, it is not always clear whether human perceptual judgments of synthetically-generated data match the generative process used to create them. Aligning networks to match humans’ perceptual inferences could be a way to further ensure model reliability, trustworthiness, downstream performance, and robustness [Nanda et al., 2021, Chen et al., 2022, Fel et al., 2022, Sucholutsky and Griffiths, 2023] . If these data are not aligned with human percepts, then performance potentially could be improved by altering such signals to better match the richness of human judgments: this has proven effective when aligning Proceedings of the 39th Conference on Uncertainty in Artificial Intelligence (UAI 2023), PMLR 216:454–464 . models with human probabilistic knowledge and perceptual uncertainty [Collins et al., 2022a, Sanders et al., 2022, Sucholutsky et al., 2023] . We argue that one ought to verify whether synthetic data aligns with human perception, and if not, explore whether training with human-relabeled examples improves model performance. In this work, we take a step in this direction by focusing on m

FAQs

Why does the paper investigate perceptual alignment for synthetic labels?

It addresses uncertainty about whether human perceptual judgments match the synthetic generative process, since misalignment can undermine reliability and performance improvements expected from synthetic-data training.

Why is mixup chosen as the focus of the study?

Mixup has a simple, controlled generative mechanism via an explicit mixing coefficient, is widely used and effective as a regularizer, and its linear category boundaries contrast with known nonlinear human perceptual warping.

What does the study contribute in terms of data collection and tools?

It introduces HILL MixE Suite to elicit perceptual judgments with uncertainties from 159 participants and releases all elicited judgments in a new data hub called H-Mix.

Aug 3, 2026

Upload by Finn