Open science perspectives on machine learning for the identification of careless responding - Anew hope or phantom menace?

Aug 3, 2026

Upload by Eliana

Powerful methods for identifying careless respondents in survey data are crucial not only for the validity of downstream analyses, but also for advancing understanding of the psychological mechanisms behind careless responding. Machine learning, despite its growing prominence across scientific fields, remains largely underused for this detection problem. This review evaluates both the potential and pitfalls of machine learning under an open science lens, highlighting likely reproducibility challenges and offering practical guidance to mitigate them. A proof-of-concept simulation further demonstrates strong promise of an unsupervised approach.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages21

Reading Time53 min

AccessFree to download

Introduction

Careless responding and threats to validity

Screening approaches and relevance

Supervised vs. unsupervised learning

Received: 14 December 2022 Revised: 16 November 2023 Accepted: 22 January 2024 DOI: 10. 1111/spc3 .12941 REVIEW ARTICLE Open science perspectives on machine learning for the identification of careless responding: Anew hope or phantom menace? Andreas Alfons1 | Max Welz1,2 1Department of Econometrics, Erasmus University Rotterdam, Rotterdam, Netherlands 2Department of Public Health, Erasmus MC—University Medical Center Rotterdam, Rotterdam, Netherlands Correspondence Max Welz, Department of Econometrics, Erasmus University Rotterdam, PO Box 1738, 3000 DR Rotterdam, Netherlands. [Email: welz@ese.eur.nl](Email: welz@ese.eur.nl) Funding information Nederlandse Organisatie voor Wetenschappelijk Onderzoek, Grant/Award Number: VI.Vidi.195.141 Abstract Powerful methods for identifying careless respondents in survey data are not just important to ensure the validity of subsequent data analyses, they are also instrumental for studying the psychological processes that drive humans to respond carelessly. Conversely, a deeper understanding of the phenomenon of careless responding enables the development of improved methods for the identification of careless respondents. While machine learning has gained substantial attention and popularity in many scientific fields, it is largely unexplored for the detection of careless responding. On the one hand, machine learning algorithms can be highly powerful tools due to their flexibility. On the other hand, science based on machine learning has been criticized in the literature for a lack of reproducibility. We assess the potential and the pitfalls of machine learning approaches for identifying careless respondents from an open science perspective. In particular, we discuss possible sources of reproducibility issues when applying machine learning in the context of careless responding, and we give practical guidelines on how to avoid them. Furthermore, we illustrate the high potential of an unsupervised machine learning method for the identification of careless respondents in a proof-of-concept simulation experiment. Finally, we stress the necessity of building an open data repository This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited. © 2024 The Authors. Social and Personality Psychology Compass published by John Wiley & Sons Ltd. Soc Personal Psychol Compass. 2024;e12941 . [https://doi.org/10.1111/spc3.12941](https://doi.org/10.1111/spc3.12941) [wileyonlinelibrary.com/journal/spc3](wileyonlinelibrary.com/journal/spc3) 1 of 21 2 of 21 ALFONS and WELZ with labeled benchmark data sets, which would enable the evaluation of methods in a more realistic setting and make it possible to train supervised learning methods. Without such a data repository, the true potential of machine learning for the identification of careless responding may fail tobe unlocked. KEYWORDS careless responding, guidelines, machine learning, open science, reproducibility, unsupervised learning 1 | INTRODUCTION Participants in surveys may not comply with survey instructions due to, for instance, fatigue, lack of motivation, or failure to correctly understand the instructions. This phenomenon is known as careless responding in the psychology literature and has been identified as a major threat to the validity of research results (e.g., Arias et al., 2020; Huang, Liu, & Bowling, 2015; McGrath et al., 2010; Meade & Craig, 2012; Woods, 2006) . If sufficiently many responses are careless, any knowledge drawn from these responses is potentially biased and at worst inaccurate. Considering the ubiquity of survey data in psychology together with the societal impact of the field, this can have dire consequences for future research, policy decisions, and societal outcomes. For example, careless responding may inflate or deflate the effect size of

FAQs

What is careless responding and why does it threaten survey research validity?

Careless responding occurs when participants do not follow instructions due to factors like fatigue or misunderstanding. If many responses are careless, conclusions drawn from the data become biased or even inaccurate, affecting effect sizes and statistical decisions.

How can understanding careless responding improve methods for identifying it?

Careless responding is linked to personal characteristics and personality traits. Better psychological insight can guide the development of improved screeners for detecting careless respondents.

What open science issues may arise when applying machine learning to careless responding?

The review focuses on sources of reproducibility problems that can occur when machine learning is applied in this context. It provides practical guidelines to avoid these issues and emphasizes the need for open data repositories to evaluate methods realistically.

Aug 3, 2026

Upload by Eliana