Best Practices in Supervised Machine Learning: A Tutorial for Psychologists

Aug 5, 2026

Supervised machine learning (ML) is increasingly used in psychology and other social sciences, yet core ML concepts and predictive-modeling techniques are still not widely taught in psychology curricula. This tutorial provides an intuitive, thorough four-module primer for psychologists: resampling and model evaluation, random-forest modeling for prediction, benchmark comparisons across datasets, and interpretation with variable importance, effect plots, and fairness. Minimal formulas are used, with R code examples and practical guidance on the PhoneStudy dataset plus a reporting checklist and online materials.

LanguageEnglish

CategoryTechnology

FormatPDF

Pages35

Reading Time1 h 28 min

AccessFree to download

Tutorial modules

Module 1: resampling and evaluation

Module 2: random forest prediction

Module 3: benchmark experiments

Module 4: model interpretation and fairness

~~ ~~ ASSOCIATION FOR Tutorial PSYCHOLOGICAL SCIENCE Best Practices in Supervised Machine Learning: A Tutorial for Psychologists Florian Pargent1, Ramona Schoedel1, and Clemens Stachl1,2 1Department Psychology, Ludwig-Maximilians-Universität München, Munich, Germany, and 2Institute of Behavioral Science and Technology, University of St. Gallen, St. Gallen, Switzerland Advances in Methods and Practices in Psychological Science July-September 2023, Vol. 6, No. 3, pp. 1–35 © The Author(s) 2023 Article reuse guidelines: [sagepub.com/journals-permissions](sagepub.com/journals-permissions)[ ](sagepub.com/journals-permissions)[DOI: 10.1177/25152459231162559](DOI: 10.1177/25152459231162559)[ ](DOI: 10.1177/25152459231162559)[www.psychologicalscience.org/AMPPS](www.psychologicalscience.org/AMPPS) Abstract Supervised machine learning (ML) is becoming an influential analytical method in psychology and other social sciences. However, theoretical ML concepts and predictive-modeling techniques are not yet widely taught in psychology programs. This tutorial is intended to provide an intuitive but thorough primer and introduction to supervised ML for psychologists in four consecutive modules. After introducing the basic terminology and mindset of supervised ML, in Module 1, we cover how to use resampling methods to evaluate the performance of ML models (bias-variance trade-off, performance measures, k-fold cross-validation) . In Module 2, we introduce the nonlinear random forest, a type of ML model that is particularly user-friendly and well suited to predicting psychological outcomes. Module 3 is about performing empirical benchmark experiments (comparing the performance of several ML models on multiple data sets) . Finally, in Module 4, we discuss the interpretation of ML models, including permutation variable importance measures, effect plots (partialdependence plots, individual conditional-expectation profiles), and the concept of model fairness. Throughout the tutorial, intuitive descriptions of theoretical concepts are provided, with as few mathematical formulas as possible, and followed by code examples using the mlr3 and companion packages in R. Key practical-analysis steps are demonstrated on the publicly available PhoneStudy data set (N = 624), which includes more than 1,800 variables from smartphone sensing to predict Big Five personality trait scores. The article contains a checklist to be used as a reminder of important elements when performing, reporting, or reviewing ML analyses in psychology. Additional examples and more advanced concepts are demonstrated in online materials ([https://osf.io/9273g/](https://osf.io/9273g/)). Keywords tutorial, supervised machine learning, cross-validation, interpretable machine learning, random forest, open data, open materials Received 7/13/22; Revision accepted 2/17/23 Over the past decade, supervised machine learning (ML) has appeared with increasing frequency in psychology and other social sciences. In psychology, ML has been used to tackle such diverse topics as predicting psychological traits from digital traces of online and offline behavior (Kosinski et al. , 2013; Stachl, Au, et al. , 2020; Youyou et al. , 2015), modeling consistency in human behavior (Shaw et al. , 2022), or investigating the empirical structure of self-regulation (Eisenberg et al. , 2019) . This popularity can be traced to a number of features: a focus on prediction, which complements traditional methods that emphasize description and explanation (Shmueli, 2010); the flexibility to account for nonlinear patterns in large quantities of data (Kosinski et al. , 2016); and an increase in the generalizability of research findings by evaluating predictive performance on new data (Yarkoni, 2022) . Together, these features hold the promise of elevating the understanding of the processes that connect human behavior, cognition, and experience while being able to account for real-world complexity (Ro

FAQs

How does the tutorial help readers interpret supervised ML models?

Module 4 covers interpretation methods such as permutation variable importance, effect plots (partial dependence and individual conditional expectation profiles), and the concept of model fairness.

Aug 5, 2026

Upload by Lucas Martin