You Are What You Click - Using Machine Learning to Model Trace Data for Psychometric Measurement

Aug 4, 2026

Upload by Jiven

Assessment trace data such as mouse movements and timing can reflect individual differences, but testing practice underuses them. This article proposes a 10-step development procedure to improve the success of trace-data modeling, from grounding in psychometric theory and building data-collection infrastructure to developmental validation, holdout validation, feature engineering, algorithm selection, hyperparameter tuning with internal cross-validation, and diagnostic checks for overfitting or underfitting. It also frames challenges from weak trait signals, arguing that valid models can support decision-making systems.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages21

Reading Time53 min

AccessFree to download

Abstract & Motivation

Trace data as digital traces for measurement

Need for combined psychometrics and data science

10-step framework for successful modeling

Psychometric grounding and data infrastructure

Validation strategy and feature engineering

Algorithm selection, model tuning, diagnostics

Final evaluation against conceptual, technical, and psychometric goals

InternatIonal Journal of testIng 2022, Vol. 22, nos. 3–4, 243–263 [https://doi.org/10.1080/15305058.2022.2134394](https://doi.org/10.1080/15305058.2022.2134394) You are what you click: using machine learning to model trace data for psychometric measurement Richard N. Landersa , Elena M. Auera , Gabriel Mersya,b, Sebastian Marina and Jason Blaikc aDepartment of Psychology, university of Minnesota, Minneapolis, Mn, usa; bDepartment of Computer science, university of Chicago, Il, usa; cCappfinity, sydney, australia ABSTRACT Assessment trace data, such as mouse positions and their timing, offer interesting and provocative reflections of individual differences yet are currently underutilized by testing professionals. In this article, we present a 10-step procedure to maximize the probability that a trace data modeling project will be successful: 1) grounding the project in psychometric theory, 2) building technical infrastructure to collect trace data, 3) designing a useful developmental validation study, 4) using a holdout validation approach with collected data, 5) using exploratory analysis to conduct meaningful feature engineering, 6) identifying useful machine learning algorithms to predict a thoughtfully chosen criterion, 7) engineering a machine learning model with meaningful internal cross-validation and hyperparameter selection, 8) conducting model diagnostics to assess if the resulting model is overfitted, underfitted, or within acceptable tolerance, and 9) testing the success of the final model in meeting conceptual, technical, and psychometric goals. If deemed successful, trace data model predictions could then be engineered into decision-making systems. We present this framework within the broader view of psychometrics, exploring the challenges of developing psychometrically valid models using such complex data with much weaker trait signals than assessment developers have typically attempted to model. KEYWORDS Machine learning; trace data; data science; mousetrap; psychometric CONTACT richard n. landers [rlanders@umn.edu](rlanders@umn.edu) Department of Psychology, university of Minnesota, Minneapolis, Mn, usa. Copyright © 2022 the author(s) . Published with license by taylor & francis group, llC. this is an open access article distributed under the terms of the Creative Commons attribution-nonCommercialnoDerivatives license ([http://creativecommons.org/licenses/by-nc-nd/4.0/](http://creativecommons.org/licenses/by-nc-nd/4.0/)), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited, and is not altered, transformed, or built upon in any way. 244 R. N. LANDERS ET AL. As digital assessment technologies grow more complex, so do the trace data that they produce and the potential of those trace data to be repurposed for additional valid assessment. The term, trace data, sometimes called digital traces, digital footprints, digital exhaust or behavioral surplus, refers to usually-incidental behavioral metadata created and recorded in the wake of other digital activities. Although the science of repurposing such data is in early stages, it capitalizes on two specific technological advances. First, the devices that people typically use to complete remote digital assessments, such as smartphones with mobile broadband, now have such powerful data processing capabilities and high data transfer speeds that the additional burden of collecting, preprocessing, and streaming trace data back to assessment servers has become trivial throughout the developed world. Second, although the sheer volume and variety of such data would have even ten years ago been difficult to meaningfully process, recent advances in machine learning have made the development of predictive models using such data within reach of even modestly priced personal computers. Combining an ever-growing variety of potentially useful data with advanced analytic techniques

FAQs

What types of data are considered “trace data” in this work?

Trace data refers to behavioral metadata created during digital activities, such as mouse positions and timing recorded in the course of assessment tasks.

What is the core contribution of the article?

It provides a 10-step procedure to increase the likelihood that a trace-data modeling project succeeds, linking technical modeling stages to psychometric requirements.

Why is it difficult to build trace-data models successfully?

Researchers must balance complex measurement concepts (e.g., reliability and validity) with data-analytic skills, because lacking either perspective can lead to models that seem adequate in development but fail in real-world use.

Aug 4, 2026

Upload by Jiven