A Human-Centered Approach to Aligning Risk Constructs in Machine Learning and Social Care Practice

Aug 5, 2026

Upload by Miles

Social care organizations increasingly rely on machine learning-based predictive risk models (ML-based PRMs) to support risk-informed decisions for tasks such as welfare fraud detection, identifying children at risk of maltreatment, and determining eligibility for preventative services. Research highlights that practitioners often resist ML-enabled PRMs, and this study examines one cause: misalignment between the risk constructs used by practitioners and those operationalized by the models. Using a five-month field study, it analyzes tensions in how risk is framed and proposes human-computer interaction strategies to align these constructs.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages1

Reading Time3 min

AccessFree to download

Abstract

Research problem and context

Field study scope and questions

Tension between practitioner and PRM risk constructs

Mental models and design strategies

A Human-Centered Approach to Aligning Risk Constructs in Machine Learning and Social Care Practice Tyler Reinmund Department of Computer Science, University of Oxford, Oxford, UK [tyler.reinmund@cs.ox.ac.uk](tyler.reinmund@cs.ox.ac.uk) Abstract Social care organizations increasingly use machine learning-based predictive risk models (ML-based PRMs) for risk-informed decision-making. For example, these technological systems are used to identify welfare fraud, children at risk of maltreatment, or service users eligible for preventative care programs. Risk management, as these examples show, is a major component of contemporary social care practice. Yet, studies have found that social care practitioners meet ML-enabled PRMs with resistance. In this research, I explore one reason for this phenomenon: a misalignment between risk constructs. Drawing from a 5-month field study on the implementation of a ML-based PRM within asocial care organization in England, I address two questions: (1) how does the construct of risk, as it isoperationalized in a ML-based PRM, resemble the way social care practitioners engage with risk in their own practice? and (2) how can designers align these constructions of risk? Towards the first question, I show a tension between the two conceptions of risk. Social care practitioners are taught to see risk as contextual, subjective, and contingent: respectively, aspects ofa person’s environment may either exacerbate or alleviate a risk; the person for whom the risk assessment is being conducted should be able to determine what is, and is not, a risk; and risks should be thought of in relation to proposed interventions. Meanwhile, the PRM assigns risks to individuals generally – a given risk is treated equivalently across different individuals – and externally: an external agent ascribes risk to an individual without input from the latter. Why is this dissonance important? One answer to that question comes from the field of human-computer interaction and its concept of a “mental model.” A mental model is the understanding a user has of how some system works; when a person has an incorrect mental model, she will struggle to effectively use a system and adapt to errors in its behavior. Broadly, there are two families of solutions that designers can employ to support the development of valid mental models: increasing system transparency and basing system design on people’s work practices. The first strategy aims to relay details about system behavior to users through feedback, instructions, or explanations. Meanwhile, the second attempts to design a system’s conceptual model – the concepts used by a system, the relationships between them, and their organisation – in accordance with people’s professional practices by drawing on user expertise and experience during the design process itself. Overall, this research contributes to the discussion on machine learning in risk-informed decision-making in two ways: first, it provides one explanation to the phenomenon of user resistance to ML-based PRMs in the context of social care; second, it outlines concrete and well-established strategies from the field of human-computer interaction that designers can employ to align risk constructs in machine learning and social care practice. Keywords: Social Care, Predictive Risk Models, Human-Centered Machine Learning

FAQs

What issue does the research investigate regarding ML-based predictive risk models?

It investigates why social care practitioners resist ML-enabled PRMs, focusing on misalignment between how risk is constructed in practice versus how the model operationalizes risk.

How do social care practitioners typically understand risk, according to the study?

Practitioners are taught to view risk as contextual, subjective, and contingent, where environments can amplify or reduce risk, individuals can determine what is or is not risk, and risk relates to proposed interventions.

What design strategies are proposed to align risk constructs?

The study outlines two human-computer interaction approaches: increase system transparency through feedback and explanations, and base system conceptual models on people’s professional practices using user expertise during design.

Aug 5, 2026

Upload by Miles