Machine Learning Facial Emotion Classifiers in Psychotherapy Research - A Proof-of-Concept Study

Q: What problem does this proof-of-concept study address in psychotherapy research?

Manual coding of facial expressions is time-consuming and makes it difficult to assess emotions across many video frames and sessions. Machine learning classifiers aim to automate reliable in-session emotion identification.

Q: How was the machine learning emotion classifier evaluated?

The classifier was applied to video data from 389 psychotherapy sessions of 23 patients with borderline personality pathology, and results were validated with human ratings using the Clients Emotional Arousal Scale.

Q: What relationships did the study find between facial emotions and psychotherapy outcomes?

Machine learning ratings agreed significantly with human ratings, with medium effects for positive emotions on treatment outcomes and evidence of continuous improvement. Patients who dropped out showed more neutral expressions and less social smiling, especially early in sessions.

Aug 2, 2026

Upload by Emma Wilson

New advances in machine learning enable high-resolution tracking of facial emotional expressions, including micro-expressions, offering a faster alternative to manual coding approaches such as the Facial Action Coding System. The study tested whether machine learning can reliably identify in-session emotional expression in a naturalistic psychotherapy setting and examined how these measures relate to psychotherapy outcomes. A machine learning emotion classifier was applied to video material from 389 sessions involving 23 patients with borderline personality pathology, with validation using human ratings on the Clients Emotional Arousal Scale and analyses of treatment outcomes.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

AccessFree to download

Abstract

Background

Purpose

Method

Results

Conclusions

Introduction

Limits of manual facial coding in psychotherapy

Facial Action Coding System and its role

Indirect markers of emotional arousal

Facial emotion recognition with real-time algorithms

Psychopathology Research Article Psychopathology DOI: 10.1159/000534811 Received: September 6, 2022 Accepted: October 16, 2023 Published online: November 27, 2023 Machine Learning Facial Emotion Classiﬁers in Psychotherapy Research: A Proof-of-Concept Study Martin Steppana, b Ronan Zimmermann a, b Lukas Fürer b Matthew Southward c Julian Koenig d, e Michael Kaessd,f Johann Roland Kleinbubg Volker Roth h Klaus Schmeck b aFaculty of Psychology, University of Basel, Basel, Switzerland; bPsychiatric University Hospital, Basel, Switzerland; cDepartment of Psychology, University of Kentucky, Lexington, KY, USA; dUniversity Hospital of Child and Adolescent Psychiatry and Psychotherapy, University of Bern, Bern, Switzerland; eSection for Experimental Child and Adolescent Psychiatry, Department of Child and Adolescent Psychiatry, Centre for Psychosocial Medicine, University of Heidelberg, Heidelberg, Germany; fSection for Translational Psychobiology in Child and Adolescent Psychiatry, Department of Child and Adolescent Psychiatry, Centre for Psychosocial Medicine, University of Heidelberg, Heidelberg, Germany; gDepartment of Philosophy, Sociology, Education and Applied Psychology, University of Padova, Padova, Italy; hDepartment of Mathematics and Informatics, University of Basel, Basel, Switzerland Keywords Adolescents · Borderline personality disorder · Emotions · Facial expressions classiﬁers · Psychotherapy Abstract Background: New advances in the ﬁeld of machine learning make it possible to track facial emotional expression with high resolution, including micro-expressions. These advances have promising applications for psychotherapy research, since manual coding (e.g., the Facial Action Coding System), is timeconsuming. Purpose: We tested whether this technology can reliably identify in-session emotional expression in a naturalistic treatment setting, and how these measures relate to the outcome of psychotherapy. Method: We applied a machine learning emotion classiﬁer to video material from 389 psychotherapy sessions of 23 patients with borderline personality pathology. We validated the ﬁndings with human ratings according to the Clients Emotional Arousal Scale (CEAS) and explored associations with treatment outcomes. Results: Overall, machine learning ratings showed signiﬁcant agreement with human ratings. Machine learning emotion classiﬁers, particularly the display of positive emotions (smiling and happiness), showed medium effect size on median-split treatment outcome (d = 0.3) as well as continuous improvement (r = 0.49, p < 0.05). Patients who dropped out form psychotherapy, showed signiﬁcantly more neutral expressions, and generally less social smiling, particularly at the beginning of psychotherapeutic sessions. Conclusions: Machine learning classiﬁers are a highly promising resource for research in psychotherapy. The results highlight differential associations of displayed positive and negative feelings with treatment outcomes. Machine learning emotion recognition may be used for the early identiﬁcation of drop-out risks and clinically relevant interactions in psychotherapy. © 2023 The Author(s) . Published by S. Karger AG, Basel [karger@karger.com](karger@karger.com)[ ](karger@karger.com)[www.karger.com/psp](www.karger.com/psp) © 2023 The Author(s) . Published by S. Karger AG, Basel Correspondence to: Martin Steppan, [mhsteppan](mhsteppan@ gmail.com)[@](mhsteppan@ gmail.com)[ gmail.com](mhsteppan@ gmail.com) This article is licensed under the Creative Commons AttributionNonCommercial4.0International License(CC BY-NC)([http://www](http://www). [karger.com/Services/OpenAccessLicense](karger.com/Services/OpenAccessLicense)). Usage and distribution for commercial purposes requires written permission. Downloaded from [http://karger.com/psp/article-pdf/doi/10.1159/000534811/4051895/000534811.pdf by Universit](http://karger.com/psp/article-pdf/doi/10.1159/000534811/4051895/00053481

FAQs

What problem does this proof-of-concept study address in psychotherapy research?

Manual coding of facial expressions is time-consuming and makes it difficult to assess emotions across many video frames and sessions. Machine learning classifiers aim to automate reliable in-session emotion identification.

How was the machine learning emotion classifier evaluated?

The classifier was applied to video data from 389 psychotherapy sessions of 23 patients with borderline personality pathology, and results were validated with human ratings using the Clients Emotional Arousal Scale.

What relationships did the study find between facial emotions and psychotherapy outcomes?

Machine learning ratings agreed significantly with human ratings, with medium effects for positive emotions on treatment outcomes and evidence of continuous improvement. Patients who dropped out showed more neutral expressions and less social smiling, especially early in sessions.

Aug 2, 2026

Upload by Emma Wilson