Shortcut Learning in Legal Judgment Prediction Empirical Evidence from the UK Employment Tribunal

Jul 16, 2026

Upload by Noah

Current Legal Judgment Prediction (LJP) is limited by reliance on post-hoc judicial materials, which can shift systems toward retrospective classification rather than true forecasting. This paper empirically examines shortcut learning in a claim-level setting using 33,158 UK Employment Tribunal claims and compares TF-IDF interpretable models with black-box LLMs. Performance increases when test items contain outcome-revealing leakage cues, and masking these features yields only negligible Macro-F1 loss.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages25

Reading Time1 h 3 min

AccessFree to download

Introduction

Legal Judgment Prediction as a temporally constrained task

Data limitations and post-hoc judicial texts

Shortcut learning and validity concerns

1 Shortcut Learning in Legal Judgment Prediction: Empirical Evidence from the UK Employment Tribunal Joe Watson1,2*, Joana Ribeiro de Faria1, Marcus Tomalin3, Måns Magnusson4, Huiyuan Xie5, Hao Tian Yeung6, Christine Carter1, Jonathan Rutherford1, Felix Steffek1 1. Faculty of Law, University of Cambridge, Cambridge, United Kingdom 2. The Psychometrics Centre, Cambridge Judge Business School, University of Cambridge, Cambridge, United Kingdom 3. Faculty of English, University of Cambridge, Cambridge, United Kingdom 4. Department of Statistics, Uppsala University, Uppsala, Sweden 5. Department of Computer Science and Technology, Tsinghua University, Beijing, China 6. Department of Engineering, University of Cambridge, Cambridge, United Kingdom * Corresponding author: Joe Watson, [jmw239@cam.ac.uk](jmw239@cam.ac.uk) 2 Abstract Current Legal Judgment Prediction (LJP) is constrained by its reliance on post-hoc judicial materials, increasing the likelihood that models perform retrospective classification rather than true forecasting. This paper empirically investigates shortcut learning in this context by studying claim-level outcome prediction in UK Employment Tribunal (UKET) decisions. Using a corpus of 33,158 individual claims, we predict outcomes from claim texts and LLMextracted case summaries, evaluating models ranging from interpretable TF-IDF-based classifiers to black-box LLMs. While headline predictive performance figures appear strong, we demonstrate that such performance in LJP systems trained on post-hoc judicial text can be driven by the retrospective nature of the source material. Stratifying the test data by human judgments of leakage reveals that performance increases where outcome-revealing cues are embedded in the narrative. Moreover, a model trained on just the 4% of features identified as leakage achieves high performance, outperforming human experts. These findings substantiate concerns that LJP performance may be exaggerated by linguistic artefacts. Yet this vulnerability is not fatal to the research agenda. Instead, post-hoc judgments might be treated as potentially contaminated texts, requiring active auditing. Retraining models after masking leakage features results in only a negligible reduction in Macro-F1 . Hence, while models will opportunistically exploit shortcuts when available, they remain capable of extracting useful predictive signals when these artefacts are removed. Keywords Legal Judgment Prediction; shortcut learning; information leakage; UK Employment Tribunal 3 Main Text 1 Introduction The practical promise of Legal Judgment Prediction (LJP) rests on the possibility of forecasting court outcomes from information available before a decision is made, estimating a dispute’s likely resolution from ex ante case materials. In its strongest form, therefore, LJPis a temporally constrained task: a model should not rely on information that would only become available after the relevant legal decision has been reached. This requirement matters because LJP is often motivated by its potential to help litigants, lawyers, or policymakers assess likely outcomes before proceedings. For that promise to hold, predictions must be based on information available at that point, rather than on facts, findings, or reasoning later used to justify the result. This ideal is difficult to satisfy in practice because most LJP work relies on post-hoc judicial texts: decisions published after the court outcome has been reached. This reliance largely arises from data constraints. Pre-decisional materials, such as claim forms, response forms, comprehensive case files, and other pre-trial documents, are rarely available to researchers (Medvedeva et al. 2023), including in the context of the UK Employment Tribunal (UKET) . As a result, many LJP studies risk performing outcome identification or outcome classification rather than true forecasting (Medvedeva et al. 2023) . The problem is tha

FAQs

What problem does the paper identify in current Legal Judgment Prediction systems?

It highlights that reliance on post-hoc judicial materials can cause models to perform retrospective outcome identification or classification instead of forecasting from information available before a decision.

How does the study test for shortcut learning in the UK Employment Tribunal context?

It uses a corpus of 33,158 individual claims to predict outcomes from claim texts and LLM-extracted case summaries, and it evaluates whether performance depends on leakage cues revealed by human judgments.

What happens to model performance when leakage features are masked or removed?

Retraining after masking leakage features results in only a negligible reduction in Macro-F1, indicating models can still extract useful predictive signals even when artefacts are removed.

Jul 16, 2026

Upload by Noah