Temporal Leakage in Financial News NLP - A Multi-Architecture Audit with a Regime-Specific M&A Signal

Q: What is temporal leakage in financial-news NLP benchmarks?

Temporal leakage refers to inflated performance when training and test splits are not properly separated in time, letting models exploit regime-correlated vocabulary, entities, and label structure.

Q: How do random and chronological splits affect model performance?

Random splits significantly increase MCC relative to chronological splits, and the gap grows with model capacity and feature richness.

Q: Which event category retains a signal under near-temporal chronological evaluation?

Mergers and acquisitions (M&A) is the only audited category with a positive locked-test signal under near-temporal chronological evaluation, and the headline-level effect does not transfer to a U.S. corpus.

Sep 15, 2026

Uploaded by Ivy

Financial-news direction prediction is a widely used NLP benchmark, yet reported improvements vary sharply with train–test protocol, especially whether splits are chronological or random. This study audits temporal leakage across a 49,799-article corpus using 16 feature–model pipelines (TF-IDF, MiniLM, FinBERT, fine-tuned RoBERTa/DeBERTa) and LLM probes (zero/few-shot and LoRA). Random splits inflate MCC substantially, while only M&A shows a near-temporal locked-test signal under strict evaluation, with the effect not transferring to a separate U.S. dataset.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages34

Reading Time1 h 26 min

AccessFree to download

Abstract

Introduction

Motivation: Financial News Prediction as an NLP Benchmark

Central Research Questions

Main Findings

Temporal Leakage in Financial News NLP: A Multi-Architecture Audit with a Regime-Specific M&A Signal Chenhao Xue1,2 , Raslen Guesmi1 , Siwei Feng1 , Yucheng Gong1,3 Jacob Xavier Sundram 1,3 , Jordan Pang 1,3 , Lan Wang4 , Julian Kaljuvee* 1 1Predictive Labs Ltd 2University of Oxford 3Imperial College London 4Independent Researcher arXiv : 2608 . 17223v2 [cs .CL] 6 Sep 2026 Abstract Financial-news direction prediction has become a popular NLP benchmark, yet reported gains depend critically on whether the train–test split is chronological or random, i.e., on temporal leakage. We audit this dependence on a 49,799-article corpus across 16 feature–model combinations spanning TF-IDF, MiniLM, FinBERT, and fine-tuned RoBERTa-large / DeBERTa-v3-large, plus separate zero/few-shot and LoRA probes of Llama-3 and Qwen2.5 LLMs: random splits inflate MCC by 1.1 × to 6.5 ×, tracking model capacity and feature richness, and end-to-end FinBERT fine-tuning re-amplifies rather than closes the gap (size-matched ratio 1.75 ×) . Conditioning on event type, mergersand acquisitions (M&A) is the only audited category with a positive locked-test signal under near-temporal chronological evaluation (TF-IDF MCC = 0 . 138 train-only, 0.068 under train∪val refit; 10 ,000-permutation p < 10 −3); the signal does not transfer to FNSPID’s 2009–2020 U.S. corpus, localising the headline to our 2024–2025 European-tilted M&A semantics rather than a universal predictor. Three independent role labellers converge on acquirer-tagged articles as the signal locus, a power-limited qualitative convergence rather than a hypothesis-tested asymmetry. Chronological splitting plays for financial NLP the role characteristics-purging plays for asset pricing: it strips the predictable, stale component of news and leaves a residual that is small, event-localized, and lexically shallow. We advocate leakage audits as a required disclosure for financial-NLP benchmarks. We open-source our data and code at [https:](https:)//[github.com/ChenHX111/Temporal_](github.com/ChenHX111/Temporal_)[Leakage_in_Financial_News_NLP](Leakage_in_Financial_News_NLP.)[.](Leakage_in_Financial_News_NLP.) 1 Introduction 1.1 Motivation: Financial News Prediction asan NLP Benchmark Financial-news direction prediction is one of the most contested empirical benchmarks at the NLP– finance boundary: text-understanding capability is tested against an economically grounded outcome signal. A decade of work, spanning structured event embeddings (Ding et al., 2014, 2015), recurrent models over tweets and prices (Xu and Cohen, 2018 ; Hu et al., 2018), domain-adapted LMs (Araci, 2019 ; Yang et al., 2020b), and LLMbased approaches (Lopez-Lira and Tang, 2023 ; Liet al., 2024 ; Wang et al., 2024 ; Xie et al., 2024), has produced an optimistic narrative: per-paper directional-accuracy figures in the high-50s to mid- 60s (e.g., 58.2% binary accuracy on StockNet tweets (Xu and Cohen, 2018); 64.2% S&P directional accuracy Ding et al., 2015) . The premise is worth taking seriously; the evidence is not. Financial NLP inherits a structural vulnerability: news articles, firm identities, market regimes, and return labels are jointly temporally autocorrelated, and modern language models exploit that structure. A model trained on a random half of a year’s financial news absorbs the vocabulary, entity graph, and return correlations of the same market cycle as its test set: regime memorisation that differs qualitatively from within-sample overfitting. A random split holding out individual articles from the early- 2024 AI rally, the late-2024 rate-cut pivot, or the 2025 European M&A wave in our own corpus lets a classifier exploit regime-context cues unavailable in a prospective chronological test. This concern is well-documented in financial machine learning (López de Prado, 2018 ; Bailey et al., 2014 ; Kapoor and Narayanan, 2023 ; Hewamalage et al., 2023 ; Bergmeir and Benítez, 2012) and, separately, in NLP e

FAQs

What is temporal leakage in financial-news NLP benchmarks?

Temporal leakage refers to inflated performance when training and test splits are not properly separated in time, letting models exploit regime-correlated vocabulary, entities, and label structure.

How do random and chronological splits affect model performance?

Random splits significantly increase MCC relative to chronological splits, and the gap grows with model capacity and feature richness.

Which event category retains a signal under near-temporal chronological evaluation?

Mergers and acquisitions (M&A) is the only audited category with a positive locked-test signal under near-temporal chronological evaluation, and the headline-level effect does not transfer to a U.S. corpus.

Sep 15, 2026

Uploaded by Ivy