Domain Fine-Tuning FinBERT on Finnish Histopathological Reports - Train-Time Signals and Downstream Correlations

Sep 4, 2026

Domain fine-tuning of transformer models on unlabeled data is a common strategy when labeled examples are scarce. This paper investigates fine-tuning the Finnish BERT model on Finnish medical text and reports observations from the training process. It also evaluates whether downstream benefits of domain-specific pre-training can be predicted by analyzing embedding-geometry changes induced by fine-tuning. The focus addresses healthcare-AI scenarios where dataset and label acquisition is delayed.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages20

Reading Time50 min

AccessFree to download

Abstract

Introduction

Domain adaptation in low-resource medical languages

arXiv :2604 . 14815v1 [ cs .CL] 16 Apr 2026 Domain Fine-Tuning FinBERT on Finnish Histopathological Reports: Train-Time Signals and Downstream Correlations Rami Luisto 1,2,3,*, Liisa Pet¨ainen 1 , Tommi Gr¨onholm 1 , Jan B¨ohm4 , Maarit Ahtiainen4 , Tomi Lilja4 , Ilkka P¨ol¨onen 1 , Sami ¨Ayr¨am¨o 1, 5 1 Faculty of Information Technology, University of Jyv¨askyl¨a, Jyv¨askyl¨a, Finland. 2 Digital Workforce Services, Helsinki, Finland. 3 Heart and Lung Center, Helsinki University Hospital, Helsinki, Finland. 4 Central Finland Biobank, Jyv¨askyl¨a, Finland. 5Wellbeing Services County of Central Finland, Jyv¨askyl¨a, Finland. *[Corresponding author. E-mail:](Corresponding author. E-mail: rami.m.luisto@jyu.fi)[ rami.m.luisto@jyu.fi](Corresponding author. E-mail: rami.m.luisto@jyu.fi); Abstract In NLP classification tasks where little labeled data exists, domain fine-tuning of transformer models on unlabeled data is an established approach. In this paper we have two aims. (1) We describe our observations from fine-tuning the Finnish BERT model on Finnish medical text data. (2) We report on our attempts to predict the benefit of domain-specific pre-training of Finnish BERT from observing the geometry of embedding changes due to domain fine-tuning. Our driving motivation is the common† situation in healthcare AI where we might experience long delays in acquiring datasets, especially with respect to labels. 1 Introduction Ever since ULMFiT [1] the idea of separating the training of a language model to different stages of specialization (typically pre-training, domain fine-tuning and task-specific †First author’s professional anecdote. 1 training) has been a core tenet in any modern NLP. This approach is particularly suitable for models based on the transformer architecture, see e.g. [2–5] . Recent work has continued to study the specific benefits of domain fine-tuning, e.g. in the catchily titled Don’t stop pretraining: Adapt language models to domains and tasks [6] study how Domain-adaptive Pre-Training (DPT) compares to Task-Adaptive Pre-Training (TAPT) 1 . Especially in the realm of Large Language Models (LLMs) it can be crucial to predict from smaller exploratory training runs what would be the optimal configurations of model weights, train data amount and compute budget. The literature records various power laws that can be used to extrapolate model training behaviour. Such power laws exist both for comparing models within a data domain, see e.g. [7, 8] and the references within, or to compare behaviour across domains, see [9] . Though such work is typically focused on LLMs instead of Small Language Models (SLMs) like BERT, the underlying transformer architecture is the same, and many of the effects transfer. MLM Training Dynamics with Epoch Markers (Circles mark epoch boundaries) Loss 3.0 2.5 2.0 1.5 1.0 0 50 100 150 200 250 300 Tokens (millions) Fig. 1: Observations on the train-time loss of FinBERT on various datasets it is more or less “familiar” with. Histopathological dataset shows massive changes in loss. In the current work our aim is twofold. First we study how we might observe the effect of DFT on a dataset where we do not have training labels available. Second we try to replicate and observe the benefits that DFT have towards a classification task, with an eye on DFT training time observations that might correlate with classification task performance. We emphasize that our aim is to observe the phenomenon and make qualitative assessments. In particular, we do not aim for optimal classification algorithms as we wish to observe performance differentials rather than absolute levels. 1 In this paper we do not make a differentiation at this resolution, and just talk of Domain Fine-Tuning, DFT. 2 Our driving motivation here is NLP work in the healthcare sector, in particular with regards to Finnish medical text data. Finnish is a minority language with complex g

FAQs

Why is domain fine-tuning useful in healthcare NLP with limited labels?

When labeled data is delayed or difficult to obtain, domain fine-tuning on unlabeled text can be performed first. The paper motivates this approach by typical healthcare-AI dataset and labeling constraints.

What two aims does the paper pursue?

The study aims to (1) describe observations from fine-tuning Finnish BERT on Finnish medical text and (2) attempt to predict downstream benefits from domain-specific pre-training by examining embedding changes caused by fine-tuning.

How are train-time observations connected to downstream classification performance?

The paper uses training-time signals (including changes reflected in embeddings and loss behavior) to look for correlations with classification task performance. It reports tentative correlations and notes that replication is needed.

Sep 4, 2026

Upload by Evangeline