BERT is to NLP what AlexNet is to CV - Can Pre-Trained Language Models Identify Analogies?

Sep 14, 2026

Analogies underpin human commonsense reasoning and shape how knowledge is structured and language is understood. Yet identifying analogical proportions has received limited focus in the era of language models. This paper evaluates transformer-based pre-trained language models on an unsupervised analogy task using both educational benchmarks and widely used datasets. Results show partial analogy identification, with weak performance on abstract and complex relations and strong sensitivity to architecture and hyperparameters.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages16

Reading Time40 min

AccessFree to download

Abstract

Introduction

Related work and benchmarks

Prompting and scoring in zero-shot settings

Experimental results and sensitivity analysis

BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies? Asahi Ushio, Luis Espinosa-Anke, Steven Schockaert, Jose Camacho-Collados Cardiff NLP, School of Computer Science and Informatics Cardiff University, United Kingdom fUshioA,Espinosa-AnkeL,SchockaertS1,[CamachoColladosJ](CamachoColladosJg@cardiff.ac.uk)[g](CamachoColladosJg@cardiff.ac.uk)[@cardiff.ac.uk](CamachoColladosJg@cardiff.ac.uk) Abstract Analogies play a central role in human commonsense reasoning. The ability to recognize analogies such as “eye is to seeing what ear is to hearing”, sometimes referred to as analogical proportions, shape how we structure knowledge and understand language. Surprisingly, however, the task of identifying such analogies has not yet received much attention in the language model era. In this paper, we analyze the capabilities of transformer-based language models on this unsupervised task, using benchmarks obtained from educational settings, as well as more commonly used datasets. We ﬁnd that off-the-shelf language models can identify analogies to a certain extent, but struggle with abstract and complex relations, and results are highly sensitive to model architecture and hyperparameters. Overall the best results were obtained with GPT-2 and RoBERTa, while conﬁgurations using BERT were not able to outperform word embedding models. Our results raise important questions for future work about how, and to what extent, pre-trained language models capture knowledge about abstract semantic relations.1 1 Introduction One of the most widely discussed properties of word embeddings has been their surprising ability to model certain types of relational similarities in terms of word vector differences (Mikolov While the title is probably self-explanatory, this is a small note explaining it. BERT is to NLP what AlexNet is to CV is making an analogy on what the BERT and AlexNet models represented for Natural Language Processing (NLP) and Computer Vision (CV), respectively. They both brought a paradigm shift in how research was undertaken in their corresponding disciplines and this is what the analogy refers to. 1 Source code and data to reproduce our experimental results are available in the following repository: [https://github.com/asahi417/](https://github.com/asahi417/)[ ](https://github.com/asahi417/)analogy-language-model | Query: | | word:language | | --- | --- | --- | | Candidates: | (1)
(2)
(3)
(4)
(5) | paint:portrait
poetry:rhythm
note:music
tale:story
week:year | Table 1: An example analogy task from the SAT dataset. The third candidate is the answer to the query. et al., 2013a ; Vylomova et al., 2016 ; Allen and Hospedales, 2019 ; Ethayarajh et al., 2019) . The underlying assumption is that when “a is to b what c is to d” the word vector differences b 􀀀 a and d 􀀀 c are expected to be similar, where we write x for the embedding of a word x. While this assumption holds for some types of syntactic relations, for semantic relations this holds to a much more limited degree than was suggested in early work (Linzen, 2016 ; Schluter, 2018) . Moreover, the most commonly used benchmarks have focused on speciﬁc and well-deﬁned semantic relations such as“capital of”, rather than the more abstract notion of relational similarity that is often needed for solving the kind of psychometric analogy problems that can be found in IQ tests and educational settings. An example of such a problem is shown in Table 1. Given the central role of analogy in human cognition, it is nonetheless important to understand the extent to which NLP models are able to solve these more abstract analogy problems. Besides its value as an intrinsic benchmark for lexical semantics, the ability to recognize analogies is indeed important in the contexts of human creativity (Holyoak et al., 1996), innovation (Hope et al., 2017), computational creativity (Goel, 2019) and education (Pardos and Nam, 2020) .

FAQs

What does the paper investigate about pre-trained language models?

It analyzes how well transformer-based pre-trained language models can identify word analogies in an unsupervised, zero-shot setting.

What datasets and benchmarks are used for the analogy evaluation?

The study uses benchmarks obtained from educational settings and also more commonly used public analogy datasets, including an example from the SAT dataset.

How do different model architectures affect analogy identification?

The results are highly sensitive to model architecture and hyperparameters, with the best performance reported for GPT-2 and RoBERTa, while BERT configurations do not outperform word embedding models.

Sep 14, 2026

Uploaded by eBook King