David vs. Goliath - comparing conventional machine learning and a large language model for assessing students’ concept use in a physics problem

Aug 4, 2026

Upload by Noah

Large language models offer new opportunities for educational research, especially for assessment, yet they face limitations such as hallucinating knowledge, weak explainability of predictions, and high resource expenditure. Conventional machine learning may be preferable when researchers need tighter control over modeling decisions. This study evaluates how conventional machine learning algorithms compare with a recently advanced large language model for assessing students’ concept use in a physics problem-solving task. Results show conventional machine learning outperforms the large language model, with analyses of classification behavior supporting the conclusion that both approaches can be complementary when labeled data is available.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages18

Reading Time45 min

AccessFree to download

Introduction

Opportunities of large language models in education

Challenges: hallucination, explainability, bias, and resource use

Study focus and comparison goal

TYPE Original Research PUBLISHED 18 September 2024 DOI 10. 3389/frai.2024.1408817 OPEN ACCESS EDITED BY Georgios Leontidis, University of Aberdeen, United Kingdom REVIEWED BY Michael Flor, Educational Testing Service, United States Antonio Sarasa-Cabezuelo, Complutense University of Madrid, Spain *CORRESPONDENCE Fabian Kieser kieser@ph-heidelberg.de RECEIVED 15 April 2024 ACCEPTED 30 August 2024 PUBLISHED 18 September 2024 CITATION Kieser F, Tschisgale P, Rauh S, Bai X, Maus H, Petersen S, Stede M, Neumann K and Wul􀀀 P (2024) David vs. Goliath: comparing conventional machine learning and a large language model for assessing students’concept use in a physics problem. Front. Artif. Intell. 7:1408817 . doi: 10.3389/frai.2024.1408817 COPYRIGHT © 2024 Kieser, Tschisgale, Rauh, Bai, Maus, Petersen, Stede, Neumann and Wul􀀀 . This isan open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms. David vs. Goliath: comparing conventional machine learning and a large language model for assessing students’ concept use in a physics problem Fabian Kieser1*, Paul Tschisgale2 , Sophia Rauh3 , Xiaoyu Bai3 , Holger Maus2 , Stefan Petersen2 , Manfred Stede3 , Knut Neumann2 and Peter Wul􀀀1 1 Physics and Physics Education Research, Heidelberg University of Education, Heidelberg, Germany, 2 Department of Physics Education, Leibniz Institute for Science and Mathematics Education, Kiel, Germany, 3Applied Computational Linguistics, University of Potsdam, Potsdam, Germany Large language models have been shown to excel in many di􀀀erent tasks across disciplines and research sites. They provide novel opportunities to enhance educational research and instruction in di􀀀erent ways such as assessment. However, these methods have also been shown to have fundamental limitations. These relate, among others, to hallucinating knowledge, explainability of model decisions, and resource expenditure. As such, more conventional machine learning algorithms might be more convenient for speciﬁc research problems because they allow researchers more control over their research. Yet, the circumstances in which either conventional machine learning or large language models are preferable choices are not well understood. This study seeks to answer the question to what extent either conventional machine learning algorithms or a recently advanced large language model performs better in assessing students’ concept use in a physics problem-solving task. We found that conventional machine learning algorithms in combination outperformed the large language model. Model decisions were then analyzed via closer examination of the models’ classiﬁcations. We conclude that in speciﬁc contexts, conventional machine learning can supplement large language models, especially when labeled data is available. KEYWORDS large language models, machine learning, natural language processing, problem solving, explainable AI 1 Introduction The introduction of ChatGPT, a conversational arti􀀂cial intelligence (AI)-based bot, to the public in November 2022 directed attention to large language models (LLMs) . As of 2023, ChatGPT is based on a LLM called Generative Pre-trained Transformer (GPT; versions 3.5, 4, 4V, or 4o) and has proven to perform surprisingly well on a wide range of di􀀓erent tasks in various disciplines—including medicine, law, economics, mathematics, chemistry, and physics (Hallal et al., 2023; West, 2023; Surameery and Shakor, 2023; Sinha et al., 2023) . A number of tasks in education (research) can be tackled using LLMs in general, or ChatGPT more speci􀀂cally

FAQs

What question does the study aim to answer?

It examines to what extent conventional machine learning algorithms or a recently advanced large language model performs better for assessing students’ concept use in a physics problem-solving task.

What were the main findings from the comparison?

Conventional machine learning algorithms, when combined, outperformed the large language model for the targeted assessment task.

Why might conventional machine learning be preferred in some educational research contexts?

Conventional methods can provide more control over research design, and they may supplement large language models particularly when labeled data is available, addressing concerns such as hallucination and lack of explainability.

Aug 4, 2026

Upload by Noah