Assessing AI’s problem solving in physics - Analyzing reasoning, false positives and negatives through the force concept inventory

Oct 15, 2026

Uploaded by OmBimo

GPT-4 is evaluated on the force concept inventory (FCI) to assess accuracy, reasoning patterns, and error types including false positives and false negatives. Across multiple sessions, results show strong performance on several FCI items, especially those tied to Newton’s third law, with perfect scores on many questions. Performance drops on items requiring figure interpretation and spatial reasoning, producing more false negatives where reasoning was correct but answers were wrong. Conceptual issues also appear, such as errors about friction and reliance on the outdated impetus theory, underscoring the need to refine AI tools and address common physics misconceptions to improve educational outcomes.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages16

Reading Time40 min

AccessFree to download

Introduction

AI in Education

AI in Physics Education

OPEN ACCESS Research Article Contemporary Educational Technology, 2024, 16(4), ep538 ISSN: 1309-517X (Online) Assessing AI’s problem solving in physics: Analyzing reasoning, false positives and negatives through the force concept inventory Salima Aldazharova 1 0009-0004-6986-1885 Gulnara Issayeva 1 0000-0002-4860-3797 Samat Maxutov 2 0000-0002-4453-5512 Nuri Balta 2* 0000-0002-6364-5346 1 Abai Kazakh National Pedagogical University, Almaty, KAZAKHSTAN 2 SDU University, Almaty, KAZAKHSTAN * Corresponding author: [baltanuri@gmail.com](baltanuri@gmail.com) Citation: Aldazharova, S., Issayeva, G., Maxutov, S., & Balta, N. (2024). Assessing AI’s problem solving in physics: Analyzing reasoning, false positives and negatives through the force concept inventory. Contemporary Educational Technology, 16(4), ep538. [https://doi.org/10.30935/cedtech/15592](https://doi.org/10.30935/cedtech/15592) ARTICLE INFO ABSTRACT | Received: 2 Aug 2024
Accepted: 23 Sep 2024
INTRODUCTION | This study investigates the performance of GPT-4, an advanced AI model developed by OpenAI, on the force concept inventory (FCI) to evaluate its accuracy, reasoning patterns, and the occurrence of false positives and false negatives. GPT-4 was tasked with answering the FCI questions across multiple sessions. Key findings include GPT-4’s proficiency in several FCI items, particularly those related to Newton’s third law, achieving perfect scores on many items. However, it struggled significantly with questions involving the interpretation of figures and spatial reasoning, resulting in a higher occurrence of false negatives where the reasoning was correct, but the answers were incorrect. Additionally, GPT-4 displayed several conceptual errors, such as misunderstanding the effect of friction and retaining the outdated impetus theory of motion. The study’s findings emphasize the importance of refining AI-driven tools to make them more effective in educational settings. Addressing both AI limitations and common misconceptions in physics can lead to improved educational outcomes.
Keywords: AI assisted learning, force concept inventory, GPT-4, physics education | | --- | --- | The integration of artificial intelligence (AI) into educational settings has gained significant attention in recent years (Yilmaz et al., 2023) . Advanced AI models, such as GPT-4, have shown remarkable capabilities in generating human-like responses and reasoning. However, understanding how effectively these models can replicate human cognitive processes, particularly in complex domains like physics, is crucial for their application in enhancing educational outcomes. This study addresses the educational challenge of accurately assessing students’ conceptual understanding in physics. By analyzing AI-generated responses, we aim to identify and investigate misconceptions, thereby informing targeted instructional strategies and improving the integration of AI-driven educational tools into curricula (Anderson et al., 2019) . Furthermore, examining the reasoning patterns of AI can point out the potential of AI to promote higher-order learning and critical thinking skills. The study focuses on analyzing the accuracy of responses, identifying false positives and false negatives, and understanding the reasoning patterns of AI. Similar previous studies (Kortemeyer, 2023; West, 2023) employed an earlier version of GPT, which could not read figures. Therefore, they generated text versions of the questions that included figures and then assessed the performance of GPT. However, this study used GPT-4, which can read and analyze figures directly. The GPT-4’s responses on the force concept inventory (FCI) is not only relevant to AI developers but also holds significant educational value for science educators. By examining responses, we can learn common misconceptions and reasoning patterns that both AI exhibit. This information is crucial for educators to develop

FAQs

What is the main goal of the study evaluating GPT-4 on the FCI?

To assess GPT-4’s accuracy, reasoning patterns, and the occurrence of false positives and false negatives when answering force concept inventory questions.

Which FCI question types did GPT-4 perform best on?

GPT-4 performed particularly well on items related to Newton’s third law, achieving perfect scores on many of those questions.

Why did GPT-4 produce more false negatives in some cases?

It struggled with questions requiring interpretation of figures and spatial reasoning, leading to situations where the reasoning was correct but the final answers were incorrect.

Oct 15, 2026

Uploaded by OmBimo