Will Artificial Intelligence Nurse Practitioners Become True? - Performance Evaluation of ChatGPT in the American Association of Nurse Practitioners Exams

Sep 21, 2026

Uploaded by Eliana

Nurse practitioners underpin achievement of the UN Sustainable Development Goals and Universal Health Coverage, particularly in managing chronic noncommunicable diseases. The study evaluates whether ChatGPT can function effectively as a nurse practitioner by testing ChatGPT-3.5 and GPT-4 on American Association of Nurse Practitioners (AANP) practice examination questions. Accuracy, answer concordance across five discipline exam sets (535 questions), and retest consistency are analyzed. Results show 80.6% accuracy for ChatGPT-3.5 and 90.7% for ChatGPT-4, with strong set consistency and retained outputs in retests, supporting potential decision support and efficiency gains.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages15

Reading Time38 min

AccessFree to download

Abstract

Introduction

Citation Lang Peng, Yi Wu, Jiayi Sun, Yihong Xing, Mingqin Li and Mingzi Li (2025), Will Artiﬁcial Intelligence Nurse Practitioners Become True? Performance Evaluation of ChatGPTin the American Association of Nurse Practitioners Exams. AI, Computer Science and Robotics Technology 4(1), 1–15. DOI [https://doi.org/10.5772/acrt.20240045](https://doi.org/10.5772/acrt.20240045) Copyright © The Author(s) 2025 . This is an Open Access article distributed under the terms of the Creative Commons Attribution License ([https://creativecommons](https://creativecommons). org/licenses/by/4.0/), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited. Received: 6 August 2024 Accepted: 10 February 2025 Published: 28 February 2025 R E S E A R C H PA P E R WillArtiﬁcial Intelligence Nurse Practitioners Become True? Performance Evaluation of ChatGPTin the American Association of Nurse Practitioners Exams Lang Peng1 , Yi Wu1 , Jiayi Sun2 , Yihong Xing3 , Mingqin Li4 and Mingzi Li1 , * 1 School of Nursing, Peking University, Beijing, China 2 Bellevue High School, Bellevue, USA 3 School of Pharmaceutical Sciences, Peking University, Beijing, China 4 Microsoft Corporation, Bellevue, USA *Corresponding author. E-mail: [limingzi@bjmu.edu.cn](limingzi@bjmu.edu.cn) Abstract Nurse practitioners play a vital role in contributing to the UN’s Sustainable Development Goals, and Universal Health Coverage, especially the management of chronic noncommunicable diseases. Artiﬁcial intelligence tools such as ChatGPT are becoming promising resources for healthcare professionals. This study aimed to explore the capability of ChatGPT as a nurse practitioner by validating the performance of ChatGPT-3.5 and GPT-4 in the American Association of Nurse Practitioners (AANP) practice examinations. Questions from exams for ﬁve nurse practitioner disciplines were used to evaluate the accuracy and consistency of the responses in two phases. In the ﬁrst phase, the accuracy rates and concordance of answers between the two versions with the ﬁve exam sets, totaling 535 questions were analyzed. In the second phase, the consistency of ChatGPT-4 performance in six retests, each involving ﬁve random questions from each set. ChatGPT-3.5 achieved an overall accuracy rate of 80.6%, while ChatGPT-4 achieved 90.7% . ChatGPT-3.5 and ChatGPT-4 showed strong consistency within all sets, while 1/15 ChatGPT-4 performed better than ChatGPT-3.5. In the retests, ChatGPT-4 provided exactly the same answers as generated initially, including the incorrect ones. In conclusion, ChatGPT demonstrated excellent performance in AANP practice exams, with high levels of accuracy and consistency. This suggests that ChatGPT may support nurse practitioners in making clinical decisions and improving efficiency. Further studies could explore ways to integrate artiﬁcial intelligence tools with nurse practitioner practice to enhance the advanced practice nursing workforce. Keywords: artiﬁcial intelligence, ChatGPT, large language model, nurse practitioners, performance 1. Introduction The escalating global burden of diseases resulting from the increase in population aging, coupled with the scarcity and unequal distribution of medical resources, presents a formidable challenge. In light of this crisis, nurses, especially nurse practitioners (NPs), have assumed an indispensable and irreplaceable role in healthcare services [1] . NPs, a type of advanced practice nurse, are required to earn a master’s or doctoral degree in advanced clinical practice, pass a national certiﬁcation exam in the specialty of choice, and combine clinical skills assimilated from both nursing and medicine to assess, diagnose, and manage patients. Their practice spans primary healthcare, acute care, and the long-term management of chronic illnesses [2] . NPs have played a vital role within the healthcare system, offering primary

FAQs

How was ChatGPT evaluated for nurse practitioner capability in the AANP exams?

The study validated ChatGPT-3.5 and GPT-4 using questions from AANP practice examinations across five nurse practitioner disciplines, analyzing accuracy and answer concordance in two phases.

What were the accuracy results for ChatGPT-3.5 and ChatGPT-4?

ChatGPT-3.5 achieved an overall accuracy rate of 80.6%, while ChatGPT-4 achieved 90.7%.

Did ChatGPT show consistent answers during retests?

Yes. In retests, ChatGPT-4 produced exactly the same answers as in the initial generation, including the incorrect ones, indicating high consistency.

Sep 21, 2026

Uploaded by Eliana