ChatGPT sits the DFPH exam - large language model performance and potential to support public health learning

Sep 18, 2026

Uploaded by William

Artificial intelligence-based large language models, such as ChatGPT, are evaluated for risks and opportunities in health-related assessment and learning. Applications of these models in public health professional examinations have not yet been studied. This paper evaluates ChatGPT performance on the Faculty of Public Health’s Diplomat exam (DFPH) using past-paper question parts and examiner scoring to assess insight, understanding, and learning relevance.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages7

Reading Time18 min

AccessFree to download

Background

Methods

Results

Conclusions

Keywords

[eprints@whiterose.ac.uk](eprints@whiterose.ac.uk)[ ](eprints@whiterose.ac.uk)[https://eprints.whiterose.ac.uk](https://eprints.whiterose.ac.uk) Universities of Leeds, Sheffield and York Deposited via The University of Sheffield. White Rose Research Online URL for this paper: [https://eprints.whiterose.ac.uk/id/eprint/207667/](https://eprints.whiterose.ac.uk/id/eprint/207667/) Version: Published Version Article: Davies, N.P. , Wilson, R. , Winder, M.S. et al. (2024) ChatGPT sits the DFPH exam: large language model performance and potential to support public health learning. BMC Medical Education, 24 (1) . 57. ISSN: 1472-6920 [https://doi.org/10.1186/s12909-024-05042-9](https://doi.org/10.1186/s12909-024-05042-9) Reuse This article is distributed under the terms of the Creative Commons Attribution (CC BY) licence. This licence allows you to distribute, remix, tweak, and build upon the work, even commercially, as long as you credit the authors for the original work. More information and the full terms of the licence here: [https://creativecommons.org/licenses/](https://creativecommons.org/licenses/) Takedown If you consider content in White Rose Research Online to be in breach of UK law, please notify us by emailing [eprints@whiterose.ac.uk](eprints@whiterose.ac.uk) including the URL of the record and the reason for the withdrawal request. Davies et al. BMC Medical Education (2024) 24:57 [https://doi.org/10.1186/s12909-024-05042-9](https://doi.org/10.1186/s12909-024-05042-9) BMC Medical Education RESEARCH Open Access ChatGPT sits the DFPH exam: large language model performance and potential to support public health learning Nathan P Davies 1*, Robert Wilson2, Madeleine S Winder 1, Simon J Tunster 1, Kathryn McVicar 1, ShivanThakrar3, Joe Williams4 and Allan Reid2 Abstract Background Artificial intelligence-based large language models, like ChatGPT, have been rapidly assessed for both risks and potential in health-related assessment and learning. However, their applications in public health professional exams have not yet been studied. We evaluated the performance of ChatGPT in part of the Faculty of Public Health’s Diplomat exam (DFPH) . Methods ChatGPT was provided with a bank of 119 publicly available DFPH question parts from past papers. Its performance was assessed by two active DFPH examiners. The degree of insight and level of understanding apparently displayed by ChatGPT was also assessed. Results ChatGPT passed 3 of 4 papers, surpassing the current pass rate. It performed best on questions relating to research methods. Its answers had a high floor. Examiners identified ChatGPT answers with 73 . 6% accuracy and human answers with 28 . 6% accuracy. ChatGPT provided a mean of 3.6 unique insights per question and appeared to demonstrate a required level of learning on 71.4% of occasions. Conclusions Large language models have rapidly increasing potential as a learning tool in public health education. However, their factual fallibility and the difficulty of distinguishing their responses from that of humans pose potential threats to teaching and learning. Keywords Public health, Examination, Artificial intelligence, Theory *Correspondence: Nathan P Davies [Nathan.davies@nottingham.ac.uk](Nathan.davies@nottingham.ac.uk) 1Nottingham Centre for Public Health and Epidemiology, University of Nottingham, Nottingham City Hospital, Hucknall Rd, Nottingham NG5 1PB, England 2NHS England, Seaton House, City Link, London Road, Nottingham NG2 4LA, England 3Leicester City Council, Public Health, 115 Charles Street, Leicester LE1 1FZ, England 4School of Health and Related Research (ScHARR), The University of Sheffield, 30 Regent St, Sheffield S1 4DA, England Background Several use cases for artificial intelligence (AI) have recently been set out for medicine and life sciences [1]. ChatGPT is an artificial intelligence (AI) chatbot that runs on OpenAI’s Generative Pre-

FAQs

What is the main purpose of the study?

To evaluate ChatGPT performance on the Faculty of Public Health’s Diplomat exam (DFPH) and assess whether it could support public health learning.

How was ChatGPT evaluated in the exam context?

ChatGPT was given 119 publicly available DFPH question parts from past papers, and performance was assessed by two active DFPH examiners.

What were the key results regarding passing and insight?

ChatGPT passed 3 of 4 papers, performed best on research methods questions, and provided a mean of 3.6 unique insights per question.

Sep 18, 2026

Uploaded by William