Evaluation of ChatGPT-4o and DeepSeek as tools for orthodontic health literacy in public dental education

Q: What were the key aspects used to evaluate ChatGPT-4o and DeepSeek?

The study evaluated accuracy, consistency of responses, and response behavior, including response time (latency), on orthodontic health literacy tasks.

Q: What were the main findings when comparing the two models?

ChatGPT-4o showed slightly higher overall accuracy than DeepSeek and demonstrated significantly faster response times in both English and Chinese, while accuracy and consistency differences were not statistically significant across models.

Sep 30, 2026

Uploaded by Noah

Study compares and evaluates ChatGPT-4o and DeepSeek for orthodontic health literacy tasks, focusing on accuracy, consistency, and response behavior. A set of 50 Delphi-generated multiple-choice, text-based questions across five orthodontic domains was used. Each question was presented to both models three times daily over three days in English and Chinese versions. Accuracy, consistency, and latency were analyzed with SPSS and R. ChatGPT-4o achieved slightly higher accuracy with significantly faster response times, supporting clinical efficiency.

LanguageEnglish

CategoryHealthcare

FormatPDF

Pages10

Reading Time25 min

AccessFree to download

Evaluation design and objectives

Question development and bilingual testing

Metrics and statistical analysis

Results and comparative performance

Accuracy and consistency

Response time and practical implications

[www. nature.com/scientificreports](www. nature.com/scientificreports) OPEN Evaluation of ChatGPT-4o and DeepSeek as tools for orthodontic health literacy in public dental education ZhaoxiangWen1,2, Jiaxin Huang1,2, KeerYu1,2, Yaqi Li1,2, Zhenhui Wang1,2, Xiaozhu Liao1,2, Biao Li1,2, Zhendong Tao1,2 & Hong He1,2,3􀀍 This study aims to compare and evaluate the accuracy, consistency, and response behavior of ChatGPT- 4o and DeepSeek in orthodontic health literacy tasks. A total of 50 multiple-choice, text-based questions covering five orthodontic domains were developed using the Delphi method by a panel of five experienced orthodontic professionals. Each question was asked to both models three times daily (morning, noon, evening) over three days, in both English and Chinese versions. Accuracy, consistency, and response time were analyzed using SPSS and R software. The overall accuracy of ChatGPT-4o (90.4%) was slightly higher than that of DeepSeek (88.0%), though the difference was not statistically significant in either English (p = 1.000) or Chinese (p = 0.263). Across individual question groups, no significant differences in accuracy were found between models (p > 0.05). The time of day and testing day had no effect on model performance (p > 0.05 for all comparisons) . Both models demonstrated similar consistency in their responses (p = 1.000 for English; p = 0.412 for Chinese) . However, ChatGPT- 4o showed significantly faster response times than DeepSeek in both languages (p < 0.01) . Both ChatGPT-4o and DeepSeek demonstrate comparable accuracy and consistency in orthodontic health literacy tasks. However, ChatGPT-4o’s significantly faster response time makes it more suitable for clinical applications requiring efficient information delivery. Keywords Artificial intelligence (AI), ChatGPT, DeepSeek, Health literacy, Orthodontics, Delphi method Artificial intelligence (AI), a subfield of computer science, seeks to develop systems capable of performing cognitive functions traditionally associated with human intelligence, including reasoning, learning, and decision-making1. In healthcare, AI technologies have been increasingly employed to enhance diagnostic accuracy, optimize treatment protocols, reduce clinical workload, and support patient education2. One of the most transformative areas of AI application lies in education, where it offers the potential to deliver personalized learning experiences, simulate complex clinical scenarios, and reinforce domain-specific knowledge. Among recent advances in AI, transformer-based language models have become particularly influential due to their ability to process and generate human-like language. These models are underpinned by selfattention mechanisms that allow them to evaluate relationships between all elements in a sequence, thereby producing coherent, context-aware outputs3. The transformer architecture enables efficient parallel processing and contextual representation, making it foundational to large language models (LLMs) such as ChatGPT and DeepSeek. ChatGPT-4o, developed by OpenAI and released in 2024, incorporates multimodal capabilities and improved response accuracy, especially in non-English contexts4. DeepSeek, a more recent open-source alternative, emphasizes transparency and domain customization, offering enhanced adaptability for specialized educational tasks5. Orthodontic health literacy, defined as the ability to comprehend, evaluate, and apply information related to orthodontic conditions, appliances, and treatment protocols, is fundamental for the general public and patients alike6. It plays a critical role in fostering treatment compliance, obtaining truly informed consent, and sustaining appropriate self-care practices throughout orthodontic therapy7. Despite its established importance, effectively 1State Key Laboratory of Oral & Maxillofacial Reconstruction and Regeneration, Key Laboratory of Oral Biomedicine Ministry of Education, Hubei Key

FAQs

What were the key aspects used to evaluate ChatGPT-4o and DeepSeek?

The study evaluated accuracy, consistency of responses, and response behavior, including response time (latency), on orthodontic health literacy tasks.

How was the question set for orthodontic health literacy created?

A total of 50 multiple-choice, text-based questions covering five orthodontic domains were developed using the Delphi method by five experienced orthodontic professionals.

What were the main findings when comparing the two models?

ChatGPT-4o showed slightly higher overall accuracy than DeepSeek and demonstrated significantly faster response times in both English and Chinese, while accuracy and consistency differences were not statistically significant across models.

Sep 30, 2026

Uploaded by Noah