Performance of ChatGPT on USMLE: Unlocking the Potential of Large Language Models for AI-Assisted Medical Education

by   Prabin Sharma, et al.

Artificial intelligence is gaining traction in more ways than ever before. The popularity of language models and AI-based businesses has soared since ChatGPT was made available to the general public via OpenAI. It is becoming increasingly common for people to use ChatGPT both professionally and personally. Considering the widespread use of ChatGPT and the reliance people place on it, this study determined how reliable ChatGPT can be for answering complex medical and clinical questions. Harvard University gross anatomy along with the United States Medical Licensing Examination (USMLE) questionnaire were used to accomplish the objective. The paper evaluated the obtained results using a 2-way ANOVA and posthoc analysis. Both showed systematic covariation between format and prompt. Furthermore, the physician adjudicators independently rated the outcome's accuracy, concordance, and insight. As a result of the analysis, ChatGPT-generated answers were found to be more context-oriented and represented a better model for deductive reasoning than regular Google search results. Furthermore, ChatGPT obtained 58.8 questions and 60 approaching the passing range for logical questions and has crossed the threshold for ethical questions. The paper believes ChatGPT and other language learning models can be invaluable tools for e-learners; however, the study suggests that there is still room to improve their accuracy. In order to improve ChatGPT's performance in the future, further research is needed to better understand how it can answer different types of questions.


page 1

page 2

page 3

page 4


Can ChatGPT pass the Vietnamese National High School Graduation Examination?

This research article highlights the potential of AI-powered chatbots in...

Performance of ChatGPT-3.5 and GPT-4 on the United States Medical Licensing Examination With and Without Distractions

As Large Language Models (LLMs) are predictive models building their res...

Evaluation of AI Chatbots for Patient-Specific EHR Questions

This paper investigates the use of artificial intelligence chatbots for ...

"Correct answers" from the psychology of artificial intelligence

Large Language Models have vastly grown in capabilities. One proposed ap...

Using Large Language Models for Cybersecurity Capture-The-Flag Challenges and Certification Questions

The assessment of cybersecurity Capture-The-Flag (CTF) exercises involve...

Please sign up or login with your details

Forgot password? Click here to reset