Original Article
Performance of Chat Generative Pre-Trained Transformer on Personal Review of Learning in Obstetrics and Gynecology
Abstract
Objectives: Chat Generative Pre-Trained Transformer (ChatGPT) is a popular natural-language processor that is able to analyze and respond to a variety of prompts, providing eloquent answers based on a collection of Internet data. ChatGPT has been considered an avenue for the education of resident physicians in the form of board preparation in the contemporary literature, where it has been applied against board study material across multiple medical specialties. The purpose of our study was to evaluate the performance of ChatGPT on the Personal Review of Learning in Obstetrics and Gynecology (PROLOG) assessments and gauge its specialty specific knowledge for educational applications.Methods: PROLOG assessments were administered to ChatGPT version 3.5, and the percentage of correct responses was recorded. Questions were categorized by question stem order and used to measure ChatGPT performance. Performance was compared using descriptive statistics.
Results: There were 848 questions without visual components; ChatGPT answered 57.8% correct (N = 490). ChatGPT performed worse on higher-order questions compared with first-order questions, 56.8% vs 60.5%, respectively. There were 65 questions containing visual data, and ChatGPT answered 16.9% correctly.
Conclusions: The passing score for the PROLOG assessments is 80%; therefore ChatGPT 3.5 did not perform satisfactorily. Given this, it is unlikely that the tested version of ChatGPT has sufficient specialty-specific knowledge or logical capability to serve as a reliable tool for trainee education.
This content is limited to qualifying members.
Existing members, please login first
If you have an existing account please login now to access this article or view purchase options.
Purchase only this article ($25)
Create a free account, then purchase this article to download or access it online for 24 hours.
Purchase an SMJ online subscription ($75)
Create a free account, then purchase a subscription to get complete access to all articles for a full year.
Purchase a membership plan (fees vary)
Premium members can access all articles plus recieve many more benefits. View all membership plans and benefit packages.