Artificial Intelligence in Diagnostic Radiology: Evaluating the Performance of ChatGPT on Radiology Core Examination

Purpose: Artificial Intelligence (AI) has shown promising advancements in recent years. ChatGPT is a powerful large AI language model with potential in medical practice, but its performance in radiology remains unclear. ChatGPT currently has around 180.5 million users and the website generated 1.7 billion visits in October 2023. This study aims to assess the performance of ChatGPT’s newest and most advanced version, GPT-4, on diagnostic radiology core examination questions with and without images and to explore its strengths and limitations.

Methods/Materials: This study utilized a set of 74 multiple-choice questions, including text-based and image-based questions. The selected questions were drawn from The American Board of Radiology Diagnostic Radiology Qualifying (Core) Examination Study Guide 2013. The responses generated by GPT-4 were systematically compared against the provided answer keys. The overall performance was evaluated by question type and type of radiology images. Statistical analysis was performed to determine any association.

Results: Overall, GPT-4 answered 56.8% of questions correctly (42 out of 74) and did not specifically answer 27.0% of the questions (20 out of 74). The model performed significantly better on text-based questions (76.9%, 20 out of 26) compared to image-based questions (45.8%, 22 out of 48) (P = 0.0197). When comparing different image-based questions, the model performed best with other types of images including PET scan, V/Q scan, and thyroid scan with 66.7% correct (2 out of 3), followed by MRI (60.0%, 6 out of 10), ultrasound (44.4%, 4 out of 9), CT (40.0% correct, 6 out of 15), and X-rays (36.4%, 4 out of 11).

Conclusions: GPT-4 demonstrated notable performance in the radiology core examination, particularly excelling in text-based questions. However, its ability to process and respond to radiology image questions remains limited. The ever-changing field of AI holds the potential to transform diagnostic radiology, providing invaluable support to radiologists in image interpretation. While GPT-4 showcases the ability to recognize diverse image types, its accuracy is suboptimal and there is still room for more development and advancement in the future.