Testing a Popular Large Language Model on Image Rich Diagnostic Radiology Board Exam Practice Questions

Purpose: Use of large language models (LLM) has grown exponentially in recent years. Previous work has shown their capability for impressive performance on text-based diagnostic radiology (DR) board questions. The purpose of this study is to evaluate the accuracy of a widely used LLM capable of processing images in answering image rich DR board exam practice questions.

Methods/Materials: A set of 107 publicly available DR board exam practice questions, including images and answer choices, was used. The LLM was given a standardized prompt to respond as a DR resident, answering multiple choice questions with explanations and confidence levels. Accuracy and adjusted accuracy (accounting for significant flaws in logic) were computed. Sub-analyses assessed differences in accuracy between text-only and image-based questions, and confidence variations between correct and incorrect answers. Means were compared using 2-sample T-tests. LLM performance was benchmarked against national averages of DR residents at various postgraduate year (PGY) levels.

Results: The LLM achieved 58.5% overall accuracy, lower than the PGY-3 average (61.9%) but higher than the PGY-2 average (52.8%). Adjusted accuracy was 52.8%. The LLM showed significantly higher (p = 0.012) confidence for correct answers (87.1%) compared to incorrect (84.0%). Performance on image-based questions was notably poorer (p < 0.001) at 45.4% compared to text-only questions (80.0%), with adjusted accuracy for image questions of 36.4%. There was no difference in confidence levels for image-based questions (p = 0.986), regardless of answer correctness.

Conclusions: The LLM performed between PGY-2 and PGY-3 levels on DR board practice questions, but significantly poorer on image-based questions. This study underscores the potential and risks of using minimally-prompted LLMs in interpreting radiologic images as a diagnostic tool. A significant limitation is the likelihood of the question set being within the LLM training data, raising concerns about its real-world applicability. Implementers of general AI radiology systems should exercise caution given the possibility of spurious yet confident responses.