Performance of ChatGPT in Radiology: A Systematic Review Study

Purpose: To systematically review the reported performance of ChatGPT across clinical radiology applications.

Methods/Materials: After a comprehensive review of PubMed, Web of Science, Embase, and Google Scholar, we derived a cohort of published studies utilizing ChatGPT for clinical radiology applications. (Fig. 1)

Results: Out of 859 studies derived, 27 studies met the criteria (Table 1) and reported 77.8% (21/27) performance of using ChatGPT in simplifying radiology reports (55.5%, 15/27), aiding radiologist decisions (48.2%, 13/27), enhancing patient outcome (48.2%, 13/27) and increasing processing speed and cost-effectiveness (40.7%, 11/27). None suggested unsupervised use in medical education or clinical practice (Figs. 2 and 3). Six studies (22.3%, 6/27) found ChatGPT to be inaccurate in discussing cancer surveillance, diagnosis, patient education, and complex radiology questions. Three studies employed different Likert scale variations to assess ChatGPT's performance in diverse scenarios, using a Linkert scale 2 (-2 to +2) for evaluating text generation, [4.27 (1 to 5)] for translating radiology reports, and [1.75 (0 to 5)] for identifying diagnostic imaging methods. In six studies, ChatGPTv4 outperformed v3.5, showing notable enhancements in addressing higher-order thinking questions, better comprehension of radiology terms, and improved accuracy in describing images. Risks and concerns about using ChatGPT included biased responses, limited originality, inaccurate information leading to hallucination or unwarranted confidence, improper citations and fake references, cybersecurity vulnerabilities, and patient privacy risks.

Conclusions: Although ChatGPT has shown a 77.8% performance in radiology, there are still multiple pitfalls and limitations to address. It's too soon to confirm its complete proficiency and accuracy, and more extensive multicenter studies utilizing diverse datasets and pre-training techniques are required to verify ChatGPT’s role in radiology.