Evaluating ChatGPT and Alpaca on the ACR Appropriateness Criteria to Help Ordering Physicians

Purpose: ChatGPT, a bot utilizing large language model (LLM), is occasionally used in healthcare settings for clinical decision support. In this study, we evaluate the ability of open source LLMs to select appropriate medical imaging as compared to physician decision makers. The purpose of this study is to evaluate if AI large language models choose appropriate medical imaging with similar accuracy to physicians.

Methods/Materials: Cases (n=42) from various specialties were collected from ACR Cortex. ACR Appropriateness Criteria and 212 records were downloaded. LLMs used include ChatGPT 3.5 and Alpaca 7B. All models were tested using the records from the collected data set. The first imaging modality output by the model was used for comparison. Responses were graded on a scale of 1-3 based on ACR appropriateness criteria guidelines. A one-way ANOVA test was run in R comparing the physician decision maker’s imaging decisions to Alpaca and ChatGPT LLMs. T-test comparing the outputs each model to each other, the expected results, and the physician decision maker’s imaging decisions was run in R.

Results: Difference between ChatGPT and physician decision makers was not significant (p=0.615, CI -0.210 0.353)
Alpaca was significantly less accurate than physician decision makers (p=0.0399, CI -0.697 -0.0167)
All categories performed significantly different from ACR Gold Standard criteria
ChatGPT was significantly different from Gold Standard ACR criteria (p=0.000147, CI -0.600 -0.209)
Alpaca was significantly different from Gold Standard ACR criteria (p=2.83e-07, CI -1.107 -0.558)
Physician decision makers were significantly different from Gold Standard ACR criteria (p=4.05e-05, CI -0.685 -0.266)
ChatGPT had the highest mean score (2.595) compared to physician decision makers (2.524), and Alpaca (2.167)

Conclusions: ChatGPT and physicians performed with similar accuracy in determining appropriate imaging. Both ChatGPT and physicians performed superior to Alpaca. All categories showed significant deviation from ACR Appropriateness Criteria. This indicates that significant modification must be made to the AI LLMs before clinical implementation is appropriate.