Context-Based GPT-4 surpasses Base GPT-4 in adhering to ACR Appropriateness Criteria for Ordering Neuroradiology Imaging

Purpose: Current Clinical Decision Support Mechanism (CDSM) offerings often do not effectively exploit the full corpus of knowledge offered by the American College of Radiology (ACR) Appropriateness Use Criteria’s (AUC) which contains both rankings tables and a fulltext synopsis. As a result, substantial variability in ordering patterns persists, often due to a lack of adoption, inconsistent delivery of content, and variable use among providers. Foundational large language models (LLMs) show promise in augmenting medical decision making by delivering supplemental specialized content to providers using natural language instead of controlled rules based selection.

A novel context-based GPT (cbGPT) assistant tailored for neuroradiology imaging recommendations utilizing OpenAI’s latest GPT-4 model, will align more closely with ACR 'Usually Appropriate' and ‘May be Appropriate' recommendations than its current base GPT-4 counterpart.

Methods/Materials: cbGPT is a custom assistant created within ChatGPT. cbGPT was supplemented with ACR AUC documents containing freetext and tables, as well as custom prompts (Figure 3). Fifty-one synthetic clinical vignettes, typical of neurological conditions, were used to test the alignment of each model with ACR AUC. Outputs were scored in accordance with the grading schema (Figure 1) and the final score was an average of three runs per scenario.

Results: cbGPT assistant demonstrated superior performance in aligning with ACR criteria compared to its base model counterparts. It exhibited enhanced accuracy (73% vs 42.5%, p< 0.0001) in recommending appropriate imaging procedures.

Conclusions: cbGPT significantly improves adherence to ACR AUC compared to base GPT-4 for recommending neuroradiology studies. These findings point towards a role for investigating the use of LLMs to enhance patient safety, optimize resource utilization, and potentially augment decision making in healthcare settings. Further studies on deterministic and reproducible outputs are warranted. Exploration into fine-tuned LLMs should also be considered for enhancing consistency and accuracy.