Accurate Cervical Spine Fracture Detection from Radiology Report: Harnessing Privacy-preserving Open Source Large Language Models (LLMs)

Purpose: Open-source Language Models (LLMs) have emerged as powerful tools for data abstraction from radiologic reports. These models leverage their natural language processing capabilities to extract critical information from complex medical texts, enhancing the efficiency of healthcare professionals. The open-source approach provides a more secure and compliant option for medical data abstraction, as it eliminates the need to transmit sensitive patient information through a commercial API. Here, we use an open-source LLM to determine the presence and level of acute cervical spine fractures in radiologic reports.

Methods/Materials: We acquired radiology reports from 700 non-contrast cervical spine CT scans conducted between January and February 2022. Out of these reports, 110 were utilized for initial model prompting and fine-tuning, the remaining 590 served as a test dataset to assess the model's performance. The primary aim of the model was to generate a JSON file that could indicate the presence or absence of an acute cervical vertebral fracture and specify its associated location. This specific challenge was chosen because the model needed to distinguish among various anatomical regions and discern between chronic and acute fractures. To accomplish this, we employed an open-source Large Language Model (LLM) known as the Zephyr-7B-Alpha and deployed it locally. We applied a combination of chain-of-thought (CoT) and few-shot prompting techniques to instruct the model in producing the final output containing the results.

Results: The model attained a 98% accuracy rate, correctly identifying acute cervical spine fractures in 583 out of 590 cases. In cases where reports indicated the "possibility" of fracture, the model reported the presence of fracture. Particularly noteworthy is that it accurately predicted multiple fracture sites in which multiple fractures were present in one report.

Conclusions: ChatGPT limitation lies in offering solely a commercially hosted API, which mandates the transmission of patient data beyond the confines of the healthcare institution. By employing proper prompts, open-source LLMs can excel in their performance without the necessity of sharing patient data.