Multimodality-based convolutional neural network capturing intra- and inter-observer variability for GTV delineation of sarcomas

Purpose: Develop a multimodality-based convolutional neural network (CNN) for automatic gross tumor volume (GTV) delineation of sarcomas while accounting for intra- and inter-observer variability among different observers.

Methods/Materials: The patient cohort was composed of 61 patients with multiple sarcoma histology subtypes who underwent pre-operative CT, MR and FDG-PET imaging within a 4-week time window. 14 patients were scanned at Massachusetts General Hospital. The imaging data for the other 47 patients was obtained from the public The Cancer Imaging Archive dataset.

6 observers (2 radiologists, 1 radiation oncologist, 1 nuclear physician and 2 surgeons) performed between 1 and 3 GTV delineation trials per patient using MIMVista software (MIM Software Inc., Cleveland, OH) as per the American Society for Therapeutic Radiology and Oncology guidelines. The GTV delineations were converted into confidence maps (ranging between 0 and 1) calculated by averaging the GTV masks (e.g. a confidence of 1 corresponds to perfect agreement amongst all observers).

A deep learning model was designed to predict GTV confidence maps from the CT, MR and PET images. A deep convolutional network following the U-Net architecture was used. Multi-modality image patches were stacked as network inputs and GTV confidence maps were used as labels. The network was trained on 46 patients (leaving 15 for testing) using a cross-entropy loss.

Results: The average continuous Dice score between predicted and ground truth confidence maps over the testing population was 81.5% +/- 13.6%, demonstrating a good agreement between human and deep learning GTVs in different sarcoma subtypes (e.g. soft-tissue sarcoma, osteosarcoma, chordoma). The 95th percentile Hausdorff distance estimated from confidence maps thresholded at 50% was 9.03 mm +/- 7.35 mm.

Conclusions: While most available networks are based on delineations provided by a single observer, often neglecting both intra- and inter-observer variability, the proposed CNN can accurately predict GTV delineations of sarcomas while accounting both intra- and inter-observer variability thereby mimicking real-life scenarios.