Deep learning based 3D brain metastasis synthesis with configurable parameters for 3D data augmentation

To develop and evaluate deep learning methods for synthesizing 3D brain metastases (BM) on magnetic resonance (MR) images to improve downstream BM detection and segmentation performances for robust clinical detection and streamlined treatment-planning workflows, T1-weighted MR images of 1832 patients with 10,276 BM were divided into training (80%), validation (10%), and test (10%) datasets for training the BM synthesis models and downstream models for BM detection and segmentation. A 3D-2D generative adversarial network with controllable configurations for 3D BM synthesis was proposed. The network consists of two 3D generators to create 3D lesion intermediate representations controlling the lesion’s characteristics and 3D continuity, and a 2D generator with a 2D perceptual loss to generate realistic lesion images slice by slice. Training with synthetic data (SD) and conventional augmentation (CA) on the downstream tasks were compared using two-sided pairwise Wilcoxon signed rank test. SD showed improvement on the BM detection and segmentation downstream tasks with limited training data. Especially, when 10% of the original training data was available, adding 176 SD improved BM segmentation Dice by 2.9% relative to CA alone (0.665 vs. 0.646, p < 0.0001). With 8000 SD, segmentation boundary accuracy further improved, reducing HD95 by 10.5% and ASSD by 23.5% compared with CA alone.