Background & Summary

Image guidance with computerized navigation based on preoperative magnetic resonance imaging (MRI) was introduced as a surgical adjunct in the 1990s. It facilitates greater resection accuracy by helping the surgeon plan the approach and locate the boundaries of the intended resection. Neuronavigation loses validity as the surgery progresses due to non-linear deformations of the resection cavity and brain shift1,2,3,4,5. Intraoperative imaging such as intraoperative MRI (iMRI) and intraoperative ultrasound (iUS) serve to alleviate this issue. While iMRI provides high-contrast images, they can take several minutes up to an hour to acquire. On the other hand, iUS provides relatively lower-contrast images, but they can be acquired within minutes.

Research in the field of medical imaging and image-guided therapy is increasingly seeking large datasets to develop and test machine learning-based algorithms. Specifically, these algorithms aim at improving precision in image guided neurosurgery and interpretability by performing tasks such as image segmentation6,7 (e.g., tumor, ventricles, cerebrum, resection cavity), image registration8 (e.g., MRI-iMRI, MRI-iUS, iUS-iUS), image synthesis9 (e.g., MR to iUS and iUS to iMRI), and visualization10 (e.g., uncertainty of registration). However, sharing patient datasets for public research use is challenging due to the significant resources required for data curation as well as the need to ensure patient privacy. Moreover, datasets that combine multimodal imaging from both preoperative and intraoperative acquisitions in the same patient are particularly scarce. To address this gap, we have curated a database from neurosurgical procedures conducted in the Advanced Multimodality Image Guided Operating (AMIGO) suite at the Brigham and Women’s Hospital. This database represents the largest publicly-available collection of preoperative MRI, intraoperative MRI, and intraoperative ultrasound data from surgically treated brain tumors. It contains 92 gliomas, 11 metastases, and 11 non-glioma pathologies. The database includes 369 preoperative MRI series, 320 three-dimensional iUS sweeps, 301 iMRI series, and 356 segmentations obtained from 114 consecutive patients who underwent image-guided resection at a single institution. Additionally, each case contains segmentations of the preoperative tumor, the pre-resection cerebrum, and the previous resection cavity derived from the preoperative MRI (if applicable), as well as any residual tumor identified on the iMRI. For reference, Fig. 1 provides an illustrative example of the contents of each dataset and all available segmentations.

Fig. 1
figure 1

Illustrative example of one dataset - a right frontal lobe recurrent WHO Grade II Oligodendroglioma (IDH-positive, 1p/19q co-deleted). (a) Preoperative contrast-enhanced T1-weighted MR; (b) Preoperative T2-weighted MR; (c) Preoperative T2 FLAIR MR; (d) Intraoperative US prior to dural opening; (e) Intraoperative US post dural opening; (f) Intraoperative US prior to iMRI; (g) Intraoperative contrast-enhanced T1-weighted MRI; (h) Intraoperative T2 FLAIR MRI; (i) Intraoperative T2-weighted MRI (BLADE); (j) Cerebrum segmentation on the preoperative contrast-enhanced T1-weighted MRI; (k) Tumor segmentation on the preoperative T2 FLAIR MRI; (l) Residual tumor segmentation on the intraoperative T2 FLAIR MRI.

Full size image

With this work, we build upon the effort initiated by the Montreal Neurosurgical Institute (MNI) and St. Olav University Hospital (Trondheim, Norway) to publicly share MRI and iUS images acquired for brain tumor patients through the BITE11,12,13 (Brain Images of Tumors for Evaluation) and RESECT14,15 (REtroSpective Evaluation of Cerebral Tumors) databases (Table 1) respectively.

Table 1 Comparison against existing publicly available databases of preoperative and intraoperative imaging in brain cancer patients.
Full size table

Methods

This section describes all the procedures followed to acquire and curate the data within the Brain Resection Multimodal Imaging Database (ReMIND) collection16, including experimental design, data acquisition, data annotation, and computational processing (e.g., format conversion, defacing).

Patient cohort

The ReMIND database16 comprises 123 consecutive patients who were surgically treated with image-guided tumor resection in the AMIGO Suite at the Brigham and Women’s Hospital (Boston, USA), between November 2018 and August 2022, using both intraoperative ultrasound (iUS) and intraoperative MRI (iMRI). Of the 123 cases, 9 were excluded due to corrupted or poor-quality data, resulting in a final cohort of 114 cases. Specifically, the reasons for exclusion include the presence of large artifacts in iMRI (N = 1), the corruption of 3D iMRI (N = 1), or poor quality of intraoperative US prior to dural opening (N = 7). Summary demographics can be found in Table 2.

Table 2 Summary of the clinical metadata of the ReMIND data collection.
Full size table

The patients in the study were treated according to the current standard of care, augmented with intra-operative imaging. Collection, analysis, and release of the ReMIND database have been performed in compliance with all relevant ethical regulations. The Institutional Review Board at the Brigham and Women’s Hospital approved the protocol (2002-P-001238), and informed consent was obtained from all participants, including for public sharing of data.

Surgical setup with neuronavigation

Neuronavigation systems aim at providing intraoperative guidance to neurosurgeons by allowing them to visualize the position of their surgical tools relative to the tumor and critical brain areas visible in preoperative Magnetic Resonance Imaging (MRI). Specifically, intraoperative neuronavigation was performed using the optical tracking version of the “Curve” Dual Display system (Brainlab AG, Munich, Germany). The patient reference frame was established by rigidly securing the MRI-safe “Standard Cranial Reference Array with 4 Marker Spheres” (Brainlab AG, Munich, Germany) to the MRI-safe IMRIS head holder (IMRIS Inc., Minnesota, USA). Patients were positioned either supine or lateral, with variable head positions ranging from neutral to approximately 90 degrees to one side. No patients were positioned prone.

The image-to-patient registration was performed in two steps. In the first step, the precalibrated “Softouch Pointer” (Brainlab AG, Munich, Germany) was used to perform an initial image-to-patient registration with the “Cranial Navigation” module (Brainlab AG, Munich, Germany). The nasion, left lateral canthus, and right lateral canthus were located on the patient using the tracked pointer to establish an initial registration. In the second step, a dense set of points was acquired on the skin to refine the patient registration to the skin surface on preoperative imaging. For this step, the “Cranial Navigation” employs a registration approach based on the iterative closest point algorithm with a custom point-to-surface distance function17. Surface-based registration was consistently achieved in all cases by ensuring that the patient’s face was visible to the camera during registration. Finally, the quality of the registration was visually verified by the surgeon and improved when necessary by collecting additional registration points.

Clinical, demographic, and pathology data

Demographic information, including age, sex, and ethnicity, was obtained from the corresponding patient medical records. The age range of the included population was 20–76. The ratio of male:female was equal to 61:53. Moreover, clinico-pathologic data such as the tumor type, tumor grade, radiological characteristics upon contrast administration, tumor location, and the reoperation status were assessed by the treating neurosurgeons. Tumor type and grade were specified according to the World Health Organization (WHO) 2021 Classification of Tumors of the Central Nervous System18,19,20,21. Additionally, tumors were classified into one of 3 categories based on proximity to the functional cortex (non-eloquent, near eloquent, and eloquent).

Table 2 shows a summary of the clinical metadata. A number of the surgeries were reoperations (44.7%) and a minority were performed awake (4.4%). There was an equal number of right- and left-sided craniotomies. The majority of the patients treated were classified as gliomas, glioneuronal, and neuronal tumors (83.3%) and were primarily Astrocytomas (34.7%). IDH-mutations, 1p/19q-codeletions, and MGMT promoter methylation status were reported in 100%, 67.5%, and 94.3% of the adult-type diffuse gliomas, respectively. The majority of the gliomas, glioneuronal, and neuronal tumors were CNS WHO Grade 4 (38.9%). (67.5%) were located in eloquent areas based on the fMRI, anatomic substrate, and neurophysiological monitoring. The full metadata table can be downloaded from our repository16 on The Cancer Imaging Archive (TCIA).

Imaging data

The ReMIND database16 corresponds to a collection of preoperative MRI, intraoperative MRI, and intraoperative ultrasounds. A summary of the acquisition parameters for preoperative and intraoperative MRI is provided in Table 1. The imaging data can be found and downloaded from our TCIA repository16 as well.

Preoperative MRI

Preoperative MRI were accessed, selected, and co-registered using the “Elements” software (Brainlab AG, Munich, Germany), a CE-certified/FDA-cleared medical device software used routinely in clinical practice. Specifically, “Image Fusion”, a mutual information-based registration algorithm22, was used to rigidly co-register preoperative images. Preoperative MRI comprises four structural MRI sequences: native T1-weighted (T1), contrast-enhanced T1-weighted (ceT1), native T2-weighted (T2), and T2-weighted fluid-attenuated inversion recovery (T2-FLAIR). These scans were acquired before surgery using various scanners at multiple institutions, making their acquisition parameters heterogeneous, as shown in Table 1. More details about the acquired sequences are presented in Table 3. Most of the preoperative imaging was performed on a 3 T (71.1%) MRI scanner with Siemens (87.7%) being the most common manufacturer.

Table 3 Summary of data characteristics of the ReMIND MR sets.
Full size table

Intraoperative MRI (iMRI)

Unlike preoperative MRI, all intraoperative MRI were acquired in the AMIGO suite at the Brigham and Women’s Hospital (Boston, USA) using a 3 T wide-bore (70 cm) MRI scanner (Magnetom Verio, Siemens Healthineers, Erlangen, Germany) with an 8-channel head coil to evaluate the presence of residual targeted tissue once the planned portion of targeted tissue was removed. Before the acquisition, a temporary closure of the craniotomy was performed. The entire acquisition process required 1–1.5 hrs. and included MRI safety procedures, instrument counts, preparation for scanning, and redraping to resume surgery after iMRI.

Intraoperative Ultrasound (iUS)

All iUS series were acquired using a sterilizable 2D neuro-cranial curvilinear transducer on a cart-based ultrasound system (N13C5, BK5000, GE Healthcare, Peabody, MA, USA) in the AMIGO suite. The ultrasound probe had a contact area of 29 mm × 10 mm and a frequency range of 5–13 MHz. The “Ultrasound Navigation Adapter Array” together with the “Ultrasound Navigation Adapter Base - BK N13C5” (Brainlab AG, Munich, Germany) were attached to the iUS probe to enable the Curve platform to track the probe relative to the patient. The imaging plane was chosen to be as parallel as possible to one of the three cardinal axes of the head (axial, sagittal, coronal). However, this was often limited by the size and shape of the craniotomy. The transducer was swept unidirectionally at a slow, consistent speed through the craniotomy. This specific motion, in conjunction with the tracking, enabled the reconstruction of a 3D volume from the tracked 2D sweeps using the “Ultrasound” module within the “Elements” software platform on the “Curve” hardware system. Similar to the RESECT database14, 3D iUS acquisition was aimed to be performed at three distinct surgical time points:

  1. 1.

    Pre-dura iUS: between the craniotomy and the dural opening.

  2. 2.

    Post-dura iUS: after the dural opening but before any tumor resection was performed.

  3. 3.

    Pre-iMRI iUS: immediately before the iMRI was acquired, i.e. after substantial tumor resection was completed to the degree that either the surgeon was satisfied with the microscopically-visible extent of resection or that iMRI was needed to identify the remaining portion of the tumor.

When more than one iUS volume was acquired at a specific surgical time point, the acquisition with the best image quality, field of view, and maximal tumor coverage was included in this collection. In 22 cases, iUS acquisitions were unavailable at some surgical time points due to surgeon preference (15/22) or durotomies during craniotomies or previous surgeries (7/22). Such cases have an explanation detailed in the clinical metadata table available on TCIA.

Segmentation data

Various segmentations were created to assist the surgical resection. These typically include manual segmentations of the preoperative whole tumor, preoperative tumor target (i.e., the radiologically identifiable tumor specifically targeted for resection), resection cavity resulting from prior surgery (i.e. in case of reoperation) and intraoperative residual tumor. These segmentations were performed using the BrainLab “Elements” planning software by a neurosurgical fellow and refined if needed by the attending neurosurgeon. For enhancing lesions, the preoperative and residual tumor segmentation encompass the enhancing region and any necrotic core if present. In the case of non-enhancing lesions, the preoperative and residual tumor segmentations correspond to the solid/nodular component within the T2/T2-FLAIR hyperintensity region.

Moreover, automatic segmentations of cerebrum and ventricles were obtained using the “Object Manipulation” module (Brainlab AG, Munich, Germany). The segmentation of these two structures serves both clinical and research purposes. The three-dimensional model of the cerebrum helps neurosurgeons to assess the accuracy of the navigation using gyral anatomy and correct any small errors and to select the operative trajectory. Additionally, the segmentation of ventricles is useful to assess registration and any brain shift since they are readily visible on iUS; clinically these can be helpful for avoiding inadvertent entry during procedures. Only structures deemed necessary for surgical resection by the attending neurosurgeon were automatically segmented. For research purposes, the segmentation of the cerebrum and ventricles may be valuable as the starting point for the development and validation of algorithms, such as those for brain structure segmentation in the presence of lesions or image registration.

In total, manual preoperative whole tumor segmentations are provided for 113 cases; preoperative tumor target segmentations are provided for 3 cases in which subtotal resection was planned; manual previous resection cavity segmentations are provided for 21 cases; residual tumor segmentations are provided for 58 cases; and automated segmentations of the cerebrum and ventricles are provided for 89 and 54 cases respectively. All cerebrum, ventricle, and tumor segmentations were created preoperatively during the surgical planning stage. In contrast, residual tumor segmentations were created intraoperatively from iMRI.

Data export

Preoperative MRI were accessed, selected, and co-registered using the “Image Fusion” module within the “Elements” software platform (Brainlab AG, Munich, Germany). After visual inspection, some MR series were excluded due to poor quality, the presence of artifacts, or a small field of view. Manual and automated segmentations were performed using the “Object Manipulation” module. Intraoperative ultrasounds were tracked using the “Curve” neuronavigation system, allowing them to be roughly registered with the preoperative images. Finally, intraoperative MRI were automatically registered with preoperative scans also using the “Image Fusion” module in “Elements”. Note that image registration was performed by only updating image headers to avoid resampling errors. Images and segmentations were finally exported as NRRD files from the “Curve” neuronavigation system using 3D Slicer via OpenIGTLink23.

Data transmission, de-identification and format conversion

The de-identified imaging data and segmentation data were submitted to The Cancer Imaging Archive (TCIA) in DICOM format (imaging and segmentation) and NRRD format (segmentation). Experienced quality-control reviewers inspected images to ensure the data are fully de-identified and well-curated.

Data were fully de-identified by removing all health information identifiers and by applying a de-facing algorithm. Specifically, each MR scan was defaced by: 1/ affinely registering an MR template to the MR scan using NiftyReg24; 2/ applying the obtained affine transformation to the face mask of the template; 3/ applying the resampled face mask to the MR scan. The code of the algorithm is publicly available at https://github.com/ReubenDo/pydeface-niftyreg. All the defaced scans were visually inspected, and 100% of them were successfully defaced.

The DICOM format is increasingly used in research as it is standardized, preserves all metadata (e.g. patient, session), and allows for long-term archival. For that reason, our imaging data was released in DICOM format on TCIA. Specifically, the preoperative and intraoperative MR images were converted from NRRD to DICOM series using 3D Slicer25. Moreover, the reconstructed 3D iUS images were converted to multi-frame DICOM using the dicom3tools software26. Finally, each segmentation was converted using DCMqi27 with the segmented MR DICOM data as the reference image.

Data Records

All the imaging data and the metadata described here as the “ReMIND” collection are available as a publicly available repository of The Cancer Imaging Archive (TCIA)28 (TCIA: https://doi.org/10.7937/3RAG-D070)16. The DICOM files conform to DICOM standards and are organized on TCIA based on anonymized patient ID numbers (CaseID). Each patient is associated with two studies: preoperative and postoperative, each comprising images and segmentations. Note that ultrasound images are exclusively part of the intraoperative study. As a result, the downloaded DICOM database follows the folder structure outlined in Box 1. Segmentation NRRD files are organized similarly, as depicted in Box 2.

Technical Validation

Quality control steps are performed before, during, and after surgery. These include assessing image quality as well as quality of image-to-image registration and image-to-patient registration, image deidentification, segmentation, and data conversion. We present these steps below.

Image selection and registration steps

During surgical planning, a neurosurgical fellow selected the preoperative MR data included in the surgical plan. These images correspond to the ReMIND’s preoperative data. In particular, the fellow ensured data integrity by checking for corruption or significant artifacts. Selected preoperative scans were then co-registered using the Brainlab “Image Fusion” module. The neurosurgical fellows and the attending neurosurgeon visually assessed the registration quality during surgical planning, confirming successful co-registration for all cases.

Image-to-patient registration

At the start of the procedure, image-to-patient registration was performed and assessed by the attending neurosurgeon using surface landmarks. If needed, the neurosurgeon collected additional registration points on the patient’s skin surface to refine the registration. The surgeons assessed the registration accuracy by visually checking the position of the tracked pointer with surface landmarks on the pre-operative MRI. If a substantial inaccuracy was observed, the registration step was repeated until acceptable accuracy was obtained.

Image de-facing

The derived de-facing masks for all cases were visually checked for quality by a research fellow and experienced quality-control TCIA reviewers. The developed tool was robust with 100% of the preoperative and intraoperative MR scans successfully defaced.

Imaging data conversion

To ensure that the data conversion step (from NRRD to DICOM) did not deteriorate raw imaging data, DICOM images were converted back to NRRD files using 3D Slicer. Then, the 3D image arrays of the original NRRD and the converted NRRD files were compared using the Simple Insight Toolkit29 (SimpleITK). The conversion process was found to preserve data information, i.e. the same values were found at each voxel location.

Automated segmentation data

Automated brain structure segmentations were subject to visual inspection around the surgical areas of interest by the neurosurgical fellows and the attending neurosurgeon during surgical planning. Although these automated segmentations were generated by a widely trusted commercial software routinely used in clinical practice, inaccuracies may occur, especially for cases involving reoperations of recurrent tumors. For that reason, these segmentations should not be used as a gold-standard for segmentation problems, but would serve well as starting points which could be further refined.

Manual segmentation data

Preoperatively, the tumor annotations were performed by the neurosurgical fellow during surgical planning. These annotations were assessed by the attending neurosurgeon and refined if needed. Intraoperatively, the neurosurgical fellow and the attending neurosurgeon segmented the residual tumor visible in intraoperative MRI. As part of the public release of this collection, interobserver variability was measured for the manual tumor segmentations on 10 cases (5 enhancing and 5 non-enhancing tumors) for further validation. Preoperative T1 gadolinium enhanced MRI was used to segment enhancing tumors and preoperative T2-weighted MRI (SPACE) were used for non-enhancing tumors were segmented independently by two neurosurgical research fellows. We quantitatively measured the Dice score and average symmetric surface distance (ASSD) between the two sets of manual segmentations. For manual tumor segmentation on enhanced T1-weighted MRI, interobserver variability yielded a Dice score of 92.2% (SD 2.8%) and an ASSD score of 0.68 mm (SD 0.34 mm). For manual non-enhancing tumor segmentation on T2-weighted MRI, interobserver variability yielded a Dice score of 86.4% (SD 3.7%) and an ASSD score of 1.24 mm (SD 0.22 mm).

Usage Notes

To view the imaging data, we recommend 3D Slicer available at https://www.slicer.org, which is a free and open-source platform for medical image informatics, image processing, and three-dimensional visualization. 3D Slicer supports the DICOM format. We also developed a custom 3D Slicer extension to create landmarks on images. The MRUSLandmarking extension is freely available in the 3D Slicer Extension Manager or at https://github.com/koegl/SlicerMRUSLandmarking.