Introduction

Medical imaging plays an important role in clinical cancer care for diagnosis, radiation therapy, treatment planning, and cancer management. Researchers have developed various analytical medical imaging methods, such as image segmentation, registration, pattern recognition, and multivariate pattern classification. One of these, radiomics1,2,3,4, has recently emerged as a promising medical image analysis tool for diagnosis and prediction of response to treatment of various diseases. Radiomics involves the high-throughput extraction of large numbers of quantitative features from medical images and analysis of these features to predict patients’ outcome and support clinical decision-making, such as classifying benign and malignant tumors, determining molecular subtypes and/or mutation status, and predicting overall survival.

Several radiomics analyses have been used with various imaging modalities in oncology, such as computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET), and results showed that a large number of radiomics features have prognostic power in several studies, such as lung and head and neck cancer patients on CT images3,5, prognosis of recurrence and survival in lung cancer patients on PET/CT images6,7, and in brain tumor and breast cancer patients on MRI images8,9,10,11,12. Radiomics features are sensitive to several factors, however, such as reconstruction settings13,14, tumor delineation15, scanning protocols16,17, different scanners18, and various noise sources. Several radiomics studies have investigated reproducibility and repeatability19. For example, Peerlings et al.20 investigated on stability of radiomics features in apparent diffusion coefficient (ADC) maps. Schwier et al.21 investigated on repeatability of multiparametric prostate MRI radiomics features. Fave et al.22 evaluated how different image preprocessing techniques may impact both the volume dependence and prognostic potential of the features of non-small cell lung cancer in CT and investigated the variability in voxel size, slice thickness, and convolution kernels in CT23. Also, Mackin et al.24 investigated variability in radiomics features with the x-ray tube current used in CT. In a recent study, Shiri et al.25 investigated the impact of image reconstruction settings on radiomics features using two PET/CT scanners. They found that the variability and robustness of PET/CT images are dependent on different features and concluded that radiomics features with a low coefficient of variation (COV) are good candidates for reproducible tumor quantification in multicenter studies. In a similar study of PET, Bailly et al.26 investigated the variability of 15 textural features according to reconstruction parameters in multicenter trials and found that Homogeneity, Entropy, Dissimilarity, High Gray-Level Run Emphasis (HGRE), High Gray Level-Zone Emphasis (HGZE), and Zone Percentage (ZP) features are robust and suitable for use in multicenter trials.

However, not many studies have investigated the repeatability (variations when a patient is scanned twice on the same scanner with the same parameters) and variability when different scanning protocols are used for MRI radiomics studies. MRI is an important diagnostic imaging modality and has been widely used as a major diagnostic tool in both clinical imaging and scientific research, and quantitative radiomics analysis using MRI has increased recently.

Therefore, in the present study, we created an MRI radiomics phantom and used it to assess the robustness of MRI radiomics features with various MRI scanning protocols and two MRI scanners. First, we evaluated radiomics features of the MRI phantom by comparing each feature value with patient population data using the two-sigma range of feature values extracted from 97 T1- and T2-weighted MR images of patients with brain lesions. We then investigated the robustness of magnetic resonance imaging (MRI) radiomics features with various MRI scanning protocol parameters and scanners using an MRI radiomics phantom.

Results

We determined the suitability of the MRI phantom materials by comparing the radiomics feature values from the phantom materials with those of the brain lesions of the patient data (mean values ± two standard deviations [SDs]) (Table 1). Figure 1 illustrates this analysis, showing the values of the inverse variance texture feature for the phantom materials over various settings in a number of excitations (NEX). The orange solid lines and orange dashed line in the figure represent the mean ± two SDs bounds and mean patient population data for the inverse variance feature, respectively. Averages of 92.5% and 79.6% phantom radiomics features for the 20 materials were within the established patient population bounds for T1- and T2-weighted images, respectively.

Table 1 The percentages of radiomics features for the MRI phantom within the established patient population bounds (mean ± 2 SDs).
Full size table
Figure 1
figure 1

GLCM inverse variance feature values for the MRI phantom materials with various settings in NEX. The orange solid lines and orange dashed line are the mean ± 2 SDs bounds and the mean of the patient population data for the inverse variance feature, respectively.

Full size image

We used the COV to assess the variability of radiomics features for the impact of different MRI parameter settings and plotted a heat map of the COV for both the phantom and volunteer. We repeated this with image intensity normalization, without normalization, with smoothing filter, and without smoothing filter as a preprocessing, respectively (Fig. 2). We used 3 sigma method27 for the intensity normalization and the butterworth algorithm28,29,30 for the smoothing filter. We also investigated the variability of radiomics features with different ROI size (diameter of 1.2 cm) (Fig. 2). Based on the COV, we categorized the features in terms of variation using three groups: large variation (COV > 30%), intermediate variation (10% < COV ≤ 30%), and small variation (COV ≤ 10%)25. Without any image reconstruction such as normalization and filtering process, the average COVs in these three groups were 6.1%, 18.5%, and 45.5%, respectively, for T1-weighted images and 4.5%, 17.2%, and 51.4%, respectively, for T2-weighted images. Tables 2 and 3 summarize the radiomics features in the three groups for T1- and T2-weighted images, respectively. With normalization and filtering process, the average COVs for three groups summarized in Table 4. The detailed radiomics features in the three groups for T1 and T2- weighted images are listed in Tables S4, S5, S6, S7, S8 and S9 in the supplement information.

Figure 2
figure 2

Heat map of the variability of radiomics features with different MRI scanning settings for (a) T1-weighted and (b) T2-weighted images without normalization and filtering. (c,d) shows the heat map for T1- and T2-weighted images with normalization. (e,f) shows the heat map for T1- and T2-weighted images with filtering, respectively. We also test the variability of radiomics features with different ROI size (regular ROI = diameter of 1.8 cm and small ROI = diameter of 1.2 cm). (g,h) shows the heat map for T1- and T2-weighted images with smaller ROI size. On the x-axis, _A, _B, and _C represent phantom scanned using the Siemens 1.5 T scanner, phantom scanned using the Philips 1.5 T scanner, and volunteer scanned using the Siemens 1.5 T scanner, respectively. The COV value is rescaled from 0 to 1 with blue representing high COV values. The hierarchical clustering on the y-axis was computed using a Euclidean distance measure. The full feature names for each plot listed in the tables S13, S14, S15, and S16 in the supplementary information.

Full size image
Table 2 Variations of radiomics features over different MRI scanning settings for T1-weighted images without normalization and smoothing.
Full size table
Table 3 Variations of radiomics features over different MRI scanning settings for T2-weighted images without normalization and smoothing.
Full size table
Table 4 Average COV for T1- and T2-weighted images.
Full size table

Figure 3 shows intraclass correlation coefficient (ICC) plots for T1- and T2-weighted images of the phantom and volunteer for a test–retest scheme on a single scanner. We found that the T1- and T2-weighted image repeatability measures for the phantom (average ICC, 0.963 and 0.959, respectively) were higher than those for the volunteer (average ICC, 0.856 and 0.849, respectively). In this study, we categorized repeatability variations using three groups: high repeatability (ICC ≥ 0.9), intermediate repeatability (0.6 ≤ ICC < 0.9), and poor repeatability (ICC < 0.6)31. Tables 5 and 6 summarize the repeatability of the radiomics features for various MRI scanning parameters for all three groups for the phantom and volunteer, respectively. For the phantom, the ICC for all features except the Gray Level Non-uniformity (T1), Inter Quartile Range (T2), and Information Measure Corr 1 (T2) was greater than 0.6 for both T1- and T2-weighted images. For the feature comparison between with and without normalization, with and without smoothing effects, and different ROI sizes, we summarized the results in Tables S10, S11, S12, respectively. Based on these results, we can see that features in GLRL and NID are more invariant compared to other feature categories.

Figure 3
figure 3

ICC plots in (a) T1-weighted and (b) T2-weighted images of the phantom and volunteer. The blue circles and orange squares represent the ICC values for the phantom and volunteer, respectively. The order of the features’ names is shown in Table 9.

Full size image
Table 5 Repeatability of the radiomics features with different MRI scanning settings using the same scanner for the phantom.
Full size table
Table 6 Repeatability of the radiomics features with different MRI scanning settings using the same scanner for the volunteer.
Full size table

Discussion

In recent years, radiomic studies have become increasingly important for medical image analysis to assist the diagnosis, prognosis, and prediction of treatment response within clinical-decision making systems. However, radiomics features are sensitive to different image reconstruction settings, scanning protocols, scanners, and noise sources, so we must identify the radiomics features that remain stable to provide accurate and reliable decision support for patient care. In the present study, we made our phantom with 20 homogeneous and heterogeneous materials selected carefully (Fig. 4B). So, our phantom is similar to the human brain as brain has both homogeneous and heterogeneous regions for fair comparison. We showed the suitability of the phantom materials by comparing radiomics features obtained from phantom materials with those of the brain lesions of patients. We used the brain MRI data over other patient anatomies because of its stable movement. Various studies showed that respiratory motion was a major factor leading to irreproducibility in various modalities such as MRI, PET, and CT32. Next, we investigated the variability and repeatability in radiomics features extracted from T1- and T2-weighted MR images of an MRI phantom and a healthy volunteer to identify radiomics feature robustness for various scanning protocols and different scanners. Our results showed that the robustness of the MRI radiomics features across the different scanning protocols varies depending on radiomics features. According to our results, most intensity-based and gray level co-occurrence matrix (GLCM) features were in the intermediate or small variation group, whereas most neighborhood gray-tone difference (NGTD) features were in the high variation group. NGTD features are extracted from an image inside the region of interest (ROI), and intensity difference is computed in a two-dimensional neighborhood. NGTD features provide fundamental texture properties, such as coarseness, contrast, busyness, complexity, and texture strength33. Of the GLCM features, variance, cluster shade, cluster tendency, and cluster prominence varied highly across different MRI scanning settings for both the volunteer and phantom, implying that these features are associated with poor robustness. Yang et al.34 investigated on the impact of contouring variability on PET radiomics features in the lung. They reported that the impact of contouring variability is present to varying degrees. In this study, we used the same uniform ROI size for both the volunteer and phantom. Our results showed that some features vary more than other features with different settings. The reason is that each feature has its own formula to express its characteristics of the image and some features are dealing with pixel-wise changes such as NGTD features that describe the differences between each voxel and the neighboring voxels, while other features are dealing with overall (average) changes in an image such as sum average that quantify the mean of the sum histogram of an image. Although NGTD features and these four GLCM features are sensitive to different scanning parameters, they have high reproducibility if the parameters are kept the same. These features, therefore, may be useful for intrascanner studies with fixed protocol settings.

Figure 4
figure 4

The MRI phantom and the healthy volunteer’s brain (a) The MRI phantom consisting of 20 different materials. (b) A T1-weighted image of the phantom. (c) A T1-weighted image of the healthy volunteer’s brain with 20 ROIs.

Full size image

In this study, we performed several scans with various scanning protocol parameters such as NEX, slice thickness, phasing steps, and FOV for T1 and T2 respectively with a multi-center scanner. We also performed all scans twice for each setting to evaluate the reliability of scans. Although we limited the number of scans, our results of repeatability showed highly reproducible. For the repeatability measures, we computed the ICC for radiomics features obtained using the two MRI scanners and showed that the repeatability for the phantom was very high (average ICC, 0.963 and 0.959 for T1- and T2-weighted images, respectively) but that the repeatability for the volunteer was intermediate (average ICC, 0.865 and 0.849 for T1- and T2-weighted images, respectively). The repeatability of the volunteer is slightly lower than that of the phantom. This is not surprising, as humans have factors such as patient movement, respiration, and blood flow that can affect radiomics features, and also highlights the fact that phantom measurements alone are not sufficient for understanding variabilities in MRI-based radiomics features. Also, we showed that for the volunteer, the overall repeatability for T1-weighted images was slightly lower than that for T2-weighted images. Of note is that 39 radiomics features were highly reproducible for T1-weighted images of the volunteer, and 41 radiomics features were highly reproducible for T2-weighted images. The variability results for the normalization and filtering effect (Table 4) did not show much difference between them in average COV values.

We also found that radiomics features have different effects depending on the scanning parameters, which similar studies by other groups also demonstrated. For example, Ford et al.35 investigated the impact of pulse sequence parameter selection (i.e., echo time [TE] and repetition time [TR]) on MRI textural features of the brain. They found that the variability in radiomics features with the choice of pulse sequence and imaging parameters was feature-dependent and can be substantial. In another study, Saha et al.17 assessed the impact of various MRI scanner parameters on the radiomics features in breast MRI studies. They found that the feature group related to variation in fibroglandular tissue enhancement was the most sensitive to the scanner manufacturer and parameters.

Our study had some limitations. First, we could not remove the effect of the volunteer’s movement including blood flow, which influences radiomics feature values. We sought to minimize this effect by using an immobilization mask to fix the volunteer’s head in place during the scan. Also, we simulated a movement effect with the phantom on an MR image. For example, we shifted an image 1 mm to the right and generated a new image by averaging this shifted image with the original image to simulate an image for NEX 2. However, this simulation study did not change the radiomics feature values and does not explain the effect of the volunteer’s motion artifacts including blood flow. For repeatability measures, we took about a 30-min break between two scans for the volunteer. This may have resulted in uncertainties when the volunteer returned to the original position. In this study, we performed image preprocessing to reduce uncertainty in the feature analysis and used a uniform ROI size. However, there is an uncertainty remaining in the lesion segmentation procedure of the patient data, which may affect the suitability test for our phantom materials. Lastly, it should be noted that our previous study and other work reported volume-dependent and gray level-dependent features22,36, respectively. In the current study, Tables S2 and S3 are provided in the supplementary information to show the corrected formulas along with the original formulas for the volume-dependent and the gray level-dependent GLCM features, respectively. In this study, corrected formulas were used for the volume-dependent features (Table S2) but original formulas were used for the gray level-dependent GLCM features (Table S3). Please note that since our analysis is based on the same gray levels with various MRI parameter settings for GLCM features, different gray levels with different MRI parameter settings could have different results although the GLCM features in the large variation (COV > 30%) would still be in the same category. Also, it should be noted that our repeatability test will not be affected since the repeatability analysis used the same parameter settings.

In this study, we aimed to identify the robustness of MRI radiomics features with various scanning parameters and multi-scanner variation using an MRI radiomics phantom, which is very useful for calibrating, testing, and evaluating new MRI techniques and variability and repeatability measurements. In this study, we focused on the scanning parameters such as NEX, slice thickness, phasing steps, and FOV, which are the most commonly used in MRI scanning and we fixed all other parameters including filtering, smoothing, and coil sensitivity to avoid introducing other uncertainty factors in this study. We showed that all of the materials in the phantom were suitable by comparing its radiomics features with the patient data from the 97 T1- and T2-weighted MR images and investigated the robustness of various radiomics features with different MRI scanning protocols and two scanners.

Conclusions

In the present work, an MRI phantom was constructed with 20 MRI materials covering a wide range of radiomics feature values and several scans were performed with various scanning protocol parameters such as NEX, slice thickness, phasing steps, and FOV for T1 and T2 respectively. The ICC showed high repeatability for the phantom but intermediate repeatability for the volunteer, while the COV revealed little difference in variability between normalization and filtering effect.

We believe that this study is very useful for practice in the radiomics community, especially in MRI radiomics studies. Our results demonstrated that various radiomics features have different effects depending on the different scanning parameters and scanners. Furthermore, we identified the robust MRI radiomics features with various scanning parameters and multi-scanner variation using an MRI radiomics phantom. The radiomics features with a low COV and high ICC can be considered good candidates for MRI radiomics studies, whereas those with a high COV and low ICC must be used with caution.

Methods

MRI phantom and volunteer

An MRI phantom was created and used to investigate the repeatability and robustness in quantitative radiomics features with various MRI scanning protocol parameters, preprocessing (normalization and image filtering), and scanners. Figure 4 shows the MRI phantom, which was made of acrylic with dimensions of 14.5 × 17.8 × 10.3 cm. Inside the phantom, there were 20 cylinders and each cylinder had a diameter of 2.4 cm and length of 10.3 cm. The phantom could be filled with water through the hole on top of it (Fig. 4A). The MRI phantom was constructed of 20 MRI materials covering a wide range of radiomics feature values (Table 7).

Table 7 The 20 materials used in the MRI phantom.
Full size table

The phantom and the brain of the healthy volunteer were scanned using a 1.5 T Siemens MRI system (SIEMENS Magnetom Aera, Erlangen, Germany) with three-dimensional T1-weighted gradient echo sequence and T2-weighted fast spin echo sequence. A fixed TR (11 ms) and TE (4.77 ms) and flip angle of 30° with various scanning protocol parameters were used for T1-weighted images. For T2-weighted images, a TE of 281 ms, TR of 1530 ms, and flip angle of 160° with various scanning protocols were used. For comparison, scanning of the MRI phantom was also performed using a 1.5 T Philips MRI system (PHILIPS Marlin, Finland). For this scanner, a fixed TR (11 ms) and TE (4.61 ms) and flip angle of 30° were used for T1-weighted images, and a TE of 281 ms, TR of 1535 ms, and flip angle of 90° were used for T2-weighted images. We then varied the following scanning protocol parameters: number of excitation (NEX), slice thickness, phasing steps, and field of view (FOV). The detailed scanning protocols are listed in Table 8. Each scan was performed twice with the same setting for both scanners for the repeatability test. The phantom was removed from the scanner after the first scan and repositioned for the second scan. For the volunteer, the scan was also performed twice with the same setting and the volunteer took about a 30-min break between the two scans. The scans were performed each week for multi-scanner variability. In order to determine the variability from different scanning parameters and scanners accurately, we did not perform any intensity normalization on MR images to prevent another uncertainty on radiomics features or diminishing the effects of various scanning settings.

Table 8 The scanning protocols used with the Siemens and Philips 1.5 T MRI scanners.
Full size table

Patient data for the suitability test

First, we investigated the suitability of our phantom materials with brain lesions. A total of 97 patient data identified as having necrosis or progression of brain lesions were used to evaluate the suitability of each phantom material37. The use of all patient data were approved and written informed consent was waived by The MD Anderson Cancer Center Institutional Review Board. All MR images of these patients were acquired using a GE 1.5 T MRI scanner with a slice thickness of 5 mm, slice spacing of 6.5 mm, and field-of-view of 22 cm for T1- and T2-weighted images. The brain lesions were segmented on the post-contrast T1 images by a radiation oncologist because the lesions were easier to identify. The post-contrast T1 contour was then rigidly mapped to the other scan sequences such as pre-contrast T1- and T2-weighted images for each patient at each time point using the Velocity AI software (version 3.0.1; Varian Medical Systems, Atlanta, GA, USA).

Phantom and a healthy volunteer data for the repeatability and variability

For the repeatability and variability of the radiomics features, we used the features from the phantom and a healthy volunteer from two scans. All ROIs on the phantom and a healthy volunteer were delineated semiautomatically using a contour tool available with our in-house imaging software program IBEX23,38. Each ROI had a cylindrical shape with a diameter of 1.8 cm and a height of 10 cm for both the phantom and the volunteer. We used axial images where the height is along the z-axis. We used this uniform ROI size on MR images of the phantom and the volunteer to avoid uncertainty between the ROI size and radiomics features. Twenty ROIs on the phantom and volunteer’s brain were delineated (Fig. 4B,C, respectively); Twenty ROIs on a healthy volunteer’s brain were evenly selected over the brain. For patient data, each lesion on MR images for each patient was delineated by ValocityAI software (version 3.0.1; Varian Medical Systems, Atlanta, GA, USA). The radiation oncologist reviewed the contours on the MR images to ensure correct mapping and modified them when necessary.

In this study, we performed image preprocessing before extracting radiomics features to reduce uncertainty in the feature analysis; an edge-preserving smoothing filter was applied to the tumor volume before the feature calculations to preserve meaningful edge information while smoothing out undesirable imaging noise29. Then, we extracted a total 76 radiomics features from delineated ROIs from MR images of the phantom, volunteer, and patients, respectively. The radiomics features consisted of 7 Gradient orient histogram features, 22 GLCM features, 11 GLRL features, 31 intensity features, 5 neighborhood gray-tone matrix (NGTDM). The detailed features are listed in Table 9 and Table S1 in the supplementary information. All quantitative image features were calculated and extracted using IBEX23,38,39. This software was designed based on MATLAB (version 8.1.0; MathWorks, Natick, MA), and available at http://bit.ly/IBEX_MDAnderson. Our previous study and other work reported volume dependent and gray level dependent features22,36. In this study, corrected formulas were used for the volume-dependent features and original formulas were used for the gray level-dependent GLCM features as shown in the Table S2 and S3.

Table 9 The examined radiomics features extracted from delineated ROIs on MR images.
Full size table

Data analysis

First, we investigated the suitability of each phantom material to see whether the range of radiomics features of each material was similar to the range of radiomics features of the brain lesions of patients. This was done by comparing each feature value from the phantom with those from brain lesions using mean values ± 2 SDs, where this range covers 95% of an approximately normal data set and excludes outliers of the data. This brain lesions of patients only used for the suitability of the phantom materials. Next, we investigated the robustness of the radiomics features obtained from the 20 phantom materials in T1- and T2-weighted images using various scanning protocols and the two scanners. To assess the robustness of the various radiomics features with the different MRI scanning protocol parameters, the COV was computed for each radiomics feature in each scan using Eq. (1)

$$\mathrm{COV}=\frac{\sigma }{\mu }\times 100$$
(1)

where σ is the standard deviation and μ is the mean when applying different scanning settings for each MRI parameter (i.e., NEX = 1, 2, and 3).

Next, the repeatability of the radiomics features in two scans was investigated. This was performed with the Siemens 1.5 T MRI scanner twice under the same conditions, such as the same range of whole scanning parameter settings. The repeatability of the radiomics features extracted from normalized images was assessed using the ICC, a measure of the reliability of measurements that can demonstrate how strongly measurements with the same settings resemble each other. For our test–retest scheme with two repeated scans, the ICC was computed using Eq. (2) 40

$$ICC\left(\mathrm{1,1}\right)= \frac{BMS-WMS}{BMS+WMS}$$
(2)

where BMS is the between-subjects mean square and WMS is the within-subjects mean square. Therefore, the ICC considers the variation in repeated scans in relation to the total variation in the population40.