Temporal subtraction CT with nonrigid image registration improves detection of bone metastases by radiologists: results of a large-scale observer study

To determine whether temporal subtraction (TS) CT obtained with non-rigid image registration improves detection of various bone metastases during serial clinical follow-up examinations by numerous radiologists. Six board-certified radiologists retrospectively scrutinized CT images for patients with history of malignancy sequentially. These radiologists selected 50 positive and 50 negative subjects with and without bone metastases, respectively. Furthermore, for each subject, they selected a pair of previous and current CT images satisfying predefined criteria by consensus. Previous images were non-rigidly transformed to match current images and subtracted from current images to automatically generate TS images. Subsequently, 18 radiologists independently interpreted the 100 CT image pairs to identify bone metastases, both without and with TS images, with each interpretation separated from the other by an interval of at least 30 days. Jackknife free-response receiver operating characteristics (JAFROC) analysis was conducted to assess observer performance. Compared with interpretation without TS images, interpretation with TS images was associated with a significantly higher mean figure of merit (0.710 vs. 0.658; JAFROC analysis, P = 0.0027). Mean sensitivity at lesion-based was significantly higher for interpretation with TS compared with that without TS (46.1% vs. 33.9%; P = 0.003). Mean false positive count per subject was also significantly higher for interpretation with TS than for that without TS (0.28 vs. 0.15; P < 0.001). At the subject-based, mean sensitivity was significantly higher for interpretation with TS images than that without TS images (73.2% vs. 65.4%; P = 0.003). There was no significant difference in mean specificity (0.93 vs. 0.95; P = 0.083). TS significantly improved overall performance in the detection of various bone metastases.


1.
A large-scale observer study was performed for detection of bone metastases. Numbers of radiologists and patients were 18 and 100, respectively. 2. To validate the robustness of detection with TS images, the radiologists with a variety of backgrounds and the patients with various primary tumors and various bone metastases were included. To include various bone metastases, osteoblastic, osteolytic, intertrabecular, and mixed types of newly-developed and preexisting bone metastases at various locations were included. 3. Although the studies of Onoue et al. 18 and Sakamoto et al. 14 did not show significant improvement between with and without TS images, the current study shows significant improvement.

Materials and methods
This retrospective study was approved by the institutional review board (Kyoto University Graduate School and Faculty of Medicine, Ethics Committee), and requirement for informed consent was waived. This study conformed to the Declaration of Helsinki and Ethical Guidelines for Medical and Health Research Involving Human Subjects in Japan (https:// www. mhlw. go. jp/ file/ 06-Seisa kujou hou-10600 000-Daiji nkanb oukou seika gakuka/ 00000 80278. pdf).
Subject selection. Six board-certified radiologists (M.Y., M.N., T.K., Y.E., K.O., T.A.; all of these are authors of this paper) with 9-22 years of experience in interpreting CT images selected subjects meeting pre-defined criteria (Supplementary Information A) from a clinical database, sequentially scrutinizing CT images. Briefly, the criteria are as follows.
(i) The six board-certified radiologists included subjects with a history of malignancy who were examined with at least three CT studies (previous, current, and future CT). (ii) The subjects had a history of examinations of 18F-fluoro-2-deoxy-d-glucose positron emission tomography and/or bone scintigraphy which was performed for evaluation of bone metastases. (iii) Positive subjects (subjects with bone metastases) had at least one bone metastasis measuring 5 mm or more in diameter. (iv) Negative subjects (subjects without bone metastases) had no bone metastasis. Supplementary Information B shows the procedure of subject selection. With reference to images from CT and other imaging modalities, they selected 50 positive subjects and 50 negative subjects. Furthermore, they selected a pair of CT images (previous and current CT) for each subject that satisfied predefined criteria (see Supplementary Information A). Negative subjects were selected to match the background characteristics (e.g., age and sex) of positive subjects. The 6 radiologists detected and reviewed all suspicious lesions, and identified lesions over 5 mm or more to create the reference standard. Finally, lesions were determined to be bone metastases with sufficient confidence by consensus. In this procedure, future CT was used for confirming the reference standard.
The three-dimensional region of each bone metastasis was manually segmented on current CT images by consensus. Subject-and lesion-based attributes were investigated as shown in Tables 1 and 2, respectively. Table 1 shows that the CT scan conditions, such as slice thickness and use of contrast media, were different between previous and current CT in some subjects. The following CT scanners (Canon Medical Systems, Otawara, Japan) were used; Aquilion 16 (16-detector row CT), Aquilion 64 (64-detector row CT), Aquilion Prime (80-detector row CT), and Aquilion One (320-detector row CT). www.nature.com/scientificreports/ TS image generation. The process for generating TS images is almost identical to that of Onoue et al. 18 .
Previous CT images for each subject were non-rigidly transformed to match current CT images. The non-rigid image transformation was performed fully automatically. Subsequently, transformed previous CT images were subtracted from current CT images to generate TS images using the Intel Xeon E5-1650v4 processor (Clock, 3.50 GHz, number of cores 6; memory, 32 GB). Processing time for TS generation was recorded. Projection images, which were the average of the maximum and minimum intensity projections of TS images, were also generated to enable observers to immediately grasp osseous temporal changes across the whole area.
Observer enrollment. This experiment was a fully crossed multi-observer multi-subject study. Based on Sakamoto's study 14  Observer study. To reduce memory bias, observers were randomly assigned to two groups of equal size (n = 9). One group independently interpreted the image pairs for each subject first without and then with TS images. The other group interpreted the image pairs first with and then without TS images. The interval between two sessions without and with TS for each observer was more than 30 days. Moreover, the order of subjects was randomized for each observer. www.nature.com/scientificreports/ Observers used a medical monitor (Radiforce RX440, EIZO) and a dedicated image viewer ( Fig. 2) with multi-planar reconstruction and window level/width modification functions to view CT and TS images. To control practice effects, observers were trained to use the viewer with training data of ten subjects prior to the actual study. Observers were blinded to all clinical data except the age and sex of each subject and the interval between previous and current studies.
Observers were asked to mark the location of any suspicious lesions measuring 5 mm or more on current images and to rate the percentage likelihood of bone metastasis. The interpretation time for each subject was automatically recorded by the viewer excluding the time for rating. After interpretation of each subject, observers were asked to subjectively rate on a five-point scale the confidence level for their interpretation (1, very low; 2, low; 3, moderate; 4, high; 5, very high) and the usefulness of TS images (1, useless; 2, not very useful; 3, somewhat useful; 4, very useful; 5, extremely useful).
After completion of all assessments, the marked locations of lesions were compared against the reference standard for lesion identification. A lesion with a likelihood rating of 51% or higher was considered positive in lesion-based analyses. A subject with at least one positive lesion was considered positive in subject-based analyses. TS images were considered beneficial for identifying lesions where at least one observer could correctly identify and positively rate only with TS images. Meanwhile, they were deemed detrimental to identifying lesions where www.nature.com/scientificreports/ at least one observer could correctly identify and positively rate only without TS images. All false positives were further reviewed by the six radiologists.
Statistical analyses. JAFROC analysis 22,23 was conducted with JAFROC software with random-observersand-random-subjects models and the figure of merit (FOM) was calculated to evaluate overall observer performance. Sensitivity at lesion-based, false positive count (FPC) per subject, sensitivity and specificity at subjectbased, interpretation time, and confidence levels were compared between sessions (with TS images vs. without TS images) with the Wilcoxon signed rank test. SAS (Version 9.4, SAS Institute, Cary, North Carolina) was used for statistical analyses, and P < 0.05 was considered to indicate a significant difference.  Table 1 and Supplementary Information D. In total, the reference standard consisted of 160 bone metastases. Their detailed characteristics are shown in Table 2 and Supplementary Information E. TS images were generated for the image pairs of the 100 subjects. The mean processing time per image pair was 973 s (range 322-2310, standard deviation 405). TS images were not generated for one metastasis because it was out of the scan area of the previous CT image.

Results
Observer characteristics. All 18 enrolled observers were board-certified radiologists, with the following specialties in radiology: general radiology (n = 4), nuclear medicine (n = 2), neuroradiology (n = 2), cardiovascular radiology (n = 1), respiratory radiology (n = 2), upper abdominal radiology (n = 6), gastrointestinal radiology (n = 2), and urological radiology (n = 2). They had 10-36 years of experience in the interpretation of CT images. In clinical practice, they interpreted 3000 to 10,000 CT examinations each year. Two radiologists had previously used computer-aided diagnosis system. None had previously used TS-CT. Image interpretation. The 18 observers evaluated the 100 image pairs with and without TS. In total, 3600 reading sessions were performed. Figure 3 and Table 3 show the main results for image interpretation. Representative cases are shown in Figs. 4 and 5. Compared with interpretation without TS, TS images were associated with a significant increase in mean FOM from 0.658 to 0.710 (JAFROC analysis, P = 0.0027). Mean sensitivity at lesion-based was significantly higher for interpretation with TS compared with that without TS (46.1% [73.8 Median confidence levels ranged from 2 (low) to 5 (very high) for interpretations without TS and from 3 (moderate) to 5 (very high) for those with TS. The median ratings for usefulness of TS images ranged from 3 (somewhat useful) to 5 (very useful), indicating that all observers evaluated TS as useful.
Subjects were divided into subgroups according to the type, location, and preexistence of bone metastases (Table 4). Sensitivity with TS was higher than or equal to that without TS for all subgroups. The gain in sensitivity for interpretation with TS compared with that without TS was small in metastases in the scapulae. Moreover, the gain for metastases in extremities was zero because sensitivity for both interpretations without and with TS were also zero.

Effects of TS images on metastases detection.
Of the 160 metastases, a beneficial effect of TS images was observed for 118 and a detrimental effect was observed for 82. In particular, there were eight notable metastases for which detection was improved by TS images for 10-15 of the 18 observers, while a detrimental effect was observed for 0-1 observer. These metastases comprised not only three small metastases but also five larger ones, measuring 21.8-32.9 mm. These larger lesions were "lost" on current CT images, disguised by commonlyobserved degenerative changes and sterically complex structures of the sternum, ribs, or pelvic bones. . When an observer clicks on a suspicious lesion, the dialog box appears to rate its likelihood (low to high) of being a bone metastasis. These representative images are obtained from a 55-year-old male patient with renal cell carcinoma who developed two osteolytic metastases in a thoracic vertebra (red circle) and the left iliac bone (blue circle). Both metastases are clearly visualized. www.nature.com/scientificreports/ In contrast, there were seven notable metastases for which detection was detrimentally affected by TS images for 5-8 observers, while a beneficial effect was observed for 0-2 observers. These metastases resembled commonly-observed benign findings on TS images, especially the projection images, such as degenerative changes of the vertebrae and joints, healing fractures of the ribs and pelvic bones, and subtraction artifacts around the scapulae.

Confidence levels
Scientific Reports | (2021) 11:18422 | https://doi.org/10.1038/s41598-021-97607-7 www.nature.com/scientificreports/ The review of false positive marks without and with TS images identified 161 and 212 bone lesions, respectively. In most of the false positives (n = 130 without TS and 148 with TS), the number of observers who marked was one, while the number was 5 to 15 in some lesions (n = 7 without TS and 23 with TS). It is speculated that these lesions represent degenerative changes (n = 4 without TS and 10 with TS), healing fractures (n = 1 without TS and 6 with TS), post-operative changes (n = 1 without TS and 1 with TS), and other benign bone lesions (n = 1 without TS and 6 with TS) such as bone islands.

Discussion
This study investigating the effects of TS on bone metastases detection in CT images indicated that TS images could be made available at follow-up CT without any extra physical burden on patients. Moreover, TS images significantly improved overall performance in detection of various types of bone metastases at various locations by radiologists without additional interpretation time. This study recruited a relatively large number of radiologists to assess CT images from a large number of subjects. Furthermore, considering the frequency of CT scans in oncology patients, we believe that our TS method could bring considerable benefit to clinical diagnostic imaging. This is the first study to report a significant improvement in overall radiologist performance at detecting various types of newly-developed and preexisting bone metastases at various locations by using TS images. Table 4 suggests that TS was beneficial for all types of bone metastases unlike 18 F-fluoro-2-deoxy-d-glucose positron emission tomography or bone scintigraphy, which are reported to only have benefits for specific metastases [25][26][27] . Moreover, TS retains the advantages of CT, which has finer resolution and is more frequently performed in oncology patients than other imaging modalities. All these advantages are essential for earlier detection of bone metastases.
TS method is clinically applicable because our study evaluated TS images without excluding subjects for inconsistencies between previous and current CT images in posture, breathing depth, and other study attributes (Table 1), which are inevitable with real-world application. Furthermore, these results were obtained with the 18 radiologists who have various backgrounds and no previous experience of TS for bone metastasis detection. Moreover, TS is likely to be accepted by radiologists based on their usefulness ratings. As such, clinical application of TS could enable early detection of bone metastases, reducing SRE and cancer-related mortality and improving quality of life of cancer patients. www.nature.com/scientificreports/ There were some detrimental effects of TS on detection, which can presumably be attributed to conspicuous visualization of commonly-observed degenerative and traumatic changes, premature judgment of such changes or bone metastases on TS images, and abbreviated observation of CT images based on this judgment. To minimize these effects, radiologists should be educated about TS. Some visualization aids might also be helpful to minimize such effects, including image fusion and synchronized scrolling to assist radiologists in exploiting both CT and TS information.
It was observed that sensitivity for intertrabecular metastases was lower than that for other types even with TS. Although TS improved sensitivity, the improvement was smaller presumably due to a smaller density change. To increase the advantage gained with TS images for such metastases, computer-aided detection might be developed.
By its nature, the TS method used here exploits follow-up CT images and requires prior images, which are unavailable at initial imaging assessments. In such situations, another modality should also be considered because some cancer patients already have bone metastases at initial diagnosis [28][29][30] . However, follow-up evaluations, as well as detection of bone metastases, are important for their management, including the prevention of SRE. Based on Table 4, TS appears to assist radiologists in identifying both preexisting and newly-developed bone metastases. Follow-up evaluation of bone metastases with CT is generally considered difficult in some cases 31 . Further research is therefore required to investigate the use of TS for follow-up evaluations.
Although the processing time in this study was much shorter than that of Sakamoto's study 14 , it would be preferable to further shorten it for clinical application of TS, especially in emergency CT assessments for SRE. According to preliminary results using in-house software, processing time can be reasonably expected to be reduced to less than 10 min with the use of a graphics processing unit.
There were several previous studies for investigating the usefulness of TS for detection of bone metastases [14][15][16]18,21 . To the best of our knowledge, the current study was the first to show that TS was useful for detecting bone metastases even when inconsistent CT sets (such as slice thickness) were included for generating TS.
There were several limitations to this study. First, despite repeated scrutinization of CT images by the 6 board-certified radiologists, reference to all available images including those obtained after current images, and determination with sufficient confidence by consensus, the definition of the reference standard might be incomplete because any use of clinical information other than images was not accepted by the Japanese regulatory body (Pharmaceuticals and Medical Device Agency). This study was conducted as a clinical performance www.nature.com/scientificreports/ test for which the results were to be submitted to the body for approval of TS for clinical use 32 . Although TS images would have considerably assisted the definition of the reference standard, they were not referred to for the definition. Second, TS effects were not sufficiently evaluated for metastases in the skull, scapulae, and extremities due to the small number of subjects with these metastases. All three metastases in extremities happened to be too difficult to detect and differentiate with CT without reference to other modality images. Therefore, further studies focusing on specific types of metastases are also required. Third, the effect of bone metastasis therapy on detectability with TS was not examined in the current study. The therapy can change CT density of bone metastases 33,34 . Therefore, detectability with TS may be changed with the bone metastasis therapy. Because the access of medical records was severely restricted in performing the current study 32 , we could not examine the effect of bone metastasis therapy on detectability with TS.
In conclusion, TS images obtained from serial CT scans using nonrigid image registration significantly improved radiologist performance in the detection of bone metastases.