Reliability and Influence on Decision Making of fully-automated vs. semi-automated Software Packages for Procedural Planning in TAVI

Precise procedural planning is crucial to achieve excellent results in patients undergoing Transcatheter aortic valve implantation (TAVI). The aim of this study was to compare the semi-automated 3mensio (3 m) software to the fully-automated HeartNavigator3 (HN) software. We randomly selected 100 patients from our in-house TAVI-registry and compared aortic annulus and perimeter as well as coronary distances between 3m-measurements and post-hoc HN-measurements. Finally, we retrospectively simulated prosthesis choice based on HN-measurements and analyzed the differences compared to routinely used 3 m based strategy. We observed significant differences between the two software packages regarding area (3 m 464 ± 88 mm², HN 482 ± 96 mm², p < 0.001), perimeter (3 m 77 ± 7 mm, HN 79 ± 8 mm, p < 0.001) and coronary distances (LCA: 3 m 13 ± 3 mm, HN 12 ± 3 mm, p < 0.001; RCA: 3 m 16 ± 3 mm, HN 15 ± 3 mm, p < 0.001). Prosthesis choice simulation based on newly obtained HN-measurements would have led to a decision change in 18% of patients, with a further reduction to 4% following manual adjustment of HN-measurements. The fully-automatic HN-software provides higher values for annular metrics and lower annulus-to-coronary-ostia distances compared to 3m-software. Measurement differences did not influence clinical outcome. Both, the HN-software and the 3m-software are sophisticated, reliable and easy to use for the clinician. Manual adjustment of HN-measurements may increase precision in complex aortic annulus anatomy.

www.nature.com/scientificreports www.nature.com/scientificreports/ inter-observer variability, its semi-automatic nature comes along with a certain time effort 5 . Therefore, particular efforts were targeted towards the development of fully-automatic tools for procedural planning in TAVI. The HeartNavigator ((HN), 3.0, Philips, Amsterdam, NL) was released recently, providing all important planning measurements like annulus area and perimeter, coronary ostia distances as well as optimal implantation angle in a fully-automatic fashion with highest reproducibility. Although, HN covers all aspects of device landing zone assessment in a reliable and sophisticated manner, it lacks a tool for access site planning so far.
In this manuscript, we sought to compare the new and fully-automatic HN package to the established and semi-automated 3 m image analysis tool regarding the impact on procedural planning for TAVI.

Results
Baseline and procedural characteristics are summarized in Table 1. The implantation strategy solely depended on the semi-automated 3m device landing zone assessment. Semi-automatic measurements using the 3m software were achieved in all cases. Fully-automatic measurements were obtained in 99% (99/100 patients) without any problem. In one MSCT series segmentation needed to be manually corrected: in this series, contrast dye was predominantly on the right heart side with a low concentration on the left heart side. Instead of the aorta, the pulmonary trunk has been segmented.
Agreement of semi-automated and fully-automated measurements. Measurement results and agreement metrics are summarized in Table 2. Bland-Altman plots of agreement are shown in Fig. 1. All measured parameters were significantly different among both software packages. Measurements of annular area (HN 482 ± 96 mm², 3 m 464 ± 88 mm², p < 0.001) and annular perimeter (HN 79 ± 8 mm, 3 m 77 ± 7 mm, p < 0.001) were consistently and significantly larger when obtained with the fully-automated HN software package, compared to the semi-automated 3 m software package. In contrast, the annulus-to-coronary-ostia distances were consistently larger when measured with the semi-automated 3 m package compared to the HN package (LCA: 3 m 13 ± 3 mm, HN 12 ± 3 mm, p < 0.001; RCA: 3 m 16 ± 3 mm, HN 15 ± 3 mm, p < 0.001) (Fig. 2). The agreement of annular metrics, as quantified with the concordance correlation coefficient, was markedly higher than the agreement of the annulus-to-coronary-ostia metrics. Among the annular metrics, the measurement of the area shows the highest agreement of both methods. Interobserver agreement of 3m-measurements was obtained from three independent measurements of 50 randomly selected MSCT studies (Fig. 3) and is summarized in Table 3. The agreement of the annular metrics was higher than the agreement of the annulus-to-coronary-ostia distances. This was mainly driven by a lower precision (degree of variation) 6 . Accuracy (degree of location or scale shift) was high in both, the annular metrics measurements as well as the annulus-to-coronary-ostia distances. Analysis of intraclass correlation coefficient (ICC) supported the high interrater reliability of 3m-measurements ( Simulation of TAVI and impact on strategy. Based on our TAVI sizing strategy (Appendix B) we retrospectively simulated prosthesis choice considering the HN-measurements and analyzed the differences with the actual used 3 m based strategy. Consistent with the statistically significant larger annulus dimension measurements, prosthesis choice based on HN would have led in 13 out of 100 cases (13%) to a decision-change to a larger prosthesis. Interestingly, in 4 out of these 13 patients, the operator opted for one size larger based on clinical intuition, considering sinus dimensions, calcification patterns, other anatomical factors and gut feeling. In those where HN-measurements would have suggested a smaller prosthesis size selection (5 cases), in none of the cases a change of strategy based on clinical intuition was observed. The retrospective TAVI simulation and the impact on the implantation strategy are illustrated in Fig. 4. The overall rate of changes in valve size was reduced to 4%, by applying manual adjustment of HN-measurements.

Discussion
The present study represents the first head to head comparison of the fully-automated HN software package and the semi-automated 3 m software package in terms of important CT-measurements for procedural planning prior TAVI. The main findings are: (1) the fully-automatic sizing tool HN-measures significantly and consistently larger annuli, but smaller annulus-coronary-ostia distances compared to the established semi-automated 3 m tool; (2) the agreement between HN and 3 m is high for annular metrics (area and perimeter) and moderate for annulus-to-coronary-ostia distances; (3) a retrospective simulation reveals that differences in annular metrics of HN compared to 3 m would have translated in a larger prosthesis-size selection in 13% and in a smaller prosthesis-size in 5% of the cases; (4) the overall rate of changes in valve size can be reduced to 4% with manual adjustment of HN-measurements; (5) the identified measuring differences do not translate into a higher rate of severe complications such as annular rupture or valve migration. Taken together, our findings suggest that both the HN and the 3 m software represent a sophisticated, reliable and easy to use option for procedural planning in TAVI.
The rapid technological evolution of devices and the gain of interventional experience, let to a reduction of life threatening complications after TAVI over time 7 . Nevertheless, rigorous and accurate risk stratification including anatomical aspects of the device landing zone remains essential. Multi-slice-computed-tomography represents a powerful tool to specifically assess the 3D shaped aortic annulus, the left ventricular outflow tract as well as annulus-to-coronary-ostia distances, in order to prevent severe procedural complications 8 . Nowadays, several software packages are available for this purpose. Recently, the fully-automated HN software was established, with the advantage of automatic annular assessment and transcatheter heart valve prosthesis selection in less time compared to traditional CT-measurements 9 . Although, the 3 m and the HN software packages are used www.nature.com/scientificreports www.nature.com/scientificreports/ N = 100 mean/count (%/std) Continued www.nature.com/scientificreports www.nature.com/scientificreports/ routinely for procedural planning prior to TAVI, our study is the first, providing comparative results investigating a real-world all-comers population.
The present study shows statistically significant higher values for aortic annulus area and perimeter using the HN software, while annulus-to-coronary-ostia distances are lower, compared to 3 m. We hypothesize that these differences are associated to a slightly different annulus plane definition of both softwares. In case of the semi-automated 3 m software the annulus plane is defined according to manually placed reference points at the corresponding nadirs, while HN follows a standardized algorithm were the reference points are placed under the first CT-slice were aortic valve tissue disappeared. The exact reason for these differences cannot be fully clarified, based on our analyses. Nevertheless, we may hypothesize that there is a tendency to define the annular plane more towards the left ventricular outflow tract based on deeper manual placement of the nadir reference points using the 3m-software (Appendix A1), compared to the HN software (Appendix A2). Due to the conical form of the left ventricular outflow tract, a lower definition of the annular plane may lead to smaller annulus area and perimeter as well as to higher annulus-to-coronary-ostia distances. In this context, it is particularly important to emphasize that very small differences in the annular plane definition lead to relatively large differences in annulus area and perimeter, even though these differences are too small to be noticed in a side-to-side visual comparison of both softwares (Appendix A3).
Although our results demonstrated significant measurement differences between the two softwares, both provided the same results in terms of prosthesis size selection in 82% of the cases. Differences in annulus area and perimeter only turn into therapeutic consequences when they are closely located to the cut-off point between two different valve sizes. Interestingly, in the 13 patients where a HN sizing approach suggested a larger prosthesis size compared to the original 3m-measurement, the operator opted in 4 patients (31%) for one size larger based on her/his clinical experience. For the 5 patients were HN-measurements would have suggested a smaller prosthesis size selection, no change of strategy based on clinical intuition was observed. These results stress the importance of a patient-specific procedural planning considering sinus dimensions, calcification patterns and other anatomical factors, with annulus area and perimeter only being one component of the decision-making process. The fact that the agreement between 3 m and HN regarding valve size selection increased to 96%, when manual adjustments of the HN-measurements were applied, is indicating the importance of reviewing the appropriateness of automatically obtained HN-measurements, especially in cases of complex annular anatomy (e.g. severe annular calcification) or reduced CT quality. Fatal complications such as prosthesis migration or annular rupture because of severely wrong annular assessment were not observed in our cohort. Therefore, both softwares appear to be reliable tools for procedural planning prior to TAVI.
Although, we confirmed a robust annular measurement using the semi-automated approach, with a relatively high accuracy and precision, it is not as reproducible as the deterministic algorithm of the fully-automated software. The advantage of the 3m-software is a full assessment workflow, where a detailed device landing zone assessment and access planning can be performed. But this comes at the price of a more demanding user experience. On the other hand, the HN software offers a very easy, highly reproducible, fast and reliable device landing N = 100 mean/count (%/std) www.nature.com/scientificreports www.nature.com/scientificreports/  www.nature.com/scientificreports www.nature.com/scientificreports/ zone assessment, which however comes at the price of the currently lacking integrated access planning workflow. Nevertheless, a manual segmentation of the access route with individual measurements can be edited easily.
Limitations. We provide robust data regarding CT-based measurement differences between the HN and the 3 m software. Nevertheless, the following limitations merit to be mentioned: 1) The present study is a non-randomized single center study, with all the disadvantages coming along with such a study design. Although the interpretation of clinical outcomes is limited, our results are hypotheses generating for future investigations, ideally with a randomized study design, in order to draw final conclusions; 2) The impact of softwares on the postoperative grade of paravalvular regurgitation was beyond the scope of the study, and was also described in recently published manuscripts already 9 . However, in a prospective setting, we would consider to include paravalvular regurgitation as an endpoint in the analyses; 3) The inclusion of different valve types in an overall analysis may have a potential bias on the results and no valve specific conclusion (i.e. self-or balloon expandable) can be drawn from our analysis; 4) Results of semi-automated 3m-measurements may differ among raters. However, based on the high interobserver reliability we assume no significant influence on comparative results of our study.  Table 3. Agreement of 3mensio measurements. To assess the agreement of the semi-automated 3mensio measurements aortic valve annular area and perimeter and coronary distances to the aortic valve annular plane were measured by three independent observers on the same patient and according to the same guidelines, respectively. The table lists for each measurement site the overall CCC which summarized both of its components the precision (degree of variation) and the accuracy (degree of location or scale shift) and the Friedman rank sum p-value for repeated measurements.
www.nature.com/scientificreports www.nature.com/scientificreports/ Conclusion. In summary, HN-software provides higher values for annular area and perimeter but lower annulus-to-coronary-ostia distances compared to the 3m-software. However, the identified measuring differences do not translate into a higher rate of severe complications such as annular rupture or valve migration in our cohort. Both available software tools are sophisticated, reliable and easy to use in daily clinical routine. Although, the HN-software is fully-automated, a small percentage of patients may benefit from manual adjustment, especially in complex annular anatomy. With still ongoing technical improvements of CT scans as well as softwares, procedural planning in TAVI will be further facilitated.

Methods
Patients. We randomly selected 100 patients out of all TAVI patients within in the last 12 month (March 2016 -February 2017) from our in-house prospective transcatheter aortic valve registry. Patients without a complete MSCT record were excluded from the selection process. Other exclusion criteria were not applied. Pre and post-operative characteristics of the study cohort are listed in Table 1. The study complies with the declaration of Helsinki and was approved by our institutional review board (Charité's Ethics Committee, EA1/062/19). Informed consent was obtained from all patients.
Multislice computed tomography and landing zone assessment. For device landing zone assessment and access planning cardiac MSCT examinations of the aortic valve, the thoracic aorta and the iliofemoral vessels were acquired with a dual source 2 × 128 slice scanner (Siemens Medical Solutions, Erlangen, Germany) using a standardized protocol which is optimized for subsequent device landing zone analysis. In a minority of the cases the MSCT examination was not performed in our hospital, but was already performed in the referring center. In all cases the semi-automated software application 3 m (3 mensio 8.0, Pie Medical Imaging BV, The Netherlands) was used for the assessment of the device landing zone and access site. All measurements were performed exclusively by an experienced member of our dedicated TAVI team. The implantation strategy of all cases was based on this semi-automated analysis.
Device landing zone assessment and manual parameter measurement was performed after the definition of the virtual annular plane as shown in Appendix A. In this paper we focused on the measurements of the annular area, annular perimeter and the left and right coronary distances to the annular plane.
In order to obtain comparative results between the semi-automated and the fully-automated software package, we assessed the four mentioned parameters for this study sample again with the fully-automated HN (HeartNavigator 3.0, Philips, Amsterdam, NL). In brief, a diastolic cardiac MSCT series is loaded into the HN application. The segmentation, the definition of the virtual annular plane and the measurements are subsequently performed fully automatically.
Simulation of TAVI prosthesis choice and its impact on implantation strategy. We simulated the choice of prosthesis size for TAVI by assuming a fixed prosthesis choice. That means, when for a patient in our sample prosthesis A has been chosen, then we restricted the sizing boundary to that prosthesis model for the HN-measurements. The detailed sizing algorithm is described in Appendix B. We analyzed the hypothetical impact on implantation strategy given the results of the two different measurement approaches. Patients where a change in valve size was proposed based on HN-measurements were reanalyzed applying manual adjustment, where appropriate.
Statistics. Continuous variables were expressed as mean ± SD. Categorical data were presented as numbers and percentage. Comparison of the differences in annular dimensions and coronary distances were analyzed with the paired t-test. The agreement between the HN and 3m-measurements was analyzed with the Bland-Altman test and further quantified with the concordance correlation coefficient 10 . For quantification of interobserver variability of the semi-automated 3m-measurements, 50 randomly selected patients within our study cohort were completely and independently analyzed again by two physicians with profound 3 m experience and who were . Retrospective TAVI prosthesis sizing simulation. The waffle plot is showing each sizing decision and the difference to the actual 3mensio based sizing is color-coded. In 73 of 100 cases no change (light-green) of the sizing strategy was observed. In 9 cases the HeartNavigator measurements were not in the valid sizing range of the prosthesis which was actually used during implantation (i.e. annulus area was too big for a sapien 329 mm prosthesis) -depicted as dark-green squares. In 13 cases HeartNavigator based sizing would have led to a bigger (light-red) and in 5 cases to a smaller prosthesis (dark-red).
www.nature.com/scientificreports www.nature.com/scientificreports/ blinded to the clinical data. We then analyzed the degree of agreement of the original 3 m case planning analysis and the two new analyses by calculating the overall concordance correlation coefficient, which represents both precision (degree of variation) and accuracy (degree of location or scale shift), and the Friedman rank sum test. The interobserver reliability of 3m-measurements was further assessed using intraclass correlation coefficient (ICC). A two-tailed P-value of <0.05 was considered significant. Statistical analyses were performed using R 3.3.2 (R Development Core Team. R: A Language and Environment for Statistical Computing 2011); data preparation, processing and plotting was supported by the tidyverse package.