Repeatability and Reproducibility of Quantification of Superficial Peri-papillary Capillaries by four Different Optical Coherence Tomography Angiography Devices

This study was performed to test the repeatability and reproducibility of measurements of peri-papillary capillaries from four optical coherence tomography angiography (OCTA) devices. 109 healthy eyes were imaged with four OCTA devices (Spectralis, Optovue, Triton and Cirrus). A 3 × 3 mm scan pattern centered on the disc was repeated twice by each device. En face images of superficial capillary plexus were screened and processed for calculation. Vessel length density (VLD) was calculated on four equally divided parts of a ring between two concentric circles manually centered on the disc. General linear model (GLM) was used to test the impact of device and location on VLD. Intraclass correlation coefficient (ICC) of VLD between repeated scans was calculated. Of 218 acquisitions, 36%, 92%, 76% and 88% were eligible for analysis from Spectralis, Optovue, Triton and Cirrus, respectively. ICC was 0.94, 0.90, 0.84 and 0.87 for the four devices. GLM showed measurements significantly varied among devices (P < 0.001) and locations (P < 0.001). Pairwise comparison showed Triton = Spectralis >Optovue >Cirrus, and temporal = nasal >superior = inferior in measuring capillary VLD. This study revealed the repeatability of measuring peri-papillary capillaries was high for all four devices, while the reproducibility among the machines was unfavorable.

advanced OCT, Nidek Co. Ltd. Japan). By far, no data was available on the comparison of different OCTA devices in measurement of pericapillary vasculature. This is particularly important if different OCTA devices are involved in a multi-center clinical trial for whatever disease affecting peri-papillary capillaries.

Reproducibility.
For the comparative study, we had achieved at least one adequate image from each of the four machines on 39 eyes (31 females). The mean age was 27 ± 8 (21-50) years and the mean axial length was 24.8 ± 1.1 (21.8-26.6) mm, with 11 eyes included in group 2 (AL ≥ 25.5 mm). The mean VLD and the CV among individuals on the four locations for each instrument was listed in Table 1. The general linear model showed that instrument (F = 69, P < 0.001) and location (F = 35, P < 0.001) had significant impact on the measurement of VLD of peri-papillary capillaries, while AL did not affect the outcome measurements significantly. Pairwise comparison in this model demonstrated that Triton and Spectralis detected the most skeletonized vessels, followed by Optovue, and Cirrus detected the least ( Table 2). The estimated means of VLD measured by each instrument on different locations is displayed in Fig. 2. The measurements approximated in superior and inferior peri-papillary area, which were significantly less than that in the temporal and nasal area, which were also similar ( Table 2). The CVs and ICCs between each pair of instruments on the four locations are shown in Table 3.
Correlation between axial length and VLD. Pearson's correlation tests (Table 4) showed that the axial length was positively correlated with vessel length density on temporal peri-papillary area for both Optovue and Triton. There was a weak positive correlation on superior location for Spectralis and on inferior location for Cirrus.

Discussion
This study showed that all the four examined OCTA devices had high repeatability, suggesting their reliability in measuring peri-papillary capillaries, provided that adequate quality and centration of image was achieved. Generally speaking, the CVs between pairs of instruments were approaching or even higher than that among participants, indicating a low reproducibility among different types of machines.
In previous studies, the repeatability between scans has been reported to be high on individual commercialized instruments [11][12][13][14][15] . However, few studies compared their repeatability on the same cohort. It turned out in our study that the four instruments did not vary significantly in terms of repeatability (95% confidence interval of ICC overlapped). Although the ICC of Spectralis is the highest (0.94), variation of measurements among individuals was also the highest among the four machines, which might contribute to the high ICC. We also noticed that the proportion of eligible images was quite low for Spectralis. The main explanation maybe the long acquisiting time, which could lead to more motion artifacts and blinks. Besides, the participants would feel too stressed to cooperate well on Spectralis after two repeated scans, when the technician wished to get more scans if the previous one was unsatisfied. Another reason is that large patch of dark shadow was more frequent for Spectralis probably due to severe segmentation error or blink.  Table 2. Pairwise comparison of peripapillary vessel length density among different instruments and locations based on estimated marginal means using a general linear model. Munk et al. 9 reported a similar macular superficial vessel density measured from those four instruments. However, although we examined exactly the same area on the four devices, there were still large disparities among them, and there was no overall good agreement among them. Thus, we suggest using the same instrument in future multi-center studies on peri-papillary capillaries. We believe the main reasons are the different calculating algorithms and segmentations of the peri-papillary superficial layer for each OCTA device. The definitions of the segmented layers for each machine varied somehow. But we didn't manually adjust the segmentation, so that it provided a real-world data. When looking into each device closely, we found that the measurement from Spectralis disagreed with the remaining devices, so did the measurement between Triton and Cirrus. Agreement between Optovue and Triton was fairly good on each of the four locations (ICC ranging from 0.75 to 0.89), which could result from a similar definition of the segmentation. Agreement between Optovue and Cirrus on superior and nasal area was also acceptable, but not on temporal or inferior area. Noticeably, the peri-papillary retinal atrophy at the temporal and inferior side of the disc margin was not uncommon in this cohort because of the myopic shift. In some cases, the two segmentation lines overlapped in this atrophic zone for Cirrus and Spectralis resulting in a dark area. However, for Optovue and Triton, the lower segmentation boundary was a fixed offset below the ILM, which could result in the detection of choroidal capillaries and increased vessel density in this zone. This may also explain why there was a large variation among healthy participants on the temporal location for Cirrus (CV = 22%). These findings might be meaningful for future OCTA studies, especially on Asian population, due to the high prevalence of myopia.
Overall, Triton and Spectralis detected more skeletonized capillaries and Cirrus detected the least. We did not use the capillary perfusion density (PD) in this study because a blurred image or a decreased resolution might give a false perception of increased vessel density, which has been demonstrated in a study comparing the 3 × 3 mm and the 6 × 6 mm modes using Angioplex 12 . The PD within the central 3mm-ring measured by 6 × 6 mm mode was higher than that by 3 × 3 mm mode, while the VLD was lower in the same area by 6 × 6 mm mode. In our study, we used a multi-scale Hessian filter to decrease the noise and also eliminated the influence of large vessels. Thus, a higher amount of VLD could result from a better ability in detecting capillaries. More interestingly, the sequence of the amount of VLD seemed to be in accordance with the sequence of the A-scan rate for the four instruments.The swept source OCT with a longer wavelength and higher A-scan speed might have advantage in detecting peri-papillary capillaries. Previous studies have revealed that the OCTA with swept source and longer wavelength could define a larger area of choroidal neovascularization compared to a spectral domain OCTA 16,17 .   In this study, we also found out that the location significantly influenced the measurements, with higher VLD in the temporal and nasal peri-papillary areas, which could be explained by a lessdistribution of large vessels in these areas. From Fig. 3, we could see that the mean estimated VLD in the temporal area was higher than that in the remaining areas for Triton and Optovue. As we have mentioned above, choroidal capillaries could be segmented into the studied layer for eyes with peri-papillary atrophy.
Previous studies have demonstrated a negative correlation between peri-papillary vessel density and elongation of axial length [18][19][20] . In this study, we also tested the influence of axial length on all four different types of OCTA instruments. It turned out that the measurements were not significantly affected by variation of axial length. Compared to previous studies, we only included healthy participants, and only a small proportion of them have high myopia. Besides, eyes with obvious myopic maculopathy or with vitreous haze were excluded. When a specific peri-papillary location was evaluated separately, we found that the vessel length density at temporal side had a positive correlation with the axial length for Optovue and Triton, which might be explained by the enhancement of choroidal capillaries in this area with retinal atrophy commonly seen in myopic eyes. There was also a weak positive correlation on superior side for Spectralis and inferior side for Cirrus. We did not correct the magnification from elongation of the eyeball, which could result in higher calculation readings of the vessels.
There were several limitations of our study. One major concern is the measurement of vessels was based on the intensity of pixels. Though denoising was applied during image processing, it is still possible that the non-vascular signal was calculated as vessel density. Thus, a higher vessel length density does not necessarily mean a better resolution. Another limitation is the small sample size for the multiple comparison, although the sample was larger than previous comparative studies on different machines. Finally, the majority of the participants were young females, thereby we could not evaluate the influence of age or gender on the measurements of each device.
Despite those limitations, our study is not without strengths. We've taken strict measures to exclude images with inadequate quality in order to minimize the influence of artifacts that might lead to incorrect calculation. And that is why only 39 out of 109 eyes were finally included in the study. In conclusion, our study revealed that the repeatability was high for all the four instruments, reassuring their reliability in evaluation of peri-papillary capillaries. The reproducibility among the four instruments was unfavorable, while the agreement between Optovue and Triton was fairly good, usually with a higher measurement of VLD for the latter. Longer axial length ( ≥ 25.5 mm) did not influence the measurement. However, the increased VLD in the temporal peri-papillary area with longer axial length for Optovue and Triton could result from the atrophy in this area and choroidal capillaries could be segmented into this layer. Those findings may provide valuable information for future clinical trials and practices.

Methods
Study Design and Image Acquisition. In this cross-sectional study, healthy volunteers over 18 years of age with no history of systemic or eye diseases were recruited. Best corrected visual acuity of all eyes was at least 20/20. Eyes with posterior vitreous detachment and a Weiss's ring anterior to the disc were excluded. High myopia was not listed in the exclusion criteria, however, the severity of myopic maculopathy should be no more than category 1 (tessellated fundus only) 21 . Any myopic retinal lesion that beyond category 1 or there is "plus" lesion 21 was excluded. All studied eyes were divided into two groups according to their axial lengths (AL) (group 1, AL < 25.5 mm; group 2, AL ≥ 25.5 mm). Of all enrolled participants, one eye was randomly chosen and was imaged using four OCTA devices in a random sequence. The four devices were Spectralis OCT2 module (version 6. Image Processing. En face images of peri-papillary vasculature were generated automatically by each instrument software. Retinal slab of superficial capillary plexus (defined by the instrument as region spanning from the internal limiting membrane (ILM) to the inner plexiform layer on Spectralis and Cirrus) was used in the study. The slab of optic nerve head was used on Optovue (spanning from ILM to a boundary 150 μm below the ILM) and Triton (spanning from ILM to a boundary 130 μm below the ILM). No segmentation was manually corrected.
Image with inadequate quality was excluded if any of the following criteria was met. (1) there was obvious artifact identified as saccades (horizontal misalignment of vessel segments) ≥3, or at least one patch of dark shadow that obscures retinal vessels outside the disc area. (2) Disc central deviation prevented from achieving the target area after matching the four images from the different machines. (3) The signal strength was below the minimum demand (described above). A resident in ophthalmology (C.W.) went through all the images and screened for adequate ones according to the image exclusion criteria. Questionable images (the first grader had less than 90% of the confidence in judging the image quality) were reviewed by a retinal specialist (J.L.) and a final decision was made through open adjudication.
The scan with a better image quality from the two repeated acquisitions was included in the comparative study so that each study eye had four en face images of superficial retinal vasculature centered on the disc from the four different instruments. The four images were exported and transformed to the same size (1024 × 1024 pixels corresponding to 3 × 3 mm).
Image J software (U. S. National Institutes of Health, Bethesda, Maryland)was used for all image processing. Registration was conducted by manually matching four distinct anatomical features (for example, bifurcation of a certain vessel) so that the four images displayed exactly the same area (Fig. 3, upper row). The average intensity was also adjusted to the same level for the four images. Then, the handled image underwent Hessian filter (smoothing = 3) followed by an auto-threshold (Otsu) to get the initial binarized image. Large vessels were also extracted using Hessian filter (smoothing = 15) and auto-threshold (Otsu), followed by removing outliers (radius = 15). The large vessels were then subtracted from the initial binarized image to get the binarized image of peri-papillary capillaries (Fig. 3, middle row).
Skeletonization was further applied to get a single line of capillaries. Vessel length density (VLD) (defined as vessel length per unit area based on the skeletonized image) was calculated on four equally divided locations (superior, nasal, inferior and temporal) of a ring between two concentric circles (1.5 and 2.25 mm in diameter) manually centered on the disc (Fig. 3, bottom row).
We also tested the repeatability between the two consecutive scans on each of the four devices. No image registration was needed for this study. The VLD was calculated within the ring between the 3-mm and 1-mm circles centered on the image.

Statistical Analysis.
To test the general effect of different instruments on the outcome measurements, the general linear model (repeated measures) was used, with instrument and location being as within-subject effect and axial length being as between-subject effect. We also evaluated the association between axial length and the metrics on each location of the four instruments using Spearman correlation test. To test the reproducibility among different instruments, intraclass correlation coefficient (ICC)and coefficient of variation (CV) was calculated between each pair of instruments on each location. The CVs among healthy subjects were also calculated. For the repeatability test, ICC between the repeated scans on each device was calculated. SPSS 23.0 (IBM SPSS Statistics for Windows, Version 23.0. Armonk, NY) was used for all statistical analyses.