Introduction

Transcutaneous bone conduction devices (BCDs) are implantable devices that enable acoustic transmission via a magnetic interface with an external sound processor1,2,3. By direct vibratory stimulation of the inner ear, it is widely employed for conductive or mixed hearing loss associated with abnormalities of the middle or external ear in which traditional amplification devices cannot be worn1,2,3. Examples include the BoneBridge floating mass transducer (BB-FMT) made by MED-EL (Innsbruck, Austria) and the Osia osseointegrated steady-state implant (O-OSI) made by Cochlear™ (Sydney, Australia). Although conceptually similar, design and surgical procedures differ.

Exact placement of BCDs requires careful evaluation of the temporal bone contour and thickness. More importantly, critical structures such as the sigmoid sinus or underlying dura should be avoided. Sigmoid sinus injury can lead to substantial bleeding and air embolisms, while dural injury can result in CSF leaks and the development of arteriovenous malformations3,4,5. The use of three-dimensional (3D) planning software allows surgeons to accurately determine ideal BCD positioning while safeguarding against potential injuries and potentially reducing operating times. However, a chasm exists between preoperative planning and surgical performance as manual measurements based off anatomical landmarks introduce room for error1.

Several technologies for pre-operative planning and placement of BCDs have been explored. 3D printed transparent stencils interfacing with the surface of the one translated into a mean accuracy near 1.0 mm3. However, widespread adoption is lacking possibly due to cost, accessibility, and familiarity with 3D printing. Surgical simulation software paired with intraoperative navigation (IN) tools have also been utilized2,6. A major challenge of IN technologies is the omission of relevant information from the surgical field requiring surgeons to switch their focus to and from the surgical field7. Augmented reality (AR) guided surgery has been proposed to overcome this by placing relevant information directly into the surgical field8.

From guiding bony cuts in oncologic maxillofacial surgery to outlining intradural spinal tumor margins during resection, projected AR technology can help guide surgeons where difficult, atypical anatomy is encountered9,10. Given the prospect of underdeveloped anatomy, lack of soft tissue landmarks, and subsurface critical anatomy, AR may be poised to improve BCD surgical performance. We assessed the accuracy of a novel augmented reality (AR) technology to guide placement on cadaveric specimens10,11.

Methods

Specimen preparation

Whole cadaveric heads were prepared with bilateral curvilinear post-auricular incisions with elevation of a soft tissue flap for exposure of the zygomatic root, posterior external auditory canal, and the mastoid tip. Eight 2 mm bone wells were drilled outside of the surgical field to act as fiducial references for eventual image guidance calibration within the experimental arm. Areas of placement included the zygomatic root, bony external auditory canal, and the mastoid tip.

Pre-operative planning

Using a prototype intraoperative cone-beam computed tomography scanner (Powermobil, Siemens, Germany), the cadaver heads were obtained, with an isotropic voxel size of 0.78 mm12. Scans were evaluated for abnormal anatomy or evidence of previous surgery. Both the O-OSI and BB-FMT devices were imaged for surgical modelling by creating the virtual rendering of hearing device for projecting the overlay during the procedure. Materialise Mimics Medical 19.0 (Materialise NV, Belgium) was used to identify optimal placement of the devices with creation of virtual heads rendered from CT imaging using pre-set bony segmentation sequencing.

Implants were imported into Materalise Mimics as optimized triangulated surface meshes that moved independently from the bone. The experimental design is outlined in Fig. 1. Each surgeon’s pre-operative planning included placement of four O-OSI devices and four BB-FMT devices in two separate sessions. Bone depth and avoidance of critical structures, such as the sigmoid sinus were major factors. O-OSIs were placed within the mastoid and clearance around the implant was ensured to avoid inadvertent contact with underlying bone. The three possible placements of the BB-FMTs included the mastoid, retrosigmoid, and middle fossa areas. Each surgeon underwent a brief 10-min session with surgical manuals to review optimal surgical technique for both implants. Each planning session lasted five minutes to allow for surgeons to guide exact placement.

Figure 1
figure 1

Study protocol (CBCT cone beam computed tomography, O-OSI Osia osseointegrated implant steady-state implant, BB-FMT BoneBridge floating mass transducer).

Full size image

Bone conduction implantation

Implantation followed a standardized protocol beginning with the control arm followed by the experimental AR arm (Fig. 1). Within the control arm, surgeons utilized Materialise Mimics’ built-in measurement tool for eventual intraoperative reference during implant placement. Whereas in the experimental arm, device placement was projected onto the surgical field using GTx-Eyes (Guided Therapeutics, TECHNA Institute, Canada) via a PicoPro projector (Cellon Inc., South Korea)7,11. The AR setup is demonstrated in Fig. 2 and seen in the supplementary video.

Figure 2
figure 2

Integrated augmented reality surgical navigation system. (A) the projector and surgical instruments were tracked with the optical tracker in reference to the registered fiducials on the cadaveric head. Optical tracking markers attached to the projector allows for real-time adjustments to image projection. The surgical navigation platform displaying a pre-operatively placed implant. Experimental AR projection arm setup. (B) Surgeons were encouraged to align the projector to their perspective to reduce parallax.

Full size image

Following implant placement, CT scans were obtained of the cadaveric heads to capture the location of implantation for eventual 3D coordinates measurement analysis. Each surgeon performed four O-OSI placements followed by four BB-FMTs.

AR system setup

The integrated AR surgical navigation system consists of a PicoPro projector (Cellon Inc., South Korea), a Polaris Spectra stereoscopic infrared optical tracker (NDI, Canada), a USB 2.0-megapixel camera (ICAN, China), and a standard computer. A 3D printed PicoPro projector enclosure enabled the attachment of four tracking markers, which provide real-time three-dimensional tracking information (Fig. 2). GTx-Eyes (Guided Therapeutics, TECHNA Institute, Canada) is a surgical navigation platform that utilizes open-source, cross-platform libraries included IGSTK, ITK, and VTK11,13,14,15,16. The developed AR system has demonstrated the projection accuracy at 0.55 ± 0.33 mm and has been widely adapted to the domains of Otolaryngologic and Orthopedic oncologic operations17,18,19,20. Recently, the software has evolved to include AR integration7,9.

The AR system requires two calibrations: (1) camera and instrument tracker, (2) camera and projector, which are both are outlined by Chan et al.9,11. The result allows the tracked tool to be linked with the projector’s spatial parameters allowing for both translation and rotational movements.

Camera and instrument tracker calibration

The camera and tracking tool calibration defines the relationship between the camera’s center and the tracking tool coordinates by creating a homogeneous transformation matrix, \({{}^{Tracker}T}_{Cam}\), consisting of a 3 × 3 rotational matrix (R) and a 3 × 1 translational vector (t). The rotational parameter was represented with Euler angles \(({R}_{x},{R}_{y},{R}_{z})\). This calibration process requires photographing a known checkerboard pattern from various perspectives using the camera that is affixed to the projector’s case. The instrument tracker’s position and orientation are recorded to compute the spatial transformation. The grid dimensions from each photograph are compared with actual dimensions (30 mm × 30 mm in a 9 × 7 array) using an open-source Matlab camera calibration tool21. This calibration serves as the extrinsic parameter of the camera.

The intrinsic parameters (A) of the camera include the principal point \(({u}_{0,}{v}_{0})\), scale factors (\(\alpha , \beta ),\mathrm{ and the skew of the two image axes }\left(c\right)\)22,23,24. This is denoted as:

$$\mathbf{A}=\left[\begin{array}{ccc}\alpha & c& {u}_{0}\\ 0& \beta & {v}_{0}\\ 0& 0& 1\end{array}\right]$$

When combining the extrinsic (R t) with intrinsic (A) parameters, three-dimensional space \(({\mathbf{M}=[X,Y,Z,1]}^{T})\) can be mapped to a two-dimensional camera image (\({\mathbf{m}=[u,v,1]}^{T}\)). s is defined as the scale factors. This is represented by: \(s\mathbf{m}=\mathbf{A}\left[\mathbf{R} \mathbf{t}\right]\mathbf{M}.\)

Camera and projector calibration

This link defines the spatial relationship between the camera’s centre and the projector to create a homogenous transformation matrix (\({{}^{Cam}T}_{Proj}\)). A two-dimensional checkerboard image is projected onto a planar checkerboard surface, which was used in the previous calibration step. The camera captures both images from various perspectives. Using the projector-camera calibration toolbox, the transformation of the camera and projector (\({{}^{Cam}T}_{Proj}\)) is now established25. The calibration requires linking the camera and the projector tracking markers, both of which are mounted on the projector enclosure (Fig. 2). By combining both calibration processes, the resulting transformation matrix from the AR projector to the tracking marker is denoted by \({{}^{Tracker}T}_{Proj}={{}^{Tracker}T}_{Cam}*{{}^{Cam}T}_{Proj}\) .

Projection accuracy

AR projection setup required confirmation of projection adequacy using an image guidance probe and a Polaris Spectra NDI (Fig. 2). Using the image guidance probe, coordinates from the bony fiducials (drilled bone well) and the projected fiducials (green dots) were captured. The difference between coordinates served as the measurement of projection accuracy (Fig. 3).

Figure 3
figure 3

(A) Fiducials projection onto the surgical field was matched to the drilled wells and (B) subsequent accuracy measurements were obtained with a tracking pointer tool placed within the drilled wells where x-, y-, and z- coordinates were captured.

Full size image

Post-operative measurements

Post-operative and pre-operative scans were superimposed on Materialise Mimics’ and centre-to-centre distances as well as angular differences on the axial plane were measured (Figs. 4, 5). For O-OSI placements, the centre of the O-OSI was used, whereas the centre of the FMT for BB-FMT.

Figure 4
figure 4

Accuracy measurements for center-to-center distances and angular accuracy.

Full size image
Figure 5
figure 5

Post-operative CT scans (A) BB-FMT and (B) O-OSI following AR projector guided surgery with paired pre-operative planning rendering seen in (C) and (D). In images (A) and (B), there is the pre-operative planning outline superimposed. The blue arrow denotes post- operative placement whereas the red arrow denotes pre-operative planning.

Full size image

All participants completed a NASA Task Load Index (TLX) questionnaire assessing the use of AR in addition to providing feedback in an open-ended questionnaire26. TLX results were used to generate raw TLX (RTLX) scores for the six domains and subsequently weighted workload scores were generated27.

Statistical analyses

Continuous data was examined for normality by reviewing histograms, quantile–quantile plots, and the Shapiro–Wilk test for normality. Given the lack of normality and repeated measurements, Wilcoxon signed-rank testing was used for centre-to-centre (C-C) and angular accuracies comparisons between the control and experimental arms. All analyses were performed using SPSS 26 (IBM Corp., Armonk, NY).

Ethics statement

All methods were carried out in accordance with relevant guidelines and regulations. This study was approved by the Sunnybrook Health Sciences Centre Research Ethics Board (Project Identification Number: 3541). Informed consent was obtained from all subjects and/or their legal guardian(s) by way of the University of Toronto’s Division of Anatomy–Body Donation Program. All subjects provided consent in the publication of identifying images in an online open-access publication.

Results

Surgeons

Three of the five surgeons were fellowship-trained otologists comprising an average of 15 years of experience. The remaining two surgeons were otology fellows. All participants had experience with BB-FMT and O-OSI BI300 placement, respectively.

A Kruskal–Wallis H test was run to determine if there were differences in accuracies between the five surgeons. Distributions of accuracies were not similar for all groups, as assessed by visual inspection of a boxplot. However, the distributions of C-C accuracies lacked statistically significant difference between groups (p = 0.857). Similarly, distributions of angle accuracies were not significantly different (p = 0.561).

Fiducial accuracy

The average error in fiducial landmark accuracy within the x-, y-, and z-axes were 0.3 ± 1.1 mm, 0.2 ± 0.8 mm, and 0.6 ± 0.9 mm. This translated to an overall average distance of 1.7 ± 0.6 mm between the bony fiducial markings and the AR projected fiducials.

Performance

All O-OSI devices were placed in a mastoid position while BB-FMT placement varied with seven FMTs placed in the mastoid, seven in a middle fossa position, and four in a retrosigmoid position. Outlined in Table 1, average operative times were less in the group with AR projection was nearly 2 min faster (4.3 ± 1.3 vs. 6.6 ± 3.5 min., p = 0.030). This pattern is consistent when comparing within O-OSI surgeries (p < 0.005), but not BB-FMT surgeries (p = 0.169).

Table 1 Surgical outcomes of both the control and experimental arms (BB-FMT BoneBridge floating mass transducer, O-OSI Osia osseointegrated steady-state implant).
Full size table

Overall C-C distances between the two arms were significantly different (9.0 ± 5.3 mm vs. 1.9 ± 1.6 mm, p < 0.001). This trend was seen for both devices with C-C distances consistently less within the experimental arm. For BB-FMT surgeries, the difference between the two arms was 9.8 ± 3.9 mm (p < 0.007). For O-OSI surgeries, this difference was less pronounced at 4.5 ± 2.8 mm (p = 0.007). In contrast, angular accuracies were not statistically significant between AR and control arms (p = 0.490). A significant difference between the two arms when stratifying by device was not evident.

NASA-TLX and feedback

The raw TLX (RLTX) scores for each domain are shown in Fig. 6. Ranked from lowest to highest was frustration (very low: 12), effort (very low: 19), performance (perfect: 21), mental demand (low: 24), physical demand (very low: 25), and temporal demand (low: 33). When factoring in paired-choice weighting scores, the average weighted workload was 19.9. This translates to a “medium” workload when using the AR setup28.

Figure 6
figure 6

NASA Task Load Index (TLX). Average surgeon raw TLX scores are plotted.

Full size image

Other comments with the technology highlighted the concerns associated with the projector’s light intensity and clarity of the projection borders. Suggestions to overcome blocking of optical tracking markers with bone dust or surgeon placement were to draw projections prior to performing the surgery.

Discussion

AR guided projection poses a novel technique to bridge the divide between pre-operative planning software and performance of bone conduction implantation surgery (Fig. 7). When comparing C-C distances, AR projection has a clear advantage. The AR system is fully integrated into a surgical navigation platform that utilizes an optically tracked projector and registered fiducial landmarks to avoid bony fixation11. The projected image automatically adjusts with any head movement and provides real-time intraoperative reference. Despite being a prototype setup, the AR platform utilizes widely available image-guidance tracking software. Moreover, a central projector prevents the need for wearable technology and allows for multiple viewers from all angles.

Figure 7
figure 7

Pre-operative planning of O-OSI and BB-FMT placements. (A) Osia osseointegrated implant steady-state implant on coronal imaging with right lateral view. Projection of O-OSI onto the surgical field. (B) BoneBridge floating mass transducer on axial imaging with a right oblique view. Projection of the BB-FMT onto the surgical field.

Full size image

Other existing technologies that aim to address pre- and intra-operative planning include 3D printed stencils that provide guidance for intraoperative placement1,3,5. Given the ease of sterilization and the added benefit of attaching the guide to the surgical field, these stencils have achieved a high degree of accuracy. Following a successful cadaveric feasibility study, the BB-STAMP (Bone Bridge Surface template-assisted marker positioning) has been performed on three patients with a high degree of C-C accuracy of 0.3 ± 0.1 mm and a rotational accuracy of 4.1 ± 4.7°5. A key distinguishing factor, however, was the number of hours dedicated to simulation on physical models and the multi-day process model creation. In contrast, AR projection circumvents the need for a 3D printer or the need for sterilization processing. Moreover, the ability to project critical anatomy such as the sigmoid sinus provides an added layer of patient safety.

For both BB-STAMP and AR projection technologies, rotational accuracy remains a domain requiring continuing improvement. Further iterations of AR projection will include drill trajectory information from optical markers attached to the drill hand pieces as well as a tracking probe that are linked to color projections indicating adequate or inadequate trajectories. Kong et al. described a case study in which CT-guided navigation utilizing an electromagnetic tracker as another means of adapting intraoperative planning for intraoperative reference. However, accuracy and time comparisons were not established2.

Other AR technologies have been applied to the head and neck including the use of optical see-through (OST) head-mounted devices (HMD) including the HoloLens™ Gen 1 and 2 (Microsoft Corporation, Redmond, WA)8. Advantages of HMD technology over the AR projection includes the convergence of the surgeon’s perspective and field of view without manipulation of the projector or occupying valuable, physical space29. Although OST technology preserves the direct view of the world within the HMD, a fixed focal length challenges a user’s ability to maintain focus of both AR and real-world content with more proximal tasks8,30. Moreover, performance of precise manual tasks with OST HMD guidance is inferior to performance without AR assistance30. The inability to wear surgical loupes or a headlight, battery life concerns, patient privacy, and signal interference have challenged widespread adoption29. The use of a projector addresses these shortcomings by enabling multiple viewers from numerous angles and avoiding the need for wearables.

Several challenges exist with AR projection including the need for pre-operative registration to pre-established surface landmarks. In this investigation, 2 mm wells served as bony landmarks for registration. Further research will aim to eliminate this pre-operative preparation with surface registration only. A potential confounder of our results may be attributed to the design of our study order. Specifically, the experimental arm followed the control arm for both implant surgeries. This may have primed surgeons with techniques or nuances that translated to the experimental arm to improve performance.

One of the inherent challenges facing AR projection arises from parallax error8. When a surgeon’s perspective and the projection axis are misaligned, the accuracy of surgical implantation accuracy is threatened. The parallax effect, however, was minimized in our AR setup by allowing for surgeons to freely adjust the projector to their viewpoint since the projector’s image would automatically adjust for any movement given its attached location tracker. As a result, recalibration and reregistration were not required following displacement. Given the placement of bone conduction devices along surface anatomy, the parallax has a reduced effect8.

The impact of cognitive loading from this novel technology remains consistent across surgeons that AR projection remains low on within the mental, physical, temporal cognitive demands. Both effort and frustration were considered very low with performance considered high. Technological adaptation challenges remain low, enabling surgeons to focus on the surgical task with minimal disruption to their cognitive load.

Future directions include projection of critical anatomy beyond the sigmoid sinus and enable periodic reference of the facial nerve, inner ear, or even the internal auditory canal during other temporal bone procedures (Fig. 8). Additionally, real-time visualization of the insertion trajectory of BCDs and drill tracking may improve angular and placement accuracy, respectively31. Future development and research will focus on surface anatomy registration to exclude the need for surface fiducials as well.

Figure 8
figure 8

Projection of critical structures encountered in a cortical mastoidectomy including the sigmoid sinus and horizontal semicircular canal.

Full size image

Conclusion

Projection of pre-operative planning onto the surgical field proves to be an exciting opportunity to interact with medical information in a meaningful way. With direct intraoperative reference, AR guided surgery enhances surgical precision when compared to conventional surgical planning while maintaining a low cognitive load on surgeons. In comparison to existing BCD planning technologies such as 3D printed stencils, AR projection does not require time-consuming model production and circumvents the need for sterilization. Its integration into existing surgical navigation platforms while avoiding the challenges associated with head-mounted AR technology reinforces its promise as a useful intraoperative tool.