Live volumetric (4D) visualization and guidance of in vivo human ophthalmic surgery with intraoperative optical coherence tomography

Minimally-invasive microsurgery has resulted in improved outcomes for patients. However, operating through a microscope limits depth perception and fixes the visual perspective, which result in a steep learning curve to achieve microsurgical proficiency. We introduce a surgical imaging system employing four-dimensional (live volumetric imaging through time) microscope-integrated optical coherence tomography (4D MIOCT) capable of imaging at up to 10 volumes per second to visualize human microsurgery. A custom stereoscopic heads-up display provides real-time interactive volumetric feedback to the surgeon. We report that 4D MIOCT enhanced suturing accuracy and control of instrument positioning in mock surgical trials involving 17 ophthalmic surgeons. Additionally, 4D MIOCT imaging was performed in 48 human eye surgeries and was demonstrated to successfully visualize the pathology of interest in concordance with preoperative diagnosis in 93% of retinal surgeries and the surgical site of interest in 100% of anterior segment surgeries. In vivo 4D MIOCT imaging revealed sub-surface pathologic structures and instrument-induced lesions that were invisible through the operating microscope during standard surgical maneuvers. In select cases, 4D MIOCT guidance was necessary to resolve such lesions and prevent post-operative complications. Our novel surgical visualization platform achieves surgeon-interactive 4D visualization of live surgery which could expand the surgeon’s capabilities.


Results
4D MIOCT system. The integrated 4D MIOCT microsurgical imaging system is shown in Fig. 1. A portable cart housed the custom OCT hardware, computer, and software-user interface (Fig. 1A). The OCT sample arm design employed a customized mechanical enclosure for seamless integration into a Leica M844 ophthalmic surgical microscope (Fig. 1B) 31 to enable OCT data acquisition during live surgery without disturbing surgical visualization through the operating microscope. To achieve fast volumetric OCT imaging, an ultra-high speed swept-frequency source centered at 1040 nm (Axsun Technologies; Billerca, MA) was used, and the interferograms were digitized at 800 MS/s and processed with custom software on a graphics processing unit (GPU). The volumetric data was rendered to a two dimensional image using optimized ray casting, edge enhancement, and depth-based shading algorithms 37 . The system was capable of acquiring and processing OCT volumes at up to 10 volumes per second, although in human surgical trials volume rates of 3.33 Hz and 2 Hz with higher sampling density were selected for retinal and anterior eye surgery, respectively. A detailed description of the optical design and software is provided in the Methods section.
4D MIOCT data acquisition and visualization. Surgery-tailored MIOCT scan control methods and visualization techniques were developed to facilitate image acquisition and feedback ( Fig. 1D-F). During surgery, the operator could rotate the MIOCT scan axis to align the B-scan direction to a particular maneuver, tool, or region of interest, and used a manual tracking module ( Supplementary Fig. S1) to relocate the OCT scanning site within the surgical field in real-time. The volumetric rendering perspective was also adjustable by the operator in real-time. Mixed mode volumes, in which only the B-scan of interest was densely sampled and averaged while the rest of the data was sparsely sampled, were used to preserve a fast volumetric rate and high-quality B-scans. The acquired volumetric data was displayed in four different formats: rendered volumes, B-scans (cross-sections), maximum intensity projection (MIP) en face views, and two rotationally-offset volume renders for stereoscopic visualization. The surgeon visualized the stereo volumes and B-scans during surgery (detailed below) while the volumes, MIPs, and B-scans were projected on a wall-mounted display in the operating suite and a computer monitor for feedback to the surgical team.
Scientific RepoRts | 6:31689 | DOI: 10.1038/srep31689 To provide immediate stereo 4D MIOCT visualization to the surgeon with surgeon control of the volumetric rendering perspective, we utilized a novel heads-up display (HUD) integrated into the operating microscope (Fig. 1B) 38 . The dual-channel stereoscopic HUD allowed simultaneous projection of the rotationally-offset MIOCT volumes into the surgical oculars using a single micro-display ( Fig. 1F) (optical design detailed in the Methods section). The location of data projected within the oculars using the HUD was controlled by the operator to ensure that the surgical field was not obstructed. A foot-operated joystick allowed the surgeon to change the rendering orientation of the MIOCT volumes projected with the HUD. This feature enabled the surgeon to inspect the surgical field with an adjustable volumetric visual perspective. A B-scan was also projected into the surgical oculars for real-time visualization by the surgeon, and the location of the displayed B-scan within the volume was denoted by a white rectangle overlaid on the volume render (Fig. 1E). The OCT operator controlled the location of the B-scan with guidance from the surgeon or surgical assistant and could turn on/off the B-scan and volume view or add the MIP if desired by the surgeon. Alternatively, the surgeon could control the location of the entire volume and displayed B-scan by rotating the patient's eye or repositioning the microscope laterally. Preliminary 4D MIOCT ex vivo studies. The integrated 4D MIOCT system was tested with a replica of the operating microscope in a wet laboratory setting before translation into the human operating suite ( Fig. 2A). MIOCT visibility of commonly used ophthalmic surgical instruments (including different instrument gauges) was characterized during retinal surgery in ex-vivo porcine whole eyes (Supplementary Table S1). Simulated surgical maneuvers were also performed in cadaveric porcine eyes and human donor corneas to test MIOCT volumetric visualization of anterior segment and retinal surgery at different volume rates and sampling densities. Using a 5 × 5 mm and 10 × 10 mm field of view for retinal and anterior eye surgery, respectively, the volumetric frame rate was varied between 1-10 volumes per second by varying the number of A-lines per volume to determine the fastest volume rate that would still allow sufficient lateral resolution for human surgical visualization. The optimal scan parameters selected were 300 A-lines/B-scan and 100 B-scans for retinal surgery, and 500 A-scans/B-scan and 100 B-scan for anterior eye surgery, which resulted in 3.33 Hz and 2 Hz volume rates, respectively. However, if desired for particular applications in future studies, higher volume rates with similar sampling density can be achieved over smaller fields of view.
4D MIOCT quantitatively enhances performance of depth-based microsurgical skills in mock surgical trials. Because 4D MIOCT offers improved depth perception compared to the operating microscope, we hypothesized that our system could enhance the performance of depth-based microsurgical maneuvers. We tested this hypothesis for two conventional microsurgical skills: suture placement at target depths within tissue and control of instrument positioning relative to tissue (Fig. 2B).
To test suture placement with 4D MIOCT guidance, a group of 14 ophthalmic surgeons-in-training were recruited from Duke University ophthalmology residents to perform suture passes at target depths within ex vivo porcine corneas 39 . The subjects were randomized into two groups: one that used MIOCT for guidance, and one without MIOCT (N = 7). Each subject was asked to perform a corneal suture pass at both 50% and 90% depth and was allowed up to three trials before performing two gradable attempts for each depth. During the trials, only the group with MIOCT guidance was allowed direct depth visualization of the suture needle within corneal tissue using volumes and B-scans after passing the suture (Fig. 3A,B) (see Methods section for more details).
To test control of instrument placement with MIOCT guidance, 3 ophthalmic surgeons were recruited from Duke University ophthalmology retinal surgery fellows. The surgeons were asked to place a surgical instrument (nitinol surgical flex loop (Alcon; Ft. Worth, TX)) as close to the retinal surface as possible without contacting tissue during simulated retinal surgery in cadaveric porcine eyes. Each surgeon performed the maneuver 8 times without MIOCT guidance and 8 times with MIOCT guidance (N = 8) after 3 practice trials. When performing the maneuvers under MIOCT guidance, the surgeon was given the option of visualizing the MIOCT data in real-time either through the microscope oculars (using the HUD) or on a monitor adjacent to the microscope ( Fig. 4A-C), both with surgeon control of the volume perspective (see Methods section for more details).
MIOCT guidance improved the control of instrument placement relative to tissue of all 3 surgeons in the study (Fig. 4D-F  Duke University Health System Institutional Review Board. All in vivo volumes and B-scans were processed in real-time and did not require additional post-processing for presentation. Additional details on maximum permissible light exposure to the patient and imaging protocols are provided in the Methods section.

4D MIOCT imaging of human retinal surgery. Retinal microsurgery involves restoration of macro and
micro-architectural retinal alterations that arise from pathologic conditions. In one such condition, a proliferation of abnormal tissue membrane on the surface of the retina (epiretinal membrane) contracts and causes visual distortion and loss of central vision. Macular holes (full thickness retinal breaks) and lamellar holes (partial thickness retinal breaks) can also result from a combination of traction from collagen fibers in the vitreous gel, contraction of surface membranes and of the inner retinal surface basal lamina (internal limiting membrane). Microsurgical forceps and/or scrapers are used to peel these pathologic and/or native membranes to relieve the underlying retinal contraction and close the retinal defect.
MIOCT imaging was performed on patients undergoing standard 3-port vitrectomy with three surgeons at a single site. Inclusion criteria for imaging included macular surgery for membrane removal, macular hole or lamellar hole closure, and/or retinal detachment repair. MIOCT imaging was attempted in a total of 35 retinal surgeries. To evaluate success of MIOCT imaging, a trained technician graded the acquired data based on the identification of pathology of surgical interest in MIOCT B-scans or volumes in concordance with preoperative diagnosis. A summary of results organized by pathology is provided in Table 1. Note that multiple pathologies could be present in one surgical case. The pathology of interest was successfully visualized with MIOCT in 93% In this trial, the surgeon inserted the suture needle at a 44% corneal depth, 6% away from the 50% target depth. The crosssectional view allowed accurate grading of the suture needle placement within corneal tissue. (C,D) Box plots summarizing suture placement with and without MIOCT guidance for 50% (C) and 90% (D) target depths, respectively. The data is plotted in percent difference from the target depth; closer to 0% difference is more accurate. At both target depths, the surgeons achieved increased suture placement accuracy with MIOCT. of attempted cases. MIOCT imaging was unsuccessful in 3 retinal detachment cases because of insufficient field of view (1 case) and inability to focus the OCT beam on retina due to the severity of the detachment (2 cases).
4D MIOCT revealed previously unrecognized micro-architectural alterations and instrument-tissue interactions that occurred during conventional surgical maneuvers. Excerpts from a 4D MIOCT recording ( Fig. 5A and Supplementary Movie S1) show visualization of a diamond dust-coated membrane scraper brushing the retina around a full-thickness macular hole and producing a transient indentation of the inner retinal contour adjacent to the hole. Corresponding frames from a surgical camera that records the surgeon's view through the operating microscope are shown above each MIOCT volume. The depth of the macular hole (partial versus full-thickness) and tissue deformation by the surgical instrument could not be identified using the conventional view through the operating microscope alone. After brushing was completed, residual retinal deformation was noted but was not as prominent compared to during instrument contact, which emphasized the need for real-time imaging during live maneuvers. 4D MIOCT imaging was also performed during retinal brushing with the microsurgical Both this interaction and elevation of retinal blood vessels above the retinal surface (black arrows) were not visible via conventional microscope viewing. The proximity of the instrument to retina could also be determined more precisely in 4D MIOCT volumes compared to the surgical microscope view. In contrast to the diamond-dusted scraper, the loop appeared to be less abrasive and resulted in transient focal retinal depression during time of contact but not afterward. 4D MIOCT also improved real-time visualization during the pathologic tissue removal from the retinal surface. Removal of these membranes and underlying basal lamina (internal limiting membrane) is challenging through the operative microscope because they are translucent, hundreds-of-micrometers thick, and often require staining 40,41 to facilitate their identification. 4D MIOCT enabled clear visualization of the elevated portion of the membrane (red arrows) that was grasped with surgical forceps (purple arrows) to initiate peeling (Fig. 6A, and Supplementary Movie S3) without tissue staining. Important parameters that the surgeon must consider to minimize retinal damage during peeling-including membrane tension, associated deformation of the retinal surface, and angle of the membrane relative to retina-were also better appreciated with 4D MIOCT than through the conventional microscope view. Moreover, the membrane architecture and its proximity to underlying retina was consistently visible with 4D MIOCT irrespective of visible light endoscopic illumination angle and position, which was required only for conventional microscope viewing.
In addition to live volumetric visualization, the surgeon's ability to view 4D MIOCT B-scans and static volumes at various rendering perspectives via the HUD provided important feedback. In a separate membrane peeling case, 4D MIOCT images (Fig. 6C-E) allowed enhanced visualization of partially separated surface membrane tissue (red arrows)-which was attached at the margin of a partial thickness macular hole (blue arrow)-compared to the limited conventional en face view (Fig. 6B). The different perspectives of the 3D micro-architecture of the membrane and retina were controlled by the surgeon in real-time with a foot-operated joystick. The altered viewpoint in Fig. 6E improved visualization of the membrane attachment at the margin of the hole in contrast to the viewpoint in Fig. 6D. After tissue removal, 4D MIOCT volumetric images and B-scans showed that the membrane was successfully removed and that retinal tissue (photoreceptors) was present at the base of the hole (below blue arrow), verifying that the hole had remained partial thickness and did not progress to full thickness during surgery (Fig. 6G,H). The surgeon-controlled viewpoint also augmented the view of retinal vessels elevated across the inner surface and the residual hole (Fig. 6H,I  caused by protein build up in the native ocular lens that leads to reduced visual acuity. In cataract surgery, the native lens is fragmented using an ultrasonic probe or femtosecond laser and replaced with an artificial intraocular lens. Corneal transplantation may be required to treat a number of conditions, including injury from trauma, corneal edema, or tissue scarring. During corneal transplantation, at least a portion of the patient's diseased cornea is replaced with a donor graft. Proper graft placement reduces the risk of rejection and results in improved visual acuity for the patient. However, the graft/native cornea interface is not readily visible in conventional surgery using the operating microscope due to its limited depth visualization. Additionally, corneal edema can limit the surgeon's view of the graft under standard operating microscope illumination 42 . 4D MIOCT imaging was performed on patients undergoing anterior eye surgery with two surgeons at a single site. Inclusion criteria included full corneal transplantation (penetrating keratoplasty), partial thickness corneal transplantation (Descement's stripping automated endothelial keratoplasty (DSAEK)), and cataract surgery. MIOCT imaging was attempted in 13 anterior segment surgeries. Successful MIOCT imaging was defined as clear visualization of all the surgical sites of interest in B-scans or volumes, and grading was performed by a trained technician. A summary of results organized by surgical procedure is provided in Table 2. Note that multiple procedures could be performed during one surgery. Successful MIOCT imaging was achieved in 100% of the attempted cases.
4D MIOCT imaging was necessary to reveal and guide the treatment of select lesions during a full corneal transplantation. During the procedure, 4D MIOCT revealed an abnormal adhesion (red circle) of the iris to the graft/cornea interface after securing the graft (Fig. 7B,C). This adhesion was not visible with the operating microscope (Fig. 7A) and could have led to post-operative complications-such as wound leakage, local corneal endothelial loss, increased inflammation, and glaucoma-if not resolved. Using 4D MIOCT for localization and guidance, the surgeon directed a cannula (blue arrows) to inject viscoelastic and release the iris (Fig. 7D and Supplementary Movie S4). The surgeon then verified the resolution of the abnormal adhesion by identifying clear intervening space between the iris and cornea on the 4D MIOCT data (Fig. 7D time: 1.00 s).
4D MIOCT imaging also helped guide the placement of the corneal graft in a partial thickness corneal transplantation. During the DSAEK procedure, only the posterior 10% cornea is replaced with a donor graft; therefore, the surgeon must insert, unfold, and manipulate the graft beneath the remaining native cornea. Figure 8 and Supplementary Movie S5 show a 4D MIOCT recording of the graft unfolding after insertion with the corresponding surgical camera frames. While the folded graft (blue arrows) could be identified in the operating microscope, the axial proximity of the graft to native cornea was only visible with MIOCT. The surgeon then injected air to unfold the graft and compress the intervening space between the graft and cornea. MIOCT was used during the maneuver to monitor graft placement and confirm graft/host apposition. Failure to eliminate graft/host interface fluid would result in post-operative graft detachment and need for additional surgery to reattach the graft.

Discussion
While tomographic modalities can image subsurface anatomy, most intraoperative implementations suffer from low resolution, instrument incompatibility, and/or the inability to image in real-time [7][8][9][10][11][12][13] . OCT is a real-time volumetric modality that uses interferometric detection to achieve micron-scale, structural imaging of tissue. Since its inception in 1991 21 , OCT has become the gold standard diagnostic modality for preoperative evaluation and postoperative assessment of surgical endpoints in most retinal and many corneal procedures. The development of handheld OCT (HHOCT) also allowed imaging of supine, anesthetized patients and the first OCT imaging at predefined time points of ophthalmic surgeries; however, these probes required gross displacement of the microscope away from the surgical field and could not be used during live surgery [43][44][45] . Our previous MIOCT work laid the foundation for real-time tomographic feedback to the surgeon and achieved B-scan imaging of live surgery for the first time 46,47 . Since, similar research-grade and commercial systems has been developed, but all are limited to B-scans for real-time imaging and therefore cannot comprehensively image the dynamic 3D surgical field 16,48,49 .
We described a 4D MIOCT system that, together with a custom HUD and surgeon control of visualization, offers a novel integrated 4D imaging platform that enhances the feedback available to the surgeon. For example, surgeons currently rely on indirect visual cues such as shadows or instrument deflection to estimate the pressure exerted on tissue with their instruments. Our 4D MIOCT system allows surgeons to precisely determine the instrument-tissue distance and volumetric tissue deformation during instrument contact (Fig. 5). Such new information may become critical for evaluating the surgeon's instrument control and residual tissue damage caused by instrument contact. Additionally, dyes such as indocyanine green 40   and dissection of translucent pathologic membranes through the operating microscope, but a recent study correlated their use with decreased visual acuity for patients in the first postoperative year 41 . We demonstrated that 4D MIOCT could help identify and guide the dissection of such membrane without the need for dyes (Fig. 6). Additionally, 4D MIOCT was necessary to identify and guide the treatment of select lesions, and to monitor the volumetric localization of grafts relative to native corneal tissue in select anterior eye surgeries (Figs 7 and 8). This improved surgical feedback afforded by our system may facilitate development of future ocular surgical therapies based on volumetric, micron-scale visualization of surgery, including precise removal or movement of layers of retinal or corneal tissue and the placement of biomedical devices 50 (such as retinal prostheses) in the proper location relative to native tissues. Compared to high-resolution MIOCT B-scan imaging alone 49,18,17,51,52 , fast volumetric imaging may improve OCT visualization of complete surgical maneuvers and 3D alterations of tissue anatomy. B-scan imaging of complete surgical maneuvers requires the surgeon to restrict his/her motion to the B-scan axis-which is unfeasible in an in vivo setting-or automated instrument tracking to continuously reposition the B-scan axis during the maneuver. The latter proposition however would add cost and complexity to the system, and recently demonstrated techniques have been restricted to anterior segment mock surgery 53,54 . In our study, fast volumetric OCT imaging relaxed these constraints and facilitated imaging of transient tissue deformation (Figs 5 and 7) and surgical maneuvers (Figs 6 and 8). Moreover, volumetric imaging may provide to the surgeon lateral context that more closely resembles the view through the operating microscope compared to B-scans alone, which could also facilitate real-time interpretation of the OCT data. Preoperative and postoperative OCT assessment of the microsurgical field is crucial to analyze the progression of pathologies and evaluate surgical success. Our 4D MIOCT system successfully imaged either the pathology or surgical site of interest in 95% of human surgeries (Tables 1 and 2) and provided comparable diagnostic information during live surgery. This intraoperative feedback enables surgeons to continuously monitor regions of interest throughout the surgical procedural progression and can potentially alter the surgeon's decision-making to optimize outcomes. We expect this benefit to be immediately impactful in the ophthalmic surgical community and are currently conducting in-depth studies examining the utility of 4D MIOCT to monitor microsurgeries beyond those described here.
Our mock surgery results make a compelling case that 4D MIOCT guidance can improve the accuracy of important microsurgical skills such as suture placement at target depths within tissue and control of instrument positioning relative to sensitive tissue (Figs 3 and 4). In both cases, surgeons exploited the augmented depth perception provided by MIOCT to improve their surgical accuracy. Although cross-institutional large-scale studies will be required to evaluate the long-term impact of MIOCT on surgical training 39 and performance, the results presented here demonstrate that for maneuvers requiring precise dexterity in the depth dimension MIOCT can improve the performance of surgeons with varying levels of experience.
While our initial clinical target for 4D MIOCT guided surgery was ophthalmic microsurgery due to the expertise of our team, the underlying intrasurgical imaging technology has much broader potential application to many surgical specialties. Live 3D OCT is equally applicable to imaging neural tissue in the retina or at a craniotomy Figure 8. 4D MIOCT-guided unfolding of graft below native cornea during human partial thickness corneal transplantation. 4D MIOCT imaging was necessary to determine the axial distance between the graft and native cornea, which was invisible through the operating microscope. (A) Frames from the surgical camera (top) and excerpts from a 4D MIOCT recording shown in B-scans (middle) and volumes (bottom) during graft insertion and unfolding (Movie S5). As the graft (GR) (blue) unfolded, the surgeon used 4D MIOCT data to monitor the graft/cornea interface (GCI) (orange) and to ensure graft/cornea apposition. The volumetric rate for was 0.5 volumes/second. An image artifact due to specular reflection (SR) (yellow) from the corneal apex is present in the middle of the volumes. Time stamps (black) are in seconds. The white rectangle in the volumes denotes the location of the B-scan. The green dashed box denotes the lateral MIOCT field of view. The volumetric MIOCT field of view was 6 × 10 × 10 mm.
Scientific RepoRts | 6:31689 | DOI: 10.1038/srep31689 site, whether in the hands of an ophthalmic surgeon, an otolaryngologist, or a neurosurgeon. Intraoperative 3D surgical visualization with direct surgeon control of the viewpoint of the tissue volume revolutionizes the surgeon's ability to act on real-time changes in the surgical field regardless of the tissue or specialty. Additionally, quantitative metrics derived from OCT volumes-including attenuation coefficients of tissue and thickness measurements-have been demonstrated to differentiate normal tissue from tumor or high-grade malignancy in the kidney, prostate, vulva, breast and brain in animal models or after excision during surgical pauses 28,[55][56][57][58] . An expected future application of 4D MIOCT could be the implementation of such quantitative measurements to differentiate normal and malignant tissue at a microsurgical margin and guide human tumor resection in real-time rather than evaluating the excised tissue during pauses in surgery.
Although 4D MIOCT presents substantial advantages over current microsurgical visualization techniques, there are important limitations that must be addressed. Namely, the maximum volumetric frame rate (~10 volumes per second) in the presented prototype system was inherently limited by the sweep rate of the laser. Fast volumetric frame rates are important to mitigate motion artifacts resulting from rotation/translation of the patient's eye and to capture fast surgical maneuvers. Research-grade swept-source OCT systems achieving volumetric rates of > 20 volumes/second have been demonstrated, but these require significant development before use in large-scale human subject studies and cannot be integrated into operating microscopes [59][60][61][62] . Nonetheless, rapid commercial development of swept-source lasers is ongoing, and near-future increases in the line-scan and volumetric imaging rates may be anticipated using lasers with faster sweep rates and laser sweep buffering schemes 63 . Additionally, the optical properties of current ophthalmic surgical instruments (e.g. forceps, picks, surgical blades) are not optimized for OCT imaging (Table S1). Many instruments are made from metallic alloys that provide stability and precision required to perform microsurgery but cast shadows on underlying tissues 64 . Moreover, highly reflective instruments may result in saturation when the instrument is oriented orthogonally to the OCT optical axis. Recent work demonstrated OCT-optimized surgical tools that are compatible with our MIOCT system, which achieve sufficient OCT visibility while minimizing shadowing and saturation artifacts 16 .
In summary, we have reported on the first micrometer-scale 4D visualization of human microsurgery. The 4D MIOCT prototype improved the accuracy of widely-applicable surgical maneuvers during mock surgery. In vivo 4D MIOCT results from 48 human surgeries enhanced the visual feedback available to the surgeon. Comprehensive 4D visualization of microsurgery with our system can improve current ophthalmic surgery and may catalyze the development of novel surgical techniques in the future.

Methods
Design of 4D MIOCT scanner. The MIOCT scanner optics comprised a collimating lens followed by an afocal magnifying telescope (Thorlabs, Inc.; New Jersey, USA) to match effective numerical aperture (NA) of the OCT system to the NA of the surgical microscope 31 (Supplementary Fig. S2). A turning mirror on a kinematic mount redirected and folded the optical path and was used to ensure that the OCT scan location was centered within the surgical field. A dichroic mirror with a reflectivity cutoff of 990 nm coupled the OCT and microscope optical paths. The OCT beam was then focused by the operating microscope objective. To adjust the OCT focus plane relative to the microscope, the OCT fiber terminator before the collimator was placed on a micrometer-based linear translator (Thorlabs, Inc.; Newton, New Jersey). Either the OCT operator or surgical assistant actuated the micrometer to change the beam vergence prior to the objective and refocus the OCT. For retinal imaging, a surgical contact lens (Alcon; Ft. Worth, TX) was placed on the patient's cornea to relay the OCT beam to the retinal plane. For anterior segment imaging, the OCT beam was directly focused on the subject's cornea. The axial and lateral resolutions of the MIOCT system were 7.8 μ m and 14 μ m, respectively. The lateral imaging range was limited to about 5 × 5 mm by the patient's ocular pupil in retinal surgery and 10 × 10 mm by the optical design in anterior segment surgery. The system achieved imaging at a depth-scan (A-line) rate of 100 kHz with high sensitivity (102 dB peak signal to noise standard deviation ratio), up to 7.4 mm depth imaging range, and 4.9 mm − 6 dB signal falloff. The anti-reflection (AR) coating of the commercial microscope objective was not optimized for near IR, which resulted in sub-optimal (65%) transmission of the OCT light and an associated reduction in imaging sensitivity compared with using lenses with an appropriate anti-reflective coating.

GPU-accelerated volumetric rendering with lighting, edge, and depth-enhanced ray casting.
Custom software enabled real time acquisition, processing, and rendering of volumetric data sets acquired at 100 kHz line rates ( Supplementary Fig. S3) 37 . The software was written in C/C+ + and was comprised of three concurrent threads: a data collection thread, a data processing and rendering thread, and a display thread. The data collection thread communicated with the digitizer and collected 4000 spectral samples of data for each A-scan. Blocks of 16 B-scans were processed at a time through the use of custom GPU code written in CUDA and executed on a GTX Titan (NVIDIA; Santa Clara, CA). The 4D MIOCT control software provided real time 3-view rendering including full ray-cast volumetric visualization at up to 10 volumes per second. The volumetric view was created using an enhanced, real time, integrated volumetric rendering engine for OCT data that mimics the quality of those produced in commercial post-processing software. The rendering pipeline incorporated high performance volumetric median (3 × 3 × 3 pixels) and 2D Gaussian filtering (5 × 5 pixels), boundary and feature enhancement, depth encoding, and lighting into a ray casting volume rendering model. A CUDA implementation of this enhanced ray casting model generated real time dynamic renders that dramatically improved depth perception and visualization of microsurgical volumetric features. A customizable histogram transfer function also allowed the operator to assign opacity and color based on the voxel intensities to the final volume render. The NVIDIA profiler timing output for a typical high-density scan setting of 2752 (axial) x 300 (lateral) pixels per B-scan is illustrated in Supplementary Fig. S3.
The maximum system latency, defined as the maximum delay between acquisition and display of data from any particular sample location, was governed by the time required to acquire, process, and render that data. As described in ref. 37, our approach to minimize latency was to process data in blocks of 16 B-scans while the next block was being acquired, thus limiting latency to 116.5 ms for our longest B-scans comprising 512 A-scans (86 ms acquisition +30 ms processing/rendering). For B-scans containing less than 512 A-scans, the computational time scaled down linearly with the number of A-scans per B-scan 37 . For repetitive volumetric imaging, the maximum time between updates of data from any particular sample location was given by the total volume acquisition time, which in our system was 0.3 seconds for retinal imaging and 0.5 seconds for anterior segment imaging.
Design of stereoscopic HUD. The stereoscopic HUD was designed to be mechanically and optically integrated into the surgical microscope ( Supplementary Fig. S4) 38 . An organic LED microdisplay (Olightek Co., Ltd.; Kirkland, WA) with high contrast ratio (10,000:1) was relayed to an intermediate imaging plane with a 50 mm focal length imaging lens (Edmund Optics, Inc., Barrington, NJ) and a ~190 mm focal length lens located in the operating microscope binocular head. The intermediate image plane was then relayed to the surgeon's retina with a 10x microscope eyepiece (Leica Microsystems; Wetzlar, Germany). The optical path of the HUD was coupled into the path of both surgical oculars using two beam splitters in the microscope infinity space. The beam splitters were tilted by 1.5 degrees relative to the HUD optical axis so that only the left half of the microdisplay was projected to the right ocular, while the right half of the microdisplay was projected to the left ocular. Therefore, separate images could be displayed into the two oculars using only one microdisplay. To project stereoscopic OCT data, two rotationally-offset volumetric renders were projected into the left and right ocular. A foot-operated joystick allowed the surgeon to control the rendering perspective of the volumes to facilitate visualization of the surgical field.
4D MIOCT mock surgery set up. For anterior segment and retinal porcine eye surgery, fresh enucleated porcine eyes were obtained from a local slaughterhouse, stored at 4 °C, and used within 12-18 hours to minimize corneal edema. For retinal surgery, standard 3-port vitrectomy was utilized: one port was used to insert the infusion line to regulate intraocular pressure, the second port was used to insert a fiber-optic endoilluminator, and the third port was used to insert the surgical instruments. Alternatively, fresh research grade donor corneas were obtained from a local eye bank (Miracles in Sight; Winston-Salem, NC).
Quantitative studies during mock surgery. For the suture depth study, each group (MIOCT vs. no MIOCT) consisted of 3 first-year, 2 second-year, and 2 third-year residents (N = 7) to remove potential bias from level of training and surgical experience. Both groups performed the gradable attempts without MIOCT feedback. The maximal depth of the needle and suture were determined after acquiring volumetric MIOCT scans encompassing a 7.4 × 10 × 10 mm field of view centered on the pass. To grade the suture placement, B-scans at the point of maximal depth of the suture needles were marked by a masked trained grader.
For the instrument positioning study, the group consisted of 1 first-year and 2 second-year surgical fellows to remove potential bias from level of training. All maneuvers were recorded with MIOCT volumetric scans encompassing a 3 × 5 × 5 mm field of view centered on the surgical instrument. To grade the distance of the instrument to retinal surface, B-scans showing the instrument cross section closest to the retinal surface were marked by a masked grader.
Grading and statistical analysis. Custom MATLAB software (Mathworks; Natick, MA) was created to grade the suture needle placement. On averaged B-scans at the point of maximal suture needle depth, a masked grader marked the surface of the corneal endothelium, epithelium, and suture needle after refraction correction to compensate for image distortion 65 . The corneal thickness was then identified as the shortest path between the endothelial and epithelial layers that included the marked suture needle position. The percent suture depth was then calculated as the ratio of the depth of the needle of suture (distance from the epithelium to the suture or needle) over the corneal thickness (distance from epithelium to endothelium). The difference between the percent suture depth and target suture depth was then calculated. Commercial software (Meazure, C Thing Software) was used to grade the distance from the instrument to retinal surface in the instrument positioning study. The distance in number of pixels was measured by a masked grader on B-scans at the point of closest instrument proximity to retinal surface and converted to millimeters using a calibration factor.
Statistical comparisons between the groups performing suture placement with and without MIOCT guidance were analyzed with the two-tailed Wilcoxon Rank Sum test, with the null hypothesis that the different samples are from distributions with equal medians. The samples were considered independent since suture placements with and without MIOCT guidance were performed on different cadaveric porcine eyes. P < 0.05 was classified as statistically significant. Statistical comparison between instrument placement trials in porcine retinal surgery performed with and with 4D MIOCT guidance were analyzed with the two-tailed Wilcoxon Sign Rank test, with the null hypothesis that the differences between paired samples come from a distribution with zero median. P < 0.05 was classified as statistically significant.
4D MIOCT translation to human subjects. All human participant studies (ClinicalTrials.gov Clinical Trial Identifier: NCT01588041) were performed after obtaining informed consent and in accordance with protocols approved by the Duke University Health System Institutional Review Board. Prior to imaging, informed consent was received from all subjects after explanation of the possible consequences and nature of the study. All methodology and portions of this study followed the tenets of the Declaration of Helsinki and were conducted in compliance with the Health Insurance Portability and Accountability Act. During imaging, MIOCT optical power on the eye was below 1.7 mW and the surgical visible illumination was reduced by 20% to maintain the total irradiance below the maximum permissible exposure for ocular illumination 66 . Unless otherwise specified, retinal MIOCT scans were 3 mm axially and were composed of 300 A-lines/B-scan and 100 B-scans per volume, resulting in a volumetric rate of 3.33 Hz. All anterior segment MIOCT scans were 6-7.4 mm axially and were composed of 500 A-lines/B-scan and 100 B-scan per volume, resulting in a volumetric rate of 2 Hz.
While the functionality of the operating microscope was unchanged after integrating the MIOCT scanner and HUD, the distance between the surgical oculars and microscope objective was increased and resulted in an increase in the height of the surgical oculars relative to the patient. To mitigate potentially negative ergonomic effects due to the height increase, one of the two sets of oculars for surgical assistants was removed before surgery to decrease the height of the main oculars for the surgeon, since only one surgical assistant was required in all cases of interest.