Abstract
Due to loss of tactile feedback the assessment of tumor margins during robotic surgery is based only on visual inspection, which is neither significantly sensitive nor specific. Here we demonstrate time-resolved fluorescence spectroscopy (TRFS) as a novel technique to complement the visual inspection of oral cancers during transoral robotic surgery (TORS) in real-time and without the need for exogenous contrast agents. TRFS enables identification of cancerous tissue by its distinct autofluorescence signature that is associated with the alteration of tissue structure and biochemical profile. A prototype TRFS instrument was integrated synergistically with the da Vinci Surgical robot and the combined system was validated in swine and human patients. Label-free and real-time assessment and visualization of tissue biochemical features during robotic surgery procedure, as demonstrated here, not only has the potential to improve the intraoperative decision making during TORS but also other robotic procedures without modification of conventional clinical protocols.
Similar content being viewed by others
Introduction
Robotic surgery is becoming a preferred means of treatment for numerous cancers and diseases in areas such as head and neck, urologic, gynecologic, and thoracic surgery. This technology can provide high surgical precision and improve recovery times for the patient1. To date, most robotic interventions are performed with the da Vinci Surgical System2,3.
Irrespective of the surgical approach, the success of surgery relies on the complete excision of the diseased tissues. In most cancers, including head and neck cancer, surgery is often the first choice of treatment, depending on the prognostic factors of the patient, the tumor classification, and the potential morbidity associated with acquiring complete resection with negative surgical margins4,5. Frozen section biopsy is the most commonly employed technique for rapid margin assessment. Positive margins with disease present at the periphery of the resected specimen are associated with worse disease specific survival in many tumors6,7. Preventing positive margins is a key surgical objective and is now recognized as a quality metric for surgeons in many fields. While standard pre-operative imaging can provide tumor location, surgeons still rely on visual and/or tactile guidance to determine the interface between normal and abnormal tissue and histology to confirm microscopic clearance of abnormal disease. In robotic surgery3,8 the lack of tactile feedback limits the surgeon to visual assessment through the robot camera, thus increasing the possibility of positive margins and residual diseased tissue.
To minimize such risk, the da Vinci Surgical System employs advanced technologies including highly-magnified 3D vision and precisely controlled instruments3. The most recent integration of the FireFly module into the da Vinci Surgical System has resulted in a paradigm shift in surgical navigation. This near-infrared fluorescence (NIRF) imaging technology allows for the visualization of blood flow and related tissue perfusion employing indocyanine green (ICG) as the fluorescent agent. Recent studies have employed this technology for numerous fluorescence-guided robot-assisted interventions including, but not limited to, robotic oncologic surgery, lymph nodes identification, vascularization and organ and tissue perfusion imaging9,10.
Nevertheless, the fluorescence-guided surgery, primarily with ICG, encompasses two significant limitations. First, the field of view augmented with ICG fluorescence information provides a qualitative rather than quantitative assessment of the tumor delineation. ICG imaging is based on perfusion and accumulation in tissue due to leaky capillaries seen in many diseased states such as neoplasms. However, it is not a molecularly targeted imaging probe and thus lacks specificity for early neoplastic lesions or field cancerization11,12. Second, ICG fluorescence imaging requires the lights of the operating room (OR) to be turned off, with only a few recent studies trying to address this limitation13.
A different approach that has the potential to guide surgical interventions is based on tissue autofluorescence. This technique has demonstrated potential to provide diagnostic contrast for diverse cancers, including head and neck cancer14,15. Steady-state (intensity or spectral) autofluorescence techniques have been reported as potential diagnostic tools for oral cancer. Such measurements, however, are affected by the experimental conditions, including changes in excitation-collection coupling due to tissue motion, non-uniform illumination or fluorescence collection due to surface geometry, and photobleaching16. In contrast, time-resolved autofluorescence techniques can address these limitations by resolving the dynamics of the fluorescence decay (lifetime)17,18,19. Additionally, fluorescence lifetime provides a contrast mechanism that can be employed for the discrimination of endogenous fluorophores with overlapping spectra. By exploiting alterations in tissue structural and metabolic characteristics resulting from cancer autofluorescence, lifetime imaging has the potential to provide information about tissue molecular composition including enzyme cofactors involved in cellular metabolism, matrix proteins and inflammatory activity20,21. Recent studies demonstrate the ability of this approach to delineate head and neck tumors from surrounding normal tissues in patients14,15.
Current study demonstrates the first-in-human proof-of-principle for integration of an autofluorescence technique with a surgical robot and real-time augmentation of the surgical field of view with parameters encoding tissue diagnostic information for surgical guidance. Specifically we show (1) the synergetic integration of a scanning multispectral time-resolved fluorescence spectroscopy (ms-TRFS) system22,23 into the da Vinci Surgical System that enable real-time continuous evaluation of tissue biochemical features; (2) the evaluation of the functionality of the integrated system in vivo in swine models; and (3) the ability of the integrated ms-TRFS system to operate in human subjects undergoing TORS procedures and to provide real-time diagnostic contrast based on intrinsic tissue autofluorescence properties. While TORS is not as common as other applications of robotic surgery, there are many advantages of using it as a model to demonstrate the feasibility of integrating the ms-TRFS system into the da Vinci Surgical System workflow. The unique attributes of TORS, working with multitude of tissue types and providing robotic access to challenging anatomic locations, demonstrate the advantages of the robotic approach and the utility of the ms-TRFS system to query tissue composition and surgical margins during surgery. The integrated system is designed to collect and analyze ms-TRFS data in real-time from areas of interest in the oropharynx prior to and after surgical excision of cancer. The system also extracts fluorescence parameters related to tissue biochemical properties that are used to augment the robot’s field of view. In addition, current approach can serve as a framework for the clinical translation of other point-scanning optical techniques, such as elastic-scattering spectroscopy, diffused reflectance spectroscopy, or Raman spectroscopy24,25,26, to name a few.
Results
Integration of ms-TRFS and da Vinci Surgical Systems
The ms-TRFS system used for this study is packaged in a compact cart that allows minimal interference with OR procedures and is integrated into the da Vinci Surgical System (see Fig. 1 and Methods). The system can provide spectroscopic images registered with the operating field by combining freehand scanning of the optical fiber-probe with the white-light video stream acquired by the endoscopic camera (see Methods).
Delivery of an additional visible aiming beam through the optical probe, as described in earlier work from our group23, enables dynamic tracking of the location probed by ms-TRFS. An illustration of how the information acquired by a point-measurement technique is transformed into a two-dimensional image with “memory” capability, resulting in detailed visualization of the fluorescence lifetime contrast, is displayed in Fig. 2. The two distinct formats to display the ms-TRFS derived parameters on the surgeon console (the two panels labeled with (1) and (2) in Fig. 2a) were demonstrated in vivo in porcine animal models. Supplementary Video 1 displays a scanning sequence inside the oral cavity from the animal’s tongue. For each location of the probe, it takes ~0.04 sec (practically instantaneously) to acquire, process/analyze and display the ms-TRFS parameters on the surgeons’ console. The augmented frame of Fig. 2b showcases the ability of the ms-TRFS system to display the tissue autofluorescence features on the da Vinci surgeon console once integrated into the da Vinci Surgical System. The endogenous fluorescence contrast originates from the different tissue constituents emitting fluorescence under the same excitation. Figure 2c depicts the transient signals acquired simultaneously for the four spectral bands of the system (see Methods) from two distinct areas (pixels) in the oral cavity. The wealth of fluorescence parameters retrieved from each fluorescence pulse transient during the scanning procedure is depicted in Fig. 2d. This includes the spectral ratio (i.e. intensity of one channel over the sum of intensities from all channels), lifetime, and the 12 Laguerre coefficients27.
Efficient and safe imaging in animal models and patients was achieved with several additional functionalities described in the following section. This includes means to account for the wide variations of signal observed during freehand scanning, the potential fouling of the probe with biological matter during imaging, and the limitation of tissue exposure to UV light during ms-TRFS measurements.
Dynamic gain control
Acquisition of robust ms-TRFS data during surgery depends on the system’s ability to account for sudden variations in fluorescence intensity. Such differences are due to variations in the fluorescence signal generated by different types of tissues encountered during surgery and changes in excitation-collection geometry caused by variations in the probe-to-tissue distance when performing freehand scanning. The fluorescence signal generated by tissue is collected by the optical probe, spectrally resolved and sent to a photodetector (microchannel plate photomultiplier tube) where it is converted to an electrical signal, amplified and sampled by a high-speed digitizer (see Methods). High signal-to-noise ratio ms-TRFS data are obtained for the highest signal voltage that can be generated by the photodetector and amplifier without causing saturation of the digitizer. This constraint can be reconciled with the wide variations of collected fluorescence signal noted above by modulating in real-time the bias voltage - and hence gain - of the photodetector, based on the intensity of the measured signal. This high dynamic range measurement scheme based on closed loop control of the detector gain is adapted from a method reported by our group22. Figure 3a depicts representative results from the evaluation of this method during an in vivo scan and demonstrates its ability to compensate for rapid changes in acquired intensity. The scanned area consists of 500 consecutive points and includes normal mucosa, taste buds, and charred tissue. The amplitude of the acquired transient signals over the 500 points considered is depicted in Fig. 3a (red dotted line), along with the applied gain (blue solid line). As seen, the gain varies between ~1700 Volts and ~1900 Volts, whereas the amplitude remains relatively constant and always within the digitizer’s dynamic range (i.e. 0.05 and 0.8 Volts, red transparent region). Despite the difference in tissue types and probe position, the SNR for all 500 points remained stable (48.2 ± 2.4 dB) throughout the scan, validating the robustness of the dynamic gain control. This is further confirmed by the relatively constant amplitude of the acquired transient signals (0.54 ± 0.1 V). The variation of fluorescence signal intensity is recovered by correcting the measured signal (red dotted line) with the applied high voltage bias (blue line) and the gain/bias voltage characteristic of the photomultiplier tube (0.03 V−1) and shows the signal that would be obtained without adaptive gain control (solid orange line). We observe that without the adaptive gain control this signal would fall outside the dynamic range of the system for a relatively large number of measured points.
Optical fiber probe contamination (blood, cauterization byproducts) and cleaning strategy
Besides the system’s ability to account for rapid changes in fluorescence signal, acquisition of robust ms-TRFS data also relies on the ability to keep the distal end of the fiber clean throughout the entire intraoperative procedure. Fiberoptic contamination due to adhesion of either blood or surgical debris (i.e. cautery byproducts) may affect the characteristics of the measured fluorescence. The optical probe distal end was treated with a hydrophobic coating (see Methods). The ability of this coating to limit contamination of the probe following contact with blood was evaluated in vivo in the oral cavity of a swine. The ms-TRFS measurements were performed in 4 configurations: clean fiber reference measurement (n = 54 measurements), in contact with blood (n = 204 measurements), in air following contact with blood (n = 207 measurements) and close from tissue (n = 75 measurements). The signal collected in each configuration for all four channels of the instrument is displayed in Fig. 3b. Imaging in blood led to a small increase in detected signal due to the underlying tissue, with no significant difference in the signal before and after exposure to blood. Channel 1 shows a minor increase in the intensity before exposure to blood, compared to the one after, which is due to possible proximity to tissue with high collagen or elastin content (i.e. although in air the fiber is still inside the oral cavity). Additionally, strong signal could be acquired from tissue following exposure of the optical probe to blood, thus demonstrating the efficacy of the hydrophobic layer at the fiber’s distal end in preventing blood contamination.
Cauterization byproducts can also contaminate the fiber resulting in unwanted increase of fluorescence background. A sequence showing the ability to clean the fiber in the oral cavity using saline is demonstrated in Supplementary Video 2. A clean fiber placed inside the oral cavity of the swine was situated a few centimeters away of any tissue before cauterization, after cauterization, and after cleaning using a saline solution. The smoke and projections from cauterization resulted in a fluorescence signal that can be observed in all 4 ms-TRFS detection channels. Saline flushing removed such fluorescent debris. The cleaning process requires only a few seconds and does not affect the standard surgical procedure or present any risk for the patient (Supplementary Video 2).
Tracking of Maximum Permissible Exposure (MPE)
Exposure of human tissue to laser light must be performed in accordance with the American National Standard for Safe Use of Lasers (ANSI Z136.1-2014, Laser Institute of America)28. Based on light pulse characteristics, an MPE value is determined to prevent thermal damage to tissue (see Methods). The exposed locations of the tissue are identified to create the ms-TRFS augmented video streams and a cumulative exposure for each pixel of the field of view is determined. When the cumulative exposure reaches 50% of the permissible exposure, the corresponding pixels are marked in black to warn the surgeon that such areas should not be further exposed (Fig. 3c).
Integrated ms-TRFS and da Vinci Surgical Systems: In vivo evaluation in human patients (case study)
The ability of the system to allow the surgeon to acquire and display ms-TRFS measurements during a TORS procedure was evaluated in four patients – case study. Figure 4 shows the results from ms-TRFS interrogation of surgical margins prior to tumor resection in one of the patients. The measurements, as captured from the surgeon’s console, that produced the results shown in Fig. 4 can be seen in Supplementary Video 3. Results from the other three patients can be seen in Supplementary Data 2 and Supplementary Fig. 2.
The data collected from this patient demonstrate the potential of ms-TRFS measurements to provide diagnostic information and delineate tumor from normal tissue during TORS procedures. The results are summarized in Fig. 4a. Specifically, three regions of interest were identified by the pathologist and co-registered with the lifetime maps constructed during the scanning procedures: carcinoma in situ, carcinoma in situ over normal lymphoid tissue, and normal tissues, (Fig. 4a, left). The lifetime distribution of these areas (Fig. 4a, middle) reveals that the carcinoma in situ over normal lymphoid tissue presents higher lifetime values than the other two tissue types in the 390/40 nm channel of the ms-TRFS system. The normal tissue, on the other hand, presents significantly higher lifetimes with respect to the carcinoma in situ in the other two spectral channels. The contrast between the three tissue types and the intrapatient variability is further demonstrated in the scatter plot of Fig. 4a (right), where the lifetime values are plotted for all three channels. Two representative histologic sections used to identify these three regions are shown in Fig. 4b,c.
Discussion
Here we present the first integration of a multispectral label-free fluorescence lifetime technique into the da Vinci Surgical System and demonstrate the ability of this technique to (1) record robust time-resolved fluorescence data during TORS procedures in human patients, (2) augment autofluorescence lifetime data onto conventional white-light images seen on the da Vinci surgeon console in real-time, and (3) provide contrast between distinct tissue types, including alteration by surgical procedures in human patients. Although TORS presents numerous benefits for the patient, it removes tactile guidance leaving only visual inspection for the assessment of tumor margins. The technique reported here can provide a novel surgical guidance technique with potential to become a valuable tool during robotic surgical procedures.
We showed that the ms-TRFS technique can be integrated into the da Vinci Surgical System without affecting the standard functions of either system. The fiberoptic probe can be easily coupled with the da Vinci 5Fr Introducer; ms-TRFS data acquisition and visualization is implemented with the standard video output connections, already available in the robotic system. After the ms-TRFS system preparation, the ms-TRFS measurement is controlled through a single foot-switch, which is placed at the surgeon console and directly accessed by the surgeon. Since the ms-TRFS data visualization is in real-time, the surgeon can dynamically assess the signal quality and interact accordingly (i.e. reduce the probe-surface distance or reduce/increase the scanning speed). Also, by recording each pixel’s exposure, the MPE can be approximated and the surgeon can visually be notified to avoid the regions where the MPE was reached. Future plans include the disabling of the excitation source in case these regions are visited even though the MPE threshold has been reached. This measure, in combination with the conservative definition of the applied MPE threshold will ensure that the true MPE will never be reached during a scanning session. Visualization of the acquired data is implemented through the robot’s TilePro function; thus, it does not interfere with the standard robotic platform currently used in clinical practice. None of the two surgeons participating in the study presented herein had any issue or difficulty in performing the measurements, even though one of them had no prior experience with the system (see Methods and Supplementary Data 2), which further increases its prospect for clinical translation. Moreover, the same approach can be adapted to other non-TORS procedures conducted with the robot (Supplementary Fig. 1 and Supplementary Data 1).
One important aspect of the technology described herein is its ability to acquire robust autofluorescence data from tissue. In part this is due to the high dynamic range of the ms-TRFS system, which is enabled by the dynamic control of the gain applied to the detector. This feature ensures that data are acquired regardless of rapid changes in quantum efficiency or light collection geometry. In addition, the saline flushing method developed enables the rapid cleaning of the distal end of the fiber from cauterization byproducts or smoke. Finally, the hydrophobic coating of the fiber’s distal end prevents blood contamination, minimizing signal attenuation risks.
The developed approach for intraoperative guidance presents one basic limitation challenge; the tissue motion impact on acquired data. While tissue motion can be controlled with instrumentation in TORS, this may not be as possible during laparoscopic procedures. This challenge can be addressed reasonably well with the development of methods that could detect the motion and/or tissue deformation and correspondingly transform the lifetime maps. Moreover, during intraoperative guidance, surgeons rely on the assessment of relatively small areas, close to the interrogated locations. As demonstrated by this study, the technique described herein can image such areas without being degraded from motion blurring.
Earlier fluorescence lifetime instrumentation based on fiberoptic probes with direct access to distinct areas in the oral cavity has demonstrated potential to differentiate between cancerous and normal tissue in the oral cavity14,15,29. However, these techniques were not able to either provide real-time feedback on tissue biochemical features or to augment the fluorescence parameters on tissue surface. They lack the “memory” function or the ability to trace-back the evaluated areas. This function is important for intraoperative guidance, as well as for further validation with standard histopathology analysis. In this work, we demonstrate for the first time a technique able to provide continuous real-time visualization of areas interrogated by a fiber optic probe, streamed to the surgeon console and to generate diagnostic contrast based on tissue autofluorescence properties.
While the scope of this study was primarily to demonstrate the integration of ms-TRFS system into the da Vinci Surgical System and its functionality during conventional TORS procedures, we have also shown the potential of the ms-TRFS data to discriminate different tissue types and/or conditions, namely carcinoma in situ, carcinoma in situ over lymphoid tissue such as tonsils, and normal tissue. Importantly this was achieved based on tissue intrinsic fluorescence properties without the need of contrast agents. These findings are anticipated to steer future studies to quantify the discriminatory power of ms-TRFS.
In addition, the technique demonstrated here could be adapted to other point-scanning tissue diagnostic modalities as it can overcome the lack-of-memory limitation without the need of specialized equipment or fixed geometry requirements26. Moreover, the methodology presented herein can easily be modified to allow for integration of any fluorescence spectral band of interests and display of measured quantities derived from light-tissue (e.g. elastic scattering spectroscopy, diffuse reflectance spectroscopy, or Raman spectroscopy).
Methods
Study design
The objective of this study was to determine whether ms-TRFS, a point-scanning technique, could be employed during TORS for real-time assessment of tissue biochemical characteristics, including delineation of tumor. To test this approach, we integrated the ms-TRFS system in the da Vinci Surgical System and performed a series of in vivo measurements, both in animal models and human patients undergoing TORS. The measurements were targeted to validate the performance of ms-TRFS to acquire and visualize data online without interfering with the standard clinical practice during TORS.
Animal models
The functionality of the integrated ms-TRFS system into the da Vinci Surgical System was evaluated in vivo on three swine (N = 3). Animal studies were approved by the Intuitive Surgical Institutional Animal Care and Use Committee (protocol #009, entitled Imaging and Visualization in Surgery). All methods, facilities and transportation complied with current legal guidelines and regulations. Anesthesia was used for all measurements.
These experiments were designed to (a) investigate the capability of the ms-TRFS system to operate in an in vivo setting comparable with that used in humans, (b) optimize ms-TRFS measurements and data acquisition protocols, (c) evaluate the ability of the ms-TRFS system to collect data from different tissue types inside the oral cavity, (d) evaluate conditions (blood, surgical debris, cautery-derived effects) that might affect the ms-TRFS signals and to engineer solutions to overcome such challenges. In addition, the prospect of employing this system during laparoscopic interventions was investigated.
All measurements were implemented inside the oral cavity of the animal models, except from the laparoscopic measurement. For this, two small incisions in the abdominal wall of the animal allowed the EndoWrist Introducer and the endoscope to access the area of interest. The surgeon performing the interventions identified specific areas of interest and performed the scanning measurements with the da Vinci Surgical System. As depicted in the results and videos, there was no specific scanning pattern. Instead the surgeon could freely scan the region of interest, as long as this motion was not obscuring the field of view of the endoscope.
Human patients
The ability of the ms-TRFS system to operate synergistically with the da Vinci Surgical System was tested in four human patients (N = 4) during standard TORS interventions. The study was performed with approval from the University of California, Davis Institutional Review Board and in accordance with all relevant guidelines. All patients provided written informed consent. This study was designed to determine whether ms-TRFS could be applied intraoperatively to provide continuous feedback about tissue biochemical features, as well as to augment the conventional endoscopic white-light images. Two surgeons participated in the study. One surgeon also performed the swine procedures, while the other had no prior experience with the ms-TRFS system (see Supplementary Fig. 2).
Before the robotic surgery the endoscope and the EndoWrist instrument with the fiber from the ms-TRFS system were inserted inside the oral cavity of the patient under anesthesia. The surgeons identified the region of interest, based on pre-operative planning, and lifetime measurements were acquired by scanning over that region. As in the animal models, surgeon-directed scanning pattern with the EndoWrist instrument was easily accomplished with fluorescent analysis being performed in real time. For all patients, scanning time was less than 5 minutes without any significant delays or interruptions to the standard clinical procedures. Results of this study are presented here and to the authors’ knowledge this is the first-in-human application of intraoperative fluorescence lifetime imaging during robot-assisted interventions.
Integration of ms-TRFS into the da Vinci Surgical System
The instrumental apparatus employed in this study consisted of a prototype scanning ms-TRFS system interfaced with the da Vinci Si Surgical System via a 5Fr EndoWrist Introducer (Fig. 1).
Optical Instrumentation
In brief, the optical instrumentation consists of a wavelength selection module (WSM) composed of dichroic and band-pass filters that simultaneously spectrally-resolve the fluorescence emission in four spectral channels: 390/40 nm (Channel 1), 466/40 nm (Channel 2), 542/50 nm (Channel 3), and 629/53 nm (Channel 4)30. A continuous-wave solid-state laser (445 nm, TECBL-50G-440-USB, WORLD STAR TECH, Toronto, ON, Canada) coupled into Channel 2 of the WSM allows for injection of an aiming beam via the same path as the fluorescence excitation beam23. The incident power of the aiming beam is approximately 3 mW. The four channels are connected to optical fibers of distinct lengths (acting as delay channels) coupled at the distal ends into a single microchannel plate photomultiplier tube (MCP-PMT, R3809U-50, 45 ps FWHM, Hamamatsu, Japan). After being detected by the MCP-PMT and boosted by an RF amplifier (AM-1607-3000, 3 GHz bandwidth, MITEQ, USA), the fluorescence signals are temporally resolved by a high sampling frequency digitizer (12.5 GS/s, 3 GHz, 8-bit, 512 Mbytes, PXIe-5185, National Instruments, Austin, TX, USA) at 80 ps time intervals. Auto-fluorescence signal induced by the aiming beam (continuous-wave) is filtered-out by the AC-coupled amplifier. A micro Q-switched laser frequency tripled to 355 nm with a repetition rate equal to 2 KHz (Teem Photonics, France) is used for fluorescence excitation. To enable easy transportation from/to the operating room the system is integrated in a movable cart (Fig. 1).
da Vinci Surgical System
The da Vinci Surgical System (Intuitive Surgical, Sunnyvale, CA, USA) is a teleoperated surgical system for minimally invasive operations. It consists of three major parts: (i) the surgeon console, through which the surgeon performs the operation while viewing a high resolution three-dimensional image of the interrogated tissues; (ii) the patient cart, which is docked to the patient during surgery and has four robotic arms; and (iii) the vision tower which is equipped with a high-definition, three-dimensional endoscope and image processing equipment that provides images of the patient’s anatomy in real-time. The system received its first clearance from the FDA in 2000. In 2008 the system received clearance to perform TORS for selected malignant lesions of the pharynx and larynx. During this study, the ms-TRFS system was integrated into the da Vinci Si Surgical System (Fig. 1), but other models are also suitable.
Hardware Integration
A customized fiberoptic probe enabled the integration of the ms-TRFS system into the da Vinci Surgical System. A single multimode 400 μm core diameter silica fiber (BFH48-400, Thorlabs Inc., Newton, NJ, USA) was introduced to the working channel of the 5FR EndoWrist Introducer and used for both fluorescence excitation and collection. This introducer consists of a wrist with three degrees of freedom that allows aiming of the fiberoptic probe for any tissue orientation. The wrist has a tight bend (~12 mm radius), thus fatigue testing was performed to evaluate whether the imaging fiber can withstand the strain. Although no fiber damage was noticed, a >25 mm radius was maintained during scanning sessions to match the fiber’s short-term bend radius (22 mm). In addition, the employed fibers are compatible with low temperature gas plasma sterilization, while the tough tefzel fiber buffer allowed for several sterilization cycles (i.e. use of the same fiber for multiple patients) without any fiber damage. A sterile sheath (4F IMPRESS Vertebral, Merit Medical Systems, Inc. Salt Lake City, UT, USA) was used to protect the fiberoptic probe during measurements.
Preliminary testing showed that the fiberoptic distal end is prone to contamination due to adhesion of either blood or surgical debris (i.e. cautery byproducts). To address this, the fiber was treated with a hydrophobic coating (Aculon, San Diego, CA, USA) and a syringe port for a water flush was added to the sheath. In the presence of hydrophobic coating, simple retraction of the fiber into the sheath and low volume flush is sufficient to clean the fiber tip.
To streamline the flushing process and better protect the fiber outside of the imaging sequence, a motorized telescope functionality was integrated into the system. A module that clips to the sheath and locks onto the fiber with a hemostat (Fig. 1) was developed. This enables retraction of the fiber within the sheath and exposure only during the imaging process. All elements of the mechanism are either sterilizable or protected within a sterile sheath (i.e. the stepper motor).
Software integration
The visualization schemes developed (i.e. the ms-TRFS interface shown in Fig. 2c and the augmented white-light images of Fig. 2b) are shared with the surgeon console through the TilePro interface and thus provide the surgeon with real-time biochemical and/or functional information by means of fluorescence decay dynamics.
To integrate the framework described in23 into the da Vinci Surgical System, a frame-grabber (Epiphan Systems Inc., Ottawa, ON, Canada) was connected between the vision tower and the imaging laptop. The transmitted frames (i.e. one of the two channels of the endoscope) are captured and processed in parallel with the fluorescence transient signals as described in23. Finally, the resulting images are displayed on the surgeon console through the TilePro interface.
Data acquisition, processing, and visualization
Online data acquisition and processing
Two features were adopted to optimize the data acquisition during TORS procedures. The first consists of a method enabling optimization of the signal amplitude against the dynamic range of the digitizer. A closed-loop control algorithm reported earlier22 provides feedback to the high voltage power supply. The latter determines the gain of photon multiplication and alters the signal amplitude dynamically during data acquisition. The second consists of remote laser triggering and data acquisition by the surgeon from the surgeon console via a foot pedal. This feature not only gives control of the system to the surgeon but also prevents unintended exposure from the excitation light.
The data processing module is incorporated into the system’s software platform using LabVIEW (National Instruments, Austin, TX, USA) and implemented in the embedded controller of the digitizer. The approximation of the lifetime values from the acquired transient signals was implemented through a non- parametric non-negative least-squares deconvolution with Laguerre expansion. This method has been described in depth by our group elsewhere27 and represents the basis for achieving real-time quantification of lifetime values.
Real-time data visualization
When the aiming beam module is enabled, the acquired transient signals are transmitted to an external laptop. The deconvolution algorithm (implemented in the laptop) is applied in parallel to the image processing. The framework of employing an aiming beam to augment white-light images with quantitative information (biochemical, functional, statistics outcomes, etc.) has been described by our group recently23. In addition to visualizing in real-time the lifetime values from the four channels of the WSM, the software also monitors the compliance to MPE as defined by the American Standard for Safe Use of Lasers ANSI Z136.128. Specifically, MPE is determined based on wavelength, energy, pulse duration and repetition rate of the UV and aiming beams. Cumulative exposure of the illuminated areas, detected from the endoscope image, is determined under the conservative assumption that the fluence on tissue corresponds to the fiber being in contact with tissue. When the cumulative exposure reaches a set fraction of the permissible exposure (50%), the corresponding pixels are marked in black to provide feedback to the surgeon that those locations should not be exposed further.
Data Availability
All data are available by the corresponding authors under reasonable request.
References
Nezhat, F. R., Pejovic, T., Finger, T. N. & Khalil, S. S. Role of Minimally Invasive Surgery in Ovarian Cancer. J Minim Invasive Gynecol 20, 754–765 (2013).
Huda, M. N., Yu, H. & Cang, S. Robots for minimally invasive diagnosis and intervention. Robot Com-Int Manuf 41, 127–144 (2016).
Leal Ghezzi, T. & Campos Corleta, O. 30 Years of Robotic Surgery. World J Surg 40, 2550–2557 (2016).
Ho, A. S. et al. Decision making in the management of recurrent head and neck cancer. Head & neck 36, 144–151 (2014).
Carrillo, J. F. et al. Retrospective cohort study of prognostic factors in patients with oral cavity and oropharyngeal squamous cell carcinoma. Head & neck 38, 536–541 (2016).
Hinni, M. L. et al. Surgical margins in head and neck cancer: A contemporary review. Head & neck 35, 1362–1370 (2013).
Patel, R. S. et al. Impact of positive frozen section microscopic tumor cut-through revised to negative on oral carcinoma control and survival rates. Head & neck 32, 1444–1451 (2010).
Ahmad, A., Ahmad, Z. F., Carleton, J. D. & Agarwala, A. Robotic surgery: current perceptions and the clinical evidence. Surg Endosc 31, 255–263 (2016).
Daskalaki, D., Aguilera, F., Patton, K. & Giulianotti, P. C. Fluorescence in robotic surgery. J. Surg. Oncol. 112, 250–256 (2015).
Zhu, B. & Sevick-Muraca, E. M. A review of performance of near-infrared fluorescence imaging devices used in clinical studies. Brit J Radiol 88, 20140547 (2015).
Koch, M. & Ntziachristos, V. Advancing Surgical Vision with Fluorescence Imaging. Annu Rev Med 67, 153–164 (2016).
Marshall, M. V. et al. Near-Infrared Fluorescence Imaging in Humans with Indocyanine Green: A Review and Update. Open Surg. Oncol. J. 2, 12–25 (2010).
van den Berg, N. S. et al. (Near-Infrared) Fluorescence-Guided Surgery Under Ambient Light Conditions: A Next Step to Embedment of the Technology in Clinical Routine. Ann Surg Oncol, 1–10 (2016).
Meier, J. D. et al. Time-resolved laser-induced fluorescence spectroscopy as a diagnostic instrument in head and neck carcinoma. Otolaryngol. Head Neck Surg. 142, 838–844 (2010).
Sun, Y. et al. Endoscopic fluorescence lifetime imaging for in vivo intraoperative diagnosis of oral carcinoma. Microsc. microanal. 19, 791–798 (2013).
Gorpas, D. & Marcu, L. In Frontiers in Biophotonics for Translational Medicine: In the Celebration of Year of Light (2015) (eds Malini Olivo & Dinish, S. U.) 1–46 (Springer Singapore, 2016).
Berezin, M. Y. & Achilefu, S. Fluorescence lifetime measurements and biological imaging. Chem. Rev. 110, 2641–2684 (2010).
Chorvat, D. Jr. & Chorvatova, A. Multi-wavelength fluorescence lifetime spectroscopy: a new approach to the study of endogenous fluorescence in living cells and tissues. Laser Phys. Lett. 6, 175–193 (2009).
Lakowicz, J. R. Principles of Fluorescence Spectroscopy. 3rd edn, (Springer, 2006).
Fatakdawala, H. et al. Fluorescence Lifetime Imaging Combined with Conventional Intravascular Ultrasound for Enhanced Assessment of Atherosclerotic Plaques: an Ex Vivo Study in Human Coronary Arteries. J. Cardiovasc. Transl. Res. 8, 253–263 (2015).
Datta, R., Alfonso-García, A., Cinco, R. & Gratton, E. Fluorescence lifetime imaging of endogenous biomarker of oxidative stress. Sci Rep 5, 9848 (2015).
Ma, D., Bec, J., Gorpas, D., Yankelevich, D. R. & Marcu, L. Technique for real-time tissue characterization based on scanning multispectral fluorescence lifetime spectroscopy (ms-TRFS). Biomed. Opt. Express 6, 987–1002 (2015).
Gorpas, D., Ma, D., Bec, J., Yankelevich, D. & Marcu, L. Real-Time Visualization of Tissue Surface Biochemical Features Derived from Fluorescence Lifetime Measurements. IEEE T Med Imaging 35, 1802–1811 (2016).
Zhang, L. et al. Light scattering spectroscopy identifies the malignant potential of pancreatic cysts during endoscopy. Nat. Biomed. Eng. 1, 0040 (2017).
Jermyn, M. et al. Intraoperative brain cancer detection with Raman spectroscopy in humans. Sci. Transl. Med. 7, 274ra219 (2015).
Nichols, B. S. et al. A Quantitative Diffuse Reflectance Imaging (QDRI) System for Comprehensive Surveillance of the Morphological Landscape in Breast Tumor Margins. Plos One 10, e0127525 (2015).
Liu, J., Sun, Y., Qi, J. & Marcu, L. A novel method for fast and robust estimation of fluorescence decay dynamics using constrained least-squares deconvolution with Laguerre expansion. Phys. Med. Biol. 57, 843–865 (2012).
American National Standard for Safe Use of Lasers, Laser Institute of America, Orlando, FL, (2014).
Chen, H.-M., Chiang, C.-P., You, C., Hsiao, T.-C. & Wang, C.-Y. Time-resolved autofluorescence spectroscopy for classifying normal and premalignant oral tissues. Laser Surg. Med. 37, 37–45 (2005).
Yankelevich, D. R. et al. Design and evaluation of a device for fast multispectral time-resolved fluorescence spectroscopy and imaging. Rev. Sci. Instrum. 85, 034303 (2014).
Acknowledgements
We acknowledge W. Smith for technical assistance. This work was supported by the United States National Institutes of Health (NIH) Grant R01 CA187427.
Author information
Authors and Affiliations
Contributions
D.G., D.G.F. and L.M. created the study design. D.G. developed the real-time visualization software. D.G. and J.P. analyzed the data and performed the statistical analysis. J.B. prepared the optical fibers. D.M. developed the ms-TRFS control software and GUI. J.B., S.D., J.P. and D.Y. optimized the ms-TRFS system. D.G., J.B. and J.S. performed the integration of the ms-TRFS system into the da Vinci Surgical System (software and hardware). D.G.F. performed in vivo procedures in swine. D.G.F. and A.B. performed TORS procedures and commented on visualization schemes for optimal clinical experience. R.G.E. performed histology analysis. D.G. and L.M. wrote the manuscript. All authors reviewed the manuscript.
Corresponding authors
Ethics declarations
Competing Interests
The authors declare no competing interests.
Additional information
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Gorpas, D., Phipps, J., Bec, J. et al. Autofluorescence lifetime augmented reality as a means for real-time robotic surgery guidance in human patients. Sci Rep 9, 1187 (2019). https://doi.org/10.1038/s41598-018-37237-8
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s41598-018-37237-8
This article is cited by
-
Robotics in Medical Domain: The Future of Surgery, Healthcare and Imaging
Wireless Personal Communications (2023)
-
Robotic Systems in Surgery
Biomedical Engineering (2022)
-
Click-on fluorescence detectors: using robotic surgical instruments to characterize molecular tissue aspects
Journal of Robotic Surgery (2022)
-
Current Trends in Precision Medicine and Next-Generation Sequencing in Head and Neck Cancer
Current Treatment Options in Oncology (2022)
-
Real-time multispectral fluorescence lifetime imaging using Single Photon Avalanche Diode arrays
Scientific Reports (2020)
Comments
By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.