Introduction

Direct visualization by endoscopy is an early component of clinical pathways for many patients with visceral cancer that directs subsequent care. At operation, direct inspection assists quantification of overall cancer burden especially in areas poorly seen by non-invasive workup and impacts surgical strategy. However, determination of tissue nature by operator judgement alone or indeed by traditional biopsy (which takes time, risks sampling errors and distorts residual tissue) is imperfect1. Improved intraprocedural tissue characterisation could allow better in situ primary tumour profiling and confirmation/exclusion of metastatic deposits acuminating cancer outcomes and healthcare efficiency2.

Surgical colorectal cancer care is ideally suited for digital advance with both endoscopy and laparoscopy embedded in its diagnosis and therapy. Its endolaparoscopic systems are capable of providing multispectral illumination including tissue-penetrating near-infrared (NIR) spectral vision3. Such capacity can help characterise tissue intraoperatively with most clinical application thus far for tissue perfusion assessment using the non-selective fluorophore indocyanine green (ICG)4,5,6. For this, the NIR-ICG visualization is dynamic, occurring within seconds of agent administration, fitting current surgical practice very well. In contrast, NIR cancer identification strategies have concentrated on advanced preprocedural fluorophore administration with visual interrogation planned for many hours or days later at a time when tissue agent concentrations are relatively static7.

Malignant tumours distinguish from normal tissue and benign neoplasia by hallmark metabolic behaviours including their neovascularization (angiogenesis) as well as cell–cell and host-tumour interstitial interactions8,9. Their irregular vascular architecture and lack of a lymphatic recovery system leads to the tumoritropic accumulation of macromolecules such as albumin in solid tumours (in contrast to their rapid clearance in normal tissue)10. Such characteristics mean even a chemically non-selective perfusate, such as ICG, could functionally discriminate cancer at any site allowing in situ optical delineation. However, when applied simply by preoperative administration and later snapshot observation, high false positive rates pertain (a problem that also affects cell-targeting agents deployed similarly)11.

We hypothesized that kinetic characterisation of the evolving dynamic fluorescence intensity (FI) signature at and within cancer deposits could provide discrimination of cancer versus non-cancer even with a passive, non-targeted fluorophore such as ICG12. As the processes being visualized are more complex than perfusion alone and non-tumour background noise may manifest, video-based analytics could de-risk image interpretation permitting greater interpretative clarity and even potentially a ‘digital tissue biopsy’ through software. Herein we pilot the use of systemic NIR-ICG as a licensed available-now agent for endoscopic colorectal neoplasia perfusion interrogation with the goal of identifying discriminant tissue characterisation features. The automated operative perfusion quantification methodology developed and used for this study is available for download so others may test and add to this exploratory work while we continue to expand our clinical series on a collaborative basis.

Methods

This clinical experience involved adult patients already undergoing endolaparoscopic diagnostic or therapeutic procedures for colorectal neoplasia under general anaesthesia. Informed consent was obtained from all participants in accordance with relevant guidelines and, regulations and with institutional (Mater Misericordiae University Hospital, Dublin, Ireland) ethics approval (1/378/2092, ClinicalTrials.gov Identifier: NCT04220242). ICG is licensed internationally for tissue perfusion characterisation and has an excellent safety profile. NIR Endoscopic Systems are commercially available with the one used in this study (PINPOINT Endoscopic Fluorescence System, Stryker Corp, Kalamazoo, MI, USA) being capable simultaneous white light and NIR display. Therefore, no experimental agents or devices were used and no clinical care decisions changed based on NIR-ICG appearances.

Clinical study

Patients with colorectal tumors were administered ICG intravenously at a dose of 0.25 mg/kg intraprocedurally. The endoscopic NIR system was used to directly visualise rectal tumours continuously transanally for up to 20 subsequent minutes and enabled a video recording to be made of endoscopic white light and NIR appearances simultaneously. Ex vivo resection specimens from a separate cohort also receiving the same dose of ICG systemically were opened and their mucosal surfaces examined post-operatively with the same camera for still image analysis. In addition, to examine for photobleaching effects in tissue, two opened specimens were examined continuously postoperatively for 20 min at the site of neoplasia and the recording analysed similarly to the in situ rectal recordings (see below).

Data collection

The videos along with similar high-resolution photographs of the opened mucosal surface of any resected specimens ex vivo underwent post-hoc fluorescence intensity (FI) analysis. All imagery contained both the neoplastic lesion and some adjacent normal tissue. Patient demographics and reactions possibly related to ICG usage were recorded prospectively along with the routine pre and postoperative pathological and radiological reports to capture cancer-relevant details.

ICG static image analysis

Fluorescence photographs of resected neoplastic specimens were processed post-hoc using ImageJ (NIH, USA)13 after annotation of regions of interest (ROI) on the endomucosal surface regions representing the tumour and adjacent normal tissue (control) avoiding obvious areas of surface blood by a surgeon. Spot mean FI in 25 × 25 pixel areas in the tumour were contrasted with normal mucosa (see Fig. 1).

Figure 1
figure 1

Photograph of mucosal surface of an ex vivo opened, resection specimen in a patient with rectal adenocarcinoma being assessed for tumour static signal intensity using ImageJ. ICG was administered at the time of surgery commencement with the photograph being taken postoperatively using the Pinpoint endoscopic NIR system.

Full size image

ICG dynamic data analysis

For recorded videos (all rectal tumors and the two postoperative specimens being examined re photobleaching effects), a surgeon again postoperatively annotated ROIs as above also avoiding those areas only variably seen over time in the movies. Post-hoc kinetic plotting of FI dynamic patterns of in situ lesions over time, was initially performed using Image-J on grayscale still image stacks serially sampled at one frame per second (fps) from the time of fluorescence signal onset to video end. Thereafter fluorescence measurement was digitalized and automated through the creation of a bespoke, standalone Fluorescence Tracker App (FTA), developed in collaboration with MathWorks®, Galway, Ireland (software source code and executable now publicly available via the MATLAB® File Exchange, www.mathworks.com). This allowed in-app surgeon-ROI-annotation for tumour and control areas from the white-light (RGB) images prior to the onset of non-zero fluorescence intensities in the NIR images with automatic detection and frame-by-frame tracking of surface feature points in the white light images (using minimum eigenvalue14 and Kanade–Lucas–Tomasi algorithms respectively15) to account for the changing pixel locations of these areas due to camera movement. By continuously tracking in the white-light image, the app maps FI in the corresponding pixels of the complimentary NIR image for the ROIs throughout the video duration. Also, following surgeon annotation of tumour versus control ROI, the software can group the dynamic intensity results using K-Nearest Neighbour (KNN) algorithm (see Fig. 2). For automated analysis, five minutes of continuous signal quantification was set as the minimum goal. In the event that tracking was compromised (e.g. due to instruments obstructing the tracking points in the field of view or large camera movements forcing the points out of frame), ROIs were re-selected, automatically featured, and tracking resumed. Interruptions in individual patient readings were collated and gaps documented.

Figure 2
figure 2

Screenshot of Fluorescence Tracker App (FTA, MATLAB®) showing post-hoc automated video-based classification of tumour (blue cluster) versus control tissue (orange cluster) once the regions of interest have been defined via initial annotation by a surgeon on the video frame corresponding to the time of fluorescence inflow. Videos are taken in vivo endorectally using the Pinpoint endoscopic NIR laparoscope while injecting 0.25 mg/kg ICG.

Full size image

Statistical methods

Data collection and statistical analysis was carried out using Microsoft Excel for Microsoft 365(Microsoft, Redmond, WA, USA) and SPSS Software (IBM, Armonk, NY, USA) version 26 with statistical significance ascribed to p-values less than 0.05. Spot ex vivo FI measurements and dynamic mean values were compared from tumours versus control tissue using Mann–Whitney U. FI signals from in vivo videos were plotted against time and examined for distinctions regarding specific curve milestones and variables adapted from previously described methodologies16,17. An intensity rise of 5 grey units was marked as the end of the latency period and the starting block for the incline. Following the rapid inflow period, the point of maximum fluorescence intensity (Fmax), time to achieve this summit (Tmax) and respective gradient (Fmax/Tmax) were calculated along with the time taken (T½max) to achieve half the maximum (F½max) intensity. Following peak intensity, fluorescence clearance from the target tissue was flagged at 100 s (T100) from the start of the descent (see Fig. 3). Final intensity, gradients, duration of tracking, skew and kurtosis were transcribed or calculated. Mean intensities for the tumours and controls before and after KNN grouping were compared using the Wilcoxon signed rank test.

Figure 3
figure 3

Representative graph (Microsoft Excel for Microsoft 365) of in vivo NIR intensity patterns plotting fluorescence intensity versus time to show the curve variables selected for statistical comparison between malignant, benign and normal tissue in videos from patients in this series.

Full size image

Results

In total, thirty-two patients were recruited into the study (see Table 1 and Fig. 4). No patient suffered ICG reaction or toxicity. All procedures progressed as planned, including diagnostic biopsies and therapeutic endoscopic, laparoscopic and open resections, highlighting that the NIR-ICG assessment was coincidental to patient care. Twelve patients with colorectal cancer had their resected specimens examined postoperatively (therefore 4–6 h post ICG administration). Twenty patients with rectal neoplasia underwent endoscopic primary tumour site assessment under simultaneous white-light and NIR examination immediately after ICG administration continually for up to 20 min.

Table 1 Patient and case demographics overall and by analytic methodology.
Full size table
Figure 4
figure 4

Diagram (Microsoft Word for Microsoft 365) showing cohorts of patients in this study by their methods of fluorescence intensity analysis.

Full size image

With respect to spot ex vivo intensity measurement (n = 10) late after ICG administration, mean (± standard deviation) cancer intensity was non-significantly raised (57.0 ± 61.3) compared with control (36.0 ± 28.3) (Mann–Whitney U) with considerable overlap of scores suggesting poor discrimination of cancer by single point in time analysis even some hours after ICG administration (see Fig. 5). Interestingly, photobleaching was seen to occur to only a minor extent (mean 5%) when examined (n = 2) for in excised specimens using continuous examination of ROIs over twenty minutes (the same period of time tumors were visualized in situ) (see Fig. 6).

Figure 5
figure 5

Box plots (SPSS Software version 26) of mean fluorescence mucosal intensity measurements for cancers vs control tissue on postoperative, ex vivo, opened resection specimens via still image analysis with ImageJ in patients receiving ICG at operation commencement.

Full size image
Figure 6
figure 6

Photograph and graph (Microsoft Excel for Microsoft 365) showing the set-up and impact of continuous illumination on ex-vivo tissue containing cancer hours after systemic administration of ICG in vivo. Postoperatively, the specimen was opened and illuminated with a clamped NIR camera and the video recording tracked continuously to elucidate FI over time with static position and in situ dye concentration. This showed minimal induction of photobleaching both at the centre and peripheries of the image field of view.

Full size image

Regarding dynamic recordings of in-situ lesions, grossly different patterns of ICG flow through cancer versus benign tumors were visually apparent-with dye being slow in ingress into cancers early and seeming to persist for longer once penetrated versus adjacent normal tissue. Re FI, eight recordings (three cancers), were analysed manually by Image J (1 fps) and both these and the next twelve were analysed using the automated FTA. Both manual Image J analysis (n = 8) and automated FTA analysis (n = 20) of lesions in situ being dynamically profiled displayed potentially discriminative statistical differences between cancers and benign tumours from their FI temporal signature kinematics. Figure 7 shows graphs of mean FI over time for such lesions grouped by their nature using both the manual Image J and FTA methodology while Fig. 8 shows for illustration the FI time-plot for a representative benign and malignant lesion. Alongside providing cleaner, richer data due to its higher sampling frequency (30 fps), the FTA method was faster, required less user input and sampled more intensely allowing discriminative curves with shorter video analysis (average continuous FTA tracking time 601 s vs 1903s with manual ImageJ). Furthermore, only 142 of 24,051 s (0.006%) of videos had to be censored due to inability to track serially using the FTA.

Figure 7
figure 7

Graph (Microsoft Excel for Microsoft 365) of mean fluorescence intensity per frame vs time for continuous (video) in vivo fluorescence signals using (a) manual Image J methodology (1 fps) and (b) the Fluorescence Tracker App (30 fps) with automated feature detection and tracker in the video recordings of patients with both benign and malignant rectal lesions. Mean frame intensity for cancer/benign tumours and normal tissue (solid lines) and logarithmic fitted curves (dotted lines).

Full size image
Figure 8
figure 8

Illustrative example graphs (Microsoft Excel for Microsoft 365) of dynamic time vs fluorescence signal intensity plots for both (a) a benign lesion using the manual (ImageJ) method above and (b) a cancer using the automated Fluorescence Tracker App for endoluminal in vivo assessment while injecting 0.25 mg/kg ICG.

Full size image

Tables 2 and 3 show the measures of graph parameters from the lesions viewed endoscopically by both manual ImageJ and automated FTA FI curves. Statistical analysis of temporal kinetic patterns generated by the manual quantificative method with respect to flow milestones of lesions versus global control readings revealed significant peak intensity differences (p = 0.047, Kruskal Wallis). Direct Fmax comparisons showed that the cancer intensity curve zenith (Fmax 202 ± 35.68) was significantly higher than for both the benign tumours (165.8 ± 11.69, p = 0.02) and control tissue (169.63 ± 12.84, p = 0.04) (Mann–Whitney U). With respect to the FTA graphs, three-way analysis using Kruskal–Wallis comparison of cancer, benign and control regions showed significant differences in upslope (gradient from baseline to Fmax p = 0.039) and for both the downslope and fall between Fmax and T100 (p = 0.046 for each). Direct comparison with Mann–Whitney U between cancers and control regions showed that cancers peaked significantly later, took significantly longer to rise from baseline to maximum and exhibited a shallower gradient to the peak. This resulted in a significantly later T100. Skew for cancers was further to the right (− 1.69 ± 1.17 vs − 0.41 ± 1.37 p = 0.04). In comparing cancers with benign lesions, there were also significant differences by T100 in terms of both fall and downward gradient. There was no statistically significant difference in these measures between all lesions together and benign lesions alone compared with the controls.

Table 2 Tabulated measures of graph parameters from manual ImageJ quantification of fluorescence intensity curves for videos of rectal lesions viewed in vivo, endoscopically under both white light and near-infrared continuously after 0.25 mg/kg ICG administration. g.u denotes grey units.
Full size table
Table 3 Tabulated measures of graph parameters using the automated Fluorescence Tracker App (MATLAB®) of fluorescence intensity curves for videos of rectal lesions viewed in vivo, endoscopically under both white light and near-infrared continuously after 0.25 mg/kg ICG administration. g.u denotes grey units.
Full size table

Within the group analysed by FTA, six cancer cases underwent application of the KNN classification algorithm. With surgeon ROI (lesion and control) selection, the algorithm was able to cluster the data into two groups segregating cancer from normal tissue automatically, in a fashion not significantly different to the manually segregated data (Wilcoxon signed rank resting). However, the automated classification was unable to reproduce the significantly raised ‘time to rise’ demonstrated on neoplastic tissue (p = 0.041) versus control when these were manually grouped.

Discussion

Fluorescence-guided surgery is proposed as the next progressive iteration in cancer surgery, especially endo-laparoscopic operations2,3. Notably, it is already impacting decision-making and step-sequencing in reconstructive surgery both of the gastrointestinal tract18 and other tissues19. As well as a means to demarcate watersheds between perfused and non-perfused intestine, other uses include critical boundary identifications such as ureters20,21 and fascial planes22 by direct instillation. For cancer and draining lymph node identification, ICG has been placed peritumorally submucosally by endoscopic tattoo23,24,25. In situ revelation of cancer including tumour margination however requires labelling pathology rather than physiology. While cancerous tissue can retain fluorescent agents, there remains a significant issue with false positive findings26,27. Because of this lack of sensitivity, most research groups are now focusing on development of new cancer-selective agents although accuracy remains problematic11,28,29,30. Given that cancers likely act to accumulate agents in their vicinity differently to other areas that trap agents more passively, we thought to observe dynamically the processing of ICG by tumours under continuous direct observation different to others who have focused on single point in time observations.

In this experience, digital FI quantitative assessment of tumour perfusion angiograms has been demonstrated, controlling too for potential impact due to photobleaching (a well described phenomenum in vitro25,31 but one that seems less relevant in vivo due perhaps to protective effect of tissue and dynamic turnover of dye in the ROI in situ). This desktop post-hoc based assessment of the curve metadata has demonstrated possible discriminative flow milestones and timepoints related to the early phase behaviour of the dye in the peritumoral environment that preludes late stage appearances. While additional distinguishing parameters may also exist and could be identifiable via longer videos enabling analysis of dye processing and efflux patterns and/or by using more sophisticated analytical methods, this initial work is encouraging (and indeed longer in situ observation periods may be clinically impractical).

Overall, the fluorescence signal in cancers takes longer to peak and retains this intensity longer (consistent with impaired local pharmacokinetics compared with healthy tissue). Such a distinct FI signature could be visually noticed in both grey-scale, composite and threshold modes and could be quantified with post-hoc image data analysis. Manual intensity quantification allowed mathematical comparisons of the FI time patterns although such data generation was labour-intensive while episodic sampling (even at 1fps) made smoothing of data difficult and background noise impaired statistical analysis. Automating feature detection, tracking and quantification enabled higher sampling rates (30fps) and so provided richer data for analysis with processing time being a function of computational ability rather than human labour.

While further work needs to be done these findings are of great potential clinical relevance. The ability to optically characterize neoplasia through its biological hallmark behaviours could help optimize clinical diagnostic pathways by immediately indicating the nature of lesion and facilitate, in cases of benign aetiology, immediate local excision (or conversely caution such an approach in lesions with benign appearances but containing an invasive focus). Avoidance of biopsy is increasingly advocated in these cases as such an act can introduce architectural distortion in the samples and induce fibrosis in the residual lesion (confounding luminal excision)32. In lesions where biopsy is still indicated, the fluorescent signal profile could indicate the site of most relevant yield re cancer detection. With regard to visible lesions, the key issue to discriminate is cancer versus not cancer and so benign tumor behaviour being similar to normal tissue fits the clinical paradigm re lesion management.

To be maximally useful however this work of course necessitates future effort translating our dataset into a classifier which would allow validation against a gold standard such as histopathology. In this regard, KNN clustering showed encouraging discriminative abilities and which may be refined with further artificial intelligence methodologies33. Detailed region by region biopsy collated to continuous FI perfusion criteria may too yield digital tumour margin delineation and heterogeneity appreciation. Greater patient numbers is of course needed too and, with early proof of principle now provided, a larger clinical study is now ongoing to develop and validate these findings 32 while AI methods are also being developed to enable real-time deployment34.

Cell-specific targeting remains of great interest for future advances in this area but a deep understanding of tumoural functional characteristics is necessary to best inform the purpose and level of probe engineering needed. ICG accumulates quickly and can be effectively imaged with a variety of commercial cameras and so there is much to be learned from this study for new agent development. Its non-specific tumour uptake mechanisms include enhanced permeability and retention (EPR), aberrant gap junctions and dysregulated pinocytosis35,36,37. As ICG can be seen in this experience to passively concentrate in tumours, it seems likely that so too will, at least in part, a “cancer-specific” agent. Such internalisation mechanisms could prove advantageous in performance over cell-surface or ligand-specific agents which could be limited in applicability due to variance in intra and inter-tumour target expression. It is likely also that the ICG uptake mechanisms through cancer pathophysiology seen here are shared across other common cancers and could extend to sites of regional spread and metastatic disease.

While cancer cell selectivity can be engineered in cell culture, tumour imaging in vivo encompasses a variety of host:cancer interactions as well as additional cell signalling and response pathways. Furthermore, additional unrelated sites of inflammation38 or injury will equally be prone to incidental, unintended probe accumulation. This indicates that a relatively high rate of false positive findings may continue despite any additional tumour targeting features. Rigorous preclinical and clinical study controls are required to minimize heuristic mechanistic targeting assumptions. While long-in-advance (24–48 h) administration of the fluorophore has been proposed to offset non-cancer signalling (and allow signal strengthening by agent concentration), a significant downside exists with respect to the unpredictability of operative timing and patient-to-patient pharmacokinetic variances.

An inbuilt synchronous and continuous dynamic quantification algorithm and screen display can be of use to all fluorescence imaging agents and could even enable synchronous use of multiple probes of different wavelength colours. Addition of cancer-specific targeting could still accelerate agent accumulation at the ROI following systemic distribution, enabling short-window observations with the greatest sensitivity and specificity possible within minutes. This would be of use for the operator more akin to episodic perfusion assessment with the added biochemical, biological and imaging complexity hidden. “Off–on” fluorogenic compounds are especially attractive for this purpose as their signalling only switches on when they arrive at the appropriate microenvironment39.

This study is of course limited by the small cohort studied although has the strength that patients were recruited into the study without arbitary preselection. Fluorescence data is also dependent on the recording of the detected fluorescence signal. Video stability, suctioning, instrumentation and tumour oozing/desloughing impaired continuous tracking and required further user interaction. The distance and angulation of the camera as well as the depth of ICG have been found to influence the signal and need consideration in further protocols40. Furthermore, while other surgical camera manufacturers offer similar systems for ICG visualization by NIR, camera design and performance investigation has shown previously significant inter-manufacturer variation between signal and ICG concentration (including the tendency of some systems to boost the images for human interpretation) which limits the generalizability of these findings until further (ongoing) validation is performed41. While selection of the specific timepoints for analysis (e.g. 100 s post-peak timepoint) is somewhat arbitrary, they do provide milestones for statistical analysis enabling comparisons between lesions and adjacent healthy tissue as control. Statistical curve analysis is currently not calculated in real time and is limited to one curve per ROI. Simultaneous synthesis and dissection of multiple curves may further discern tumour heterogeneity. Although the automatic classifier diminished the differentiation ability of our metrics when compared to surgeon-based classification, better classification algorithmic methods may to some extent replace operator input. Other tracking algorithms exist and further software development may address field of view interruptions.

This endeavour has nonetheless shown the feasibility of ICG with NIR-illumination for continuous quantitative FI assessment. Software assessment of these perfusion curves has yielded metadata which could potentially be used to identify cancer in human patients. The previous focus on spot readings may alleviate the need for concurrent injection and recording but has overlooked the potential for incorporating dynamic ICG flow patterns to improve confidence of interpretation. Equally incorporation of computer-assistance can help and even potentially replace the need for human observer visualization promising near real-time assessment of lesions during procedures with minimal user interaction as a ‘digital biopsy’ enabled through advances in computer processing, artificial intelligence and broad cloud-based datasets42.