Hyperspectral imaging benchmark based on machine learning for intraoperative brain tumour detection

Brain surgery is one of the most common and effective treatments for brain tumour. However, neurosurgeons face the challenge of determining the boundaries of the tumour to achieve maximum resection, while avoiding damage to normal tissue that may cause neurological sequelae to patients. Hyperspectral (HS) imaging (HSI) has shown remarkable results as a diagnostic tool for tumour detection in different medical applications. In this work, we demonstrate, with a robust k-fold cross-validation approach, that HSI combined with the proposed processing framework is a promising intraoperative tool for in-vivo identification and delineation of brain tumours, including both primary (high-grade and low-grade) and secondary tumours. Analysis of the in-vivo brain database, consisting of 61 HS images from 34 different patients, achieve a highest median macro F1-Score result of 70.2 ± 7.9% on the test set using both spectral and spatial information. Here, we provide a benchmark based on machine learning for further developments in the field of in-vivo brain tumour detection and delineation using hyperspectral imaging to be used as a real-time decision support tool during neurosurgical workflows.

specialised operating theatres and equipment, increasing the time and cost of surgery 12 .Ultrasound (US) is a less expensive alternative to iMRI that provides real-time imaging not affected by navigation inaccuracy or intraoperative changes (like in IGS).However, it has problems related to artefacts (blood, haemostatic material, bones, etc.) and requires long training periods to create high-quality images, resulting in 2D images that are difficult to interpret 13 .Intraoperative fluorescence imaging, such as 5aminolevulimic acid (5-ALA) or sodium fluorescein (SF) are commonly used in brain surgeries for delineating tumour boundaries.Nonetheless, these fluorescence agents do not detect the majority of LG gliomas and must be orally administered to the patient, possibly producing side effects 14,15 .Consequently, there is still room for new research in imaging modalities and methods that could overcome these limitations, offering substitute or complementary approaches to the current state-of-the-art techniques 16 .
Hyperspectral (HS) imaging (HSI) is an emerging technique capable of providing label-free, non-contact, near real-time, and minimally-invasive intraoperative guidance by using non-ionizing illumination and without employing any contrast agent 17 .HS images are formed by hundreds of narrow spectral channels within and beyond the visual spectral range (Fig. 1.a).This technique provides, for each pixel, a continuous spectrum that allows the identification of the tissue, material or substance present in the captured scene based on its chemical composition 18 .HSI has been widely investigated in many different medical specialities (e.g., oncology 19,20 , digital and computational pathology 21 , ophthalmology 22 , dermatology 23,24 or gastroenterology 21,25 ), being also studied for several particular applications 26 , such as biomarkers discovery and validation 27 or tissue perfusion measurement 28 .Moreover, HSI allows the possibility of being used during awake brain surgery to identify eloquent brain areas adjacent to tumours (already explored in functional ultrasound imaging 29 ).
Previous works from this group have evaluated, as a proof-of-concept, the use of HSI and data processing frameworks, particularly machine learning (ML) and deep learning (DL) algorithms, for intraoperative brain tumour detection and delineation using a limited set of images and patients, employing a leave-one-patient-out cross-validation, and focused in glioblastoma detection [30][31][32] .In this work, a more exhaustive spectral characterization of different tissue and tumour types with an increased dataset is provided, as well as a more robust generation and validation of the classification results obtained using both spectral and spatial information for tumour delineation and identification targeting pathology-assisted surgery with real-time performance.Proposed intraoperative HSI approach for surgical assistance.a, HSI concept explanation.b, HS data acquisition and labelling procedure during surgery.In the ground-truth map, red represents tumour labelled pixels, green normal pixels, blue hypervascularized pixels, and black background pixels.Meanwhile, white represents nonlabelled pixels.c, Synthetic RGB images of a surface-layer tumour (left) and a deep-layer tumour (right).Tumour area is surrounded in yellow by a clinical expert.d, Patient/image flow scheme of this work and data partition.n: number of HS images; m: number of patients.e, Proposed processing framework to generate the density maps for intraoperative pathology-assisted surgery.

Spectral characterization of brain tissue
Sixty-two HS images obtained from 34 different patients (Supplementary Table 1) were captured and labelled following a specific protocol for data collection (Fig. 1b-d) (see Methods).Four classes were established: tumour tissue (TT), normal tissue (NT), blood vessel (BV), and background (BG).Raw HS data were pre-processed (see Methods) to standardize and reduce the noise of the spectral signatures.Statistical differences were found between all the medians of each spectral channel when comparing TT vs. NT (Fig. 2a) and TT vs. BV (Fig. 2b) using the Wilcoxon Rank Sum test at 5% of significance level.High standard deviation values were obtained in the spectral signatures due to the interpatient variability and also the different tumour types included in the database.Additionally, the intraoperative HS data acquisition during surgery is a complex procedure that can be affected, in some cases, by the non-flat brain surface (Fig. 1b).These irregular surfaces can affect the illumination conditions, and, hence, the image focus in certain areas, reducing the reflectance values and increasing the noise of the spectral signature respect to the more focused areas.For this reason, a complete pre-processing chain (see Methods) was applied to the HS data, where each spectral signature was normalized as minimum 0 and maximum 1 to only consider the shape of the signature in the processing algorithms computation.Additionally, a decimation process was applied to reduce the dimensionality of the data in the spectral dimension and, hence, the computational cost of the processing algorithms 38 .
Fig. 2. Spectral characterization of different brain tissue and tumour types.Mean and standard deviation (std) of the entire labelled dataset after applying a basic pre-processing (calibration, extreme band noise removal, and noise filtering) and separated by classes, including the corresponding p-value computed for each spectral channel using the Wilcoxon Rank Sum test at 5% of significance level between the two compared classes.a, TT vs. NT class.b, TT vs. BV class.c, Primary vs. secondary tumours.d, HG vs. LG primary tumours.e, G1 vs. G2 primary tumours.f, G3 vs. G4 primary tumours.
The mean spectral signatures of TT, NT, BV were converted to absorbance values (Fig. 3) to be represented and compared with the molar extinction coefficient of oxyhaemoglobin and deoxyhaemoglobin 33 .Absorbance values of all classes increase between 500 and 600 nm (Fig. 3a,c,e), due to the existence of two oxyhaemoglobin absorbance peaks (~540 and ~575 nm) and one deoxyhaemoglobin absorbance peak (~555 nm) in this spectral range 37 .Particularly, oxyhaemoglobin peaks in BV are not detected (Fig. 3e), probably because we labelled veins and arteries indistinctly, involving oxy and deoxyhaemoglobin characteristics.Higher absorbance values were found in TT with respect to NT, but lower than BV.Moreover, an absorbance peak was found at ~760 nm (Fig. 3b,f) related to deoxyhaemoglobin 34,35 .Our spectral data reveal that the contribution of deoxyhaemoglobin is the highest in BV (Fig. 3f), having a lower contribution in TT (Fig. 3b).However, this contribution is not found in NT (Fig. 3d).This difference between NT and TT could be mainly related to the lack of oxygenation in the brain tissue affected by cancer 36 .Fig. 3.
Spectral characterization of tumour tissue, normal tissue, and blood vessels classes and their relationship with deoxyhaemoglobin (Hb) and oxyhaemoglobin (HbO2).Mean absorbance values of the entire labelled dataset separated by classes (solid) after applying a basic pre-processing (calibration, extreme band noise removal, and noise filtering) and molar extinction spectra (dashed) of Hb and HbO2.a, Tumour tissue (TT) between 440 and 650 nm.b, TT between 650 and 910 nm.c, Normal tissue (NT) between 440 and 650 nm d, NT between 650 and 910 nm.e, Blood vessels (BV) between 440 and 650 nm.f, BV between 650 and 910 nm.

Spectral characterization of different brain tumour types
As stated in the introduction section, brain tumours can be subdivided into different subtypes depending on their origin (primary/secondary) or the grade of malignity in the case of primary tumours.Regardless of tumour grade and origin, there is an absorbance peak (reflectance valley) around 760 nm (Fig. 2c-f) related to deoxyhaemoglobin 34 .Secondary tumours present lower standard deviation values than primary ones (Fig. 2c).However, this fact can be related to the reduced number of patients affected by secondary tumours in our database, and the reduced number of labelled pixels with respect to the primary type.Despite of this, statistical differences between the medians of each spectral channel were found at 440-599, 602-756 and 769-909 nm spectral ranges.HG and LG primary tumours present similar reflectance and standard deviation values (Fig. 2d).Nonetheless, statistical differences were found at 466-510, 522-549, 559-572 and 580-909 nm spectral ranges (Fig. 2d).Considering the tumour grades of primary tumours (Supplementary Fig. 1), statistical differences were found between the medians of all spectral channels of G1 and G2 tumours (Fig. 2e), whereas in the case of G3 and G4 tumours (Fig. 2f), only the 440-460, 578-644, 745-764 and 779-909 nm spectral ranges were found to be statistically different.

Brain tissue classification based on spectral information
The HS data collected intraoperatively was used as input for ML and DL-based algorithms (i.e., random forest (RF), k-nearest neighbours (KNN) using Euclidean (KNN-E) and Cosine (KNN-C) distances, support vector machines (SVMs) using the linear (SVM-L) and radial basis function (SVM-RBF) kernels, and a two-layer deep neural network (DNN)) and unmixing-based methods (i.e., linear extended blind end-member and abundance extraction (EBEAE) and nonlinear extended blind end-member and abundance extraction (NEBEAE)) to generate classification models capable of distinguishing between the four different classes (TT, NT, BV, and BG).Labelled data were used to train and optimize the hyperparameters of the algorithms and to quantitatively evaluate the results on the test set (see Methods).Due to the high computational cost required to train the ML/DL algorithms, training data were reduced to 1,000, 2,000, and 4,000 pixels per class, following the method proposed in a previous work 38 .In general, during the optimization process using the macro F1-Score metric independently to the training data reduction used (Supplementary Fig. 2-4), the results tend to stabilize at a certain hyperparameter for all the five folds (Supplementary Table 2 -4).
No statistically significant differences were found between the three training data reductions (Fig. 4a).Hence, the use of 1,000 pixels per class allowed to reduce the time required for training the model (particularly for the SVM-based implementations) without compromising the classification performance.For this reason, we selected this training data reduction for the subsequent experiments.Additionally, our results show that statistically significant differences were found between the unmixing-based methods and the ML-based ones, obtaining lower classification performance.The highest median macro F1-Score result was obtained with the SVM-RBF model (78.4±5.1%),but no statistically significant differences were observed between this algorithm and the others (except for EBEAE and NEBEAE).The highest average overall accuracy (OA) was also reached by the SVM-RBF (91.5±4.7%),but the highest TT sensitivity (65.9±13.1%)was obtained with the DNN (Fig. 4b).Average specificity results were higher than 90% for the ML and DL-based approaches.

Fig. 4.
Spectral classification results of brain tissue.a, Boxplots of the macro F1-Score results of the validation set for each training data reduction and each classifier, including the five folds using the optimal hyperparameters in each classifier.Two medians are significantly different at the 5% significance level if their intervals (shaded colour areas) do not overlap.b, Average OA, sensitivity and specificity results of the validation set from the 5 folds using the data reduction of 1,000 pixels per class.c, Examples of synthetic RGB (SRGB) images, ground truth (GT) maps and supervised classification maps generated using the eight algorithms with the optimal hyperparameters from different tumour types of the validation set.Approximate tumour areas were surrounded in yellow on the SRGB image by the operating surgeon according to the intraoperative neuronavigation and the definitive pathological diagnosis of the resected tissue.Rubber ring markers were employed in some cases (e.g., Op8C1) to indicate the area where the biopsies for pathology were resected.Opx: Operation number; Cy: Capture number.
Qualitative results, extracted from the validation set and obtained after applying the supervised classification model (generated using 1,000 pixels per class and the optimal hyperparameters) to the entire HS image, show the pixel-wise identification of both labelled and non-labelled pixels (Fig. 4c).As expected, according to the quantitative results (Fig. 4a,b), the unmixing-based methods (EBEAE and NEBEAE) increase the number of false positives and false negatives in the classification maps, particularly in Op35C1 employing EBEAE, where the normal tissue is identified as tumour, and Op57C1 using NEBEAE, where tumour areas are identified as normal tissue.The remaining classifiers achieve more consistent results, although the SVM-based and DNN algorithms improve the identification of the tumour areas in Op42C2 and Op57C1 (only using SVM-L and DNN).

Brain tumour identification and delineation based on spatial-spectral information
Following the approach proposed in a previous work 30 , the use of the spatial information available in the HS images was included to evaluate the possible improvement of the classification results and to reduce the false positives found in the supervised classification maps computed based only on the spectral information (see Fig. 1e and Methods).We have compared the quantitative results of the validation set (Fig. 5a) using only the spectral information (Spectral), applying the KNN filtering to include also the spatial information (Spatial/Spectral) and combining the spatial-spectral supervised classification with an unsupervised segmentation through a majority voting (MV) approach (Majority Voting).Our results reveal that the inclusion of the spatial information increases the median macro F1-Score results (0.4-7.7%), reducing the standard deviation (0.2-3.7%), in all algorithms, except for the unmixing-based approaches.However, no statistical differences were found between these results.Additionally, it is worth noticing that the Majority Voting results achieved lower median results and increased the std.Nonetheless, this lower performance could be motivated by the construction of the output classification map in the MV approach, which is obtained by considering only the majority class assigned to each cluster of the unsupervised hierarchical k-means (HKM) map.At the Spatial/Spectral stage, the SVM-RBF reached the highest average OA (92.3±4.6%),but the DNN obtained the best average TT sensitivity (68.9±14.3%),closely followed by the SVM-L algorithm (67.7±19.3%)(Fig. 5b).
The qualitative results of each step of the proposed algorithm have been analysed, where the supervised map represents, as an example, the classification map generated using only spectral information with the DNN method (Fig. 5c).The PCA (Principal Component Analysis) map represents, in a false colour intensity map, the first principal component where the more important information contained in the HS image is relocated in a low dimensional space.For example, in Op8C1, the tumour area is partially highlighted with more intensity values (between the two rubber ring markers on the right of the image).The KNN-Filtered map offers a smoothed version of the supervised map, where the spatial properties of the HS image are used (by combining the information of the probability maps generated by the supervised classifier and the PCA map).This approach reduces the granularity of the supervised map, providing more homogeneous class regions.This Spatial/Spectral classification was combined with an unsupervised segmentation (HKM map) that identifies 24 different regions (or clusters) in the HS image according to their similar spectral characteristics, providing a very accurate delineation of different structures but without any identification of the tissue, material or substance that each cluster represents.For this reason, the information provided by the HKM map was merged with the KNN-Filtered map by means of a MV approach 30 , where each cluster is labelled by the majority class within it.In the MV map (Fig. 5c), the boundaries between different class regions are determined by the HKM map, while the identification of each cluster class is defined by the KNN-Filtered map.However, in these maps, only the information relative to the class with the majority number of pixels in each cluster is shown.Hence, as a surgical visualization tool, we proposed to combine the information provided by the three maximum probability values (classes NT, TT, and BV) of the MV approach, by mixing the RGB colours in each cluster according to the percentage of pixels covered by each class in such cluster (i.e., the R channel corresponds to the percentage of TT pixels, the G channel to NT pixels, and the B channel to BV pixels).For example, a cluster represented by a bright red, green or blue colour denotes it belongs to only one single class (TT, NT or BV, respectively).In contrast, any other colour represents a combination of classes in a cluster (e.g., purple colour represents a mixture between TT and BV classes that commonly happens in certain blood vessels, hypervascularized areas or extravasated blood, see Op42C2, Op35C2 or Op57C1).This resulting map is called three maximum density (TMD) map 30 (Fig. 5c).
After performing all the analysis and hyperparameter optimizations of the algorithms using the validation set, the test sets of the different k-folds were evaluated (Fig. 6a).Quantitative results of the macro F1-Score metric show, as expected, a performance reduction in the test set of 0.5-1% respect to the validation one, providing the best median score of 70.2±6.3%using the DNN algorithm in the Spatial/Spectral approach.Similar average OA results are obtained using SVM-L (86.6±5.5%) and DNN (86.8±3.4%) as supervised classifiers, while a slight increase of the SVM-L average TT sensitivity (57.8±23.7%)respect to the DNN (54.7±21.9%) is obtained (Fig. 6b).Specificity average results are in general higher than 90% in all ML and DL-based approaches for all classes.
Some examples of the TMD maps of the test set (Fig. 6c) show that the GBM cases (Op12C1, Op15C1, Op39C2, Op43C1, and Op43C2) delineate in red the tumour areas, as expected by neurosurgeons (marked in yellow over the synthetic RGB images).Particularly, Op15C1 presents some decoloured red/orange/purple areas that could represent the infiltrative nature of GBM tumours in the surrounding tissue.Moreover, the surrounding blue areas could be related to the hypervascularized tissue that surrounds the tumour, also including the blood vessels in such regions (Op15C1 in Fig. 6c).The same fact can be visualized in Op12C1, Op43C1, and Op43C2.In the case of Op20C1 and Op39C1, the tumour is somehow revealed but not as a red area, since the tumours are located in a deep layer of the brain tissue.Op20C1 has not an additional image captured after the resection started, since the tumour resection in such location of the brain could cause serious damages and side effects to the patient, and, additionally, the tumour boundaries were not clear enough to perform a secure and effective resection.For such reason, the operating surgeon decided not to operate the patient, prevailing the QoL of the patient.On the contrary, after Op39C1 was captured, the operating surgeon continued the tumour resection, and a second image (Op39C2) was captured during resection, where it is possible to observe the correct delineation of the tumour area in a bright red colour.This was also the case of Op43, but before starting the resection, the tumour was clearly visualized in the brain surface, showing a possible infiltration in the surrounding tissue (orange/purple colour in the upper-left part of the tumour area).
Moreover, we qualitatively evaluated some examples of test cases not related to high-grade gliomas (Fig. 6c).Firstly, Op35C1 presents a healthy brain surface, since the tumour was in a deep layer, where no false positives are present in the parenchymal area, only those related to extravasated blood surrounding the parenchymal area.In Op35C2 and Op41C2, it is possible to observe that the proposed algorithm can identify not only high-grade tumours but also low-grade tumours, a G2 oligodendroglioma and a G1 ganglioglioma, respectively.Finally, secondary tumours are also detected by the proposed algorithm, as shown in Op35C1 where a metastatic breast carcinoma is identified, although some false positives surrounding the parenchymal area are also presented.These false positives could be produced because of the low quality of the image, where an optimal focus was not achieved.

Fig. 5.
Quantitative and qualitative results at the different stages of the proposed framework in the validation set.a, Macro F1-Score of the validation set using the eight different classifiers at the three different stages.Two medians are significantly different at the 5% significance level if their intervals (shaded colour areas) do not overlap.b, Average OA, sensitivity and specificity results of the validation set from the 5 folds using the Spatial/Spectral approach.c, Example of SRGB images and output maps from different tumour types of the validation set at the different stages of the proposed framework (based on the DNN as supervised algorithm using the optimal hyperparameters).Fig. 6.
Quantitative results at the different stages of the proposed framework and qualitative TMD classification maps in the test set.a, Macro F1-Score of the test set using the eight different classifiers at the three different stages.Two medians are significantly different at the 5% significance level if their intervals (shaded colour areas) do not overlap.b, Average OA, sensitivity and specificity results of the test set from the 5 folds using the Spatial/Spectral approach.c, Examples of SRGB images, GT maps and TMD maps from different tumour types (based on the DNN as supervised algorithm using the optimal hyperparameters).

Discussion
This work demonstrates the high potential of HSI for in-vivo identification of brain tumour tissue and its boundaries during neurosurgical operations.We have developed an intraoperative customized HS acquisition system, employed in three data acquisition campaigns, to collect 62 HS images of exposed brain surface from 34 different patients that underwent surgery due to brain cancer or another disease that required surgery.Using this extended database with respect to previous works [30][31][32]39,40 , we have analysed the spectral characteristics of the brain tissue (normal and tumour) and blood vessels, and the different tumour /15 types according to their malignancy grades (G1 to G4) and origin (primary and secondary), performing a statistical analysis between all the medians of each spectral channel when comparing the different classes and tumour grades and origins. Moreoer, a robust 5-fold cross-validation approach was used to evaluate eight different processing algorithms, first using only spectral information, and then using both spatial and spectral information following a processing framework that we previously developed 30 .
The spectral-based classification results obtained using the validation set (Fig. 4a) showed that SVM-based and DNN methods provided the best macro F1-Score results, although no statistical differences were found among the other classifiers (except for the unmixing-based methods, which provided less accurate results).The qualitative results (Fig. 4c) demonstrate the ability of the proposed HSI-based system to identify not only high-grade gliomas (Op8C1), but also other low-grade tumours (Op42C2 and Op35C2) and secondary tumours (Op57C1).Moreover, these results show the capability of HSI to accurately highlight the vascularization of the brain surface, being especially remarkable in Op35C1 and Op42C2.
It is worth noticing that HS images captured in suboptimal acquisition conditions, such as a lack of correct focus or illumination, can introduce inappropriate spectral signatures for training the algorithms and can produce inaccurate classification maps.This limitation is particularly evident in deep-layer tumours (Fig. 7a), where it was not possible to correctly focus the entire area of interest by using our pushbroom-based HSI system.In the case of Op37C2 (Fig. 7a), due to uncertainty at the time of labelling the tumour pixels in the centre of the image, only NT, BV, and BG classes were labelled.The average spectral signatures (Fig. 7b) reveal an acquisition problem, possibly related to a lack of proper illumination, as the reflectance values in the three classes decrease dramatically in the infrared range (>700 nm).However, the DNN method seems to overcome this handicap and correctly identify the tumour area even using this non-optimal HS image.
The inclusion of spatial information improved the macro F1-Score medians respect to using only spectral information, although no statistical differences were found between these results (Fig. 5a).After performing the hyperparameter optimization process using the validation set, the test data of each k-fold were processed providing both quantitative and qualitative results (Fig. 6).The processing framework based on the DNN algorithm in the Spatial/Spectral approach provided a macro F1-Score of 70.2±7.9%,representing, as expected, a performance reduction of 3.6% respect to the validation results.Qualitative test results demonstrate the ability of the proposed framework to identify not only HG gliomas (e.g., GMB), but also LG and secondary tumours (e.g., G2 oligodendroglioma, G1 ganglioglioma, metastatic breast carcinoma) (Fig. 6c) and also extra-axial tumours (e.g., G1 meningioma).
The processing of the test dataset allowed us to identify some HS image cases where the data acquisition conditions were not optimal, producing some errors in the classification results (Fig. 7c), which may degrade the quantitative results of the test sets.We found that in Op55C1 and Op55C2 the classification results identified most of the pixels as tumour, and only some parts related to background (Fig. 7c).After evaluating the spectral signatures of the labelled pixels in such HS images, we found some differences in the infrared region (from 700 to 900 nm) with respect to the other images.This unusual behaviour was found also in Op56C2, where there is a decrease in the reflectance values of the labelled spectral signatures in the same infrared region (Fig. 7d), also producing wrong classification results where the parenchymal area is identified as background (Fig. 7c).The low sensitivity of the HS sensor in this spectral range, coupled with a possible misalignment of the light beam with the lens (due to an improper focusing), could lead to this decrease in reflectance.
Despite these limitations, we have demonstrated with a robust classification validation approach, the potential benefits of HSI for brain tumour tissue identification, targeting a diagnostic support system for guiding neurosurgical interventions in real-time.
In previous works, we demonstrated that it is possible to achieve near real-time HS data processing using graphical processing units, achieving processing times of ~6 s 41 .The proposed intraoperative HSI-based acquisition system must be optimized in further works by reducing the HS camera size, employing a snapshot-based HSI technology (which is able to capture the entire HS cube in a single shoot, providing also real-time performance) and integrating it into a surgical microscope.This new experimental setup will guarantee an improvement of the HS image quality to solve the focus problems, especially for deep-layer tumours.Additionally, an extensive clinical validation of the proposed framework must be carried out, employing a large number of patients and a multi-centre trial.This clinical validation will perform a comprehensive pathological analysis of the entire tumour area outlined by the TMD map (especially in the boundaries between tumour and the surrounding normal tissue), as well as correlate the results with the MRI information to verify that the system can adequately identify tumour infiltration into normal brain tissue, especially in HG gliomas.Additionally, the relation between the improvement of the patient outcomes and the use of the proposed system during the surgery could be studied through the clinical validation.

Fig. 7.
Examples of the limitations of the proposed framework after processing the test set.a, Example of synthetic RGB images, GT maps and supervised classification maps created using the eight algorithms with the optimal hyperparameters from a deep-layer tumour captured in non-optimal conditions in the validation set.b, Average spectral signatures of the GT pixels from a. c, Example of SRGB images, GT maps and TMD maps (based on the DNN algorithm) from HS images captured in non-optimal conditions in the test set.d, Average spectral signatures of the GT pixels from c.

Methods
In-vivo Hyperspectral Brain Database HS images were captured following the procedure (Fig. 1c) previously described in detail 42 .First, a craniotomy was performed to the patient by using IGS neuronavigation and then, the durotomy was accomplished to expose the brain surface.Next, the acquisition system was placed over the patient's brain to acquire the HS image.In some particular cases, rubber ring markers were placed over tumour and normal tissue areas according to the IGS system information to further identify the tissue type.After that, tumour tissue was resected for neuropathological evaluation to achieve the definitive diagnosis of the tumour.When possible, more than one HS image were acquired while the tumour was being resected.
HS images were manually cropped to select the region of interest where the parenchymal area was exposed.Afterwards, the data were labelled by using the information provided by the neuropathologists and the knowledge of the operating surgeons, using a semi-automatic labelling tool based on the spectral angle mapper (SAM) algorithm developed for this purpose 42 .The groundtruth (GT) maps (Fig. 1c) were composed by four classes: tumour tissue (TTred colour), normal tissue (NTgreen colour), blood vessels (BVblue colour), and background (BGblack colour).White pixels in the GT maps represent the non-labelled pixels, as only those with a high confidence of belonging to a particular class were labelled.Several images in the database do not contain tumour pixels due to the impossibility of performing a reliable labelling or due to the patient underwent surgery for another pathology, such as a blood clot or epilepsy.
Three data acquisition campaigns (Fig. 1d) were carried out at the University Hospital of Gran Canaria Doctor Negrin, Spain.Written informed consent was obtained from all the participant subjects.The study protocol and consent procedures were approved by the Comité Ético de Investigación Clínica-Comité de Ética en la Investigación (CEIC/CEI) for the first (from March 2015 to June 2016) and the second (from October 2016 to April 2017) data campaigns, and the Etica de la Investigacion / Comite /15 de Etica de la Investigacion con Medicamentos (DEI/CEIM) for the third data campaign (from July 2019 to October 2019).Both committees were from the University Hospital Doctor Negrin (130069 and 2019-001-1, respectively).All the research methodologies were performed in accordance with relevant guidelines/regulations.Supplementary Table 1 summarizes the number of patients (identified as Op: operation number) and images (identified as C: capture number) captured in each data campaign (including those excluded due to the inadequate capturing conditions), as well as their image dimensions, number of labelled pixels and the definitive pathological diagnosis.
Intraoperative Hyperspectral Acquisition System HS images of the in-vivo brain were obtained intraoperatively using a custom HS acquisition system (Fig. 1b) previously described 31 and later improved 43 .In brief, the system was composed by a visual and near-infrared (VNIR) HS pushbroom camera (Hyperspec ® VNIR A-Series, Headwall Photonics Inc., Fitchburg, MA, USA) able to capture 826 spectral channels in the 400-1000 nm spectral range, having a spectral resolution of 2-3 nm and a maximum spatial resolution of 741×1004 pixels, due to the pushbroom mechanism for the data acquisition process 31 .An illumination system based on a quartz tungsten halogen (QTH) lamp of 150 W coupled to a fibre optic cold light illuminator was employed, avoiding the incidence of the high temperatures of the QTH lamp in the exposed brain tissue.The working distance between the HS camera lens and the brain surface was 40 cm, with a pixel size of 128.7 µm and a maximum acquisition time of 60 s.
HS Data Pre-processing Raw HS images were pre-processed to avoid the influence of ambient illumination and dark currents of the HS sensor, and to reduce dimensionality and noise following the procedure previously described 38 .In brief, a raw HS image was calibrated following Eq.( 1), where  is the calibrated image,  is the raw image, and  and  are the white and dark reference images, respectively.The white reference was acquired in the same illumination conditions that the raw HS image was captured, using a standard white reference tile that reflects the 99% of the incident light.The dark reference was obtained by keeping the camera shutter closed, being used to correct the dark currents produced by the HS sensor.Finally, a data smoothing approach based on a moving average filter was applied for reducing the high-frequency noise.Each smoothed value was averaged using a window of five data points.Moreover, the extreme spectral channels of the HS cube (the first 56 and the last 126 spectral channels) were removed due to the low capabilities of the HS sensor in these channels, which produces high noise in the spectral signatures 42 .At this point, the HS cube is formed by 645 channels with an operating bandwidth between 440.5-909.1 nm.HS cubes with this preprocessing chain were used for the spectral characterization of the different tissue and tumour types.The spectral signatures were also converted to absorbance () following Eq.( 2) to compare the spectra of the different classes with the molar extinction coefficients of oxygenated and deoxygenated haemoglobin, where  is the reflectance value and  represents each wavelength.
In addition to this pre-processing, a dimensionality reduction of the HS cube was performed by decimating the spectral channels to reduce redundant information in the HS data due to the high dimensionality, also allowing a drastic reduction of the execution time of the processing algorithms without losing diagnostic performance.As studied in a previous work 38 , the optimal sampling interval was 3.61 nm, allowing the number of spectral channels to be reduced to 128.Finally, the spectral signatures were normalized independently to minimum and maximum values of [0, 1].
() = −(()) Supervised Classification Algorithms ML algorithms used in this work were based on SVM, RF, and KNN classifiers, while the DL algorithm employed was a twolayer one-dimensional DNN.Moreover, two unmixing-based algorithms were studied (EBEAE and NEBEAE) using their MATLAB implementations 40,44 .SVMs are widely used for classification and regression purposes 45 .The objective of this classifier is to separate different data classes by finding out the best separation hyperplane with a maximum margin.In this study, the optimal hyperplane was computed employing linear and radial basis function (RBF) kernels.The LIBSVM library was used for the SVM-L and SVM-RBF implementations 46 .The hyperparameter to be optimized for the SVM-L was the cost (C) parameter, which controls the decision limit that separates the positive and negative classes, while for SVM-RBF the hyperparameters optimized were cost and gamma ().RF is based on decision trees, identifying the new data class by taking a vote of their predictions from an aggregation of decision trees 47 .The optimization of the RF model was performed by searching for the most appropriate number of trees (T).KNN compares each incoming sample with all their neighbours using a distance metric to find the closest neighbors 48 .For the KNN classifier, we employed the Euclidean and Cosine distance metrics and the hyperparameter to be optimized in each case was the number of nearest neighbours (N).The MATLAB Statistics and Machine Learning Toolbox was employed for the RF and KNN-E and KNN-C implementations.The DNN was composed by two hidden layers, followed by a batch normalization layer, using the rectified linear unit as an activation function.The learning rate was established as 0.1, and the network was trained for 300 epochs.The output size (L) of the hidden layers was optimized.The MATLAB Deep Learning Toolbox was used for the DNN implementation.This DNN structure was studied in a previous work and compared with a two-/15 dimensional convolutional neural network (CNN) implementation, achieving the DNN the best performance 32 .The EBEAE is employed in non-negative datasets using a linear mixing model to perform the estimation of characteristic spectral endmembers and their abundances 44 .The NEBEAE is a nonlinear version of EBEAE, capable of quantifying non-linear optical interactions during the acquisition process, which is also robust against noise 40 .In both cases, different hyperparameters can be modified to adjust the similarity among endmembers (ρ) and the entropy of the abundances ().These algorithms have been previously used to identify glioblastoma tumour in pathological slides and in-vivo tissue using HS data 32,39,40,44,49 .In this case, the characteristic endmembers were estimated by the EBEAE and NEBEAE algorithms, respectively.The estimation process was performed using the labelled pixels from the training set.The representative number of endmembers was two for NT, two for TT, one for BV and three for BG, while the ρ hyperparameter was set as 0.3 for NT, 0.2 for TT, and 0.01 for BG 39 .The endmember of the BV class was obtained by calculating the average of all labelled pixels in that class.In both algorithm the entropy weight () hyperparameter was optimized during the estimation of the complete abundance matrix.

Data Partition and K-Fold Cross-Validation
To correctly evaluate the classification performance of the proposed approach, a three-way data partition was carried out at patient level, dividing the HS database into training (60%), validation (20%), and test (20%) sets.Additionally, five different folds were created to achieve more robust results due to the limited number of patients.This data partition was performed randomly using the patients' identifiers as instances, where each patient could have more than one HS image (Fig. 1d and Supplementary Table 5).Labelled data were employed to train the classification models (training set), to optimize their hyperparameters (validation set), and to quantitatively evaluate the results using unseen HS data (test set).The hyperparameter optimization of each algorithm was performed in each fold independently, evaluating the results with their respective validation sets and using the macro F1-Score metric and performing a coarse search (Supplementary Fig. 2-4).The optimal hyperparameters were selected using the best macro F1-Score result of each fold without considering the BG class.
Training Data Reduction Approach Due to the high computational cost required to train several of the employed classifiers, a methodology based on K-Means algorithm was used to reduce the number of pixels in the training set, balancing the classes, avoiding the inclusion of redundant information, and drastically reducing the training execution time 38 .Three different training sets were obtained using this methodology composed by 1,000, 2,000, and 4,000 pixels per class.The total number of labelled pixels in the HS images from the validation and test sets was used for the quantitative evaluation.

Proposed Processing Framework for TMD map generation
The spatial-spectral approach is based on a combination of a dimensionality reduction, a supervised classifier, a spatial filtering, an unsupervised segmentation, and a MV algorithm to merge the results from both supervised and unsupervised approaches (Fig. 1e).This approach was employed in previous works 30,32 to prove that the use of the spatial information available in the HS images helps to improve the classification results and to reduce the misclassified pixels found in the supervised classification maps created using only the spectral information.In this work, the PCA algorithm was employed for dimensionality reduction 31 , obtaining a one-band representation of pre-processed HS image (Fig. 5c).The spatial filtering aims to improve the supervised classification including the spatial features.The KNN filtering algorithm was employed using the previously studied parameters ( = 1 and  = 40) 30 and a window size of 8 rows using the Euclidean distance 50 .The probability maps from the supervised classifier and the one-band representation are the inputs of this algorithm.The K-means algorithm 30 was used as the unsupervised segmentation method to identify K different clusters into the HS images ( = 24 according to a previous work 30 ).Finally, the MV algorithm is used to merge the results obtained from the spatial-spectral supervised classification and the unsupervised segmentation, using a colour gradient approach to create the TMD maps 30 .MATLAB Statistics and Machine Learning Toolbox was employed to implement the K-means, PCA and KNN filtering algorithms.

Performance Metrics
The classification performance was evaluated using macro F1-Score (Eq.3), OA (Eq.4), sensitivity (Eq.5), and specificity (Eq.6) metrics, where  are true positives,  are true negatives,  are false negatives, and  are false positives.Macro F1-Score was computed with the mean of F1-Score per class (Eq.7), where  is the class index and  the number of classes.BG class was not considered to obtain the macro F1-Score result.Additionally, spectral characterization results were statistically analysed using a paired Wilcoxon Rank Sum test at the 5% significance level.

Fig. 1 .
Fig. 1.Proposed intraoperative HSI approach for surgical assistance.a, HSI concept explanation.b, HS data acquisition and labelling procedure during surgery.In the ground-truth map, red represents tumour labelled pixels, green normal pixels, blue hypervascularized pixels, and black background pixels.Meanwhile, white represents nonlabelled pixels.c, Synthetic RGB images of a surface-layer tumour (left) and a deep-layer tumour (right).Tumour area is surrounded in yellow by a clinical expert.d, Patient/image flow scheme of this work and data partition.n: number of HS images; m: number of patients.e, Proposed processing framework to generate the density maps for intraoperative pathology-assisted surgery.