Introduction

Alzheimer’s Disease (AD) is the most prevalent type of dementia in the US, and is primarily characterized by irreversible cognitive decline associated with neurodegeneration1. On account of an increasing aging population in the US, the annual incidence of AD is expected to double by 20502. However, studies have shown that the number of cases in 2050 can be reduced by 50% if the average age at the onset of the disease could be delayed by 5 years3. This may be achieved by early diagnosis and intervention with treatments that delay disease progression.

The original diagnostic criteria, known as NINCDS-ADRDA criteria, qualitatively combined information from medical history, clinical examination, neurophysiological testing and laboratory assessments to provide a sensitivity of 81% and a specificity of 70% for AD diagnosis4. In an attempt to diagnose AD earlier, the revised criteria now include two major changes: (i) addition of an intermediate diagnostic group, mild cognitive impairment (MCI) as well as (ii) guidelines for interpretation of imaging and molecular markers1. The intermediate diagnostic category, MCI, comprises of a heterogeneous group of patients who present early symptoms of cognitive impairment which do not interrupt daily life. While some MCI patients progress to AD over time, some remain stable while a few even regress back to healthy states. Given that MCI patients are at a greater risk for AD, there is an opportunity for early diagnosis of AD by identifying the subpopulation of MCI patients who progress to AD. However, to move toward this opportunity, the most immediate challenge is to accurately distinguish MCI from both HC and AD. The new diagnostic criteria therefore includes recommendations for incorporation of alternate biomarkers that have previously shown promise in predicting presymptomatic disease5.

Alongside recent developments in imaging and molecular diagnostic technologies, several studies have sought to identify biomarkers of AD. Cerebrospinal fluid (CSF) markers in particular have been extensively studied6,7,8 on account of their direct relationship with pathological characteristics of the disease such as amyloid burden and neuronal degeneration. On the genetic front, apolipoprotein E (ApoE) has been established as an indicator of risk for AD9. Structural information on 1.5 Tesla T1w Magnetic Resonance Imaging (MRI)10 such as hippocampal volume and functional information on [18F] fluorodeoxyglucose uptake (FDG-PET)11 such as changes in glucose metabolism have previously shown to be predictive of AD. Availability of multiple, complementary markers and data streams now presents an opportunity to combine different sources of information in order to potentially improve the ability to predict AD early, prior to its onset. However, qualitatively combining the vast amount of information is challenging and likely to result in subjective interpretations. On the other hand, quantitative approaches to identification of fused biomarkers is challenged by differences in data dimensionality, small sample size of most biomedical datasets and by the increase in data dimensionality associated with combining multiscale data12,13,14,15.

Several methods have previously been developed and explored to quantitatively combine multiscale biomedical data. Most data fusion approaches can generally be categorized based on the level at which information is combined: (i) raw data level (low level fusion), (ii) feature level (intermediate level fusion) or (iii) decision level (high level fusion)16. Data integration at the raw data level is limited to homogeneous data sources and is thus not directly applicable for fusion of multiscale, biomedical data. Alternatively, decision level strategies17 bypass challenges associated with fusion of heterogeneous data types by combining independently derived decisions from each data source. In doing so, relationships between the different data channels remain largely unexploited14, 15.

Most previous work in prediction of AD employ feature level integration where raw data is first converted into quantitative feature representations which are then combined using concatenation-based18, 19, kernel-based20, 21, manifold-based22 and most recently deep learning-based23, 24 methods. A brief summary of select related previous work is provided in Table 1. While feature concatenation18, 19 provides a simple method for investigating the added predictive value of each modality, it is sub-optimal for combining modalities with significantly different dimensionalities as modalities with larger feature sets are likely to dominate the joint-representation and hence the fused predictor12. Kernel-based and manifold-based methods20,21,22 alternatively transform raw data from the original space to a high dimensional embedding space where the different data types are more homogeneously represented, thereby making them more amenable for fusion. However, such methods are prone to overfitting25, 26 particularly given the small sample sizes of most biomedical datasets and the noise associated with each of the biomedical data sources which, if unaccounted for, may drown the increase in signal achievable by fusion. Suk et al.23 and Liu et al.24 presented deep learning based fusion approaches which seek to learn integrated structural and functional feature representations from MRI and PET. However, the method is limited to fusion of spatially aligned imaging data. In addition, deep learning methods generally require very large datasets in order to model complex non-linear relationships via several hidden layers. This could very easily result in overfitting on datasets with small sample size, especially in the presence of noise.

Table 1 Summary of related previous work.
Full size table

Regardless of the fusion strategy employed, most previous studies evaluate their methods by simplifying the multiclass problem (HC vs. MCI vs. AD) into the following binary classification tasks – AD vs. HC, MCI vs. HC. Recent work24 showed that multiclass classification resulted in significantly lower predictive performance as compared to that of the aforementioned binary classification tasks, suggesting that all the diagnostic classes must be considered to estimate the performance of a proposed model in a clinical setting. Generally, there are three common methods for multiclass classification – one vs. another, one vs. all (OVA) and one shot classification (OSC). For classification task with \({\mathscr{C}}=\{{c}_{1},{c}_{2},\ldots {c}_{n}\}\) classes, the one vs. another classifier attempts to independently solve binary class problems c i vs. c j , \(i\ne j\) arising from all pairwise combinations of classes. With this strategy, it is unclear how to combine results from the multiple binary problems in order to then determine the overall classifier performance. Alternatively, OVA seeks to solve c i vs {c j }, j = 1 … n, \(j\ne i\), while OSC attempts to simultaneously solve c 1 vs. c 2 vs. … vs. c n . OVA may not be able to appropriately classify intermediate classes such as MCI where the ‘all’ category comprises of data points that lie on either extrema of disease spectrum (i.e. healthy and AD). While OSC, which classifies multiple classes at once, overcomes the aforementioned limitations of the other two strategies, it assumes that the same set of modalities are optimal for separating all classes. When addressing multiclass problem in the context of data fusion, it may not be realistic to expect the same combination of modalities to be the most informative for all the various classes. In addition, some classification tasks may require information from fewer modalities to provide sufficiently accurate information while other, more challenging tasks may require additional information.

In this work, we introduce the cascaded multiview canonical correlation (CaMCCo) framework which brings together three different unique ideas; data fusion approach, modality selection concept and a cascaded classification scheme. The CaMCCo approach is employed in this paper for the problem of AD diagnosis. CaMCCo seeks to fuse a subset of modalities from T1w MRI, FDG PET, ApoE, CSF, plasma proteomics and neurophysiological exam scores in order to optimize classifier performance at each level of the cascade (Fig. 1). For data fusion, CaMCCo employs supervised multiview canonical correlation analysis (sMVCCA)27, 28 which provides a common, low dimensional representation that is discriminative of classes and allows for combining any number of heterogeneous forms of multidimensional, multimodal data. The fusion scheme operates under the assumption that information overlap increases with increasing number of data sources or ‘views’ as all views fundamentally capture information pertaining to the same object. As such, it seeks to maximize correlations between modalities and with class labels.

Figure 1
figure 1

The cascade and the modalities for fusion at each level of the cascade were determined on training set and validated on independent testing set. Neurophysiological test scores (ADAS-Cog) are fused with CSF proteomics and APOE at the first level of the cascade to identify healthy controls (HC). At the second level, ADAS-Cog scores are combined with PET to distinguish between patients with Alzheimer’s Disease (AD) and mild cognitive impairment (MCI).

Full size image

Previous work27, 28 has shown the application of sMVCCA in the context of predicting prostate cancer prognosis where sMVCCA based fusion of histologic and proteomic features was found to be more discriminative of classes as compared to individual modalities as well as several fused representations including LDA, CCA, MVCCA, PCA, regularized CCA (RCCA), supervised regularized CCA (SRCCA), and generalized embedding concatenation (GEC). Although sMVCCA is able to fuse any number of modalities, the practicality of its application in a clinical setting where the trade-off between added improvement in performance and increased burden of additional tests must be leveraged29. CaMCCo therefore extends on previous work to address clinical challenges associated with AD diagnosis by employing the fusion methodology within cascaded classification framework where only a subset of modality(ies) that maximize the performance for each classification task are fused at each level of cascade. Unlike most prior applications of multiclass classification methods to AD diagnosis, CaMCCo simultaneously considers all diagnostic classes via its cascaded classification approach. Unlike most prior applications of data fusion methods for AD diagnosis, CaMCCo seeks to identify, selectively retain and combine only the most informative data source(s) for each class label. As shown in Fig. 1, each patient is first classified as being healthy or cognitively impaired (CI) using ADAS-Cog score, CSF and APOE. If classified as CI, the ADAS-Cog and PET are used to distinguish between MCI and AD cases.

Methods

Supervised Multiview Canonical Correlation Analysis for Data Fusion

We apply supervised Multiview Canonical Correlation Analysis (sMVCCA)27, 28, an extension of canonical correlation analysis (CCA) and multiview canonical correlation analysis (MVCCA)30, to obtain a low-dimensional, shared representation of the modalities of interest. CCA31 is a linear dimensionality reduction method commonly used for data fusion as it accounts for relationships between two sets of input variables. MVCCA generalizes CCA by finding the linear subspace where pairwise correlations between multiple (more than two) modalities can be maximized. However, both CCA and MVCCA are unsupervised and therefore do not guarantee a subspace that is optimal for class separation. sMVCCA is a supervised form of MVCCA where class labels are embedded as one of the variable sets. Additional details and formulations for CCA and MVCCA are provided in the appendix and the theoretical framework for sMVCCA is provided below. Table 2 provides a summary of notations used in this section.

Table 2 Summary of Notations.
Full size table

Consider a multimodal dataset \({\bf{X}}\in \{{x}_{1},\ldots ,{x}_{k},\ldots ,{x}_{K}\}\) in \({{\mathbb{R}}}^{n\times M}\), where n is the number of subjects, K is the number of modalities and x k in \({{\mathbb{R}}}^{n\times {M}_{k}}\) refers to the feature matrix of modality k containing M k features. Additionally, X has a corresponding binary class label matrix Y in \({{\mathbb{R}}}^{n\times G}\), where G is the total number of classes. sMVCCA seeks to maximize correlation within the modalities in X and between X and Y as shown below

$$\begin{array}{ll}\mathop{\text{arg}\,\max }\limits_{{{\bf{w}}}_{1},\mathrm{...},{{\bf{w}}}_{k},\mathrm{...},{{\bf{w}}}_{K},{{\bf{w}}}_{Y}} & \sum _{k\ne 1}\sum {{\bf{w}}}_{k}^{T}{{\bf{x}}}_{k}^{T}{{\bf{x}}}_{j}{{\bf{w}}}_{j}+\sum _{k}{{\bf{w}}}_{k}^{T}{{\bf{x}}}_{k}^{T}{\bf{Y}}{{\bf{w}}}_{Y}\\ \quad \quad \quad \quad s\mathrm{.}t\mathrm{.} & {{\bf{w}}}_{1}^{T}{{\bf{x}}}_{1}^{T}{{\bf{x}}}_{1}{{\bf{w}}}_{1}=\mathrm{1,}\ldots ,{{\bf{w}}}_{K}^{T}{{\bf{x}}}_{K}^{T}{{\bf{x}}}_{K}{{\bf{w}}}_{K}=1,\,{{\bf{w}}}_{Y}^{T}{{\bf{x}}}_{Y}^{T}{{\bf{x}}}_{Y}{{\bf{w}}}_{Y}=1\end{array}$$
(1)
$$s\mathrm{.}t\mathrm{.}\quad {{\bf{w}}}_{1}^{T}{{\bf{x}}}_{1}^{T}{{\bf{x}}}_{1}{{\bf{w}}}_{1}=1,\ldots ,{{\bf{w}}}_{K}^{T}{{\bf{x}}}_{K}^{T}{{\bf{x}}}_{K}{{\bf{w}}}_{K}=1,{{\bf{w}}}_{Y}^{T}{{\bf{x}}}_{Y}^{T}{{\bf{x}}}_{Y}{{\bf{w}}}_{Y}=1$$
(2)

This can be expressed in a compact matrix form as follows:

$$\begin{array}{l}\mathop{{\rm{\arg }}\,{\rm{\max }}}\limits_{{{\bf{W}}}_{x},{{\bf{W}}}_{y}}\,trace({{\bf{W}}}_{x}^{T}\bar{{\bf{C}}}{{\bf{W}}}_{x})+2\times trace({{\bf{W}}}_{x}^{T}{{\bf{X}}}^{T}{\bf{Y}}{{\bf{W}}}_{y})\\ \quad \quad \quad \,\,=trace([\begin{array}{cc}{{\bf{W}}}_{x}^{T} & {{\bf{W}}}_{y}^{T}\end{array}]\,[\begin{array}{ll}\bar{{\bf{C}}} & {{\bf{X}}}^{T}{\bf{Y}}\\ {{\bf{Y}}}^{T}{\bf{X}} & {\bf{0}}\end{array}]\,[\begin{array}{c}{{\bf{W}}}_{x}\\ {{\bf{W}}}_{y}\end{array}])\\ \quad \quad \quad \,\,=trace({\hat{{\bf{W}}}}^{T}\hat{{\bf{C}}}\hat{{\bf{W}}})\\ s.t.\end{array}$$
$$\begin{array}{rcl}[\begin{array}{ll}{{\bf{W}}}_{x}^{T} & {{\bf{W}}}_{y}^{T}\end{array}]\,[\begin{array}{cc}{\bar{{\bf{C}}}}_{d} & {\bf{0}}\\ {\bf{0}} & {{\bf{Y}}}^{T}{\bf{Y}}\end{array}]\,[\begin{array}{c}{{\bf{W}}}_{x}\\ {{\bf{W}}}_{y}\end{array}] & = & {\bf{I}}\\ \quad \quad \quad \quad \quad \,\,\,\,\,\,\,\,\iff {\hat{{\bf{W}}}}^{T}{\hat{{\bf{C}}}}_{d}\hat{{\bf{W}}} & = & {\bf{I}}\end{array}$$
(3)
$${{\bf{W}}}_{1}^{T}{{\bf{C}}}_{11}{{\bf{W}}}_{1}=\cdots ={{\bf{W}}}_{K}^{T}{{\bf{C}}}_{KK}{{\bf{W}}}_{K}={{\bf{W}}}_{y}^{T}{{\bf{Y}}}^{T}{\bf{Y}}{{\bf{W}}}_{y}.$$
(4)

where Y is a matrix in which class labels are encoded using Soft-1-of-Class strategy32.

Solving Equation 3 consists of two steps: (i) Ignoring the constraint in (4) leaves us with a quadratic programming problem, whose W* corresponds to eigenvectors of the n-largest eigenvalues of a generalized eigenvalue system: \({{\bf{C}}}_{{\bf{xy}}}{\bf{W}}=\lambda {{\bf{C}}}_{{{\bf{d}}}_{{\bf{xy}}}}{\bf{W}}\); (ii) Imposing constraint (4) upon obtaining the optimal eigenvectors W* by normalizing the corresponding section of each modality: \({{\bf{W}}}_{{\bf{j}}}^{\ast \ast }={{\bf{W}}}_{{\bf{j}}}^{\ast }{({{\bf{W}}}_{{\bf{j}}}^{\ast {\bf{T}}}{{\bf{C}}}_{{\bf{jj}}}{{\bf{W}}}_{{\bf{j}}}^{\ast })}^{-\frac{1}{2}},j=1,\ldots ,k\).

Cascaded Multi-view Canonical Correlation Analysis (CaMCCo)

As shown in Fig. 2, CaMCCo divides the classification task for a multiclass, multimodal dataset into a cascade of multiple, sequential binary classification tasks, for each of which the optimal fused representation is independently determined and provided as input to the classifier. For the multimodal dataset X in \({{\mathbb{R}}}^{n\times M}\), consider a label matrix Y in \({{\mathbb{R}}}^{n\times G}\). A subset modalities suitable for classifying class g from all input samples can be denoted as i where \(i\subseteq k\). Features from modalities in i are concatenated to generate \({{\bf{X}}}_{{\bf{i}}}^{^{\prime} }\) in \({{\mathbb{R}}}^{n\times p}\), where \(p={\sum }_{i}{M}_{i}\) and \(p\le M\). The i modalities are fused via sMVCCA to reduce the dimensionality from p to d, where \(d\ll p\), resulting in \({{\bf{X}}}_{{\bf{i}}}^{^{\prime\prime} }\). Subsequently, \({{\bf{X}}}_{{\bf{i}}}^{^{\prime\prime} }\) serves as the input to a classifier which predicts if each sample does or does not belong to class g, \({\hat{y}}_{g}=1\) and \({\hat{y}}_{g}=0\), respectively. The multimodal dataset consisting of only samples classified as \(\hat{y}=0\) subsequently serves as the input for the next level of cascade where the modality selection, data fusion and classification steps are repeated for another class in Y.

Figure 2
figure 2

Cascaded multiview canonical correlation analysis (CaMCCo) algorithm for constructing the joint multimodal data fusion and multiclass classification framework.

Full size image

Therefore, designing the cascaded classifier for CaMCCo requires determination of (a) the sequence of classification tasks that provide the best overall classifier performance, as well as the respective (b) number and (c) type of modalities to combine at each level of the cascade. In this work, these parameters were determined experimentally on the training cohort as described in Section 3.5.

Experimental Design

Dataset Description

Data used in the preparation of this work were obtained from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database (www.loni.ucla.edu/ADNI). The ADNI was launched in 2003 with the primary goal of testing whether serial magnetic resonance imaging (MRI), positron emission tomography (PET), other biological markers, and clinical and neuropsychological assessment can be combined to measure the progression of mild cognitive impairment (MCI) and early Alzheimer’s disease (AD). The initial goal of ADNI was to recruit 800 adults, ages 55 to 90, to participate in the research – approximately 200 cognitively normal older individuals to be followed for 3 years, 400 people with MCI to be followed for 3 years, and 200 people with early AD to be followed for 2 years (see www.adni-info.org for up-to-date information). The research protocol was approved by each local institutional review board and written informed consent. In addition to raw data, the ADNI database contains several post-processed and individually evaluated biomarkers.

In this work, we consider a subset of cases for which the following was available in the database (i) pre-computed features from T1w MRI and FDG PET, (ii) neurocognitive ADAS-cog score, (iii) complete record of CSF Proteomics, Plasma Proteomics, ApoE and (iv) clinical diagnosis at baseline. 149 ADNI participants who fulfilled the criteria were included, of which 52 were diagnosed with Alzheimer’s Disease (AD), 71 were diagnosed with mild cognitive impairment (MCI), and 26 were healthy controls (HC).

Table 3 provides the clinical and demographic details of the population considered in this study as per their diagnosis at baseline. The unique ADNI database provided RID of all patients considered in this study is provided in the Appendix.

Table 3 Clinical and demographic information of the 149 ADNI subjects considered in this study, selected based on the availability of imaging, non-imaging and clinical metrics at baseline.
Full size table

Feature Description

Table 4 summarizes the number and types of features considered in this study for each modality. From imaging data, we consider volumetric features extracted from T1w MRI and measures of hippocampal glucose metabolism33, 34 from FDG PET. Considered molecular markers include proteomic measurements from cerebrospinal fluid (CSF), plasma from the biomarker consortium and geneotype ApoE data. We additionally included a neurophysiological test score as such tests serve as the primary means for diagnosis in the current clinical setting. Although the Mini-Mental State Examination (MMSE) is the most commonly performed clinical test, we avoid using MMSE scores as they were used to determine the “ground truth” labels on which we train and test CaMCCo. Therefore, we use an alternate test score, modified Alzheimer’s Disease Assessment Scale - Cognition (ADAS-Cog) which has been used to assess the effects of experimental treatments for AD in clinical trials35.

Table 4 Summary of features considered in this study from each modality.
Full size table

Classification Model

The dataset was split into training and a holdout validation set with each comprising 40% and 60% of the data, respectively. Classification and fusion parameters were determined on the training set using 10 iterations of 5-fold stratified cross validation, upon which the optimized classifier trained on the full training set was applied to the independent validation set. Naive Bayes classifier36 was used to evaluate the various fused and individual modality representations. Naive Bayes is a widely used, well-established probabilistic classifier that is known to perform well on small datasets.

Evaluation metrics

Performance measures used to evaluate each classification task include: accuracy (ACC), balanced accuracy (BACC)37, area under the receiver operating characteristic curve (AUC)38, sensitivity (SEN), specificity (SPE) and positive predictive value (PPV). The definitions and descriptions of each of these metrics are provided in Table 8 in the Appendix.

CaMCCO Model

Class groupings and modalities selected for fusion at each level of the cascaded classification design employed by CaMCCo (Fig. 1) was determined experimentally on the training set. One-vs-all (AD vs. all, MCI vs. all, HC vs. all) classifiers were constructed and evaluated independently for each considered modality. The task that most consistently resulted in the highest AUC across all modalities served as the first level of the cascade so as to reduce error propagation. Among AD, MCI and HC, the remaining classes were assigned to the second level of the cascade.

For every classification task within the cascade, each modality was ranked based on the AUC it achieved across iterations and cross validation folds within the training set. The n highest performing modalities were fused via sMVCCA, where n was varied from 2 to 6 (total number of considered modalities). The n modalities, which in combination, provided the highest training AUC were selected.

Comparative Strategies

CaMCCo represents a framework (\( {\mathcal H} \)) composed of multiple modules corresponding to modality selection (\({\mathscr{K}}\)), multimodal data fusion (\( {\mathcal F} \)), and multiclass classification (\({\mathscr{C}}\)). Accordingly, the comparative strategies against which we evaluate CaMCCo involve systematically replacing the method used for one or more of these modules with an alternative strategy. Table 5 lists the notation for each of these strategies and provides a short description.

Table 5 Summary of notations used to refer to comparative strategies evaluated in this work.
Full size table

Single Modality and Multimodality Approaches

Each modality was evaluated using a single modality framework (\({ {\mathcal H} }_{MRI}\), \({ {\mathcal H} }_{PET}\), \({ {\mathcal H} }_{CSF}\), \({ {\mathcal H} }_{PP}\), \({ {\mathcal H} }_{APOE}\), \({ {\mathcal H} }_{ADAS}\)) consisting of cascaded classification (\({{\mathscr{C}}}_{CAS}\)) to ensure fair comparison with CaMCCo. In addition, we compared classification performance of CaMCCo with that of a cascaded classification model where all modalities were fused at each level of the cascade (\({ {\mathcal H} }_{ALL}\)).

Principal Component Analysis for Data Fusion

Principal Component Analysis (PCA) is a dimensionality reduction method which projects input data onto an alternate subspace defined by orthogonal basis vectors which capture the direction of variance in the data. Consider a high dimensional, concatenated multimodal data matrix \({\bf{X}}=[{x}_{1},\ldots ,{x}_{k},\ldots {x}_{K}]\in {{\mathbb{R}}}^{n\times {M}_{1}+\ldots +{M}_{k}+\ldots +{M}_{K}}\) where K refers to the number of modalities, n refers to the number of subjects, and M k refers to the number of features in modality k.

$$[{\bf{U}},{\bf{S}},{\bf{V}}]={\rm{SVD}}(\bar{{\bf{X}}}{\bar{{\bf{X}}}}^{T})$$

Singular value decomposition is applied to mean centered data matrix, \(\bar{{\bf{X}}}\), which results in U, S, V. The columns of \({\bf{V}}\in {{\mathbb{R}}}^{({M}_{1}+\ldots +{M}_{k}+\ldots +{M}_{K})\times ({M}_{1}+\ldots +{M}_{k}+\ldots +{M}_{K})}\) are the principal components of \(\bar{{\bf{X}}}\) or the orthogonal basis vectors, ordered decreasingly by the amount of variance in the dataset explained by each component. \({\bf{U}}\in {{\mathbb{R}}}^{n\times {M}_{1}+\ldots +{M}_{k}+\ldots +{M}_{K}}\) contains the projections of \(\bar{{\bf{X}}}\) on the subspace defined by V. \({\bf{S}}\in {{\mathbb{R}}}^{({M}_{1}+\ldots +{M}_{k}+\ldots +{M}_{K})\times ({M}_{1}+\ldots +{M}_{k}+\ldots +{M}_{K})}\) is a diagonal matrix. To reduce data dimensionality, the top d principal components containing most of the variance in the data are retained, onto which the data is projected.

Multiclass Classification

For a classification task with G classes, One-vs-All (OVA) method constructs G classifiers, each tailored to separate one class from the rest. One Shot Classification (OSC) generates a single classifier designed to simultaneously distinguish between all the classes. The last comparative strategy involves the following binary classification tasks, AD vs. HC and MCI vs. HC.

Experiment 1: Single Modality and Multi-Modality Cascaded Classification

The objective of this experiment is to examine (i) classification performance achieved by combining multiple modalities as compared to any single modality for all classification tasks within the cascade design. In addition, the experiment seeks to determine if (ii) combining subsets of modalities tailored to optimize classification at each level of the cascade provides comparable and/or improved performance as compared to combining all the modalities for all tasks. Finally, it also evaluates (iii) the impact of the chosen fusion method on the findings for (i) and (ii).

To meet these objectives, we compare the data fusion (\( {\mathcal F} \)) and modality selection (\({\mathscr{K}}\)) modules in CaMCCo (\({ {\mathcal H} }_{CAMCCO}={{\mathscr{C}}}_{CAS}+{ {\mathcal F} }_{SMVCCA}+{{\mathscr{K}}}_{SEL}\)) with other fusion (\({ {\mathcal F} }_{PCA}\)), and modality selection (\({{\mathscr{K}}}_{ALL}\)) approaches, including the simple single modality (\({{\mathscr{K}}}_{MRI}\), \({{\mathscr{K}}}_{PET}\), \({{\mathscr{K}}}_{CSF}\), \({{\mathscr{K}}}_{PP}\), \({{\mathscr{K}}}_{APOE}\), \({{\mathscr{K}}}_{ADAS}\)) classification. For individual modality experiments, PCA was applied to the experiments where the number of features were larger than the number of samples to avoid curse of dimensionality. For fused and concatenated classifiers, the number of reduced dimensions was optimized on the training set. Therefore, we consider all the combinations of modalities and fusion methods listed below:

$$\begin{array}{rcl}\,\,{ {\mathcal H} }_{ALL} & = & {{\mathscr{C}}}_{CAS}+{ {\mathcal F} }_{SMVCCA}+{{\mathscr{K}}}_{ALL}\\ \,\,{ {\mathcal H} }_{PCA} & = & {{\mathscr{C}}}_{CAS}+{ {\mathcal F} }_{PCA}+{{\mathscr{K}}}_{SEL}\\ \,{ {\mathcal H} }_{PCAL} & = & {{\mathscr{C}}}_{CAS}+{ {\mathcal F} }_{PCA}+{{\mathscr{K}}}_{ALL}\\ \,\,{ {\mathcal H} }_{MRI} & = & {{\mathscr{C}}}_{CAS}+{{\mathscr{K}}}_{MRI}\\ \,\,{ {\mathcal H} }_{PET} & = & {{\mathscr{C}}}_{CAS}+{{\mathscr{K}}}_{PET}\\ \,\,{ {\mathcal H} }_{CSF} & = & {{\mathscr{C}}}_{CAS}+{{\mathscr{K}}}_{CSF}\\ \,\,\,{ {\mathcal H} }_{PP} & = & {{\mathscr{C}}}_{CAS}+{{\mathscr{K}}}_{PP}\\ { {\mathcal H} }_{APOE} & = & {{\mathscr{C}}}_{CAS}+{{\mathscr{K}}}_{APOE}\\ { {\mathcal H} }_{ADAS} & = & {{\mathscr{C}}}_{CAS}+{{\mathscr{K}}}_{ADAS}\end{array}$$

Experiment 2: Comparison of Multi-Class Classification Strategies for Fused Predictors

The objective of this experiment is to compare the cascaded classification method \({{\mathscr{C}}}_{CAS}\) used in CaMCCo (\({ {\mathcal H} }_{CAMCCO}={{\mathscr{C}}}_{CAS}+{ {\mathcal F} }_{SMVCCA}+{{\mathscr{K}}}_{SEL}\)) with other multiclass classification methods including OVA (\({{\mathscr{C}}}_{OVA}\)) and OSC (\({{\mathscr{C}}}_{OSC}\)). To ensure that only the classification module of the CaMCCo framework is evaluated, comparative classification strategies are combined with the same data fusion (\({ {\mathcal F} }_{SMVCCA}\)) and modality selection method (\({{\mathscr{K}}}_{SEL}\)) as CaMCCo. Therefore,\({ {\mathcal H} }_{OVA}={{\mathscr{C}}}_{OVA}+{ {\mathcal F} }_{SMVCCA}+{{\mathscr{K}}}_{SEL}\), and \({ {\mathcal H} }_{OSC}={{\mathscr{C}}}_{OSC}+{ {\mathcal F} }_{SMVCCA}+{{\mathscr{K}}}_{SEL}\). As with CaMCCo, the optimal set of modalities to combine for each classification task associated with OVA and OSC are determined experimentally from the training set.

Experiment 3: Evaluation of Fused Representation on Binary Classification Tasks

We perform binary classification (\({{\mathscr{C}}}_{BIN}\)) for the following two sets of classes, HC vs. AD and MCI vs. HC, in order to allow for direct comparison of the performance of fusion approach used in CaMCCo (\({ {\mathcal F} }_{SMVCCA}\) + \({{\mathscr{K}}}_{SEL}\)) with that reported in literature. As with CaMCCo, the optimal set of modalities to combine for each classification task are determined experimentally from the training set. In addition, we also report binary classification results achieved by individual modalities to examine the effect of fusion for these classification tasks and also to gain insight into the differences in classifier performance on account of the data cohort used in this study as compared to those in other studies.

Results and Discussion

Experiment 1: Single Modality and Multi-Modality Cascaded Classification

Figure 3 shows the performance of cascaded classifier when applied to (i) single modalities (\({ {\mathcal H} }_{MRI}\), \({ {\mathcal H} }_{PET}\), \({ {\mathcal H} }_{CSF}\), \({ {\mathcal H} }_{PP}\), \({ {\mathcal H} }_{APOE}\), \({ {\mathcal H} }_{ADAS}\)), (ii) fusion of all modalities (\({ {\mathcal H} }_{ALL}\), \({ {\mathcal H} }_{PCAL}\)) and (iii) fusion of selected modalities with multiple fusion methods (\({ {\mathcal H} }_{CAMCCO}\), \({ {\mathcal H} }_{PCA}\)) for prediction of HC, MCI and AD on the testing cohort. As shown in Fig. 1, ADAS-Cog, CSF and APOE were combined at the first cascade level (HC vs. All) and ADAS-Cog, and PET were combined at the second level (AD vs. MCI) in both \({ {\mathcal H} }_{CAMCCO}\) and \({ {\mathcal H} }_{PCA}\). For HC vs. all, \({ {\mathcal H} }_{CAMCCO}\) shows higher performance (AUC = 0.97) as compared to all individual modalities (max AUC = 0.93), \({ {\mathcal H} }_{PCA}\) (AUC = 0.94), \({ {\mathcal H} }_{PCAL}\) and \({ {\mathcal H} }_{ALL}\) (AUC = 0.52). Among the individual modalities, \({ {\mathcal H} }_{ADAS}\), \({ {\mathcal H} }_{CSF}\) and \({ {\mathcal H} }_{APOE}\) provided the top 3 classification AUCs, which was consistent with observations in the training set which led to these three modalities being selected for fusion in CaMCCo.

Figure 3
figure 3

Performance of single and multi modality cascaded classifiers. Area under the ROC curve (AUC) for prediction of (a) healthy control (HC) from all cognitive impairments, and (b) mild cognitive impairment (MCI) from Alzheimer’s Disease (AD).

Full size image

For AD vs. MCI, \({ {\mathcal H} }_{CAMCCO}\), \({ {\mathcal H} }_{PCA}\) and \({ {\mathcal H} }_{ADAS}\) showed similar performances (AUC = 0.89). This may, in part, be on account of the lack of orthogonality in the features being fused. A correlation test between the ADAS and PET features showed correlation coefficients between −0.49 and −0.51 with p-value < 0.01. In fact, the modality selection strategy employed in this work is limited in that it does not account for relationships between modalities to identify those that optimize the performance when fused. Instead, the selection of modalities are simplified and are based on their individual performances. However, it is interesting to note that despite the significantly poorer performance of \({ {\mathcal H} }_{PET}\) as compared to \({ {\mathcal H} }_{ADAS}\), combining PET with ADAS-Cog does not degrade the performance. We note that, similar to HC vs. All, \({ {\mathcal H} }_{CAMCCO}\) significantly outperforms \({ {\mathcal H} }_{ALL}\).

Experiment 2: Comparison of Multi-Class Classification Strategies for Fused Predictors

Table 6 shows the performance of \({ {\mathcal H} }_{CAMCCO}\), \({ {\mathcal H} }_{OVA}\) and \({ {\mathcal H} }_{OSC}\) for prediction of HC, MCI and AD. Across all 3 classes, it is evident that \({ {\mathcal H} }_{OVA}\) and \({ {\mathcal H} }_{CAMCCO}\) outperform \({ {\mathcal H} }_{OSC}\). \({ {\mathcal H} }_{OVA}\) and \({ {\mathcal H} }_{CAMCCO}\) show comparable AUCs for AD classification, although \({ {\mathcal H} }_{OVA}\) shows incrementally higher accuracy, sensitivity and specificity as compared to \({ {\mathcal H} }_{CAMCCO}\). For MCI classification however, \({ {\mathcal H} }_{CAMCCO}\) significantly outperforms \({ {\mathcal H} }_{OVA}\) in terms of all metrics (OVA AUC = 0.78 vs. CaMCCo AUC = 0.88). This is on account of the lower classification performance of MCI vs. all which is a challenging task provided the heterogeneity of the ‘all’ category which consists of both AD and HC patients. Therefore, CaMCCo provides the most optimal performance overall, across all 3 classes.

Table 6 Performance of multiclass classification strategies – one shot classifier (OSC), one vs. all (OVA), cascaded classifier in CaMCCo – upon fusion of modalities chosen from training set for each classification task.
Full size table

Experiment 3: Evaluation of Fused Representation on Binary Classification Tasks

Table 7 shows \({ {\mathcal H} }_{BIN}\) results obtained by combining select few modalities (\({{\mathscr{K}}}_{SEL}\)) via sMVCCA (\({ {\mathcal F} }_{SMVCCA}\)) for the following binary classification tasks: (i) AD vs. HC and (ii) MCI vs. HC. On the training set, the fusion of ADAS-Cog and CSF provided the best classification AUC for AD vs. HC whereas the fusion of ADAS-Cog, CSF and PET provided the best classification for MCI vs. HC. Thus, \({ {\mathcal H} }_{BIN}\) for the two classification tasks fused the respective, aforementioned modalities for the test set. For the former classification task, the performance of \({ {\mathcal H} }_{BIN}\) was similar to that of the best performing individual modality, ADAS-Cog, which already provided near perfect AUC leaving little scope for improvement. According to other evaluation metrics, ADAS-Cog outperforms the fused representation. For the more challenging MCI vs. HC classification task however, \({ {\mathcal H} }_{BIN}\) improves classifier performance slightly in terms of AUC (0.92 vs. 0.93) but more significantly in terms of BACC (0.77 vs. 0.82) and SPEC (0.65 vs. 0.71). In comparison to most previous work, our individual modality and fused modality results appear to be slightly higher possibly on account of the features that were considered in this work, all of which were quality controlled and independently proven to provide good performance previously. In addition, this work considers a neurocognitive score (ADAS-Cog), a measure that is mostly either used as a response variable or unconsidered in many fusion studies. The ADAS score appears to be strongly predictive of all classification tasks, possibly on account of a strong correlation with the MMSE scores, which were used to derive the ground truth class labels. Therefore, most gains in classification accuracy appears to be only slightly incremental. A correlation test between ADAS scores and MMSE scores across 118 patients, who had the latter data available, showed that the two were indeed highly correlated with a coefficient of −0.65 and p-value < 0.01.

Table 7 Performance of the combined fusion (\({ {\mathcal F} }_{SMVCCA}\)) and modality selection (\({{\mathscr{K}}}_{SEL}\)) modules of CaMCCo for binary classification (\({ {\mathcal H} }_{BIN}\)).
Full size table

Conclusion

In this work, we present a joint cascaded classification and radio-omics data fusion framework, called Cascaded Multiview Canonical Correlation (CaMCCo), for early diagnosis of Alzheimer’s disease. CaMCCo employs a unique strategy as compared to most previous approaches in that it accounts for multiclass classification while attempting to optimize classification accuracy by fusing a select subset of modalities for prediction of each class. As a framework, CaMCCo is comprised of three modules: (i) data fusion, (ii) modality selection and (iii) multiclass classification. Experiments were designed to investigate the choice of methods used for each CaMCCo module independently. In the first experiment, for instance, classification method was held constant while the data fusion and modality selection modules were varied and compared with that of CaMCCo. In the second experiment, the modality selection and data fusion methods were held constant and the cascaded classifier in CaMCCo was compared against other multiclass classification methods. Experimental findings on the ADNI dataset, comprising imaging, proteomics, genomics and neurophysiological data, consistently indicated that fusion of select multi-scale data channels, as in CaMCCo, outperforms fusion of all available modalities. In addition, the results showed that cascaded classification used in CaMCCo is better suited than other multi-class classification methods for MCI prediction. Finally, CaMCCo was compared against individual modalities for the two most commonly investigated binary classification tasks in most related studies, AD vs. HC and MCI vs. HC. While AD vs. HC was a simpler task well resolved by a single modality in our study, MCI vs. HC was a more challenging task where the application of CaMCCo appeared to improve classification, most significantly in terms of specificity. CaMCCo appears to be better able to distinguish between MCI and HC as compared to most previous studies, some of which are listed in Table 5.

However, the work presented in this paper is limited mainly by the method with which the modalities to be combined at each level of the cascade is determined. We only combine the modalities that independently provide the best accuracies on the training set, which may not be complementary. Nonetheless, we found that considering a subset of modalities provides improved performance over fusing all modalities. These findings indicate that incorporation of a more advanced modality selection method and additionally a feature selection method into the framework may provide further improvement in performance. Another limitation of the proposed strategy is the propagation of error from one level of the cascade to the next. To minimize this error, we therefore begin the cascade with the one-vs-all classification providing the least error. Despite these limitations, current findings indicate that the presented framework provides a promising platform for fusion of multiscale, multimodal data for early diagnosis of Alzheimer’s Disease.