Image fusion of mass spectrometry and microscopy: a multimodality paradigm for molecular tissue mapping

Journal name:
Nature Methods
Year published:
Published online


We describe a predictive imaging modality created by 'fusing' two distinct technologies: imaging mass spectrometry (IMS) and microscopy. IMS-generated molecular maps, rich in chemical information but having coarse spatial resolution, are combined with optical microscopy maps, which have relatively low chemical specificity but high spatial information. The resulting images combine the advantages of both technologies, enabling prediction of a molecular distribution both at high spatial resolution and with high chemical specificity. Multivariate regression is used to model variables in one technology, using variables from the other technology. We demonstrate the potential of image fusion through several applications: (i) 'sharpening' of IMS images, which uses microscopy measurements to predict ion distributions at a spatial resolution that exceeds that of measured ion images by ten times or more; (ii) prediction of ion distributions in tissue areas that were not measured by IMS; and (iii) enrichment of biological signals and attenuation of instrumental artifacts, revealing insights not easily extracted from either microscopy or IMS individually.

At a glance


  1. Concept of image fusion of imaging mass spectrometry (IMS) and microscopy.
    Figure 1: Concept of image fusion of imaging mass spectrometry (IMS) and microscopy.

    Image fusion generates a single image from two or more source images, combining the advantages of the different sensor types. The integration of IMS and optical microscopy is given as an example. The IMS-microscopy fusion image is a predictive modality that delivers both the chemical specificity of IMS and the spatial resolution of microscopy in one integrated whole. Each source image measures a different aspect of the content of a tissue sample. The fused image predicts the tissue content as if all aspects were observed concurrently.

  2. Example of IMS-microscopy fusion.
    Figure 2: Example of IMS-microscopy fusion.

    An ion image measured in mouse brain, describing the distribution of m/z 778.5 (identified as lipid PE(P-40:4)) at 100-μm spatial resolution (top right), is integrated with an H&E microscopy image measured from the same tissue sample at 10-μm resolution (top left). By combining the information from both image types, the image fusion process can predict the ion distribution of m/z 778.5 at 10-μm resolution (bottom).

  3. Prediction of the ion distribution of m/z 762.5 in mouse brain at 10-[mu]m resolution from 100-[mu]m IMS and 10-[mu]m microscopy measurements (sharpening).
    Figure 3: Prediction of the ion distribution of m/z 762.5 in mouse brain at 10-μm resolution from 100-μm IMS and 10-μm microscopy measurements (sharpening).

    (a) This example in mouse brain fuses a measured ion image for m/z 762.5 (identified as lipid PE(16:0/22:6)) at 100-μm spatial resolution with a measured H&E microscopy image at 10-μm resolution, predicting the ion distribution of m/z 762.5 at 10-μm resolution. Reconstruction (rec.) scores are shown. Color scales encode arbitrary ion intensity units. (b) For comparison, a measured ion image for m/z 762.5 at 10-μm spatial resolution is shown, acquired from a neighboring tissue section.

  4. Prediction of the ion distributions of m/z 646.4 and 788.5 in mouse brain at 330-nm resolution from 10-[mu]m IMS and 330-nm microscopy measurements (sharpening).
    Figure 4: Prediction of the ion distributions of m/z 646.4 and 788.5 in mouse brain at 330-nm resolution from 10-μm IMS and 330-nm microscopy measurements (sharpening).

    Measured ion images acquired in mouse brain for m/z 646.4 and m/z 788.5 at 10-μm spatial resolution are fused with an H&E microscopy image measured at 0.33-μm resolution. The resulting IMS-microscopy model is combined with the microscopy measurements to predict the ion distributions of m/z 646.4 and m/z 788.5 at 330-nm resolution with overall reconstruction scores of, respectively, 75% and 76%. Color scales encode arbitrary ion intensity units.

  5. Prediction of m/z 10,516 distribution in mouse brain areas not measured by IMS (out-of-sample prediction).
    Figure 5: Prediction of m/z 10,516 distribution in mouse brain areas not measured by IMS (out-of-sample prediction).

    (a) An IMS-microscopy model is built on a tissue subarea for which IMS is available at 100-μm resolution and an H&E microscopy image is available at 5-μm resolution. (b) The model is then used to predict the distribution of m/z 10,516 in areas where no IMS was acquired and only microscopy is available. Color scales encode arbitrary ion intensity units. (A non-sharpened version is available in Supplementary Fig. 21.)

  6. Discovery of tissue features through multimodal enrichment.
    Figure 6: Discovery of tissue features through multimodal enrichment.

    An ion image for m/z 3,345 measured by IMS at 100-μm resolution in a rat kidney section is fused with an H&E microscopy image acquired at 5-μm resolution to produce an ion distribution prediction at 5-μm resolution (reconstruction score: 85%). Annotations i–iii demonstrate multimodal enrichment. Their successful propagation through the fusion process and their presence in the final fused image confirms that they are genuine tissue features that are corroborated by another technology (in this case, microscopy). Annotation iv demonstrates multimodal attenuation. The lack of cross-modal support for this localized drop in ion intensity reduces confidence in the biological nature of this feature and at least labels it as an IMS-specific observation. Color scales encode arbitrary ion intensity units.


  1. Weissleder, R. Scaling down imaging: molecular mapping of cancer in mice. Nat. Rev. Cancer 2, 1118 (2002).
  2. Massoud, T.F. & Gambhir, S.S. Molecular imaging in living subjects: seeing fundamental biological processes in a new light. Genes Dev. 17, 545580 (2003).
  3. Jahn, K.A. et al. Correlative microscopy: providing new understanding in the biomedical and plant sciences. Micron 43, 565582 (2012).
  4. Jacobs, R.E. & Cherry, S.R. Complementary emerging techniques: high-resolution PET and MRI. Curr. Opin. Neurobiol. 11, 621629 (2001).
  5. Chughtai, S. et al. A multimodal mass spectrometry imaging approach for the study of musculoskeletal tissues. Int. J. Mass Spectrom. 325–327, 150160 (2012).
  6. Smith, C. Two microscopes are better than one. Nature 492, 293297 (2012).
  7. Caplan, J., Niethammer, M., Taylor, R.M. II. & Czymmek, K.J. The power of correlative microscopy: multi-modal, multi-scale, multi-dimensional. Curr. Opin. Struct. Biol. 21, 686693 (2011).
  8. Modla, S. & Czymmek, K.J. Correlative microscopy: a powerful tool for exploring neurological cells and tissues. Micron 42, 773792 (2011).
  9. Townsend, D.W. A combined PET/CT scanner: the choices. J. Nucl. Med. 42, 533534 (2001).
  10. Townsend, D.W., Beyer, T. & Blodgett, T.M. PET/CT scanners: a hardware approach to image fusion. Semin. Nucl. Med. 33, 193204 (2003).
  11. Masyuko, R., Lanni, E.J., Sweedler, J.V. & Bohn, P.W. Correlated imaging - a grand challenge in chemical analysis. Analyst 138, 19241939 (2013).
  12. Bocklitz, T.W. et al. Deeper understanding of biological tissue: quantitative correlation of MALDI-TOF and Raman imaging. Anal. Chem. 85, 1082910834 (2013).
  13. Clarke, F.C. et al. Chemical image fusion. The synergy of FT-NIR and Raman mapping microscopy to enable a more complete visualization of pharmaceutical formulations. Anal. Chem. 73, 22132220 (2001).
  14. Judenhofer, M.S. et al. Simultaneous PET-MRI: a new approach for functional and morphological imaging. Nat. Med. 14, 459465 (2008).
  15. Glenn, D.R. et al. Correlative light and electron microscopy using cathodoluminescence from nanoparticles with distinguishable colours. Sci. Rep. 2, 865 (2012).
  16. Josephson, L., Kircher, M.F., Mahmood, U., Tang, Y. & Weissleder, R. Near-infrared fluorescent nanoparticles as combined MR/optical imaging probes. Bioconjug. Chem. 13, 554560 (2002).
  17. Blum, R.S. & Liu, Z. Multi-sensor Image Fusion and Its Applications (CRC Press, 2005).
  18. Bretschneider, T. & Kao, O. in Proc. 1st Online Symp. Electron. Eng. 18 (2000).
  19. Pohl, C. & Van Genderen, J.L. Multisensor image fusion in remote sensing: concepts, methods and applications. Int. J. Remote Sens. 19, 823854 (1998).
  20. Price, J.C. Combining multispectral data of differing spatial resolution. IEEE Trans. Geosci. Rem. Sens. 37, 11991203 (1999).
  21. Simone, G., Farina, A., Morabito, F.C., Serpico, S.B. & Bruzzone, L. Image fusion techniques for remote sensing applications. Inf. Fusion 3, 315 (2002).
  22. Gaemperli, O. et al. Cardiac image fusion from stand-alone SPECT and CT: clinical experience. J. Nucl. Med. 48, 696703 (2007).
  23. Li, H. et al. Object recognition in brain CT-scans: knowledge-based fusion of data from multiple feature extractors. IEEE Trans. Med. Imaging 14, 212229 (1995).
  24. Yang, L., Guo, B.L. & Ni, W. Multimodality medical image fusion based on multiscale geometric analysis of contourlet transform. Neurocomputing 72, 203211 (2008).
  25. Varshney, P.K. et al. in Proc. Int. Conf. Image Proc. 3, 532536 (IEEE, 1999).
  26. Caprioli, R.M., Farmer, T.B. & Gile, J. Molecular imaging of biological samples: localization of peptides and proteins using MALDI-TOF MS. Anal. Chem. 69, 47514760 (1997).
  27. Stoeckli, M., Chaurand, P., Hallahan, D.E. & Caprioli, R.M. Imaging mass spectrometry: a new technology for the analysis of protein expression in mammalian tissues. Nat. Med. 7, 493496 (2001).
  28. Amstalden van Hove, E.R., Smith, D.F. & Heeren, R. A concise review of mass spectrometry imaging. J. Chromatogr. A 1217, 39463954 (2010).
  29. Chaurand, P. Imaging mass spectrometry of thin tissue sections: a decade of collective efforts. J. Proteomics 75, 48834892 (2012).
  30. Norris, J.L. & Caprioli, R.M. Analysis of tissue specimens by matrix-assisted laser desorption/ionization imaging mass spectrometry in biological and clinical research. Chem. Rev. 113, 23092342 (2013).
  31. Murphy, R.C., Hankin, J.A. & Barkley, R.M. Imaging of lipid species by MALDI mass spectrometry. J. Lipid Res. 50, S317S322 (2009).
  32. Van de Plas, R. Tissue Based Proteomics and Biomarker Discovery – Multivariate Data Mining Strategies for Mass Spectral Imaging. PhD thesis, KU Leuven (2010).
  33. Andersson, M., Andren, P. & Caprioli, R.M. in Neuroproteomics (ed. Azalte, O.) Ch. 7, 115134 (CRC Press, 2009).
  34. Franck, J. et al. MALDI mass spectrometry imaging of proteins exceeding 30,000 daltons. Med. Sci. Monit. 16, BR293BR299 (2010).
  35. Bradshaw, R., Bleay, S., Wolstenholme, R., Clench, M.R. & Francese, S. Towards the integration of matrix assisted laser desorption ionisation mass spectrometry imaging into the current fingermark examination workflow. Forensic Sci. Int. 232, 111124 (2013).
  36. Chavez, P.S. Jr., Sides, S.C. & Anderson, J.A. Comparison of three different methods to merge multiresolution and multispectral data- Landsat TM and SPOT panchromatic. Photogramm. Eng. Remote Sensing 57, 295303 (1991).
  37. Garguet-Duport, B., Girel, J., Chassery, J.-M. & Patou, G. The use of multiresolution analysis and wavelets transform for merging SPOT panchromatic and multispectral image data. Photogramm. Eng. Remote Sensing 62, 10571066 (1996).
  38. Lee, J. & Lee, C. Fast and efficient panchromatic sharpening. IEEE Trans. Geosci. Remote Sens. 48, 155163 (2010).
  39. Draper, N.R., Smith, H. & Pownell, E. Applied Regression Analysis 1st edn. (Wiley, 1966).
  40. Wold, S., Sjöström, M. & Eriksson, L. PLS-regression: a basic tool of chemometrics. Chemom. Intell. Lab. Syst. 58, 109130 (2001).

Download references

Author information


  1. Mass Spectrometry Research Center, Vanderbilt University, Nashville, Tennessee, USA.

    • Raf Van de Plas,
    • Junhai Yang,
    • Jeffrey Spraggins &
    • Richard M Caprioli
  2. Department of Biochemistry, Vanderbilt University, Nashville, Tennessee, USA.

    • Raf Van de Plas,
    • Junhai Yang,
    • Jeffrey Spraggins &
    • Richard M Caprioli
  3. Delft Center for Systems and Control, Delft University of Technology, Delft, the Netherlands.

    • Raf Van de Plas
  4. Department of Chemistry, Vanderbilt University, Nashville, Tennessee, USA.

    • Richard M Caprioli
  5. Department of Pharmacology, Vanderbilt University, Nashville, Tennessee, USA.

    • Richard M Caprioli
  6. Department of Medicine, Vanderbilt University, Nashville, Tennessee, USA.

    • Richard M Caprioli


R.V.d.P. conceived of and developed the methodology, designed experiments, analyzed and interpreted data and wrote the manuscript; J.Y. designed experiments, acquired data and edited the manuscript; J.S. designed experiments, acquired data, performed identifications and edited the manuscript; R.M.C. designed experiments, interpreted data, revised the manuscript and is principal investigator for the grants that fund this research.

Competing financial interests

The authors declare no competing financial interests.

Corresponding author

Correspondence to:

Author details

Supplementary information

PDF files

  1. Supplementary Text and Figures (100,250 KB)

    Supplementary Figures 1–21, Supplementary Table 1, Supplementary Notes 1–4, Supplementary Results and Supplementary Discussion

Zip files

  1. Supplementary Data (1,583 KB)

    Supplementary Data for Case Study 1
    This zip-file contains images related to Step 2 of the model building and evaluation phase, the mapping of transformed IMS and microscopy measurement sets to each other. The figures depict the IMS-to-microscopy weighted permutation mapping function (at the microscopy spatial resolution) and the integer mapping weights defined by it.

Additional data