Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Image fusion of mass spectrometry and microscopy: a multimodality paradigm for molecular tissue mapping

Abstract

We describe a predictive imaging modality created by 'fusing' two distinct technologies: imaging mass spectrometry (IMS) and microscopy. IMS-generated molecular maps, rich in chemical information but having coarse spatial resolution, are combined with optical microscopy maps, which have relatively low chemical specificity but high spatial information. The resulting images combine the advantages of both technologies, enabling prediction of a molecular distribution both at high spatial resolution and with high chemical specificity. Multivariate regression is used to model variables in one technology, using variables from the other technology. We demonstrate the potential of image fusion through several applications: (i) 'sharpening' of IMS images, which uses microscopy measurements to predict ion distributions at a spatial resolution that exceeds that of measured ion images by ten times or more; (ii) prediction of ion distributions in tissue areas that were not measured by IMS; and (iii) enrichment of biological signals and attenuation of instrumental artifacts, revealing insights not easily extracted from either microscopy or IMS individually.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Concept of image fusion of imaging mass spectrometry (IMS) and microscopy.
Figure 2: Example of IMS-microscopy fusion.
Figure 3: Prediction of the ion distribution of m/z 762.5 in mouse brain at 10-μm resolution from 100-μm IMS and 10-μm microscopy measurements (sharpening).
Figure 4: Prediction of the ion distributions of m/z 646.4 and 788.5 in mouse brain at 330-nm resolution from 10-μm IMS and 330-nm microscopy measurements (sharpening).
Figure 5: Prediction of m/z 10,516 distribution in mouse brain areas not measured by IMS (out-of-sample prediction).
Figure 6: Discovery of tissue features through multimodal enrichment.

Similar content being viewed by others

References

  1. Weissleder, R. Scaling down imaging: molecular mapping of cancer in mice. Nat. Rev. Cancer 2, 11–18 (2002).

    Article  CAS  Google Scholar 

  2. Massoud, T.F. & Gambhir, S.S. Molecular imaging in living subjects: seeing fundamental biological processes in a new light. Genes Dev. 17, 545–580 (2003).

    Article  CAS  Google Scholar 

  3. Jahn, K.A. et al. Correlative microscopy: providing new understanding in the biomedical and plant sciences. Micron 43, 565–582 (2012).

    Article  CAS  Google Scholar 

  4. Jacobs, R.E. & Cherry, S.R. Complementary emerging techniques: high-resolution PET and MRI. Curr. Opin. Neurobiol. 11, 621–629 (2001).

    Article  CAS  Google Scholar 

  5. Chughtai, S. et al. A multimodal mass spectrometry imaging approach for the study of musculoskeletal tissues. Int. J. Mass Spectrom. 325–327, 150–160 (2012).

    Article  Google Scholar 

  6. Smith, C. Two microscopes are better than one. Nature 492, 293–297 (2012).

    Article  CAS  Google Scholar 

  7. Caplan, J., Niethammer, M., Taylor, R.M. II. & Czymmek, K.J. The power of correlative microscopy: multi-modal, multi-scale, multi-dimensional. Curr. Opin. Struct. Biol. 21, 686–693 (2011).

    Article  CAS  Google Scholar 

  8. Modla, S. & Czymmek, K.J. Correlative microscopy: a powerful tool for exploring neurological cells and tissues. Micron 42, 773–792 (2011).

    Article  CAS  Google Scholar 

  9. Townsend, D.W. A combined PET/CT scanner: the choices. J. Nucl. Med. 42, 533–534 (2001).

    CAS  PubMed  Google Scholar 

  10. Townsend, D.W., Beyer, T. & Blodgett, T.M. PET/CT scanners: a hardware approach to image fusion. Semin. Nucl. Med. 33, 193–204 (2003).

    Article  Google Scholar 

  11. Masyuko, R., Lanni, E.J., Sweedler, J.V. & Bohn, P.W. Correlated imaging - a grand challenge in chemical analysis. Analyst 138, 1924–1939 (2013).

    Article  CAS  Google Scholar 

  12. Bocklitz, T.W. et al. Deeper understanding of biological tissue: quantitative correlation of MALDI-TOF and Raman imaging. Anal. Chem. 85, 10829–10834 (2013).

    Article  CAS  Google Scholar 

  13. Clarke, F.C. et al. Chemical image fusion. The synergy of FT-NIR and Raman mapping microscopy to enable a more complete visualization of pharmaceutical formulations. Anal. Chem. 73, 2213–2220 (2001).

    Article  CAS  Google Scholar 

  14. Judenhofer, M.S. et al. Simultaneous PET-MRI: a new approach for functional and morphological imaging. Nat. Med. 14, 459–465 (2008).

    Article  CAS  Google Scholar 

  15. Glenn, D.R. et al. Correlative light and electron microscopy using cathodoluminescence from nanoparticles with distinguishable colours. Sci. Rep. 2, 865 (2012).

    Article  CAS  Google Scholar 

  16. Josephson, L., Kircher, M.F., Mahmood, U., Tang, Y. & Weissleder, R. Near-infrared fluorescent nanoparticles as combined MR/optical imaging probes. Bioconjug. Chem. 13, 554–560 (2002).

    Article  CAS  Google Scholar 

  17. Blum, R.S. & Liu, Z. Multi-sensor Image Fusion and Its Applications (CRC Press, 2005).

  18. Bretschneider, T. & Kao, O. in Proc. 1st Online Symp. Electron. Eng. 1–8 (2000).

  19. Pohl, C. & Van Genderen, J.L. Multisensor image fusion in remote sensing: concepts, methods and applications. Int. J. Remote Sens. 19, 823–854 (1998).

    Article  Google Scholar 

  20. Price, J.C. Combining multispectral data of differing spatial resolution. IEEE Trans. Geosci. Rem. Sens. 37, 1199–1203 (1999).

    Article  Google Scholar 

  21. Simone, G., Farina, A., Morabito, F.C., Serpico, S.B. & Bruzzone, L. Image fusion techniques for remote sensing applications. Inf. Fusion 3, 3–15 (2002).

    Article  Google Scholar 

  22. Gaemperli, O. et al. Cardiac image fusion from stand-alone SPECT and CT: clinical experience. J. Nucl. Med. 48, 696–703 (2007).

    Article  Google Scholar 

  23. Li, H. et al. Object recognition in brain CT-scans: knowledge-based fusion of data from multiple feature extractors. IEEE Trans. Med. Imaging 14, 212–229 (1995).

    Article  CAS  Google Scholar 

  24. Yang, L., Guo, B.L. & Ni, W. Multimodality medical image fusion based on multiscale geometric analysis of contourlet transform. Neurocomputing 72, 203–211 (2008).

    Article  Google Scholar 

  25. Varshney, P.K. et al. in Proc. Int. Conf. Image Proc. 3, 532–536 (IEEE, 1999).

    Google Scholar 

  26. Caprioli, R.M., Farmer, T.B. & Gile, J. Molecular imaging of biological samples: localization of peptides and proteins using MALDI-TOF MS. Anal. Chem. 69, 4751–4760 (1997).

    Article  CAS  Google Scholar 

  27. Stoeckli, M., Chaurand, P., Hallahan, D.E. & Caprioli, R.M. Imaging mass spectrometry: a new technology for the analysis of protein expression in mammalian tissues. Nat. Med. 7, 493–496 (2001).

    Article  CAS  Google Scholar 

  28. Amstalden van Hove, E.R., Smith, D.F. & Heeren, R. A concise review of mass spectrometry imaging. J. Chromatogr. A 1217, 3946–3954 (2010).

    Article  CAS  Google Scholar 

  29. Chaurand, P. Imaging mass spectrometry of thin tissue sections: a decade of collective efforts. J. Proteomics 75, 4883–4892 (2012).

    Article  CAS  Google Scholar 

  30. Norris, J.L. & Caprioli, R.M. Analysis of tissue specimens by matrix-assisted laser desorption/ionization imaging mass spectrometry in biological and clinical research. Chem. Rev. 113, 2309–2342 (2013).

    Article  CAS  Google Scholar 

  31. Murphy, R.C., Hankin, J.A. & Barkley, R.M. Imaging of lipid species by MALDI mass spectrometry. J. Lipid Res. 50, S317–S322 (2009).

    Article  Google Scholar 

  32. Van de Plas, R. Tissue Based Proteomics and Biomarker Discovery – Multivariate Data Mining Strategies for Mass Spectral Imaging. PhD thesis, KU Leuven (2010).

  33. Andersson, M., Andren, P. & Caprioli, R.M. in Neuroproteomics (ed. Azalte, O.) Ch. 7, 115–134 (CRC Press, 2009).

  34. Franck, J. et al. MALDI mass spectrometry imaging of proteins exceeding 30,000 daltons. Med. Sci. Monit. 16, BR293–BR299 (2010).

    CAS  PubMed  Google Scholar 

  35. Bradshaw, R., Bleay, S., Wolstenholme, R., Clench, M.R. & Francese, S. Towards the integration of matrix assisted laser desorption ionisation mass spectrometry imaging into the current fingermark examination workflow. Forensic Sci. Int. 232, 111–124 (2013).

    Article  CAS  Google Scholar 

  36. Chavez, P.S. Jr., Sides, S.C. & Anderson, J.A. Comparison of three different methods to merge multiresolution and multispectral data- Landsat TM and SPOT panchromatic. Photogramm. Eng. Remote Sensing 57, 295–303 (1991).

    Google Scholar 

  37. Garguet-Duport, B., Girel, J., Chassery, J.-M. & Patou, G. The use of multiresolution analysis and wavelets transform for merging SPOT panchromatic and multispectral image data. Photogramm. Eng. Remote Sensing 62, 1057–1066 (1996).

    Google Scholar 

  38. Lee, J. & Lee, C. Fast and efficient panchromatic sharpening. IEEE Trans. Geosci. Remote Sens. 48, 155–163 (2010).

    Article  Google Scholar 

  39. Draper, N.R., Smith, H. & Pownell, E. Applied Regression Analysis 1st edn. (Wiley, 1966).

  40. Wold, S., Sjöström, M. & Eriksson, L. PLS-regression: a basic tool of chemometrics. Chemom. Intell. Lab. Syst. 58, 109–130 (2001).

    Article  CAS  Google Scholar 

Download references

Acknowledgements

This work was supported by the US National Institutes of Health grants NIH/NIGMS R01 GM058008-15 and NIH/NIGMS P41 GM103391-04. R.V.d.P. thanks E. Waelkens for his encouragement and support.

Author information

Authors and Affiliations

Authors

Contributions

R.V.d.P. conceived of and developed the methodology, designed experiments, analyzed and interpreted data and wrote the manuscript; J.Y. designed experiments, acquired data and edited the manuscript; J.S. designed experiments, acquired data, performed identifications and edited the manuscript; R.M.C. designed experiments, interpreted data, revised the manuscript and is principal investigator for the grants that fund this research.

Corresponding author

Correspondence to Richard M Caprioli.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Supplementary information

Supplementary Text and Figures

Supplementary Figures 1–21, Supplementary Table 1, Supplementary Notes 1–4, Supplementary Results and Supplementary Discussion (PDF 100622 kb)

Supplementary Data

Supplementary Data for Case Study 1 This zip-file contains images related to Step 2 of the model building and evaluation phase, the mapping of transformed IMS and microscopy measurement sets to each other. The figures depict the IMS-to-microscopy weighted permutation mapping function (at the microscopy spatial resolution) and the integer mapping weights defined by it. (ZIP 1546 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Van de Plas, R., Yang, J., Spraggins, J. et al. Image fusion of mass spectrometry and microscopy: a multimodality paradigm for molecular tissue mapping. Nat Methods 12, 366–372 (2015). https://doi.org/10.1038/nmeth.3296

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nmeth.3296

This article is cited by

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing