Letter | Published:

Image reconstruction by domain-transform manifold learning

Nature volume 555, pages 487492 (22 March 2018) | Download Citation

Abstract

Image reconstruction is essential for imaging applications across the physical and life sciences, including optical and radar systems, magnetic resonance imaging, X-ray computed tomography, positron emission tomography, ultrasound imaging and radio astronomy1,2,3. During image acquisition, the sensor encodes an intermediate representation of an object in the sensor domain, which is subsequently reconstructed into an image by an inversion of the encoding function. Image reconstruction is challenging because analytic knowledge of the exact inverse transform may not exist a priori, especially in the presence of sensor non-idealities and noise. Thus, the standard reconstruction approach involves approximating the inverse function with multiple ad hoc stages in a signal processing chain4,5, the composition of which depends on the details of each acquisition strategy, and often requires expert parameter tuning to optimize reconstruction performance. Here we present a unified framework for image reconstruction—automated transform by manifold approximation (AUTOMAP)—which recasts image reconstruction as a data-driven supervised learning task that allows a mapping between the sensor and the image domain to emerge from an appropriate corpus of training data. We implement AUTOMAP with a deep neural network and exhibit its flexibility in learning reconstruction transforms for various magnetic resonance imaging acquisition strategies, using the same network architecture and hyperparameters. We further demonstrate that manifold learning during training results in sparse representations of domain transforms along low-dimensional data manifolds, and observe superior immunity to noise and a reduction in reconstruction artefacts compared with conventional handcrafted reconstruction methods. In addition to improving the reconstruction performance of existing acquisition methodologies, we anticipate that AUTOMAP and other learned reconstruction approaches will accelerate the development of new acquisition strategies across imaging modalities.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

References

  1. 1.

    Tomography (John Wiley & Sons, 2013)

  2. 2.

    & Image reconstruction from incomplete and noisy data. Nature 272, 686–690 (1978)

  3. 3.

    Medical Image Reconstruction (Springer, 2010)

  4. 4.

    ., ., ., & Fast model-based X-ray CT reconstruction using spatially nonhomogeneous ICD optimization. IEEE Trans. Image Process. 20, 161–175 (2011)

  5. 5.

    ., ., . & Advances in sensitivity encoding with arbitrary k-space trajectories. Magn. Reson. Med. 46, 638–651 (2001)

  6. 6.

    . et al. Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process. Mag. 29, 82–97 (2012)

  7. 7.

    ., & ImageNet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 1097–1105 (2012)

  8. 8.

    ., & The neural basis of perceptual learning. Neuron 31, 681–697 (2001)

  9. 9.

    ., ., ., & Visual perceptual learning. Neurobiol. Learn. Mem. 95, 145–151 (2011)

  10. 10.

    ., ., & Extracting and composing robust features with denoising autoencoders. In Proc. 25th Int. Conf. on ‘Machine Learning’ 1096–1103, (2008)

  11. 11.

    , & Neural network based solution to inverse problems. In IEEE World Congr. on ‘Computational Intelligence’ Vol. 3, 2471–2476, (1998)

  12. 12.

    & Neural network for emulation of an inverse model operational derivation of Case II water properties from MERIS data. Int. J. Remote Sens. 20, 1735–1746 (1999)

  13. 13.

    Artificial neural networks in the solution of inverse electromagnetic field problems. IEEE Trans. Magn. 29, 1931–1934 (1993)

  14. 14.

    An artificial neural network for SPECT image reconstruction. IEEE Trans. Med. Imaging 10, 485–487 (1991)

  15. 15.

    & Fast tomographic reconstruction from limited data using artificial neural networks. IEEE Trans. Image Process. 22, 5238–5251 (2013)

  16. 16.

    ., ., . & Deep convolutional neural network for inverse problems in imaging. IEEE Trans. Image Process. 26, 4509–4522 (2017)

  17. 17.

    . et al. Learning a variational network for reconstruction of accelerated MRI data. Magn. Reson. Med. 79, 3055–3071 (2017)

  18. 18.

    ., & Sparse MRI: the application of compressed sensing for rapid MR imaging. Magn. Reson. Med. 58, 1182–1195 (2007)

  19. 19.

    . et al. MGH–USC Human Connectome Project datasets with ultra-high b-value diffusion MRI. Neuroimage 124, 1108–1114 (2016)

  20. 20.

    et al. ImageNet: a large-scale hierarchical image database. In IEEE Conf. on ‘Computer Vision and Pattern Recognition’ 248–255, (2009)

  21. 21.

    ., & Multilayer feedforward networks are universal approximators. Neural Netw. 2, 359–366 (1989)

  22. 22.

    & Cardiac PET and PET/CT Imaging (Springer, 2007)

  23. 23.

    & Mean square optimal NUFFT approximation for efficient non-Cartesian MRI reconstruction. J. Magn. Reson. 242, 126–135 (2014)

  24. 24.

    . & On the empirical effect of Gaussian noise in under-sampled MRI reconstruction. Preprint at (2016)

  25. 25.

    , , , & Magnetic Resonance Imaging: Physical Principles and Sequence Design 2nd edn (Wiley, 2014)

  26. 26.

    , & Signal but not noise changes with perceptual learning. Nature 402, 176–178 (1999)

  27. 27.

    . et al. Sparse representation for computer vision and pattern recognition. Proc. IEEE 98, 1031–1044 (2010)

  28. 28.

    & Visualizing Data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008)

  29. 29.

    in Handbook of Applied Spatial Analysis (eds . & ) 255–278 (Springer, 2010)

  30. 30.

    . et al. Radiation dose reduction in chest CT: a review. Am. J. Roentgenol. 190, 335–343 (2008)

  31. 31.

    , , , & Characterization results of EMCCDs for extreme low-light imaging. In Proc. SPIE on ‘High Energy, Optical, and Infrared Detectors for Astronomy V’ Vol. 8453, 845303, (2012)

  32. 32.

    . et al. Sparse representations and convex optimization as tools for LOFAR radio interferometric imaging. J. Instrum. 10, C08013 (2015)

  33. 33.

    ., ., & Rapid volumetric OCT image acquisition using compressive sampling. Opt. Exp. 18, 21003–21012 (2010)

  34. 34.

    & Nonuniform fast Fourier transforms using min-max interpolation. IEEE Trans. Signal Process. 51, 560–574 (2003)

  35. 35.

    , & Simple analytic variable density spiral design. Magn. Reson. Med. 50, 214–219 (2003)

  36. 36.

    , , & Berkeley advanced reconstruction toolbox. Proc. Int. Soc. Magnetic Resonance in Medicine 2486 (2015)

  37. 37.

    et al. TensorFlow: large-scale machine learning on heterogeneous distributed systems. Preprint at (2016)

  38. 38.

    . & Rectified linear units improve restricted Boltzmann machines. In Proc. 27th Int. Conf. on ‘Machine Learning’ 807–814 (ACM, 2010)

  39. 39.

    & Winner-take-all autoencoders. Adv. Neural Inf. Process. Syst. 28, 2791–2799 (2015)

  40. 40.

    et al. Practical approaches to the evaluation of signal-to-noise ratio performance with parallel imaging: application with cardiac imaging and a 32-channel cardiac coil. Magn. Reson. Med. 54, 748–754 (2005)

  41. 41.

    , , & Simple correction method for k-space trajectory deviations in MRI. J. Magn. Reson. 132, 150–153 (1998)

  42. 42.

    , , & SENSE: sensitivity encoding for fast MRI. Magn. Reson. Med. 42, 952–962 (1999)

  43. 43.

    & GMRES: a generalized minimal residual algorithm for solving nonsymmetric linear systems. SIAM J. Sci. Statist. Comput. 7, 856–869 (1986)

  44. 44.

    , et al. OSEM-3D Reconstruction Strategies for the ECAT HRRT. IEEE Symp. Conf. Record Nuclear Science 6, 3492–3496 (2004)

  45. 45.

    et al. An SPM8-based approach for attenuation correction combining segmentation and nonrigid template formation: application to simultaneous PET/MR brain imaging. J. Nucl. Med. 55, 1825–1830 (2014)

  46. 46.

    & Improved local coordinate coding using local tangents. In Proc. 27th Int. Conf. on ‘Machine Learning’ 1215–1222 (ACM, 2010)

  47. 47.

    & A general spline representation for nonparametric and semiparametric density estimates using diffeomorphisms. Preprint at (2012)

  48. 48.

    ., & Bayesian estimation of regularization and atlas building in diffeomorphic image registration. Int. Conf. Inf. Process. Med. Imaging 37–48 (Springer, 2013)

  49. 49.

    , , & Geodesic image regression with a sparse parameterization of diffeomorphisms. In 1st Int. Conf. on ‘Geometric Science of Information’ GSI 2013 (eds & ) Vol. 8085, 95–102, (2013)

  50. 50.

    ., & Manifold Learning in Regression Tasks. Statistical Learning and Data Sciences 414–423 (Springer, 2015)

  51. 51.

    Approximation capabilities of multilayer feedforward networks. Neural Netw. 4, 251–257 (1991)

  52. 52.

    & Capabilities of three-layered perceptrons. In IEEE Int. Conf. on ‘Neural Networks’ Vol. 1, 641–648 (1988)

  53. 53.

    Approximation by superpositions of a sigmoidal function. Math. Contr. Signals Syst. 2, 303–314 (1989)

  54. 54.

    Approximation and estimation bounds for artificial neural networks. Mach. Learn. 14, 115–133 (1994)

  55. 55.

    , & DeepDream—a code example for visualizing neural networks. (Google Res, 2015)

  56. 56.

    , & Simple method for MR gradient system characterization and k-space trajectory estimation. Magn. Reson. Med. 68, 120–129 (2012)

  57. 57.

    , , & Direct measurement of magnetic field gradient waveforms. Concepts Magn. Reson. 36A, 349–360 (2010)

  58. 58.

    , & Generative adversarial nets. Adv. Neural Inf. Process. Syst. 2672–2680 (2014)

  59. 59.

    , & TArgeted Motion Estimation and Reduction (TAMER): data consistency based motion mitigation for MRI using a reduced model joint optimization. IEEE Trans. Med. Imaging PP, 99, (2018)

  60. 60.

    , , , & Toeplitz-based iterative image reconstruction for MRI with correction for magnetic field inhomogeneity. IEEE Trans. Signal Process. 53, 3393–3402 (2005)

  61. 61.

    et al. Fast reconstruction for multichannel compressed sensing using a hierarchically semiseparable solver. Magn. Reson. Med. 73, 1034–1040 (2015)

  62. 62.

    , , & Superfast and stable structured solvers for Toeplitz least squares via randomized sampling. SIAM J. Matrix Anal. Appl. 35, 44–72 (2014)

  63. 63.

    , , & Fast algorithms for hierarchically semiseparable matrices. Numer. Linear Algebra Appl. 17, 953–976 (2010)

  64. 64.

    , & Augmented Lagrangian with variable splitting for faster non-Cartesian L1-SPIRiT MR image reconstruction. IEEE Trans. Med. Imaging 33, 351–361 (2014)

  65. 65.

    , , , & Maximum likelihood reconstruction for magnetic resonance fingerprinting. IEEE Trans. Med. Imaging 35, 1812–1823 (2016)

  66. 66.

    , & Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding. Preprint at (2015)

  67. 67.

    , , & Network trimming: a data-driven neuron pruning approach towards efficient deep architectures. Preprint at (2016)

Download references

Acknowledgements

We acknowledge M. Michalski and the computational resources and assistance provided by the Massachusetts General Hospital (MGH) and the Brigham and Women’s Hospital (BWH) Center for Clinical Data Science (CCDS). The CCDS is supported by MGH, BWH, the MGH Department of Radiology, the BWH Department of Radiology, and through industry partnership with NVIDIA. We also acknowledge the Center for Machine Learning at Martinos. We also thank J. Stockmann, J. Polimeni, D. E. J. Waddington and R. L. Walsworth for their comments on this manuscript, and B. Bilgic and C. Liao for their assistance in human subject data acquisition. We acknowledge C. Catana for providing raw PET data and for filtered back projection and OSEM reconstructions. We also thank M. Haskell for providing the MRI motion encoding model. B.Z. was supported by the National Institutes of Health/National Institute of Biomedical Imaging and Bioengineering F32 Fellowship (EB022390). Data were provided in part by the HCP, MGH-USC Consortium (Principal Investigators: Bruce R. Rosen, Arthur W. Toga and Van Wedeen; U01MH093765), which was funded by the NIH Blueprint Initiative for Neuroscience Research grant; the National Institutes of Health grant P41EB015896; and the Instrumentation Grants S10RR023043, 1S10RR023401, 1S10RR019307.

Author information

Affiliations

  1. A. A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Boston, Massachusetts, USA

    • Bo Zhu
    • , Stephen F. Cauley
    • , Bruce R. Rosen
    •  & Matthew S. Rosen
  2. Harvard Medical School, Boston, Massachusetts, USA

    • Bo Zhu
    • , Stephen F. Cauley
    • , Bruce R. Rosen
    •  & Matthew S. Rosen
  3. Department of Physics, Harvard University, Cambridge, Massachusetts, USA.

    • Bo Zhu
    •  & Matthew S. Rosen
  4. Department of Biostatistics, Harvard University, Cambridge, Massachusetts, USA

    • Jeremiah Z. Liu

Authors

  1. Search for Bo Zhu in:

  2. Search for Jeremiah Z. Liu in:

  3. Search for Stephen F. Cauley in:

  4. Search for Bruce R. Rosen in:

  5. Search for Matthew S. Rosen in:

Contributions

B.Z., J.Z.L., S.F.C., B.R.R. and M.S.R. conceptualized the problem and contributed to experimental design. B.Z. developed, implemented and tested the technical framework. J.Z.L. and B.Z. constructed the theoretical description. B.Z., J.Z.L., S.F.C., B.R.R. and M.S.R. wrote the manuscript.

Competing interests

The authors declare no competing financial interests.

Corresponding author

Correspondence to Matthew S. Rosen.

Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

About this article

Publication history

Received

Accepted

Published

DOI

https://doi.org/10.1038/nature25988

Further reading

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.