Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Letter
  • Published:

Image reconstruction by domain-transform manifold learning

Abstract

Image reconstruction is essential for imaging applications across the physical and life sciences, including optical and radar systems, magnetic resonance imaging, X-ray computed tomography, positron emission tomography, ultrasound imaging and radio astronomy1,2,3. During image acquisition, the sensor encodes an intermediate representation of an object in the sensor domain, which is subsequently reconstructed into an image by an inversion of the encoding function. Image reconstruction is challenging because analytic knowledge of the exact inverse transform may not exist a priori, especially in the presence of sensor non-idealities and noise. Thus, the standard reconstruction approach involves approximating the inverse function with multiple ad hoc stages in a signal processing chain4,5, the composition of which depends on the details of each acquisition strategy, and often requires expert parameter tuning to optimize reconstruction performance. Here we present a unified framework for image reconstruction—automated transform by manifold approximation (AUTOMAP)—which recasts image reconstruction as a data-driven supervised learning task that allows a mapping between the sensor and the image domain to emerge from an appropriate corpus of training data. We implement AUTOMAP with a deep neural network and exhibit its flexibility in learning reconstruction transforms for various magnetic resonance imaging acquisition strategies, using the same network architecture and hyperparameters. We further demonstrate that manifold learning during training results in sparse representations of domain transforms along low-dimensional data manifolds, and observe superior immunity to noise and a reduction in reconstruction artefacts compared with conventional handcrafted reconstruction methods. In addition to improving the reconstruction performance of existing acquisition methodologies, we anticipate that AUTOMAP and other learned reconstruction approaches will accelerate the development of new acquisition strategies across imaging modalities.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Schematic representations of AUTOMAP image reconstruction.
Figure 2: Reconstruction performance of AUTOMAP compared with conventional techniques.
Figure 3: Analysis of AUTOMAP neural networks.
Figure 4: Learning reconstruction of phase for in vivo data.
Figure 5: Performance of AUTOMAP in real-world acquisitions.

Similar content being viewed by others

References

  1. Grangeat, P. Tomography (John Wiley & Sons, 2013)

  2. Gull, S. F. & Daniell, G. J. Image reconstruction from incomplete and noisy data. Nature 272, 686–690 (1978)

    Article  ADS  Google Scholar 

  3. Zeng, G. L. Medical Image Reconstruction (Springer, 2010)

  4. Yu, Z ., Thibault, J.-B ., Bouman, C. A ., Sauer, K. D. & Hsieh, J. Fast model-based X-ray CT reconstruction using spatially nonhomogeneous ICD optimization. IEEE Trans. Image Process. 20, 161–175 (2011)

    Article  ADS  MathSciNet  Google Scholar 

  5. Pruessmann, K. P ., Weiger, M ., Börnert, P . & Boesiger, P. Advances in sensitivity encoding with arbitrary k-space trajectories. Magn. Reson. Med. 46, 638–651 (2001)

    Article  CAS  Google Scholar 

  6. Hinton, G . et al. Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process. Mag. 29, 82–97 (2012)

    Article  ADS  Google Scholar 

  7. Krizhevsky, A ., Sutskever, I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 1097–1105 (2012)

  8. Gilbert, C. D ., Sigman, M. & Crist, R. E. The neural basis of perceptual learning. Neuron 31, 681–697 (2001)

    Article  CAS  Google Scholar 

  9. Lu, Z.-L ., Hua, T ., Huang, C.-B ., Zhou, Y. & Dosher, B. A. Visual perceptual learning. Neurobiol. Learn. Mem. 95, 145–151 (2011)

    Article  Google Scholar 

  10. Vincent, P ., Larochelle, H ., Bengio, Y. & Manzagol, P.-A. Extracting and composing robust features with denoising autoencoders. In Proc. 25th Int. Conf. on ‘Machine Learning’ 1096–1103, http://www.cs.toronto.edu/~larocheh/publications/icml-2008-denoising-autoencoders.pdf (2008)

  11. Ogawa, T., Kosugi, Y. & Kanada, H. Neural network based solution to inverse problems. In IEEE World Congr. on ‘Computational Intelligence’ Vol. 3, 2471–2476, http://ieeexplore.ieee.org/document/687250/ (1998)

    Google Scholar 

  12. Schiller, H. & Doerffer, R. Neural network for emulation of an inverse model operational derivation of Case II water properties from MERIS data. Int. J. Remote Sens. 20, 1735–1746 (1999)

    Article  Google Scholar 

  13. Hoole, S. R. H. Artificial neural networks in the solution of inverse electromagnetic field problems. IEEE Trans. Magn. 29, 1931–1934 (1993)

    Article  ADS  Google Scholar 

  14. Floyd, C. E. An artificial neural network for SPECT image reconstruction. IEEE Trans. Med. Imaging 10, 485–487 (1991)

    Article  CAS  Google Scholar 

  15. Pelt, D. M. & Batenburg, K. J. Fast tomographic reconstruction from limited data using artificial neural networks. IEEE Trans. Image Process. 22, 5238–5251 (2013)

    Article  ADS  Google Scholar 

  16. Jin, K. H ., McCann, M. T ., Froustey, E . & Unser, M. Deep convolutional neural network for inverse problems in imaging. IEEE Trans. Image Process. 26, 4509–4522 (2017)

    Article  ADS  MathSciNet  Google Scholar 

  17. Hammernik, K . et al. Learning a variational network for reconstruction of accelerated MRI data. Magn. Reson. Med. 79, 3055–3071 (2017)

    Article  Google Scholar 

  18. Lustig, M ., Donoho, D. & Pauly, J. M. Sparse MRI: the application of compressed sensing for rapid MR imaging. Magn. Reson. Med. 58, 1182–1195 (2007)

    Article  Google Scholar 

  19. Fan, Q . et al. MGH–USC Human Connectome Project datasets with ultra-high b-value diffusion MRI. Neuroimage 124, 1108–1114 (2016)

    Article  Google Scholar 

  20. Deng, J. et al. ImageNet: a large-scale hierarchical image database. In IEEE Conf. on ‘Computer Vision and Pattern Recognition’ 248–255, http://www.image-net.org/papers/imagenet_cvpr09.pdf (2009)

  21. Hornik, K ., Stinchcombe, M. & White, H. Multilayer feedforward networks are universal approximators. Neural Netw. 2, 359–366 (1989)

    Article  Google Scholar 

  22. Di Carli, M. F. & Lipton, M. J. Cardiac PET and PET/CT Imaging (Springer, 2007)

  23. Yang, Z. & Jacob, M. Mean square optimal NUFFT approximation for efficient non-Cartesian MRI reconstruction. J. Magn. Reson. 242, 126–135 (2014)

    Article  CAS  ADS  Google Scholar 

  24. Virtue, P . & Lustig, M. On the empirical effect of Gaussian noise in under-sampled MRI reconstruction. Preprint at https://arxiv.org/abs/1610.00410 (2016)

  25. Brown, R. W., Cheng, Y. C. N., Haacke, E. M., Thompson, M. R. & Venkatesan, R. Magnetic Resonance Imaging: Physical Principles and Sequence Design 2nd edn (Wiley, 2014)

    Book  Google Scholar 

  26. Gold, J., Bennett, P. J. & Sekuler, A. B. Signal but not noise changes with perceptual learning. Nature 402, 176–178 (1999)

    Article  CAS  ADS  Google Scholar 

  27. Wright, J . et al. Sparse representation for computer vision and pattern recognition. Proc. IEEE 98, 1031–1044 (2010)

    Article  Google Scholar 

  28. Maaten, L. V. D. & Hinton, G. Visualizing Data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008)

    MATH  Google Scholar 

  29. Getis, A. in Handbook of Applied Spatial Analysis (eds Fisher, M. M . & Getis, A. ) 255–278 (Springer, 2010)

  30. Kubo, T . et al. Radiation dose reduction in chest CT: a review. Am. J. Roentgenol. 190, 335–343 (2008)

    Article  Google Scholar 

  31. Daigle, O., Djazovski, O., Laurin, D., Doyon, R. & Artigau, É. Characterization results of EMCCDs for extreme low-light imaging. In Proc. SPIE on ‘High Energy, Optical, and Infrared Detectors for Astronomy V’ Vol. 8453, 845303, https://doi.org/10.1117/12.926385 (2012)

    Google Scholar 

  32. Girard, J. N . et al. Sparse representations and convex optimization as tools for LOFAR radio interferometric imaging. J. Instrum. 10, C08013 (2015)

    Article  Google Scholar 

  33. Lebed, E ., Sarunic, M. V ., Beg, M. F. & Mackenzie, P. J. Rapid volumetric OCT image acquisition using compressive sampling. Opt. Exp. 18, 21003–21012 (2010)

    Article  ADS  Google Scholar 

  34. Fessler, J. A. & Sutton, B. P. Nonuniform fast Fourier transforms using min-max interpolation. IEEE Trans. Signal Process. 51, 560–574 (2003)

    Article  ADS  MathSciNet  Google Scholar 

  35. Kim, D. H., Adalsteinsson, E. & Spielman, D. M. Simple analytic variable density spiral design. Magn. Reson. Med. 50, 214–219 (2003)

    Article  Google Scholar 

  36. Uecker, M., Ong, F., Tamir, J. I. & Bahri, D. Berkeley advanced reconstruction toolbox. Proc. Int. Soc. Magnetic Resonance in Medicine 2486 (2015)

  37. Abadi, M. et al. TensorFlow: large-scale machine learning on heterogeneous distributed systems. Preprint at https://arxiv.org/abs/1603.04467 (2016)

  38. Nair, V . & Hinton, G. E. Rectified linear units improve restricted Boltzmann machines. In Proc. 27th Int. Conf. on ‘Machine Learning’ 807–814 (ACM, 2010)

  39. Makhzani, A. & Frey, B. J. Winner-take-all autoencoders. Adv. Neural Inf. Process. Syst. 28, 2791–2799 (2015)

    Google Scholar 

  40. Reeder, S. B. et al. Practical approaches to the evaluation of signal-to-noise ratio performance with parallel imaging: application with cardiac imaging and a 32-channel cardiac coil. Magn. Reson. Med. 54, 748–754 (2005)

    Article  Google Scholar 

  41. Duyn, J. H., Yang, Y., Frank, J. A. & van der Veen, J. W. Simple correction method for k-space trajectory deviations in MRI. J. Magn. Reson. 132, 150–153 (1998)

    Article  CAS  ADS  Google Scholar 

  42. Pruessmann, K. P., Weiger, M., Scheidegger, M. B. & Boesiger, P. SENSE: sensitivity encoding for fast MRI. Magn. Reson. Med. 42, 952–962 (1999)

    Article  CAS  Google Scholar 

  43. Saad, Y. & Schultz, M. H. GMRES: a generalized minimal residual algorithm for solving nonsymmetric linear systems. SIAM J. Sci. Statist. Comput. 7, 856–869 (1986)

    Article  MathSciNet  Google Scholar 

  44. Comtat, C, et al. OSEM-3D Reconstruction Strategies for the ECAT HRRT. IEEE Symp. Conf. Record Nuclear Science 6, 3492–3496 (2004)

    Google Scholar 

  45. Izquierdo-Garcia, D. et al. An SPM8-based approach for attenuation correction combining segmentation and nonrigid template formation: application to simultaneous PET/MR brain imaging. J. Nucl. Med. 55, 1825–1830 (2014)

    Article  Google Scholar 

  46. Yu, K. & Zhang, T. Improved local coordinate coding using local tangents. In Proc. 27th Int. Conf. on ‘Machine Learning’ 1215–1222 (ACM, 2010)

  47. Anderes, E. & Coram, M. A general spline representation for nonparametric and semiparametric density estimates using diffeomorphisms. Preprint at https://arxiv.org/abs/1205.5314 (2012)

  48. Zhang, M ., Singh, N. & Fletcher, P. T. Bayesian estimation of regularization and atlas building in diffeomorphic image registration. Int. Conf. Inf. Process. Med. Imaging 37–48 (Springer, 2013)

  49. Fishbaugh, J., Prastawa, M., Gerig, G. & Durrleman, S. Geodesic image regression with a sparse parameterization of diffeomorphisms. In 1st Int. Conf. on ‘Geometric Science of Information’ GSI 2013 (eds Nielsen, F. & Barbaresco, F. ) Vol. 8085, 95–102, https://link.springer.com/chapter/10.1007/978-3-642-40020-9_9 (2013)

    MATH  Google Scholar 

  50. Bernstein, A ., Kuleshov, A. & Yanovich, Y. Manifold Learning in Regression Tasks. Statistical Learning and Data Sciences 414–423 (Springer, 2015)

  51. Hornik, K. Approximation capabilities of multilayer feedforward networks. Neural Netw. 4, 251–257 (1991)

    Article  Google Scholar 

  52. Irie, B. & Miyake, S. Capabilities of three-layered perceptrons. In IEEE Int. Conf. on ‘Neural Networks’ Vol. 1, 641–648 (1988)

    Article  Google Scholar 

  53. Cybenko, G. Approximation by superpositions of a sigmoidal function. Math. Contr. Signals Syst. 2, 303–314 (1989)

    Article  MathSciNet  Google Scholar 

  54. Barron, A. R. Approximation and estimation bounds for artificial neural networks. Mach. Learn. 14, 115–133 (1994)

    MATH  Google Scholar 

  55. Mordvintsev, A., Olah, C. & Tyka, M. DeepDream—a code example for visualizing neural networks. https://research.googleblog.com/2015/07/deepdream-code-example-for-visualizing.html (Google Res, 2015)

  56. Addy, N. O., Wu, H. H. & Nishimura, D. G. Simple method for MR gradient system characterization and k-space trajectory estimation. Magn. Reson. Med. 68, 120–129 (2012)

    Article  Google Scholar 

  57. Han, H., Ouriadov, A. V., Fordham, E. & Balcom, B. J. Direct measurement of magnetic field gradient waveforms. Concepts Magn. Reson. 36A, 349–360 (2010)

    Article  Google Scholar 

  58. Goodfellow, I., Pouget-Abadie, J. & Mirza, M. Generative adversarial nets. Adv. Neural Inf. Process. Syst. 2672–2680 (2014)

  59. Haskell, M., Cauley, S. F. & Wald, L. L. TArgeted Motion Estimation and Reduction (TAMER): data consistency based motion mitigation for MRI using a reduced model joint optimization. IEEE Trans. Med. Imaging PP, 99, https://doi.org/10.1109/TMI.2018.2791482 (2018)

    Google Scholar 

  60. Fessler, J. A., Lee, S., Olafsson, V. T., Shi, H. R. & Noll, D. C. Toeplitz-based iterative image reconstruction for MRI with correction for magnetic field inhomogeneity. IEEE Trans. Signal Process. 53, 3393–3402 (2005)

    Article  ADS  MathSciNet  Google Scholar 

  61. Cauley, S. F. et al. Fast reconstruction for multichannel compressed sensing using a hierarchically semiseparable solver. Magn. Reson. Med. 73, 1034–1040 (2015)

    Article  Google Scholar 

  62. Xi, Y., Xia, J., Cauley, S. & Balakrishnan, V. Superfast and stable structured solvers for Toeplitz least squares via randomized sampling. SIAM J. Matrix Anal. Appl. 35, 44–72 (2014)

    Article  MathSciNet  Google Scholar 

  63. Xia, J., Chandrasekaran, S., Gu, M. & Li, X. S. Fast algorithms for hierarchically semiseparable matrices. Numer. Linear Algebra Appl. 17, 953–976 (2010)

    Article  MathSciNet  Google Scholar 

  64. Weller, D. S., Ramani, S. & Fessler, J. A. Augmented Lagrangian with variable splitting for faster non-Cartesian L1-SPIRiT MR image reconstruction. IEEE Trans. Med. Imaging 33, 351–361 (2014)

    Article  Google Scholar 

  65. Zhao, B., Setsompop, K., Ye, H., Cauley, S. F. & Wald, L. L. Maximum likelihood reconstruction for magnetic resonance fingerprinting. IEEE Trans. Med. Imaging 35, 1812–1823 (2016)

    Article  Google Scholar 

  66. Han, S., Mao, H. & Dally, W. J. Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding. Preprint at https://arxiv.org/abs/1510.00149 (2015)

  67. Hu, H., Peng, R., Tai, Y.-W. & Tang, C.-K. Network trimming: a data-driven neuron pruning approach towards efficient deep architectures. Preprint at https://arxiv.org/abs/1607.03250 (2016)

Download references

Acknowledgements

We acknowledge M. Michalski and the computational resources and assistance provided by the Massachusetts General Hospital (MGH) and the Brigham and Women’s Hospital (BWH) Center for Clinical Data Science (CCDS). The CCDS is supported by MGH, BWH, the MGH Department of Radiology, the BWH Department of Radiology, and through industry partnership with NVIDIA. We also acknowledge the Center for Machine Learning at Martinos. We also thank J. Stockmann, J. Polimeni, D. E. J. Waddington and R. L. Walsworth for their comments on this manuscript, and B. Bilgic and C. Liao for their assistance in human subject data acquisition. We acknowledge C. Catana for providing raw PET data and for filtered back projection and OSEM reconstructions. We also thank M. Haskell for providing the MRI motion encoding model. B.Z. was supported by the National Institutes of Health/National Institute of Biomedical Imaging and Bioengineering F32 Fellowship (EB022390). Data were provided in part by the HCP, MGH-USC Consortium (Principal Investigators: Bruce R. Rosen, Arthur W. Toga and Van Wedeen; U01MH093765), which was funded by the NIH Blueprint Initiative for Neuroscience Research grant; the National Institutes of Health grant P41EB015896; and the Instrumentation Grants S10RR023043, 1S10RR023401, 1S10RR019307.

Author information

Authors and Affiliations

Authors

Contributions

B.Z., J.Z.L., S.F.C., B.R.R. and M.S.R. conceptualized the problem and contributed to experimental design. B.Z. developed, implemented and tested the technical framework. J.Z.L. and B.Z. constructed the theoretical description. B.Z., J.Z.L., S.F.C., B.R.R. and M.S.R. wrote the manuscript.

Corresponding author

Correspondence to Matthew S. Rosen.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Additional information

Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data figures and tables

Extended Data Figure 1 Reconstruction performance of AUTOMAP in low-signal-to-noise-ratio regimes.

Reference brain images were encoded into sensor-domain sampling strategies with high levels of additive white Gaussian noise and reconstructed using both AUTOMAP and conventional approaches: ae, spiral k-space encoding compared with conjugate-gradient SENSE reconstruction with NUFFT regridding; fj, Radon projection encoding compared with model-based iterative reconstruction. Image magnitude signal-to-noise ratios (SNRs) and error maps (with root mean squared error calculations) with respect to reference ground truth images are also shown. For each encoding experiment, both error maps are windowed to the same level.

Extended Data Figure 2 Effect of training corpus on image reconstruction.

ac, AUTOMAP was trained using sensor-image pairs of Cartesian Fourier encoded corpora derived from either ImageNet, HCP brain images, or random-valued Gaussian noise without any real-world image structure. Each trained network was then used to reconstruct a noise-corrupted Cartesian k-space brain dataset. The signal-to-noise ratio (SNR) of the reconstructed images is shown. The apparent intensity discontinuity in the region above the eyes is due to the masking process used to de-identify the data in the HCP protocol (see Methods for more details).

Extended Data Figure 3 Training curves of optimizer loss convergence.

Mean squared error (MSE) loss was minimized with stochastic gradient descent using the RMSProp algorithm and plotted here against training epoch count for: a, Cartesian Fourier encoding on IMAGENET corpus; b, spiral Fourier encoding on IMAGENET corpus; and c, Cartesian undersampled Fourier encoding on HCP brain corpus. The validation error tracks the training error without upward divergence, demonstrating a stable training regime with good bias-variance tradeoff.

Extended Data Figure 4 Reconstruction of motion-corrupted MRI.

a, T2-weighted reference image acquired at 3 T with a turbo spin-echo sequence. b, Three-dimensional motion trajectories measured during an Alzheimer’s patient study. c, d, These motion trajectories were used to corrupt the k-space of this reference image, and it was reconstructed without motion compensation using inverse Fourier transform (c) and AUTOMAP (d). Both images show comparable artefact level and structure, demonstrating the stability of AUTOMAP reconstruction in the presence of unanticipated subject motion. A/P refers to anterior and posterior translational motion, L/R refers to left and right translational motion.

Extended Data Figure 5 Reconstruction of PET scanner data.

ad, Human FDG PET sinogram data (a) was reconstructed using (b) filtered back projection (FBP), (c) OP-OSEM and (d) AUTOMAP.

PowerPoint slides

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhu, B., Liu, J., Cauley, S. et al. Image reconstruction by domain-transform manifold learning. Nature 555, 487–492 (2018). https://doi.org/10.1038/nature25988

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nature25988

This article is cited by

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing