Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Learning with known operators reduces maximum error bounds

Nature Machine Intelligence volume 1, pages 373–380 (2019)Cite this article

Subjects

Abstract

We describe an approach for incorporating prior knowledge into machine learning algorithms. We aim at applications in physics and signal processing in which we know that certain operations must be embedded into the algorithm. Any operation that allows computation of a gradient or sub-gradient towards its inputs is suited for our framework. We derive a maximal error bound for deep nets that demonstrates that inclusion of prior knowledge results in its reduction. Furthermore, we show experimentally that known operators reduce the number of free parameters. We apply this approach to various tasks ranging from computed tomography image reconstruction over vessel segmentation to the derivation of previously unknown imaging algorithms. As such, the concept is widely applicable for many researchers in physics, imaging and signal processing. We assume that our analysis will support further investigation of known operators in other fields of physics, imaging and signal processing.

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Rent or buy this article

Get just this article for as long as you need it

$39.95

Learn more

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Schematic of the idea of known operator learning.
Fig. 2: Deep learning computed tomography.
Fig. 3: Improved interpretability in deep networks.
Fig. 4: Architecture of Frangi-Net over eight scales σ.
Fig. 5: Classical analytical rebinning versus derived neural networks.
Fig. 6: Towards operator discovery and sequence analysis.

Data availability

All data in this publication are publicly available. The experiments for deep learning computed tomography use data from the low-dose CT challenge39. The section on learning from heuristic algorithms uses the DRIVE database40. The data for the section on deriving networks is available in a Code Ocean Capsule available at https://doi.org/10.24433/CO.8086142.v241.

Code availability

The code and data for this Article, along with an accompanying computational environment, are available and executable online as a Code Ocean Capsule. Experiments in the section on deep learning computed tomography can be found at https://doi.org/10.24433/CO.2164960.v142. The code on learning vesselness in the section on learning from heuristic algorithms is published at https://doi.org/10.24433/CO.5016803.v241. Code for the section on deriving networks is available at https://doi.org/10.24433/CO.8086142.v243. Code capsules for experiments in the sections on deep learning computed tomography and deriving networks were implemented using the open source framework PYRO-NN44.

References

  1. Niemann, H. Pattern Analysis and Understanding Vol. 4 (Springer Science & Business Media, 2013).

  2. LeCun, Y. & Bengio, Y. in The Handbook of Brain Theory and Neural Networks (ed. Arbib, M. A.) 3361 (MIT Press, 1995).

  3. Krizhevsky, A., Sutskever, I. & Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems 25, 1097–1105 (2012).

    Google Scholar 

  4. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

    Article  Google Scholar 

  5. Dong, C., Loy, C. C., He, K. & Tang, X. Learning a deep convolutional network for image super-resolution. In European Conference on Computer Vision, 184–199 (Springer, 2014).

  6. Xie, J., Xu, L. & Chen, E. Image denoising and inpainting with deep neural networks. Advances in Neural Information Processing Systems 25, 341–349 (2012).

    Google Scholar 

  7. Wang, G., Ye, J. C., Mueller, K. & Fessler, J. A. Image reconstruction is a new frontier of machine learning. IEEE Trans. Med. Imaging 37, 1289–1296 (2018).

    Article  Google Scholar 

  8. Cohen, J. P., Luck, M. & Honari, S. Distribution matching losses can hallucinate features in medical image translation. In Proceedings of Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 (eds Frangi, A. F., Schnabel, J. A., Davatzikos, C., Alberola-López, C. & Fichtinger, G.) 529–536 (Springer, 2018).

  9. Huang, Y. et al. Some investigations on robustness of deep learning in limited angle tomography. In Proceedings of Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 (eds Frangi, A. F., Schnabel, J. A., Davatzikos, C., Alberola-López, C. & Fichtinger, G.) 145–153 (Springer, 2018).

  10. Würfl, T., Ghesu, F. C., Christlein, V. & Maier, A. Deep learning computed tomography. In Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention 432–440 (Springer, 2016).

  11. Fu, W. et al. Frangi-net: a neural network approach to vessel segmentation. In Bildverarbeitung für die Medizin 2018 (ed. Maier, A. et al.) 341–346 (Springer, 2018).

  12. Maier, A. et al. Precision learning: towards use of known operators in neural networks. In Proceedings of 2018 24th International Conference on Pattern Recognition (ICPR) (ed. Tan, J. K. T.) 183–188 (IEEE, 2018).

  13. Syben, C. et al. Deriving neural network architectures using precision learning: parallel-to-fan beam conversion. In Proceedings of German Conference on Pattern Recognition (GCPR) 503–517 (Springer, 2018).

  14. Cybenko, G. Approximation by superpositions of a sigmoidal function. Math. Control Signals Syst. 2, 303–314 (1989).

    Article  MathSciNet  Google Scholar 

  15. Hornik, K. Approximation capabilities of multilayer feedforward networks. Neural Netw. 4, 251–257 (1991).

    Article  Google Scholar 

  16. Maier, A., Syben, C., Lasser, T. & Riess, C. A gentle introduction to deep learning in medical image processing. Z. Med. Phys. 29, 86–101 (2019).

    Article  Google Scholar 

  17. Parker, D. L. Optimal short scan convolution reconstruction for fan beam CT. Med. Phys. 9, 254–257 (1982).

    Article  Google Scholar 

  18. Schäfer, D., van de Haar, P. & Grass, M. Modified parker weights for super short scan cone beam CT. In Proceedings of 14th International Meeting on Fully Three-Dimensional Image Reconstruction in Radiology and Nuclear Medicine 49–52 (Xi’an Jiaotong University, 2017).

  19. Frangi, A. F., Niessen, W. J., Vincken, K. L. & Viergever, M. A. Multiscale vessel enhancement filtering. In Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention 130–137 (Springer, 1998).

  20. Silver, D. et al. Mastering the game of go with deep neural networks and tree search. Nature 529, 484–489 (2016).

    Article  Google Scholar 

  21. Zhu, B., Liu, J. Z., Cauley, S. F., Rosen, B. R. & Rosen, M. S. Image reconstruction by domain-transform manifold learning. Nature 555, 487 (2018).

    Article  Google Scholar 

  22. Fursattel, P., Plank, C., Maier, A. & Riess, C. Accurate laser scanner to camera calibration with application to range sensor evaluation. IPSJ Trans. Comput. Vis. Appl. 9, 21 (2017).

    Article  Google Scholar 

  23. Köhler, T. et al. Robust multiframe super-resolution employing iteratively re-weighted minimization. IEEE Trans. Comput. Imaging 2, 42–58 (2016).

    Article  MathSciNet  Google Scholar 

  24. Aubreville, M. et al. Deep denoising for hearing aid applications. In Proceedings of 16th International Workshop on Acoustic Signal Enhancement (IWAENC) 361–365 (IEEE, 2018).

  25. Jaderberg, M., Simonyan, K., Zisserman, A. & Kavukcuoglu, K. Spatial transformer networks. Advances in Neural Information Processing Systems 28, 2017–2025 (2015).

    Google Scholar 

  26. Lin, C.-H., Yumer, E., Wang, O., Shechtman, E. & Lucey, S. St-gan: spatial transformer generative adversarial networks for image compositing. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 9455–9464 (IEEE, 2018).

  27. Kulkarni, T. D., Whitney, W. F., Kohli, P. & Tenenbaum, J. Deep convolutional inverse graphics network. Advances in Neural Information Processing Systems 28, 2539–2547 (2015).

    Google Scholar 

  28. Zhu, J.-Y. et al. Visual object networks: image generation with disentangled 3D representations. Advances in Neural Information Processing Systems 31, 118–129 (2018).

    Google Scholar 

  29. Tewari, A. et al. Mofa: model-based deep convolutional face autoencoder for unsupervised monocular reconstruction. In Proceedings of IEEE International Conference on Computer Vision (ICCV) 1274–1283 (2017).

  30. Adler, J. & Öktem, O. Solving ill-posed inverse problems using iterative deep neural networks. Inverse Probl. 33, 124007 (2017).

    Article  MathSciNet  Google Scholar 

  31. Ye, J. C., Han, Y. & Cha, E. Deep convolutional framelets: a general deep learning framework for inverse problems. SIAM J. Imaging Sci. 11, 991–1048 (2018).

    Article  MathSciNet  Google Scholar 

  32. Hammernik, K. et al. Learning a variational network for reconstruction of accelerated MRI data. Magn. Reson Med. 79, 3055–3071 (2018).

    Article  Google Scholar 

  33. Wu, H., Zheng, S., Zhang, J. & Huang, K. Fast end-to-end trainable guided filter. Preprint at http://arxiv.org/abs/1803.05619 (2018).

  34. Rockafellar, R. Convex Analysis (Princeton Landmarks in Mathematics and Physics, Princeton Univ. Press, 1970); https://books.google.de/books?id=1TiOka9bx3sC

  35. Zarei, S., Stimpel, B., Syben, C. & Maier, A. User Loss A. Forced-choice-inspired approach to train neural networks directly by user interaction. In Bildverarbeitung für die Medizin 2019, 92–97 (Informatik Aktuell, 2019); https://www5.informatik.uni-erlangen.de/Forschung/Publikationen/2019/Zarei19-ULA.pdf

  36. Barron, A. R. Approximation and estimation bounds for artificial neural networks. Mach. Learn. 14, 115–133 (1994).

    MATH  Google Scholar 

  37. Duda, R. O., Hart, P. E. & Stork, D. G. Pattern Classification (Wiley, 2012).

  38. Szegedy, C. et al. Going deeper with convolutions. In Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 1–9 (IEEE, 2015).

  39. McCollough, C. TU-FG-207A-04: overview of the low dose CT grand challenge. Med. Phys. 43, 3759–3760 (2016).

    Google Scholar 

  40. Staal, J., Abràmoff, M. D., Niemeijer, M., Viergever, M. A. & Van Ginneken, B. Ridge-based vessel segmentation in color images of the retina. IEEE Trans. Med. Imaging 23, 501–509 (2004).

    Article  Google Scholar 

  41. Fu, W. Frangi-Net on high-resolution fundus (HRF) image database. Code Ocean https://doi.org/10.24433/CO.5016803.v2 (2019).

    Article  Google Scholar 

  42. Syben, C. & Hoffmann, M. Learning CT reconstruction. Code Ocean https://doi.org/10.24433/CO.2164960.v1 (2019).

    Article  Google Scholar 

  43. Syben, C. Deriving neural networks. Code Ocean https://doi.org/10.24433/CO.8086142.v2 (2019).

    Article  Google Scholar 

  44. Syben, C. et al. PYRO-NN: Python reconstruction operators in neural networks. Preprint at https://arxiv.org/abs/1904.13342 (2019).

Download references

Acknowledgements

The research leading to these results has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (ERC grant no. 810316).

Author information

Authors and Affiliations

  1. Department of Computer Science, Friedrich-Alexander University Erlangen-Nürnberg, Erlangen, Germany

    Andreas K. Maier, Christopher Syben, Bernhard Stimpel, Tobias Würfl, Mathis Hoffmann, Frank Schebesch, Weilin Fu & Leonid Mill

  2. Helmholtz Zentrum Berlin für Materialien und Energie, Berlin, Germany

    Lasse Kling & Silke Christiansen

  3. Physics Department, Free University Berlin, Berlin, Germany

    Silke Christiansen

Authors
  1. Andreas K. Maier
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christopher Syben
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bernhard Stimpel
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tobias Würfl
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mathis Hoffmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Frank Schebesch
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Weilin Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Leonid Mill
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Lasse Kling
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Silke Christiansen
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

A.K.M. is the main author of the paper and is responsible for writing the manuscript, theoretical analysis and experimental design. C.S. and B.S. contributed to writing the section about deriving networks as well as its supporting experiments. T.W. and M.H. helped with writing the section on deep learning computed tomography and performed the experiments reported in that section. F.S. contributed to the mathematical analysis and the writing thereof. W.F. conducted the experiments supporting the section on learning from heuristic algorithms and contributed to their description. L.M., L.K. and S.C. contributed to the experimental design and the writing of the manuscript.

Corresponding author

Correspondence to Andreas K. Maier.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Proofs and additional experimental validation

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Maier, A.K., Syben, C., Stimpel, B. et al. Learning with known operators reduces maximum error bounds. Nat Mach Intell 1, 373–380 (2019). https://doi.org/10.1038/s42256-019-0077-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-019-0077-5

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing