We describe an approach for incorporating prior knowledge into machine learning algorithms. We aim at applications in physics and signal processing in which we know that certain operations must be embedded into the algorithm. Any operation that allows computation of a gradient or sub-gradient towards its inputs is suited for our framework. We derive a maximal error bound for deep nets that demonstrates that inclusion of prior knowledge results in its reduction. Furthermore, we show experimentally that known operators reduce the number of free parameters. We apply this approach to various tasks ranging from computed tomography image reconstruction over vessel segmentation to the derivation of previously unknown imaging algorithms. As such, the concept is widely applicable for many researchers in physics, imaging and signal processing. We assume that our analysis will support further investigation of known operators in other fields of physics, imaging and signal processing.
Access optionsAccess options
Subscribe to Journal
Get full journal access for 1 year
only $8.67 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
All data in this publication are publicly available. The experiments for deep learning computed tomography use data from the low-dose CT challenge39. The section on learning from heuristic algorithms uses the DRIVE database40. The data for the section on deriving networks is available in a Code Ocean Capsule available at https://doi.org/10.24433/CO.8086142.v241.
The code and data for this Article, along with an accompanying computational environment, are available and executable online as a Code Ocean Capsule. Experiments in the section on deep learning computed tomography can be found at https://doi.org/10.24433/CO.2164960.v142. The code on learning vesselness in the section on learning from heuristic algorithms is published at https://doi.org/10.24433/CO.5016803.v241. Code for the section on deriving networks is available at https://doi.org/10.24433/CO.8086142.v243. Code capsules for experiments in the sections on deep learning computed tomography and deriving networks were implemented using the open source framework PYRO-NN44.
Niemann, H. Pattern Analysis and Understanding Vol. 4 (Springer Science & Business Media, 2013).
LeCun, Y. & Bengio, Y. in The Handbook of Brain Theory and Neural Networks (ed. Arbib, M. A.) 3361 (MIT Press, 1995).
Krizhevsky, A., Sutskever, I. & Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems 25, 1097–1105 (2012).
LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).
Dong, C., Loy, C. C., He, K. & Tang, X. Learning a deep convolutional network for image super-resolution. In European Conference on Computer Vision, 184–199 (Springer, 2014).
Xie, J., Xu, L. & Chen, E. Image denoising and inpainting with deep neural networks. Advances in Neural Information Processing Systems 25, 341–349 (2012).
Wang, G., Ye, J. C., Mueller, K. & Fessler, J. A. Image reconstruction is a new frontier of machine learning. IEEE Trans. Med. Imaging 37, 1289–1296 (2018).
Cohen, J. P., Luck, M. & Honari, S. Distribution matching losses can hallucinate features in medical image translation. In Proceedings of Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 (eds Frangi, A. F., Schnabel, J. A., Davatzikos, C., Alberola-López, C. & Fichtinger, G.) 529–536 (Springer, 2018).
Huang, Y. et al. Some investigations on robustness of deep learning in limited angle tomography. In Proceedings of Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 (eds Frangi, A. F., Schnabel, J. A., Davatzikos, C., Alberola-López, C. & Fichtinger, G.) 145–153 (Springer, 2018).
Würfl, T., Ghesu, F. C., Christlein, V. & Maier, A. Deep learning computed tomography. In Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention 432–440 (Springer, 2016).
Fu, W. et al. Frangi-net: a neural network approach to vessel segmentation. In Bildverarbeitung für die Medizin 2018 (ed. Maier, A. et al.) 341–346 (Springer, 2018).
Maier, A. et al. Precision learning: towards use of known operators in neural networks. In Proceedings of 2018 24th International Conference on Pattern Recognition (ICPR) (ed. Tan, J. K. T.) 183–188 (IEEE, 2018).
Syben, C. et al. Deriving neural network architectures using precision learning: parallel-to-fan beam conversion. In Proceedings of German Conference on Pattern Recognition (GCPR) 503–517 (Springer, 2018).
Cybenko, G. Approximation by superpositions of a sigmoidal function. Math. Control Signals Syst. 2, 303–314 (1989).
Hornik, K. Approximation capabilities of multilayer feedforward networks. Neural Netw. 4, 251–257 (1991).
Maier, A., Syben, C., Lasser, T. & Riess, C. A gentle introduction to deep learning in medical image processing. Z. Med. Phys. 29, 86–101 (2019).
Parker, D. L. Optimal short scan convolution reconstruction for fan beam CT. Med. Phys. 9, 254–257 (1982).
Schäfer, D., van de Haar, P. & Grass, M. Modified parker weights for super short scan cone beam CT. In Proceedings of 14th International Meeting on Fully Three-Dimensional Image Reconstruction in Radiology and Nuclear Medicine 49–52 (Xi’an Jiaotong University, 2017).
Frangi, A. F., Niessen, W. J., Vincken, K. L. & Viergever, M. A. Multiscale vessel enhancement filtering. In Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention 130–137 (Springer, 1998).
Silver, D. et al. Mastering the game of go with deep neural networks and tree search. Nature 529, 484–489 (2016).
Zhu, B., Liu, J. Z., Cauley, S. F., Rosen, B. R. & Rosen, M. S. Image reconstruction by domain-transform manifold learning. Nature 555, 487 (2018).
Fursattel, P., Plank, C., Maier, A. & Riess, C. Accurate laser scanner to camera calibration with application to range sensor evaluation. IPSJ Trans. Comput. Vis. Appl. 9, 21 (2017).
Köhler, T. et al. Robust multiframe super-resolution employing iteratively re-weighted minimization. IEEE Trans. Comput. Imaging 2, 42–58 (2016).
Aubreville, M. et al. Deep denoising for hearing aid applications. In Proceedings of 16th International Workshop on Acoustic Signal Enhancement (IWAENC) 361–365 (IEEE, 2018).
Jaderberg, M., Simonyan, K., Zisserman, A. & Kavukcuoglu, K. Spatial transformer networks. Advances in Neural Information Processing Systems 28, 2017–2025 (2015).
Lin, C.-H., Yumer, E., Wang, O., Shechtman, E. & Lucey, S. St-gan: spatial transformer generative adversarial networks for image compositing. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 9455–9464 (IEEE, 2018).
Kulkarni, T. D., Whitney, W. F., Kohli, P. & Tenenbaum, J. Deep convolutional inverse graphics network. Advances in Neural Information Processing Systems 28, 2539–2547 (2015).
Zhu, J.-Y. et al. Visual object networks: image generation with disentangled 3D representations. Advances in Neural Information Processing Systems 31, 118–129 (2018).
Tewari, A. et al. Mofa: model-based deep convolutional face autoencoder for unsupervised monocular reconstruction. In Proceedings of IEEE International Conference on Computer Vision (ICCV) 1274–1283 (2017).
Adler, J. & Öktem, O. Solving ill-posed inverse problems using iterative deep neural networks. Inverse Probl. 33, 124007 (2017).
Ye, J. C., Han, Y. & Cha, E. Deep convolutional framelets: a general deep learning framework for inverse problems. SIAM J. Imaging Sci. 11, 991–1048 (2018).
Hammernik, K. et al. Learning a variational network for reconstruction of accelerated MRI data. Magn. Reson Med. 79, 3055–3071 (2018).
Wu, H., Zheng, S., Zhang, J. & Huang, K. Fast end-to-end trainable guided filter. Preprint at http://arxiv.org/abs/1803.05619 (2018).
Rockafellar, R. Convex Analysis (Princeton Landmarks in Mathematics and Physics, Princeton Univ. Press, 1970); https://books.google.de/books?id=1TiOka9bx3sC
Zarei, S., Stimpel, B., Syben, C. & Maier, A. User Loss A. Forced-choice-inspired approach to train neural networks directly by user interaction. In Bildverarbeitung für die Medizin 2019, 92–97 (Informatik Aktuell, 2019); https://www5.informatik.uni-erlangen.de/Forschung/Publikationen/2019/Zarei19-ULA.pdf
Barron, A. R. Approximation and estimation bounds for artificial neural networks. Mach. Learn. 14, 115–133 (1994).
Duda, R. O., Hart, P. E. & Stork, D. G. Pattern Classification (Wiley, 2012).
Szegedy, C. et al. Going deeper with convolutions. In Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 1–9 (IEEE, 2015).
McCollough, C. TU-FG-207A-04: overview of the low dose CT grand challenge. Med. Phys. 43, 3759–3760 (2016).
Staal, J., Abràmoff, M. D., Niemeijer, M., Viergever, M. A. & Van Ginneken, B. Ridge-based vessel segmentation in color images of the retina. IEEE Trans. Med. Imaging 23, 501–509 (2004).
Fu, W. Frangi-Net on high-resolution fundus (HRF) image database. Code Ocean https://doi.org/10.24433/CO.5016803.v2 (2019).
Syben, C. & Hoffmann, M. Learning CT reconstruction. Code Ocean https://doi.org/10.24433/CO.2164960.v1 (2019).
Syben, C. Deriving neural networks. Code Ocean https://doi.org/10.24433/CO.8086142.v2 (2019).
Syben, C. et al. PYRO-NN: Python reconstruction operators in neural networks. Preprint at https://arxiv.org/abs/1904.13342 (2019).
The research leading to these results has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (ERC grant no. 810316).
The authors declare no competing interests.
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Proofs and additional experimental validation