Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Towards a topological–geometrical theory of group equivariant non-expansive operators for data analysis and machine learning

Nature Machine Intelligence volume 1, pages 423–433 (2019)Cite this article

Subjects

Abstract

We provide a general mathematical framework for group and set equivariance in machine learning. We define group equivariant non-expansive operators (GENEOs) as maps between function spaces associated with groups of transformations. We study the topological and metric properties of the space of GENEOs to evaluate their approximating power and set the basis for general strategies to initialize and compose operators. We define suitable pseudo-metrics for the function spaces, the equivariance groups and the set of non-expansive operators. We prove that, under suitable assumptions, the space of GENEOs is compact and convex. These results provide fundamental guarantees in a machine learning perspective. By considering isometry-equivariant non-expansive operators, we describe a simple strategy to select and sample operators. Thereafter, we show how selected and sampled operators can be used both to perform classical metric learning and to inject knowledge in artificial neural networks.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Isometry equivariant non-expansive operators on MNIST.
Fig. 2: Persistent homology background.
Fig. 3: Experimental pipeline.
Fig. 4: IENEO-based metric learning on MNIST.
Fig. 5: IENEO-based metric learning on fashion-MNIST and CIFAR-10.
Fig. 6: Metric learning on transformed MNIST validation samples.
Fig. 7: Convolutional neural network architecture used in the knowledge injection experiment.

Similar content being viewed by others

Code availability

The experiments and applications can be reproduced by installing the open-source Python package, available at https://doi.org/10.5281/zenodo.3264851.

Data availability

No datasets were generated during the current study. The datasets analysed are available at

MNIST26, http://yann.lecun.com/exdb/mnist/

Fashion-MNIST27, https://github.com/zalandoresearch/fashion-mnist

CIFAR-1028, https://www.cs.toronto.edu/kriz/cifar.html.

References

  1. Silver, D. et al. Mastering the game of go without human knowledge. Nature 550, 354–359 (2017).

    Article  Google Scholar 

  2. Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017).

    Article  Google Scholar 

  3. LeCun, Y. et al. in The Handbook of Brain Theory and Neural Networks (ed. Arbib, M. A.) 3361 (MIT Press, 1995).

  4. Rawat, W. & Wang, Z. Deep convolutional neural networks for image classification: a comprehensive review. Neural Comput. 29, 2352–2449 (2017).

    Article  MathSciNet  Google Scholar 

  5. Kim, Y. Convolutional neural networks for sentence classification. Preprint at https://arxiv.org/abs/1408.5882 (2014).

  6. Niyogi, P. Manifold regularization and semi-supervised learning: some theoretical analyses. J. Mach. Learn. Res. 14, 1229–1250 (2013).

    MathSciNet  MATH  Google Scholar 

  7. Anselmi, F., Rosasco, L. & Poggio, T. On invariance and selectivity in representation learning. Inf. Inference 5, 134–158 (2016).

    Article  MathSciNet  Google Scholar 

  8. Cohen, T. & Welling, M. Group equivariant convolutional networks. Proc. 33rd Int. Conf. Mach. Learning 48, 2990–2999 (2016).

    Google Scholar 

  9. Worrall, D. E., Garbin, S. J., Turmukhambetov, D. & Brostow, G. J. Harmonic networks: deep translation and rotation equivariance. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 2 5028–5037 (IEEE, 2017).

  10. Bengio, Y., Courville, A. & Vincent, P. Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35, 1798–1828 (2013).

    Article  Google Scholar 

  11. Tishby, N. & Zaslavsky, N. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), 1–5 (IEEE, 2015); https://doi.org/10.1109/ITW.2015.7133169

  12. Mallat, S. Group invariant scattering. Commun. Pure Appl. Math. 65, 1331–1398 (2012).

    Article  MathSciNet  Google Scholar 

  13. Mallat, S. Understanding deep convolutional networks. Philos. Trans. A Math. Phys. Eng. Sci. 374, 20150203 (2016).

    Article  Google Scholar 

  14. Zhang, C., Voinea, S., Evangelopoulos, G., Rosasco, L. & Poggio, T. Discriminative Template Learning in Group-Convolutional Networks for Invariant Speech Representations, 3229–3233 (International Speech and Communication Association, 2015).

  15. Anselmi, F., Evangelopoulos, G., Rosasco, L. & Poggio, T. Symmetry-adapted representation learning. Pattern Recognit. 86, 201–208 (2019).

    Article  Google Scholar 

  16. Frosini, P. & Jabłoński, G. Combining persistent homology and invariance groups for shape comparison. Discrete Comput. Geom. 55, 373–409 (2016).

    Article  MathSciNet  Google Scholar 

  17. Frosini, P. & Quercioli, N. in Proceedings of the International Cross-Domain Conference CD-MAKE 2017, Vol. 10410 (eds Holzinger, A., Kieseberg, P., Tjoa, A. & Weippl, E.) 14–24 (Springer, 2017).

  18. Camporesi, F., Frosini, P. & Quercioli, N. in Proceedings of the International Cross-Domain Conference CD-MAKE 2018, Vol. 11015 (eds Holzinger, A., Kieseberg, P., Tjoa, A. & Weippl, E.) 265–272 (Springer, 2018).

  19. Smale, S., Rosasco, L., Bouvrie, J., Caponnetto, A. & Poggio, T. Mathematics of the neural response. Found. Comput. Math. 10, 67–91 (2010).

    Article  MathSciNet  Google Scholar 

  20. Carlsson, G. Topology and data. Bull. Am. Math. Soc. 46, 255–308 (2009).

    Article  MathSciNet  Google Scholar 

  21. Lum, P. Y. et al. Extracting insights from the shape of complex data using topology. Sci. Rep. 3, 1236 (2013).

    Article  Google Scholar 

  22. Adams, H. et al. Persistence images: a stable vector representation of persistent homology. J. Mach. Learn. Res. 18, 218–252 (2017).

    MathSciNet  Google Scholar 

  23. Pun, C. S., Xia, K. & Lee, S. X. Persistent-homology-based machine learning and its applications – a survey. Preprint at https://arxiv.org/abs/1811.00252 (2018).

  24. Gabrielsson, R. B. & Carlsson, G. Exposition and interpretation of the topology of neural networks. Preprint at http://arxiv.org/abs/1810.03234 (2018)

  25. Carlsson, G. & Gabrielsson, R. B. Topological approaches to deep learning. Preprint at https://arxiv.org/abs/1811.01122 (2018).

  26. Deng, L. The MNIST database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Process. Mag. 29, 141–142 (2012).

    Article  Google Scholar 

  27. Xiao, H., Rasul, K. & Vollgraf, R. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. Preprint at https://arxiv.org/abs/1708.07747 (2017).

  28. Krizhevsky, A., Nair, V. & Hinton, G. The CIFAR-10 Dataset https://www.cs.toronto.edu/~kriz/cifar.html (2019).

  29. Langfelder, P., Zhang, B. & Horvath, S. Defining clusters from a hierarchical cluster tree: the dynamic tree cut package for R. Bioinformatics 24, 719–720 (2007).

    Article  Google Scholar 

  30. Glorot, X. & Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. Proc. 13th Int. Conf. Art. Intell. Statist. 9, 249–256 (2010).

    Google Scholar 

  31. Hinton, G. E., Krizhevsky, A. & Wang, S. D. Transforming auto-encoders. In Proceedings of International Conference on Artificial Neural Networks 44–51 (Springer, 2011).

  32. Sabour, S., Frosst, N. & Hinton, G. E. in Advances in Neural Information Processing Systems Vol. 30 (eds Guyon, I. et al.) 3856–3866 (Curran Associates, 2017).

  33. Frosini, P. Towards an observer-oriented theory of shape comparison. In Eurographics Workshop on 3D Object Retrieval (eds Ferreira, A., Giachetti, A. & Giorgi, D.) 5–8 (The Eurographics Association, 2016).

  34. Carr, J. C et al. Reconstruction and representation of 3D objects with radial basis functions. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’01. 67–76 (ACM, 2001).

  35. Atzmon, M., Maron, H. & Lipman, Y. Point convolutional neural networks by extension operators. ACM Trans. Graph. 37, 71:1–71:12 (2018).

    Article  Google Scholar 

  36. Collins, A., Zomorodian, A., Carlsson, G. & Guibas, L. A barcode shape descriptor for curve point cloud data. In Proceedings of SPBG’04 Symposium on Point-Based Graphics 2004 (eds Gross, M., Pfister, H., Alexa, M. & Rusinkiewicz, S.) 181–191 (The Eurographics Association, 2004).

  37. Di Fabio, B. & Frosini, P. Filtrations induced by continuous functions. Topol. Appl. 160, 1413–1422 (2013).

    Article  MathSciNet  Google Scholar 

  38. Hatcher, A. Algebraic Ttopology. (Cambridge Univ.ersity Press: Cambridge, 2002.).

  39. Cerri, A., Fabio, B. D., Ferri, M., Frosini, P. & Landi, C. Betti numbers in multidimensional persistent homology are stable functions. Math. Methods Appl. Sci. 36, 1543–1557 (2013).

    Article  MathSciNet  Google Scholar 

  40. Biasotti, S. et al. Describing shapes by geometrical-topological properties of real functions. ACM Comput. Surv. 40, 12:1–12:87 (2008).

    Article  Google Scholar 

  41. Carlsson, G. & Zomorodian, A. The theory of multidimensional persistence. Discrete Comput. Geom. 42, 71–93 (2009).

    Article  MathSciNet  Google Scholar 

  42. Edelsbrunner, H. & Harer, J. Persistent homology—a survey. Contemp. Math. 453, 257–282 (2008).

    Article  MathSciNet  Google Scholar 

  43. Zomorodian, A. Fast construction of the Vietoris–Rips complex. Comput. Graph. 34, 263–271 (2010).

    Article  Google Scholar 

  44. Fabbri, R., Costa, L. D. F., Torelli, J. C. & Bruno, O. M. 2D Euclidean distance transform algorithms: a comparative survey. ACM Comput. Surv. 40, 2 (2008).

    Article  Google Scholar 

  45. Krizhevsky, A. & Hinton, G. Learning Multiple Layers of Features from Tiny Images (University of Toronto, 2009).

Download references

Acknowledgements

The authors thank H. Young for proofreading the manuscript. The research carried out by M.G.B. was supported by the European Research Council (Advanced Investigator Grant 671251 to Z.F. Mainen), the Champalimaud Foundation (Z.F. Mainen) and a GPU NVIDIA grant. The research carried out by P.F. and N.Q. was partially supported by GNSAGA-INdAM (Italy).

Author information

Authors and Affiliations

  1. Champalimaud Research, Champalimaud Center for the Unknown, Lisbon, Portugal

    Mattia G. Bergomi

  2. Department of Mathematics, University of Bologna, Bologna, Italy

    Patrizio Frosini & Nicola Quercioli

  3. Advanced Research Center on Electronic Systems ‘Ercole De Castro’, University of Bologna, Bologna, Italy

    Patrizio Frosini & Nicola Quercioli

  4. Italian National Research Council, Institute of Information Science and Technologies ‘Alessandro Faedo’, Pisa, Italy

    Daniela Giorgi

Authors
  1. Mattia G. Bergomi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Patrizio Frosini
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Daniela Giorgi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nicola Quercioli
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

P.F., M.G.B., D.G. and N.Q. devised the project and algorithms. P.F. and N.Q. formalized the mathematical model with help from M.G.B. and D.G. M.G.B. wrote the code with help from D.G. M.G.B. and D.G. analysed the data. P.F. supervised the project. M.G.B., P.F., D.G. and N.Q. contributed equally to the writing of the main text and methods.

Corresponding author

Correspondence to Patrizio Frosini.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary methods

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bergomi, M.G., Frosini, P., Giorgi, D. et al. Towards a topological–geometrical theory of group equivariant non-expansive operators for data analysis and machine learning. Nat Mach Intell 1, 423–433 (2019). https://doi.org/10.1038/s42256-019-0087-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-019-0087-3

This article is cited by

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics