Towards a topological–geometrical theory of group equivariant non-expansive operators for data analysis and machine learning

Abstract

We provide a general mathematical framework for group and set equivariance in machine learning. We define group equivariant non-expansive operators (GENEOs) as maps between function spaces associated with groups of transformations. We study the topological and metric properties of the space of GENEOs to evaluate their approximating power and set the basis for general strategies to initialize and compose operators. We define suitable pseudo-metrics for the function spaces, the equivariance groups and the set of non-expansive operators. We prove that, under suitable assumptions, the space of GENEOs is compact and convex. These results provide fundamental guarantees in a machine learning perspective. By considering isometry-equivariant non-expansive operators, we describe a simple strategy to select and sample operators. Thereafter, we show how selected and sampled operators can be used both to perform classical metric learning and to inject knowledge in artificial neural networks.

Access optionsAccess options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: Isometry equivariant non-expansive operators on MNIST.
Fig. 2: Persistent homology background.
Fig. 3: Experimental pipeline.
Fig. 4: IENEO-based metric learning on MNIST.
Fig. 5: IENEO-based metric learning on fashion-MNIST and CIFAR-10.
Fig. 6: Metric learning on transformed MNIST validation samples.
Fig. 7: Convolutional neural network architecture used in the knowledge injection experiment.

Code availability

The experiments and applications can be reproduced by installing the open-source Python package, available at https://doi.org/10.5281/zenodo.3264851.

Data availability

No datasets were generated during the current study. The datasets analysed are available at

MNIST26, http://yann.lecun.com/exdb/mnist/

Fashion-MNIST27, https://github.com/zalandoresearch/fashion-mnist

CIFAR-1028, https://www.cs.toronto.edu/kriz/cifar.html.

References

  1. 1.

    Silver, D. et al. Mastering the game of go without human knowledge. Nature 550, 354–359 (2017).

  2. 2.

    Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017).

  3. 3.

    LeCun, Y. et al. in The Handbook of Brain Theory and Neural Networks (ed. Arbib, M. A.) 3361 (MIT Press, 1995).

  4. 4.

    Rawat, W. & Wang, Z. Deep convolutional neural networks for image classification: a comprehensive review. Neural Comput. 29, 2352–2449 (2017).

  5. 5.

    Kim, Y. Convolutional neural networks for sentence classification. Preprint at https://arxiv.org/abs/1408.5882 (2014).

  6. 6.

    Niyogi, P. Manifold regularization and semi-supervised learning: some theoretical analyses. J. Mach. Learn. Res. 14, 1229–1250 (2013).

  7. 7.

    Anselmi, F., Rosasco, L. & Poggio, T. On invariance and selectivity in representation learning. Inf. Inference 5, 134–158 (2016).

  8. 8.

    Cohen, T. & Welling, M. Group equivariant convolutional networks. Proc. 33rd Int. Conf. Mach. Learning 48, 2990–2999 (2016).

  9. 9.

    Worrall, D. E., Garbin, S. J., Turmukhambetov, D. & Brostow, G. J. Harmonic networks: deep translation and rotation equivariance. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 2 5028–5037 (IEEE, 2017).

  10. 10.

    Bengio, Y., Courville, A. & Vincent, P. Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35, 1798–1828 (2013).

  11. 11.

    Tishby, N. & Zaslavsky, N. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), 1–5 (IEEE, 2015); https://doi.org/10.1109/ITW.2015.7133169

  12. 12.

    Mallat, S. Group invariant scattering. Commun. Pure Appl. Math. 65, 1331–1398 (2012).

  13. 13.

    Mallat, S. Understanding deep convolutional networks. Philos. Trans. A Math. Phys. Eng. Sci. 374, 20150203 (2016).

  14. 14.

    Zhang, C., Voinea, S., Evangelopoulos, G., Rosasco, L. & Poggio, T. Discriminative Template Learning in Group-Convolutional Networks for Invariant Speech Representations, 3229–3233 (International Speech and Communication Association, 2015).

  15. 15.

    Anselmi, F., Evangelopoulos, G., Rosasco, L. & Poggio, T. Symmetry-adapted representation learning. Pattern Recognit. 86, 201–208 (2019).

  16. 16.

    Frosini, P. & Jabłoński, G. Combining persistent homology and invariance groups for shape comparison. Discrete Comput. Geom. 55, 373–409 (2016).

  17. 17.

    Frosini, P. & Quercioli, N. in Proceedings of the International Cross-Domain Conference CD-MAKE 2017, Vol. 10410 (eds Holzinger, A., Kieseberg, P., Tjoa, A. & Weippl, E.) 14–24 (Springer, 2017).

  18. 18.

    Camporesi, F., Frosini, P. & Quercioli, N. in Proceedings of the International Cross-Domain Conference CD-MAKE 2018, Vol. 11015 (eds Holzinger, A., Kieseberg, P., Tjoa, A. & Weippl, E.) 265–272 (Springer, 2018).

  19. 19.

    Smale, S., Rosasco, L., Bouvrie, J., Caponnetto, A. & Poggio, T. Mathematics of the neural response. Found. Comput. Math. 10, 67–91 (2010).

  20. 20.

    Carlsson, G. Topology and data. Bull. Am. Math. Soc. 46, 255–308 (2009).

  21. 21.

    Lum, P. Y. et al. Extracting insights from the shape of complex data using topology. Sci. Rep. 3, 1236 (2013).

  22. 22.

    Adams, H. et al. Persistence images: a stable vector representation of persistent homology. J. Mach. Learn. Res. 18, 218–252 (2017).

  23. 23.

    Pun, C. S., Xia, K. & Lee, S. X. Persistent-homology-based machine learning and its applications – a survey. Preprint at https://arxiv.org/abs/1811.00252 (2018).

  24. 24.

    Gabrielsson, R. B. & Carlsson, G. Exposition and interpretation of the topology of neural networks. Preprint at http://arxiv.org/abs/1810.03234 (2018)

  25. 25.

    Carlsson, G. & Gabrielsson, R. B. Topological approaches to deep learning. Preprint at https://arxiv.org/abs/1811.01122 (2018).

  26. 26.

    Deng, L. The MNIST database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Process. Mag. 29, 141–142 (2012).

  27. 27.

    Xiao, H., Rasul, K. & Vollgraf, R. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. Preprint at https://arxiv.org/abs/1708.07747 (2017).

  28. 28.

    Krizhevsky, A., Nair, V. & Hinton, G. The CIFAR-10 Dataset https://www.cs.toronto.edu/~kriz/cifar.html (2019).

  29. 29.

    Langfelder, P., Zhang, B. & Horvath, S. Defining clusters from a hierarchical cluster tree: the dynamic tree cut package for R. Bioinformatics 24, 719–720 (2007).

  30. 30.

    Glorot, X. & Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. Proc. 13th Int. Conf. Art. Intell. Statist. 9, 249–256 (2010).

  31. 31.

    Hinton, G. E., Krizhevsky, A. & Wang, S. D. Transforming auto-encoders. In Proceedings of International Conference on Artificial Neural Networks 44–51 (Springer, 2011).

  32. 32.

    Sabour, S., Frosst, N. & Hinton, G. E. in Advances in Neural Information Processing Systems Vol. 30 (eds Guyon, I. et al.) 3856–3866 (Curran Associates, 2017).

  33. 33.

    Frosini, P. Towards an observer-oriented theory of shape comparison. In Eurographics Workshop on 3D Object Retrieval (eds Ferreira, A., Giachetti, A. & Giorgi, D.) 5–8 (The Eurographics Association, 2016).

  34. 34.

    Carr, J. C et al. Reconstruction and representation of 3D objects with radial basis functions. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’01. 67–76 (ACM, 2001).

  35. 35.

    Atzmon, M., Maron, H. & Lipman, Y. Point convolutional neural networks by extension operators. ACM Trans. Graph. 37, 71:1–71:12 (2018).

  36. 36.

    Collins, A., Zomorodian, A., Carlsson, G. & Guibas, L. A barcode shape descriptor for curve point cloud data. In Proceedings of SPBG’04 Symposium on Point-Based Graphics 2004 (eds Gross, M., Pfister, H., Alexa, M. & Rusinkiewicz, S.) 181–191 (The Eurographics Association, 2004).

  37. 37.

    Di Fabio, B. & Frosini, P. Filtrations induced by continuous functions. Topol. Appl. 160, 1413–1422 (2013).

  38. 38.

    Hatcher, A. Algebraic Ttopology. (Cambridge Univ.ersity Press: Cambridge, 2002.).

  39. 39.

    Cerri, A., Fabio, B. D., Ferri, M., Frosini, P. & Landi, C. Betti numbers in multidimensional persistent homology are stable functions. Math. Methods Appl. Sci. 36, 1543–1557 (2013).

  40. 40.

    Biasotti, S. et al. Describing shapes by geometrical-topological properties of real functions. ACM Comput. Surv. 40, 12:1–12:87 (2008).

  41. 41.

    Carlsson, G. & Zomorodian, A. The theory of multidimensional persistence. Discrete Comput. Geom. 42, 71–93 (2009).

  42. 42.

    Edelsbrunner, H. & Harer, J. Persistent homology—a survey. Contemp. Math. 453, 257–282 (2008).

  43. 43.

    Zomorodian, A. Fast construction of the Vietoris–Rips complex. Comput. Graph. 34, 263–271 (2010).

  44. 44.

    Fabbri, R., Costa, L. D. F., Torelli, J. C. & Bruno, O. M. 2D Euclidean distance transform algorithms: a comparative survey. ACM Comput. Surv. 40, 2 (2008).

  45. 45.

    Krizhevsky, A. & Hinton, G. Learning Multiple Layers of Features from Tiny Images (University of Toronto, 2009).

Download references

Acknowledgements

The authors thank H. Young for proofreading the manuscript. The research carried out by M.G.B. was supported by the European Research Council (Advanced Investigator Grant 671251 to Z.F. Mainen), the Champalimaud Foundation (Z.F. Mainen) and a GPU NVIDIA grant. The research carried out by P.F. and N.Q. was partially supported by GNSAGA-INdAM (Italy).

Author information

P.F., M.G.B., D.G. and N.Q. devised the project and algorithms. P.F. and N.Q. formalized the mathematical model with help from M.G.B. and D.G. M.G.B. wrote the code with help from D.G. M.G.B. and D.G. analysed the data. P.F. supervised the project. M.G.B., P.F., D.G. and N.Q. contributed equally to the writing of the main text and methods.

Correspondence to Patrizio Frosini.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary methods

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark