We provide a general mathematical framework for group and set equivariance in machine learning. We define group equivariant non-expansive operators (GENEOs) as maps between function spaces associated with groups of transformations. We study the topological and metric properties of the space of GENEOs to evaluate their approximating power and set the basis for general strategies to initialize and compose operators. We define suitable pseudo-metrics for the function spaces, the equivariance groups and the set of non-expansive operators. We prove that, under suitable assumptions, the space of GENEOs is compact and convex. These results provide fundamental guarantees in a machine learning perspective. By considering isometry-equivariant non-expansive operators, we describe a simple strategy to select and sample operators. Thereafter, we show how selected and sampled operators can be used both to perform classical metric learning and to inject knowledge in artificial neural networks.
Subscribe to Journal
Get full journal access for 1 year
only $8.25 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
The experiments and applications can be reproduced by installing the open-source Python package, available at https://doi.org/10.5281/zenodo.3264851.
Silver, D. et al. Mastering the game of go without human knowledge. Nature 550, 354–359 (2017).
Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017).
LeCun, Y. et al. in The Handbook of Brain Theory and Neural Networks (ed. Arbib, M. A.) 3361 (MIT Press, 1995).
Rawat, W. & Wang, Z. Deep convolutional neural networks for image classification: a comprehensive review. Neural Comput. 29, 2352–2449 (2017).
Kim, Y. Convolutional neural networks for sentence classification. Preprint at https://arxiv.org/abs/1408.5882 (2014).
Niyogi, P. Manifold regularization and semi-supervised learning: some theoretical analyses. J. Mach. Learn. Res. 14, 1229–1250 (2013).
Anselmi, F., Rosasco, L. & Poggio, T. On invariance and selectivity in representation learning. Inf. Inference 5, 134–158 (2016).
Cohen, T. & Welling, M. Group equivariant convolutional networks. Proc. 33rd Int. Conf. Mach. Learning 48, 2990–2999 (2016).
Worrall, D. E., Garbin, S. J., Turmukhambetov, D. & Brostow, G. J. Harmonic networks: deep translation and rotation equivariance. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 2 5028–5037 (IEEE, 2017).
Bengio, Y., Courville, A. & Vincent, P. Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35, 1798–1828 (2013).
Tishby, N. & Zaslavsky, N. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), 1–5 (IEEE, 2015); https://doi.org/10.1109/ITW.2015.7133169
Mallat, S. Group invariant scattering. Commun. Pure Appl. Math. 65, 1331–1398 (2012).
Mallat, S. Understanding deep convolutional networks. Philos. Trans. A Math. Phys. Eng. Sci. 374, 20150203 (2016).
Zhang, C., Voinea, S., Evangelopoulos, G., Rosasco, L. & Poggio, T. Discriminative Template Learning in Group-Convolutional Networks for Invariant Speech Representations, 3229–3233 (International Speech and Communication Association, 2015).
Anselmi, F., Evangelopoulos, G., Rosasco, L. & Poggio, T. Symmetry-adapted representation learning. Pattern Recognit. 86, 201–208 (2019).
Frosini, P. & Jabłoński, G. Combining persistent homology and invariance groups for shape comparison. Discrete Comput. Geom. 55, 373–409 (2016).
Frosini, P. & Quercioli, N. in Proceedings of the International Cross-Domain Conference CD-MAKE 2017, Vol. 10410 (eds Holzinger, A., Kieseberg, P., Tjoa, A. & Weippl, E.) 14–24 (Springer, 2017).
Camporesi, F., Frosini, P. & Quercioli, N. in Proceedings of the International Cross-Domain Conference CD-MAKE 2018, Vol. 11015 (eds Holzinger, A., Kieseberg, P., Tjoa, A. & Weippl, E.) 265–272 (Springer, 2018).
Smale, S., Rosasco, L., Bouvrie, J., Caponnetto, A. & Poggio, T. Mathematics of the neural response. Found. Comput. Math. 10, 67–91 (2010).
Carlsson, G. Topology and data. Bull. Am. Math. Soc. 46, 255–308 (2009).
Lum, P. Y. et al. Extracting insights from the shape of complex data using topology. Sci. Rep. 3, 1236 (2013).
Adams, H. et al. Persistence images: a stable vector representation of persistent homology. J. Mach. Learn. Res. 18, 218–252 (2017).
Pun, C. S., Xia, K. & Lee, S. X. Persistent-homology-based machine learning and its applications – a survey. Preprint at https://arxiv.org/abs/1811.00252 (2018).
Gabrielsson, R. B. & Carlsson, G. Exposition and interpretation of the topology of neural networks. Preprint at http://arxiv.org/abs/1810.03234 (2018)
Carlsson, G. & Gabrielsson, R. B. Topological approaches to deep learning. Preprint at https://arxiv.org/abs/1811.01122 (2018).
Deng, L. The MNIST database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Process. Mag. 29, 141–142 (2012).
Xiao, H., Rasul, K. & Vollgraf, R. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. Preprint at https://arxiv.org/abs/1708.07747 (2017).
Krizhevsky, A., Nair, V. & Hinton, G. The CIFAR-10 Dataset https://www.cs.toronto.edu/~kriz/cifar.html (2019).
Langfelder, P., Zhang, B. & Horvath, S. Defining clusters from a hierarchical cluster tree: the dynamic tree cut package for R. Bioinformatics 24, 719–720 (2007).
Glorot, X. & Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. Proc. 13th Int. Conf. Art. Intell. Statist. 9, 249–256 (2010).
Hinton, G. E., Krizhevsky, A. & Wang, S. D. Transforming auto-encoders. In Proceedings of International Conference on Artificial Neural Networks 44–51 (Springer, 2011).
Sabour, S., Frosst, N. & Hinton, G. E. in Advances in Neural Information Processing Systems Vol. 30 (eds Guyon, I. et al.) 3856–3866 (Curran Associates, 2017).
Frosini, P. Towards an observer-oriented theory of shape comparison. In Eurographics Workshop on 3D Object Retrieval (eds Ferreira, A., Giachetti, A. & Giorgi, D.) 5–8 (The Eurographics Association, 2016).
Carr, J. C et al. Reconstruction and representation of 3D objects with radial basis functions. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’01. 67–76 (ACM, 2001).
Atzmon, M., Maron, H. & Lipman, Y. Point convolutional neural networks by extension operators. ACM Trans. Graph. 37, 71:1–71:12 (2018).
Collins, A., Zomorodian, A., Carlsson, G. & Guibas, L. A barcode shape descriptor for curve point cloud data. In Proceedings of SPBG’04 Symposium on Point-Based Graphics 2004 (eds Gross, M., Pfister, H., Alexa, M. & Rusinkiewicz, S.) 181–191 (The Eurographics Association, 2004).
Di Fabio, B. & Frosini, P. Filtrations induced by continuous functions. Topol. Appl. 160, 1413–1422 (2013).
Hatcher, A. Algebraic Ttopology. (Cambridge Univ.ersity Press: Cambridge, 2002.).
Cerri, A., Fabio, B. D., Ferri, M., Frosini, P. & Landi, C. Betti numbers in multidimensional persistent homology are stable functions. Math. Methods Appl. Sci. 36, 1543–1557 (2013).
Biasotti, S. et al. Describing shapes by geometrical-topological properties of real functions. ACM Comput. Surv. 40, 12:1–12:87 (2008).
Carlsson, G. & Zomorodian, A. The theory of multidimensional persistence. Discrete Comput. Geom. 42, 71–93 (2009).
Edelsbrunner, H. & Harer, J. Persistent homology—a survey. Contemp. Math. 453, 257–282 (2008).
Zomorodian, A. Fast construction of the Vietoris–Rips complex. Comput. Graph. 34, 263–271 (2010).
Fabbri, R., Costa, L. D. F., Torelli, J. C. & Bruno, O. M. 2D Euclidean distance transform algorithms: a comparative survey. ACM Comput. Surv. 40, 2 (2008).
Krizhevsky, A. & Hinton, G. Learning Multiple Layers of Features from Tiny Images (University of Toronto, 2009).
The authors thank H. Young for proofreading the manuscript. The research carried out by M.G.B. was supported by the European Research Council (Advanced Investigator Grant 671251 to Z.F. Mainen), the Champalimaud Foundation (Z.F. Mainen) and a GPU NVIDIA grant. The research carried out by P.F. and N.Q. was partially supported by GNSAGA-INdAM (Italy).
The authors declare no competing interests.
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Bergomi, M.G., Frosini, P., Giorgi, D. et al. Towards a topological–geometrical theory of group equivariant non-expansive operators for data analysis and machine learning. Nat Mach Intell 1, 423–433 (2019). https://doi.org/10.1038/s42256-019-0087-3
Topology and its Applications (2020)
IEEE Access (2020)
Physical Review E (2019)