Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

# Computational challenges for multimodal astrophysics

## Abstract

In the coming decades, we will face major computational challenges, when the improved sensitivity of third-generation gravitational wave detectors will be such that they will be able to detect a high number (of the order of 7 × 104 per year) of multi-messenger events from binary neutron star mergers, similar to GW 170817. In this Perspective, we discuss the application of multimodal artificial intelligence techniques for multi-messenger astrophysics, fusing the information from different signal emissions.

This is a preview of subscription content, access via your institution

## Access options

\$32.00

All prices are NET prices.

## References

1. Abbott, B. P. et al. (LIGO Scientific Collaboration and Virgo Collaboration) GW170817: observation of gravitational waves from a binary neutron star inspiral. Phys. Rev. Lett. 119, 161101 (2017).

2. Abbott, B. P. et al. Multi-messenger observations of a binary neutron star merger. Astrophys. J. 848, L12 (2017).

3. Mészáros, P., Fox, D. B., Hanna, C. & Murase, K. Multi-messenger astrophysics. Nat. Rev. Phys. 1, 585–599 (2019).

4. LIGO Scientific Collaboration et al. Advanced LIGO. Class. Quantum Grav. 32, 074001 (2015).

5. Acernese, F. et al. Advanced Virgo: a second-generation interferometric gravitational wave detector. Class. Quantum Grav. 32, 024001 (2015).

6. Somiya, K. Detector configuration of KAGRA—the Japanese cryogenic gravitational-wave detector. Class. Quantum Grav. 29, 124007 (2012).

7. Aso, Y. et al. Interferometer design of the KAGRA gravitational wave detector. Phys. Rev. D 88, 043007 (2013).

8. Andersson, N. Whispers from the edge of physics. J. Astrophys. Astron. 38, 58 (2017).

9. Evans, M. et al. A horizon study for cosmic explorer: science, observatories, and community. Preprint at https://arxiv.org/abs/2109.09882 (2021).

10. Acharya, B. S. et al. Introducing the CTA concept. Astropart. Phys. 43, 3–18 (2013).

11. Ivezić, Ž. et al. LSST: from science drivers to reference design and anticipated data products. Astrophys. J. 873, 111 (2019).

12. Dewdney, P. E., Hall, P. J., Schilizzi, R. T. & Lazio, T. J. L. W. The Square Kilometre Array. IEEE Proc. 97, 1482 (2009).

13. Adrián-Martínez, S. et al. Letter of intent for KM3NeT 2.0. J. Phys. G 43, 084001 (2016).

14. Fialkov, A. & Loeb, A. A fast radio burst occurs every second throughout the observable Universe. Astrophys. J. Lett. 846, L27 (2017).

15. Abbott, B. P. et al. A guide to LIGO–Virgo detector noise and extraction of transient gravitational-wave signals. Class. Quantum Grav. 37, 055002 (2020).

16. Maggiore, M. et al. Science case for the Einstein Telescope. J. Cosmol. Astropart. Phys. https://doi.org/10.1088/1475-7516/2020/03/050 (2020).

17. Abbott, B. P. et al. Gravitational waves and gamma-rays from a binary neutron star merger: GW170817 and GRB 170817A. Astrophys. J. Lett. 848, L13 (2017).

18. Mogushi, K., Cavaglià, M. & Siellez, K. Jet geometry and rate estimate of coincident gamma-ray burst and gravitational-wave observations. Astrophys. J. 880, 55 (2019).

19. Pian, E. et al. Spectroscopic identification of r-process nucleosynthesis in a double neutron-star merger. Nature 551, 67–70 (2017).

20. Smartt, S. J. et al. A kilonova as the electromagnetic counterpart to a gravitational-wave source. Nature 551, 75–79 (2017).

21. Radice, D., Perego, A., Zappa, F. & Bernuzzi, S. GW170817: joint constraint on the neutron star equation of state from multimessenger observations. Astrophys. J. Lett. 852, L29 (2018).

22. Abbott, B. P. et al. A gravitational-wave standard siren measurement of the Hubble constant. Nature 551, 85–88 (2017).

23. Abbott, B. P. et al. Optically targeted search for gravitational waves emitted by core-collapse supernovae during the first and second observing runs of advanced LIGO and Advanced Virgo. Phys. Rev. D 101, 084002 (2020).

24. Ronchini, S. et al. Perspectives for multi-messenger astronomy with the next generation of gravitational-wave detectors and high-energy satellites. Preprint at https://arxiv.org/abs/2204.01746 (2022).

25. Stratta, G. et al. THESEUS: a key space mission concept for multi-messenger astrophysics. Adv. Space Res. 62, 662–682 (2018).

26. Gehrels, N. The Swift γ-ray burst mission. New Astron. Rev. https://doi.org/10.1016/j.newar.2003.12.055 (2004).

27. Bonaldi, A. et al. Square Kilometre Array science data challenge 1: analysis and results. Mon. Not. R. Astron. Soc. 500, 3821–3837 (2020).

28. Ayala Solares, H. A. et al. The Astrophysical Multimessenger Observatory Network (AMON): performance and science program. Astropart. Phys. 114, 68 (2020).

29. Nordin, J. et al. Transient processing and analysis using AMPEL: alert management, photometry, and evaluation of light curves. Astron. Astrophys. 631, A147 (2019).

30. Agayeva, S. et al. Grandma: a network to coordinate them all. In Revista Mexicana de Astronomia y Astrofisica Conference Series 198–205 (Universidad Nacional Autonoma de Mexico, 2021).

31. van der Walt, S., Crellin-Quick, A. & Bloom, J. SkyPortal: an astronomical data platform. J. Open Source Softw. 4, 1247 (2019).

32. Vianello, G. et al. The Multi-Mission Maximum Likelihood framework (3ML). In 34th International Cosmic Ray Conference, PoS(ICRC2015) 1042 (PoS, 2015).

33. The IceCube Collaboration et al. Analysis framework for Multi-messenger Astronomy with IceCube. In 37th International Cosmic Ray Conference, PoS(ICRC2021) 1098 (PoS, 2021).

34. Ritz, S. Overview of the GLAST mission and opportunities. Am. Inst. Phys. Conf. Ser. 921, 3–7 (2007).

35. Holder, J. The first VERITAS telescope. Astropart. Phys. 25, 391–401 (2006).

36. Sinnis, G. The HAWC TeV gamma-ray observatory. Nuovo Cimento C https://doi.org/10.1393/ncc/i2011-10851-8 (2011).

37. Chatterjee, D., Narayan, G., Aleo, P. D., Malanchev, K. & Muthukrishna, D. El-CID: a filter for gravitational-wave electromagnetic counterpart identification. Mon. Not. R. Astron. Soc. 509, 914–930 (2021).

38. Coughlin, M. W., Dietrich, T., Margalit, B. & Metzger, B. D. Multimessenger Bayesian parameter inference of a binary neutron star merger. Mon. Not. R. Astron. Soc. 489, L91–L96 (2019).

39. Breschi, M. et al. AT2017gfo: Bayesian inference and model selection of multicomponent kilonovae and constraints on the neutron star equation of state. Mon. Not. R. Astron. Soc. 505, 1661–1677 (2021).

40. Radice, D. & Dai, L. Multimessenger parameter estimation of GW170817. Eur. Phys. J. A 55, 50 (2019).

41. Dietrich, T. et al. Multimessenger constraints on the neutron-star equation of state and the Hubble constant. Science 370, 1450–1453 (2020).

42. Raaijmakers, G. et al. The challenges ahead for multimessenger analyses of gravitational waves and kilonova: a case study on GW190425. Astrophys. J. 922, 269 (2021).

43. Nakamura, K. et al. Multimessenger signals of long-term core-collapse supernova simulations: synergetic observation strategies. Mon. Not. R. Astron. Soc. 461, 3296–3313 (2016).

44. D’Avanzo, P. Short gamma-ray bursts: a review. J. High Energy Astrophys. 7, 73–80 (2015).

45. Metzger, B. D. Kilonovae. Living Rev. Relativ. 23, 1 (2019).

46. Allen, M. G. et al. Escape—addressing open science challenges. Preprint at https://arxiv.org/abs/2012.11534 (2020).

47. Brazier, A. SCIMMA: collaboration and information transfer cyberinfrastructure for multi-messenger astrophysics. In American Astronomical Society Meeting Abstracts 146 (BAAS, 2021).

48. Huerta, E. A. et al. Accelerated, scalable and reproducible ai-driven gravitational wave detection. Nat. Astron. https://doi.org/10.1038/s41550-021-01405-0 (2021).

49. Wilkinson, M. et al. The FAIR guiding principles for scientific data management and stewardship. Sci. Data https://doi.org/10.1038/sdata.2016.18 (2016).

50. George, D. & Huerta, E. Deep Learning for real-time gravitational wave detection and parameter estimation: results with Advanced LIGO data. Phys. Lett. B 778, 64–70 (2018).

51. Gabbard, H., Williams, M., Hayes, F. & Messenger, C. Matching matched filtering with deep networks for gravitational-wave astronomy. Phys. Rev. Lett. 120, 141103 (2018).

52. Iess, A., Cuoco, E., Morawski, F. & Powell, J. Core-collapse supernova gravitational-wave search and deep learning classification. Mach. Learn. Sci. Technol. 1, 025014 (2020).

53. Chan, M. L., Heng, I. S. & Messenger, C. Detection and classification of supernova gravitational wave signals: a deep learning approach. Phys. Rev. D https://doi.org/10.1103/physrevd.102.043022 (2020).

54. Dreissigacker, C. & Prix, R. Deep-learning continuous gravitational waves: multiple detectors and realistic noise. Phys. Rev. D https://doi.org/10.1103/physrevd.102.022005 (2020).

55. Miller, A. L. et al. How effective is machine learning to detect long transient gravitational waves from neutron stars in a real search?. Phys. Rev. D 100, 062005 (2019).

56. Morawski, F., Bejger, M., Cuoco, E. & Petre, L. Anomaly detection in gravitational waves data using convolutional autoencoders. Mach. Learn. Sci. Technol. 2, 045014 (2021).

57. Gabbard, H., Messenger, C., Heng, I. S., Tonolini, F. & Murray-Smith, R. Bayesian parameter estimation using conditional variational autoencoders for gravitational-wave astronomy. Nat. Phys. 18, 112–117 (2021).

58. Green, S. R., Simpson, C. & Gair, J. Gravitational-wave parameter estimation with autoregressive neural network flows. Phys. Rev. D https://doi.org/10.1103/physrevd.102.104057 (2020).

59. Shen, H., George, D., Huerta, E. A. & Zhao, Z. Denoising gravitational waves with enhanced deep recurrent denoising auto-encoders. In 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 3237–3241 (IEEE, 2019).

60. Torres-Forné, A., Cuoco, E., Font, J. A. & Marquina, A. Application of dictionary learning to denoise LIGO’s blip noise transients. Phys. Rev. D https://doi.org/10.1103/physrevd.102.023011 (2020).

61. Cuoco, E., Razzano, M. & Utina, A. Wavelet-based classification of transient signals for gravitational wave detectors. In 26th European Signal Processing Conference (EUSIPCO) 2648–2652 (IEEE, 2018).

62. Renner, J. et al. Background rejection in NEXT using deep neural networks. J. Instrum. https://doi.org/10.1088/1748-0221/12/01/t01004 (2017).

63. Aurisano, A. et al. A convolutional neural network neutrino event classifier. J. Instrum. https://doi.org/10.1088/1748-0221/11/09/p09001 (2016).

64. Delaquis, S. et al. Deep neural networks for energy and position reconstruction in EXO-200. J. Instrum. https://doi.org/10.1088/1748-0221/13/08/p08023 (2018).

65. Perdue, G. et al. Reducing model bias in a deep learning classifier using domain adversarial neural networks in the MINERvA experiment. J. Instrum. https://doi.org/10.1088/1748-0221/13/11/p11020 (2018).

66. Pasquet, J., Pasquet, J., Chaumont, M. & Fouchez, D. PELICAN: deep architecture for the light curve analysis. Astron. Astrophys. 627, A21 (2019).

67. Bassi, S., Sharma, K. & Gomekar, A. Classification of variable stars light curves using long short term memory network. Front. Astron. Space Sci. https://doi.org/10.3389/fspas.2021.718139 (2021).

68. Pashchenko, I. N., Sokolovsky, K. V. & Gavras, P. Machine learning search for variable stars. Mon. Not. R. Astron. Soc. 475, 2326–2343 (2017).

69. Nieto Castaño, D. et al. CTLearn: deep learning for gamma-ray astronomy. In 36th International Cosmic Ray Conference, PoS(ICRC2019) 752 (PoS, 2019).

70. Baltrusaitis, T., Ahuja, C. & Morency, L.-P. Multimodal machine learning: a survey and taxonomy. IEEE Trans. Pattern Anal. Mach. Intell. 41, 423–443 (2019).

71. Baevski, A. et al. data2vec: a general framework for self-supervised learning in speech, vision and language. In Proceedings of the 39th International Conference on Machine Learning, PMLR 1298–1312 (2022).

72. Cuoco, E. et al. Enhancing gravitational-wave science with machine learning. Mach. Learn. Sci. Technol. https://doi.org/10.1088/2632-2153/abb93a (2020).

73. Psihas, F., Groh, M., Tunnell, C. & Warburton, K. A review on machine learning for neutrino experiments. Int. J. Mod. Phys. A 35, 2043005 (2020).

74. Sen, S., Agarwal, S., Chakraborty, P. & Singh, K.P. Astronomical big data processing using machine learning: a comprehensive review. Exp. Astron. 53, 1–43 (2022).

75. Huerta, E. A. et al. Enabling real-time multi-messenger astrophysics discoveries with deep learning. Nat. Rev. Phys. 1, 600–608 (2019).

76. Ngiam, J. et al. Multimodal deep learning. In Proc. 28th International Conference on International Conference on Machine Learning, ICML’11 689–696 (Omnipress, 2011).

77. Cuoco, E., Patricelli, B., Iess, A. & Morawski, F. Multimodal analysis of gravitational wave signals and gamma-ray bursts from binary neutron star mergers. Universe https://doi.org/10.3390/universe7110394 (2021).

78. Cuoco, E., Marzini, E., Morawski, F., Petrocelli, A. & Staniscia, A. A prototype for a real time pipeline for the detection of transient signals and their automatic classification. Zenodo https://doi.org/10.5281/zenodo.3356656 (2019).

## Acknowledgements

We acknowledge support from COST Action CA17137, supported by COST (European Cooperation in Science and Technology), and from the ESCAPE project with grant number GA:824064.

## Author information

Authors

### Contributions

E.C. coordinated the work. B.P., A.I. and F.M. contributed to the writing of all the sections of the paper.

### Corresponding author

Correspondence to Elena Cuoco.

## Peer review

### Peer review information

Nature Computational Science thanks Michael Coughlin, Eliu Huerta and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Primary Handling Editor: Jie Pan, in collaboration with the Nature Computational Science team.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and Permissions

Cuoco, E., Patricelli, B., Iess, A. et al. Computational challenges for multimodal astrophysics. Nat Comput Sci 2, 479–485 (2022). https://doi.org/10.1038/s43588-022-00288-z

• Accepted:

• Published:

• Issue Date:

• DOI: https://doi.org/10.1038/s43588-022-00288-z