Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Perspective
  • Published:

Computational challenges for multimodal astrophysics

Nature Computational Science volume 2, pages 479–485 (2022)Cite this article

Subjects

Abstract

In the coming decades, we will face major computational challenges, when the improved sensitivity of third-generation gravitational wave detectors will be such that they will be able to detect a high number (of the order of 7 × 104 per year) of multi-messenger events from binary neutron star mergers, similar to GW 170817. In this Perspective, we discuss the application of multimodal artificial intelligence techniques for multi-messenger astrophysics, fusing the information from different signal emissions.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Detector timeline, running or planned (on available funds) for the next decade.
Fig. 2: Simple MML analysis workflow.
Fig. 3: The astrophysical event’s conceptual MML workflow.

Similar content being viewed by others

References

  1. Abbott, B. P. et al. (LIGO Scientific Collaboration and Virgo Collaboration) GW170817: observation of gravitational waves from a binary neutron star inspiral. Phys. Rev. Lett. 119, 161101 (2017).

  2. Abbott, B. P. et al. Multi-messenger observations of a binary neutron star merger. Astrophys. J. 848, L12 (2017).

    Article  Google Scholar 

  3. Mészáros, P., Fox, D. B., Hanna, C. & Murase, K. Multi-messenger astrophysics. Nat. Rev. Phys. 1, 585–599 (2019).

    Article  Google Scholar 

  4. LIGO Scientific Collaboration et al. Advanced LIGO. Class. Quantum Grav. 32, 074001 (2015).

  5. Acernese, F. et al. Advanced Virgo: a second-generation interferometric gravitational wave detector. Class. Quantum Grav. 32, 024001 (2015).

    Article  Google Scholar 

  6. Somiya, K. Detector configuration of KAGRA—the Japanese cryogenic gravitational-wave detector. Class. Quantum Grav. 29, 124007 (2012).

    Article  Google Scholar 

  7. Aso, Y. et al. Interferometer design of the KAGRA gravitational wave detector. Phys. Rev. D 88, 043007 (2013).

    Article  Google Scholar 

  8. Andersson, N. Whispers from the edge of physics. J. Astrophys. Astron. 38, 58 (2017).

    Article  Google Scholar 

  9. Evans, M. et al. A horizon study for cosmic explorer: science, observatories, and community. Preprint at https://arxiv.org/abs/2109.09882 (2021).

  10. Acharya, B. S. et al. Introducing the CTA concept. Astropart. Phys. 43, 3–18 (2013).

    Article  Google Scholar 

  11. Ivezić, Ž. et al. LSST: from science drivers to reference design and anticipated data products. Astrophys. J. 873, 111 (2019).

    Article  Google Scholar 

  12. Dewdney, P. E., Hall, P. J., Schilizzi, R. T. & Lazio, T. J. L. W. The Square Kilometre Array. IEEE Proc. 97, 1482 (2009).

    Article  Google Scholar 

  13. Adrián-Martínez, S. et al. Letter of intent for KM3NeT 2.0. J. Phys. G 43, 084001 (2016).

    Article  Google Scholar 

  14. Fialkov, A. & Loeb, A. A fast radio burst occurs every second throughout the observable Universe. Astrophys. J. Lett. 846, L27 (2017).

    Article  Google Scholar 

  15. Abbott, B. P. et al. A guide to LIGO–Virgo detector noise and extraction of transient gravitational-wave signals. Class. Quantum Grav. 37, 055002 (2020).

    Article  Google Scholar 

  16. Maggiore, M. et al. Science case for the Einstein Telescope. J. Cosmol. Astropart. Phys. https://doi.org/10.1088/1475-7516/2020/03/050 (2020).

  17. Abbott, B. P. et al. Gravitational waves and gamma-rays from a binary neutron star merger: GW170817 and GRB 170817A. Astrophys. J. Lett. 848, L13 (2017).

    Article  Google Scholar 

  18. Mogushi, K., Cavaglià, M. & Siellez, K. Jet geometry and rate estimate of coincident gamma-ray burst and gravitational-wave observations. Astrophys. J. 880, 55 (2019).

    Article  Google Scholar 

  19. Pian, E. et al. Spectroscopic identification of r-process nucleosynthesis in a double neutron-star merger. Nature 551, 67–70 (2017).

    Article  Google Scholar 

  20. Smartt, S. J. et al. A kilonova as the electromagnetic counterpart to a gravitational-wave source. Nature 551, 75–79 (2017).

    Article  Google Scholar 

  21. Radice, D., Perego, A., Zappa, F. & Bernuzzi, S. GW170817: joint constraint on the neutron star equation of state from multimessenger observations. Astrophys. J. Lett. 852, L29 (2018).

    Article  Google Scholar 

  22. Abbott, B. P. et al. A gravitational-wave standard siren measurement of the Hubble constant. Nature 551, 85–88 (2017).

    Article  Google Scholar 

  23. Abbott, B. P. et al. Optically targeted search for gravitational waves emitted by core-collapse supernovae during the first and second observing runs of advanced LIGO and Advanced Virgo. Phys. Rev. D 101, 084002 (2020).

    Article  Google Scholar 

  24. Ronchini, S. et al. Perspectives for multi-messenger astronomy with the next generation of gravitational-wave detectors and high-energy satellites. Preprint at https://arxiv.org/abs/2204.01746 (2022).

  25. Stratta, G. et al. THESEUS: a key space mission concept for multi-messenger astrophysics. Adv. Space Res. 62, 662–682 (2018).

    Article  Google Scholar 

  26. Gehrels, N. The Swift γ-ray burst mission. New Astron. Rev. https://doi.org/10.1016/j.newar.2003.12.055 (2004).

  27. Bonaldi, A. et al. Square Kilometre Array science data challenge 1: analysis and results. Mon. Not. R. Astron. Soc. 500, 3821–3837 (2020).

    Article  Google Scholar 

  28. Ayala Solares, H. A. et al. The Astrophysical Multimessenger Observatory Network (AMON): performance and science program. Astropart. Phys. 114, 68 (2020).

    Article  Google Scholar 

  29. Nordin, J. et al. Transient processing and analysis using AMPEL: alert management, photometry, and evaluation of light curves. Astron. Astrophys. 631, A147 (2019).

    Article  MathSciNet  Google Scholar 

  30. Agayeva, S. et al. Grandma: a network to coordinate them all. In Revista Mexicana de Astronomia y Astrofisica Conference Series 198–205 (Universidad Nacional Autonoma de Mexico, 2021).

  31. van der Walt, S., Crellin-Quick, A. & Bloom, J. SkyPortal: an astronomical data platform. J. Open Source Softw. 4, 1247 (2019).

    Article  Google Scholar 

  32. Vianello, G. et al. The Multi-Mission Maximum Likelihood framework (3ML). In 34th International Cosmic Ray Conference, PoS(ICRC2015) 1042 (PoS, 2015).

  33. The IceCube Collaboration et al. Analysis framework for Multi-messenger Astronomy with IceCube. In 37th International Cosmic Ray Conference, PoS(ICRC2021) 1098 (PoS, 2021).

  34. Ritz, S. Overview of the GLAST mission and opportunities. Am. Inst. Phys. Conf. Ser. 921, 3–7 (2007).

    Google Scholar 

  35. Holder, J. The first VERITAS telescope. Astropart. Phys. 25, 391–401 (2006).

    Article  Google Scholar 

  36. Sinnis, G. The HAWC TeV gamma-ray observatory. Nuovo Cimento C https://doi.org/10.1393/ncc/i2011-10851-8 (2011).

  37. Chatterjee, D., Narayan, G., Aleo, P. D., Malanchev, K. & Muthukrishna, D. El-CID: a filter for gravitational-wave electromagnetic counterpart identification. Mon. Not. R. Astron. Soc. 509, 914–930 (2021).

    Article  Google Scholar 

  38. Coughlin, M. W., Dietrich, T., Margalit, B. & Metzger, B. D. Multimessenger Bayesian parameter inference of a binary neutron star merger. Mon. Not. R. Astron. Soc. 489, L91–L96 (2019).

    Article  Google Scholar 

  39. Breschi, M. et al. AT2017gfo: Bayesian inference and model selection of multicomponent kilonovae and constraints on the neutron star equation of state. Mon. Not. R. Astron. Soc. 505, 1661–1677 (2021).

    Article  Google Scholar 

  40. Radice, D. & Dai, L. Multimessenger parameter estimation of GW170817. Eur. Phys. J. A 55, 50 (2019).

    Article  Google Scholar 

  41. Dietrich, T. et al. Multimessenger constraints on the neutron-star equation of state and the Hubble constant. Science 370, 1450–1453 (2020).

    Article  MathSciNet  MATH  Google Scholar 

  42. Raaijmakers, G. et al. The challenges ahead for multimessenger analyses of gravitational waves and kilonova: a case study on GW190425. Astrophys. J. 922, 269 (2021).

    Article  Google Scholar 

  43. Nakamura, K. et al. Multimessenger signals of long-term core-collapse supernova simulations: synergetic observation strategies. Mon. Not. R. Astron. Soc. 461, 3296–3313 (2016).

    Article  Google Scholar 

  44. D’Avanzo, P. Short gamma-ray bursts: a review. J. High Energy Astrophys. 7, 73–80 (2015).

    Article  Google Scholar 

  45. Metzger, B. D. Kilonovae. Living Rev. Relativ. 23, 1 (2019).

    Article  Google Scholar 

  46. Allen, M. G. et al. Escape—addressing open science challenges. Preprint at https://arxiv.org/abs/2012.11534 (2020).

  47. Brazier, A. SCIMMA: collaboration and information transfer cyberinfrastructure for multi-messenger astrophysics. In American Astronomical Society Meeting Abstracts 146 (BAAS, 2021).

  48. Huerta, E. A. et al. Accelerated, scalable and reproducible ai-driven gravitational wave detection. Nat. Astron. https://doi.org/10.1038/s41550-021-01405-0 (2021).

  49. Wilkinson, M. et al. The FAIR guiding principles for scientific data management and stewardship. Sci. Data https://doi.org/10.1038/sdata.2016.18 (2016).

  50. George, D. & Huerta, E. Deep Learning for real-time gravitational wave detection and parameter estimation: results with Advanced LIGO data. Phys. Lett. B 778, 64–70 (2018).

    Article  Google Scholar 

  51. Gabbard, H., Williams, M., Hayes, F. & Messenger, C. Matching matched filtering with deep networks for gravitational-wave astronomy. Phys. Rev. Lett. 120, 141103 (2018).

    Article  Google Scholar 

  52. Iess, A., Cuoco, E., Morawski, F. & Powell, J. Core-collapse supernova gravitational-wave search and deep learning classification. Mach. Learn. Sci. Technol. 1, 025014 (2020).

    Article  Google Scholar 

  53. Chan, M. L., Heng, I. S. & Messenger, C. Detection and classification of supernova gravitational wave signals: a deep learning approach. Phys. Rev. D https://doi.org/10.1103/physrevd.102.043022 (2020).

  54. Dreissigacker, C. & Prix, R. Deep-learning continuous gravitational waves: multiple detectors and realistic noise. Phys. Rev. D https://doi.org/10.1103/physrevd.102.022005 (2020).

  55. Miller, A. L. et al. How effective is machine learning to detect long transient gravitational waves from neutron stars in a real search?. Phys. Rev. D 100, 062005 (2019).

    Article  Google Scholar 

  56. Morawski, F., Bejger, M., Cuoco, E. & Petre, L. Anomaly detection in gravitational waves data using convolutional autoencoders. Mach. Learn. Sci. Technol. 2, 045014 (2021).

    Article  Google Scholar 

  57. Gabbard, H., Messenger, C., Heng, I. S., Tonolini, F. & Murray-Smith, R. Bayesian parameter estimation using conditional variational autoencoders for gravitational-wave astronomy. Nat. Phys. 18, 112–117 (2021).

    Article  Google Scholar 

  58. Green, S. R., Simpson, C. & Gair, J. Gravitational-wave parameter estimation with autoregressive neural network flows. Phys. Rev. D https://doi.org/10.1103/physrevd.102.104057 (2020).

  59. Shen, H., George, D., Huerta, E. A. & Zhao, Z. Denoising gravitational waves with enhanced deep recurrent denoising auto-encoders. In 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 3237–3241 (IEEE, 2019).

  60. Torres-Forné, A., Cuoco, E., Font, J. A. & Marquina, A. Application of dictionary learning to denoise LIGO’s blip noise transients. Phys. Rev. D https://doi.org/10.1103/physrevd.102.023011 (2020).

  61. Cuoco, E., Razzano, M. & Utina, A. Wavelet-based classification of transient signals for gravitational wave detectors. In 26th European Signal Processing Conference (EUSIPCO) 2648–2652 (IEEE, 2018).

  62. Renner, J. et al. Background rejection in NEXT using deep neural networks. J. Instrum. https://doi.org/10.1088/1748-0221/12/01/t01004 (2017).

  63. Aurisano, A. et al. A convolutional neural network neutrino event classifier. J. Instrum. https://doi.org/10.1088/1748-0221/11/09/p09001 (2016).

  64. Delaquis, S. et al. Deep neural networks for energy and position reconstruction in EXO-200. J. Instrum. https://doi.org/10.1088/1748-0221/13/08/p08023 (2018).

  65. Perdue, G. et al. Reducing model bias in a deep learning classifier using domain adversarial neural networks in the MINERvA experiment. J. Instrum. https://doi.org/10.1088/1748-0221/13/11/p11020 (2018).

  66. Pasquet, J., Pasquet, J., Chaumont, M. & Fouchez, D. PELICAN: deep architecture for the light curve analysis. Astron. Astrophys. 627, A21 (2019).

    Article  Google Scholar 

  67. Bassi, S., Sharma, K. & Gomekar, A. Classification of variable stars light curves using long short term memory network. Front. Astron. Space Sci. https://doi.org/10.3389/fspas.2021.718139 (2021).

  68. Pashchenko, I. N., Sokolovsky, K. V. & Gavras, P. Machine learning search for variable stars. Mon. Not. R. Astron. Soc. 475, 2326–2343 (2017).

    Article  Google Scholar 

  69. Nieto Castaño, D. et al. CTLearn: deep learning for gamma-ray astronomy. In 36th International Cosmic Ray Conference, PoS(ICRC2019) 752 (PoS, 2019).

  70. Baltrusaitis, T., Ahuja, C. & Morency, L.-P. Multimodal machine learning: a survey and taxonomy. IEEE Trans. Pattern Anal. Mach. Intell. 41, 423–443 (2019).

    Article  Google Scholar 

  71. Baevski, A. et al. data2vec: a general framework for self-supervised learning in speech, vision and language. In Proceedings of the 39th International Conference on Machine Learning, PMLR 1298–1312 (2022).

  72. Cuoco, E. et al. Enhancing gravitational-wave science with machine learning. Mach. Learn. Sci. Technol. https://doi.org/10.1088/2632-2153/abb93a (2020).

  73. Psihas, F., Groh, M., Tunnell, C. & Warburton, K. A review on machine learning for neutrino experiments. Int. J. Mod. Phys. A 35, 2043005 (2020).

    Article  Google Scholar 

  74. Sen, S., Agarwal, S., Chakraborty, P. & Singh, K.P. Astronomical big data processing using machine learning: a comprehensive review. Exp. Astron. 53, 1–43 (2022).

    Article  Google Scholar 

  75. Huerta, E. A. et al. Enabling real-time multi-messenger astrophysics discoveries with deep learning. Nat. Rev. Phys. 1, 600–608 (2019).

    Article  Google Scholar 

  76. Ngiam, J. et al. Multimodal deep learning. In Proc. 28th International Conference on International Conference on Machine Learning, ICML’11 689–696 (Omnipress, 2011).

  77. Cuoco, E., Patricelli, B., Iess, A. & Morawski, F. Multimodal analysis of gravitational wave signals and gamma-ray bursts from binary neutron star mergers. Universe https://doi.org/10.3390/universe7110394 (2021).

  78. Cuoco, E., Marzini, E., Morawski, F., Petrocelli, A. & Staniscia, A. A prototype for a real time pipeline for the detection of transient signals and their automatic classification. Zenodo https://doi.org/10.5281/zenodo.3356656 (2019).

Download references

Acknowledgements

We acknowledge support from COST Action CA17137, supported by COST (European Cooperation in Science and Technology), and from the ESCAPE project with grant number GA:824064.

Author information

Authors and Affiliations

  1. European Gravitational Observatory (EGO), Pisa, Italy

    Elena Cuoco & Barbara Patricelli

  2. Scuola Normale Superiore, Pisa, Italy

    Elena Cuoco & Alberto Iess

  3. INFN, Sezione di Pisa, Pisa, Italy

    Elena Cuoco, Barbara Patricelli & Alberto Iess

  4. University of Pisa, Pisa, Italy

    Barbara Patricelli

  5. Nicolaus Copernicus Astronomical Center, Polish Academy of Sciences, Warsaw, Poland

    Filip Morawski

Authors
  1. Elena Cuoco
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Barbara Patricelli
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alberto Iess
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Filip Morawski
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

E.C. coordinated the work. B.P., A.I. and F.M. contributed to the writing of all the sections of the paper.

Corresponding author

Correspondence to Elena Cuoco.

Peer review

Peer review information

Nature Computational Science thanks Michael Coughlin, Eliu Huerta and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Primary Handling Editor: Jie Pan, in collaboration with the Nature Computational Science team.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cuoco, E., Patricelli, B., Iess, A. et al. Computational challenges for multimodal astrophysics. Nat Comput Sci 2, 479–485 (2022). https://doi.org/10.1038/s43588-022-00288-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s43588-022-00288-z

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics