Introduction

The third edition of the ECMWF–ESA Workshop on Machine Learning for Earth Observation and Prediction (ML4ESOP) took place from 14 to 17 November 2022 at ECMWF Reading, UK (https://events.ecmwf.int/event/304/). After two exclusively online editions1,2, this edition of the workshop was run in a hybrid format, hosting about 120 people on-site and large and active virtual participation (about 700 registered participants). The attendance numbers, together with a record number of 121 abstract submissions, confirm the large interest in machine learning (ML) for Earth system science applications and also the growing popularity of the ECMWF–ESA workshop series as a reference meeting and discussion venue in this area.

As it has become traditional in this series of workshops, two leading experts were invited to provide broad overviews of the state-of-the-art, current opportunities, and challenges in their field of expertise.

Prof. Stephen Penny’s talk concentrated on the interactions and synergies between data assimilation (DA) and ML3. One fascinating suggestion was that current DA could leverage ML tools and ideas to greatly increase efficiency, which in turn could make more advanced algorithms affordable, e.g., allowing much larger ensemble sizes in ensemble DA and thus opening the way to more fully nonlinear/non-Gaussian assimilation methods.

Prof. Damien Borth’s presentation focussed on recent advances in ML tools used in Earth Observation (EO) and Remote Sensing4,5. The main focus of the talk was on efficient representation learning, i.e., the increasingly sophisticated set of techniques that allows a ML model to automatically discover the representations needed for feature detection or classification from raw data. This is a field of research in ML which is particularly relevant for EO, where there are huge quantities of unlabelled, remotely sensed data which could potentially be used.

From both these talks, it was apparent that increasingly sophisticated ML techniques have further spread into research and operational practice in the Earth sciences and, more importantly, they are being tailored to this specific domain with compelling results.

The workshop was structured according to separate thematic areas (TA) designed to cover the main application areas of ML in EO, numerical weather prediction (NWP), and climate prediction:

  • TA1: ML for Earth observations

  • TA2: Hybrid-ML in data assimilation

  • TA3: ML for model emulation and model discovery

  • TA4: ML for user-oriented Earth science applications

  • TA5: ML at the network edge and high-performance computing

The following sections will describe in more detail the presentations and discussions in the working groups on each TA.

TA1: ML for Earth observations

Current ML applications

The working group was chaired by Rochelle Schneider (ESA) and Alan Geer (ECMWF). Membership of the working group indicated a very broad interest in ML4EO that spans university, national meteorological services, and the private sector, with attendees drawn from Europe, America, Asia, Africa, and Australia. Application areas were similarly broad, including renewable energy, hydrology, clouds, environmental health, pollution, wildfires, urbanisation, geodesy, and crop classification.

Limitations, opportunities, and challenges

Much of the discussion reflected how far EO applications of ML have progressed in terms of serving real applications. To be beneficial to society, operational AI tools should ideally be reproducible, scalable, maintainable, transferable, and explainable. They need to integrate well with existing non-Artificial Intelligence (AI) tools rather than compete, i.e., aiming to fully replace state-of-the-art methods. Further, the ‘train once’ approach often needs to be replaced by continuous learning and retraining from fresh observations.

The perennial question of finding enough training data brought up the contrast between applications that can use non-expert labelling, achievable through citizen science or gamification (e.g., zooniverse.org), applications that require domain experts for labelling, and those where ‘ground truth’ observations are required6. Developments in the wider AI community could help, including the idea of ‘one shot’ learning (a human only needs to see one Zebra), meta-learning (learning how to learn) and the possibility of using non-domain foundation models in EO (could a foundation model transition from cats and dogs to crop type identification?)

Another widely recognised issue was the choice between using in-house or cloud-based computing resources7. The latter are attractive for the easy access to sophisticated TPUs and GPUs to speed up training, and the possibility of rapidly scaling up applications thereafter. But issues were raised over the expense relative to in-house hardware, the dependence on private backends into which an application can become locked-in, a perceived lack of support and doubts over data protection and security8.

Future directions

Including AI onboard satellites could speed up event detection (the idea of ‘smart satellites’) and could also help with data compression, prioritisation, and cooperation between satellites9,10. However, it is important that the full raw observational data is downlinked to earth and archived, possibly with less timeliness, in order to support future learning and development. The group also discussed federated learning, particularly for applications which rely on sensitive data (e.g., health), where AI is trained on dispersed data that is kept private.

TA2: Hybrid data assimilation—ML approaches

Current ML applications

Rossella Arcucci (Imperial College London) and Marcin Chrust (ECMWF) co-chaired this working group, which explored the utilisation of hybrid ML and DA approaches. A very broad group of members coming from academia, industry, NWP centres, and research centres took part in an active discussion about the potential use of ML with DA from a perspective of improving DA modelling. Hybrid approaches based on ML and DA have become increasingly popular in the DA field, with applications spanning from using neural networks to emulate model components within DA to entirely replacing the well-established DA algorithms with ML-based emulators or bespoke techniques. The latter include extended Elman networks11 that estimate posterior covariances and recurrent neural networks that replace 4D-Var with joint model and solver learning12. A large body of literature has been also dedicated to performing DA in latent spaces of a ML algorithm, balancing accuracy and computational cost13. The link between probabilistic ML approaches and differential equations is highlighted when the frameworks of DA and ML are combined from a Bayesian perspective. This equivalency, which demonstrates the parallels between the two areas, is presented formally in refs. 14,15.

Limitations, opportunities, and challenges

Learning full models or replacing DA algorithms with ML approaches is still considered challenging in operational settings given the difficulty posed by the high-dimensional nature of the systems involved. The development of hybrid models that combine physics-based models with statistical models has been proposed as an attractive alternative. It has been shown that statistical models that are used to correct physics-based models can be made situation dependent and trained within the framework of 4D-Var16. It was agreed that developing a common ML-DA framework would offer the potential to leverage the strengths of both approaches. ML, in particular, may present an opportunity to surpass the linearity and Gaussianity constraints imposed in current DA schemes utilised in numerical weather centres, while also significantly reducing the cost of the analysis process. The latter aspect is of growing importance as the resolution of the models and the analysis is increased in future.

Future directions

The combination of ML with DA advances the state-of-the-art of ML modelling in various fields and applications. The development trends and future challenges of this fast-growing field include learning state-observation mapping in DA or developing ML surrogates of dynamical systems assisted by DA. The working group participants reached a broad consensus that conventional DA methods can be used to improve ML algorithms, especially in addressing issues associated with noisy, incomplete or biased data. These hybrid models provide strengths in interpretability and noise reduction. Significant space for further breakthrough advances still exists, especially in applying these approaches in operational contexts.

TA3: ML for model emulation and model discovery

Current ML applications

The working group was chaired by Massimo Bonavita and Matthew Chantry (both ECMWF). Assessing the state of the field for model emulation the group saw a full spectrum of approaches, starting from learning to emulate a component of a weather or climate model up to learning to emulate realistic weather models in their entirety. On the latter approach, significant progress had been observed in the past year, with claims that some ML models have become competitive with operational state-of-the-art models for deterministic predictions17,18,19. Model discovery is currently less common, but successful work on e.g., predicting rogue waves gave a blueprint for a successful application20.

Limitations, opportunities, and challenges

Training of ML models was viewed as being overly reliant on mean-squared-error (or similarly constructed) loss functions, which has drawbacks as models trained towards this metric make cautious and overly smooth predictions which can be limiting in real world applications. Detailed discussions were held on the possibility of using Generative Adversarial Networks (GANs), diffusion models21, and others. Also, the general approach of training ML models using probabilistic instead of deterministic loss functions was indicated as a promising way forward.

Future directions

From the presentations and working group discussions it was apparent that using ML for model emulation and, more broadly, for general forecasting purposes is developing rapidly and the entry in the field of major commercial players will further accelerate progress in this area. Whether these efforts will pose a fundamental threat to the traditional Numerical Weather and Climate Prediction workflows based on physics-based weather emulators remains to be seen.

TA4: ML for user-oriented Earth science applications

Current ML applications

The working group was chaired by Claudia Vitolo and Bertrand Le Saux (both from ESA) and gathered a large and varied group of experts, as ML is now pervading all sectors of Earth System sciences and industry22. In the weather and climate domain, for instance, Deep Learning was reported to be used for nowcasting of precipitation23, the detection of extreme weather events24, post-processing of predictions (e.g., downscaling,25, as well as for the analysis of climate and weather processes at a longer time scale. In environmental applications, participants reported to have used ML to get actionable information from EO data in a variety of fields, including public health, agriculture, environmental protection (on land and at sea) just to mention a few. Several compelling industrial applications were also mentioned and included during-flight planning (aviation sector), road maintenance planning (transportation sector), energy demand and distribution planning and (re)-insurance. Participants also briefly touched upon new emerging fields in which ML may deliver promising applications, for security or policymaking, in the near future which build on federated learning, onboard processing, digital twin technologies and quantum ML.

Limitations, opportunities, and challenges

A great challenge of ML in user-oriented Earth science applications is the lack of trust in black-box models and the lack of a common language between developers and domain experts. However, many participants were confident that the development of explainable AI26, hybrid modelling, use of large pre-trained models, as well as strong community building will help to bridge the gap and lower the barrier to ML adoption. Technical issues such as costly compute resources and complex software usage are also deemed as critical and potentially limiting, and initiatives to provide accessible cloud computing, open and reusable source code are seen as potential enablers. Lastly, generative models (for image, text and data) are scrutinised with circumspection as they appear to have great potential for new applications27 but in the meantime raise problems of transparency and ethics28.

Future directions

Several perspectives emerged from the discussions and presentations. The development of new techniques and frameworks is considered highly impactful on society (for low-cost speed-up of complex numerical models, for fast simulations) and climate (for improving early warnings and identifying new sustainable environmental solutions for challenges affecting many sectors, e.g., energy). According to several participants, transformer models and explainable ML are deemed extremely promising to overcome the current lack of trust in black-box models22,29. AI is expected to be used more and more in extreme weather event predictions and digital twins modelling coupling various Earth system processes. In the longer term, there is a demand for investigating green computing, operationalisation, transparent ML and process understanding in Earth system sciences.

TA5: ML at the edge and high-performance computing

Current ML applications

During the workshop, novel computing with transformative power was a ubiquitous topic of discussion. With the rapid advances in computing technology, ML-enhanced high-performance computing has become an increasingly important tool in Earth science research, as illustrated by Carlos Alberto Gómez Gonzalez from Barcelona Supercomputing Centre in his talk on deep learning for empirical downscaling30 used to obtain e.g., estimates of nitrogen dioxide or precipitation fields at fine resolutions. At the other extremity of the computer power spectrum, on-board processing with machine learning at the edge has already proved to be useful for Earth observation, as it enables real-time processing of satellite data and instantaneous response to events such as floods10. It also helps reduce the cost of data transmission, as the data is processed on the satellite itself before being transmitted to the ground.

Limitations, opportunities, and challenges

Modular computing environments, that is, systems which integrate different types of compute resources, are seen as a way to offer a degree of flexibility and scalability to large-scale computing applications. Thus, program parts of complex simulations can be distributed over several modules so that various hardware properties can be optimally leveraged. This also allows for the addition and removal of components to meet changing demands and requirements or integrate future technologies such as quantum computing or neuromorphic modules. More powerful supercomputers built on this principle might be the way to enter the exascale era of computing. And such capacity is likely to be required to face the new challenges of this century: running numerical models with enough precision to allow forecast of weather events at a local scale or supporting the development of digital twins to enable monitoring, forecasting and assessing the impact of climate change, as mentioned by Jacqueline Lemoigne from NASA in her talk, showing examples such as IDEAS31.

Future directions

Many exciting perspectives were drawn and discussed. Distributed computing in space might offer the possibility to optimise the collaboration of small sensing satellites and satellites with computing payloads and enable cognitive cloud computing in space (C3S). Lisa Woerner from DLR highlighted the potential of quantum techniques for global Earth observation to climate change impact reduction. She stressed in particular the promises of quantum computing and quantum ML for which potential benefits could be gained from further exploration of underutilised areas of machine learning, such as reinforcement learning32. Bertrand Le Saux from ESA detailed the ongoing efforts to bring the power of Quantum Computing to Earth observation and presented two fields of research on the use of quantum ML for such classical data: quantum kernels and hybrid classical-quantum neural networks. They already yield the first proofs of concepts for image classification on gate-based quantum computers33,34 or quantum annealers35 and analysis of time-series36. These hybrid quantum-classical architectures form the foundation of the next generation of ML that can be run on modular HPC.

Conclusion

We believe the workshop continues to be a valuable environment for fostering knowledge exchange and facilitating breakthrough discoveries in the field of ML4ESOP. The range of applications of ML technologies in ESOP is truly remarkable and keeps on growing (for example, the new application areas bridging AI with Quantum Computing and the use of AI solutions close the data, e.g., onboard orbiting satellites). Themes like reproducibility and interpretability of ML outputs, together with scalability and maintainability of ML technologies still dominate the discussion towards the operational application of ML solutions. From the working groups outcomes, a clear consensus emerged of the need and opportunity to leverage ML strengths to try to fill gaps in our knowledge-based models and improve perceived weaknesses in current operational methods. On the other hand, a philosophical and practical divergence of approaches is beginning to appear in the application of ML technologies in ESOP. On one side, a gradual, incremental adoption of ML solutions in established workflows, aimed at improving results and reducing computational costs while striving to maintain a more or less complete understanding of the modelled system. This is the avenue typically chosen by domain scientists and practitioners in ESOP. On the other side, a growing corpus of research aims to show the potential of AI/ML to disrupt traditional practices through end-to-end, completely data-driven ML/AI solutions. In these applications, the modelled system is seen as a black (or grey) box, but the computational efficiencies are compelling and the quality of the predictions is becoming competitive with the state-of-the-art. This is the avenue favoured by some ML researchers approaching the ESOP world. It will be fascinating to see which approach will turn out to be more fruitful in the long run, but it can already be said that this dynamic has set in motion a far-reaching debate in the ESOP community on current methodologies and their long-term sustainability.