Two community challenges assess the correctness of cryo-EM structures; future challenges should help determine the most appropriate structure validation methods.
Interest in the structural biology technique of single-particle cryo-electron microscopy (cryo-EM) has skyrocketed in the past few years. Institutes all over the world are setting up cryo-EM facilities and rapidly hiring new staff. Even drug companies are getting into the game, as the resolution of many cryo-EM structures reaches a level of atomic detail previously attainable only by X-ray crystallography.
With such high stakes, with many novice users clamoring to use the technique, and with the field moving toward automation, it becomes ever more critical to ensure that structures produced by cryo-EM are accurate. This served as motivation for the community-based Map Challenge and Model Challenge, sponsored by EMDataBank. The results of these challenges are now reported in a special issue of the Journal of Structural Biology .
The benefits of community challenges in biological research are many. Challenges focus efforts to identify specific bottlenecks and areas in need of improvement, and can spur the development of best practices and validation standards. The benchmark datasets developed for such challenges serve as an educational touchstone for investigators who want to improve their skills, and as a valuable reference for performance testing of new algorithms.
The goal of the Map Challenge was to assess the performance of software tools (of the participants’ choice) designed to reconstruct three-dimensional density maps, starting from raw cryo-EM image datasets of diverse benchmark protein complexes with resolutions in the range of 2–5 Å. Challenge assessors examined the quality of the reconstructions, using both established and new approaches.
Although the assessors found, reassuringly, that most software tools were capable of producing high-quality reconstructions, a striking finding was that two different users using the same tool did not necessarily get the same result. This suggests that tools for reconstruction need more robust testing and clearer instructions to ensure that they are fail-safe. Another key conclusion was that the current standard metric for determining resolution, Fourier shell correlation, is inconsistently calculated by users and also inadequate for determining map resolvability. This is an issue that future challenges should tackle.
In the Model Challenge, participants tested algorithms designed to fit atomic models to benchmark density maps of diverse structures. Challenge assessors evaluated the accuracy of the models using a variety of approaches, including established tools such as MolProbity, through comparison to reference models, by assessment of the fit of the model to the original map, and by evaluating the agreement between the submitted models.
The assessors found that most software tools tested were able to build high-quality models in a fully automated manner, in some cases generating models that were even better than the published reference models. However, the results highlight a pressing need for the community to further assess which available metrics are most informative, as well as to generate new metrics to better assess model correctness. The topic of validation is slated for discussion in an upcoming meeting sponsored by the Wellcome Trust. We look forward to such developments, and will continue to work closely with the community to update our cryo-EM statistical reporting table as such metrics evolve into standards for assessing structure quality.
Though the Map and Model Challenges represent a substantial effort, many burning questions remain. For example, 50% of the map submissions were generated with a single software tool, meaning an opportunity to perform a broader assessment of algorithms was missed. The challenges also did not assess the tools’ ability to handle cases of greater structural uncertainty, including lower-resolution or highly heterogeneous datasets. And the challenges did not assess structures generated using the emerging approach of cryo-electron tomography. We hope that future studies will address some of these issues; the grant supporting these challenges has been extended to 2022, which is good news indeed.
We appreciate that scientists’ time is highly precious. However, with a little bit of effort from a large number of people, community challenges such as these can substantially propel nascent fields forward. We encourage more scientists to get involved in such efforts, and we encourage funders to support these crucial benchmark studies. After all, one can generate accurate biological results only if the available tools are at their sharpest.