Abstract
Biomolecules undergo continuous conformational motions, a subset of which are functionally relevant. Understanding, and ultimately controlling biomolecular function are predicated on the ability to map continuous conformational motions, and identify the functionally relevant conformational trajectories. For equilibrium and nearequilibrium processes, function proceeds along minimumenergy pathways on one or more energy landscapes, because higherenergy conformations are only weakly occupied. With the growing interest in identifying functional trajectories, the need for reliable mapping of energy landscapes has become paramount. In response, various dataanalytical tools for determining structural variability are emerging. A key question concerns the veracity with which each dataanalytical tool can extract functionally relevant conformational trajectories from a collection of singleparticle cryoEM snapshots. Using synthetic data as an independently known ground truth, we benchmark the ability of four leading algorithms to determine biomolecular energy landscapes and identify the functionally relevant conformational paths on these landscapes. Such benchmarking is essential for systematic progress toward atomiclevel movies of continuous biomolecular function.
Introduction
Biomolecular machines have evolved to perform specific tasks through a concerted sequence of conformational motions. There is growing recognition that such motions involve continuous conformational changes, rather than jumps between a small number of discrete states^{1,2}. Apart from disordered proteins, conformational continua span a spectrum of different energies. In thermal equilibrium, the probability of a conformational state being occupied is determined by the Boltzmann factor, which drops exponentially with increasing energy.
Conformational motions of proteins can thus be represented as lowlying (and thus strongly occupied) pathways on one or more energy landscapes (EL)^{3,4}. In principle, an unlimited number of conformational paths connect a “start” conformation A to an “end” conformation B. However, most such paths include highenergy states, which are sparsely populated under biologically relevant conditions. Due to the exponential nature of the Boltzmann inverse relationship between energy and occupation probability^{5}, lowestenergy conformational paths contribute maximally to function.
The growing recognition of the importance of energy landscapes for discerning function has spawned an increasing number of sophisticated algorithms capable of mapping continuous conformational motions. Using a synthetic dataset of cryoEM snapshots with known ground truth energy landscape, we compare the performance of four leading algorithms, specifically Relion Multibody^{6}, CryoSPARC 3DVA^{7}, ManifoldEM^{8}, and CryoDRGN VAE^{9}, in faithfully extracting the energy landscape from snapshots. We benchmark the performance of each algorithmic approach in terms of the accuracy with which the correct energy landscape is recovered from the data.
To date, no comparative benchmarking study of the strengths and weaknesses of different dataanalytical approaches for analyzing continuous conformations has been reported. The lack of comparative benchmarks hampers the assessment of the usefulness and reliability of different algorithmic tools in extracting information from experimental data.
In this paper, we use a synthetic dataset generated with an a priori known “groundtruth” energy landscape to benchmark, in silico, the above leading algorithms for conformational analysis of cryoEM data. Specifically, we quantify each method’s ability to: (a) Recover the correct energy landscape from synthetic cryoEM datasets; (b) Reveal the functionally important conformational degrees of freedom; and (c) Identify the functionally relevant conformational paths on these landscapes. Although the nature and number of potentially useful algorithms are currently evolving, the four selected approaches represent the state of the art in mapping continuous conformational motions from ensembles of singleparticle cryoEM snapshots.
The primary goals of this paper are thus twofold:

(i)
Benchmark the performance of the four leading algorithms listed above in faithfully extracting conformational energy landscapes from synthetic cryoEM snapshots, and

(ii)
Provide a wellcharacterized synthetic cryoEM dataset suitable for comparative benchmarking, in order to facilitate the development of more effective dataanalytical tools capable of identifying functionally relevant conformational landscapes and motions.
The synthetic dataset of three million cryoEM snapshots (SignaltoNoise Ratio SNR = 1) stems from a ribosomelike object with two conformational degrees of freedom, with an underlying energy landscape of 12 energy minima of various depths arranged on a 3 × 4 grid (Fig. 1a). The distribution of points (each representing a single snapshot) is determined by the underlying energy landscape. The landscape is spanned by two conformational degrees of freedom, specifically the rotations of the small subunit (SSU) about two axes named conformational coordinates 1 and 2. The SSU of the ribosomelike object is permitted to rotate in a ratchetlike manner about two mutually orthogonal axes, with the large subunit (LSU) fixed (Fig. 1b).
The performance of each of the four dataanalytical approaches is quantified in terms of the fidelity of the energy landscape recovered from the synthetic snapshots. This fidelity is quantified in terms of “Recall” and “Accuracy”, defined in terms of intrinsic distances between snapshots calculated using the Euclidean distance metric. (For details see “Methods” section entitled Accuracy metric).
Results
Each of the four dataanalytical software tools listed above was applied to the synthetic dataset described above. The groundtruth snapshot orientations were provided to each algorithm to focus the study on the ability of the four different algorithms to extract conformational information. In each case, the top two (i.e. the most “powerful”) conformational coordinates were examined. We assess the performance of each algorithm in terms of its ability to classify snapshots correctly into the 12 ground truth energy minima. As outlined below, this approach allows us to quantify algorithmic fidelity in terms of wellknown metrics from classification.
Accuracy of extracted energy landscapes
We use “Recall”^{10} defined as
where P(n) is the positive class (snapshots belonging to ground truth energy minimum ‘n’), and TP(n) the true positives i.e. snapshots correctly assigned by the algorithm to the energy minimum ‘n’. This allows us to compute a recall value for each of the twelve energy minima. (See SI Tables 3–6).
The average of Recall values, known as Balanced Accuracy^{11} or simply Accuracy, allows us to assign a quantitative score to each algorithm’s ability to accurately extract the energy landscape underlying the synthetic data. (See “Methods” section entitled Accuracy metric). Essentially, one tracks the region of snapshots around the energy minimum in the input landscape and asks, “To what extent is each energy minimum obtained by each algorithm deformed from its original groundtruth shape?” We quantify the extent of this distortion in terms of the Euclidean metric (L2norm) to calculate a welldefined region of conformational similarity in both the input and output (Fig. 2a,b). The accuracy of each method is given in Table 1.
We now summarize the outcome of our benchmarking study. Accuracy is calculated by averaging the Recall values over the 12 energy minima. Relion Multibody on average assigns only 17.54 ± 14.4% of the points to the correct region in the groundtruth energy landscape (Fig. 2a,b). The cryoSPARC 3DVA algorithm achieves an accuracy score of 51.3 ± 18.6%. cryoDRGN variational autoencoder assigns snapshots with an accuracy of 61.2 ± 9.6%. The ManifoldEM algorithm correctly assigns 77.6 ± 4.8% of the snapshots on average over the entire ground truth energy landscape.
Occupancy maps and energy landscapes
The occupation probability of a conformational state is determined by the energy of the state via the Boltzmann factor. An occupancy map (defined in^{12}) is simply an alternative representation of the conformational energy landscape of the ribosomelike object. Figure 3a shows the energy landscape for the continuous conformational landscape of the synthetic ribosome.
To date, ManifoldEM is the only method that can calculate the energy landscapes of the conformational occupancy. The corresponding energy landscape obtained from the ManifoldEM data analytical pipeline is shown in Fig. 3b. It is evident that the energy landscape obtained by ManifoldEM closely resembles the groundtruth energy landscape. This includes the high occupancy regions or, equivalently, the twelve energy minima (Fig. 3b), while preserving the modulated features (i.e. the variation in depths of the individual energy minima) across the ground truth conformational coordinates (Fig. 3a). The other benchmarked algorithms have not yet developed a way to extract thermodynamic quantities like occupancy probability or free energy. It is evident that the energy landscape obtained by ManifoldEM.
Visualizing the energy landscape
The performance of the benchmarked algorithms can be validated by using methods from datavisualization. We segment the ground truth energy minima using a multicolor approach. This procedure is known as ‘data lineage’^{13}. Each energy minimum has been assigned a unique color to identify each energy minimum. (Fig. 4a). Essentially, lineage uses the ground truth labels for each singleparticle snapshot in the energy landscape to identify its position in the output of each of the dataanalytical tools (Fig. 4b–e). The region surrounding each of the 12energy minima is rendered in a different color to elucidate the extent to which snapshots stemming from adjacent regions of the ground truth energy landscape are correctly assigned by each of the four dataanalytical techniques.
As shown in Fig. 4b, the overall groundtruth topology is recovered by ManifoldEM. It is evident from Fig. 4c–e that the alternative approaches, viz. Relion, cryoSPARC, and cryoDRGN severely distort the energy landscape, intermixing points stemming from different energy minima.
Discussion
Inferring biological function from biomolecular structure is a paramount goal of structural biology. The results presented here highlight the importance of basing functional inference on energy landscapes and conformational coordinates derived from the data. To this end, we have developed and tested a rigorous approach to benchmarking the performance of four leading dataanalytical algorithms in faithfully extracting energy landscapes underlying a collection of synthetic snapshots. This assessment is based on the accuracy with which individual snapshots are placed on the output energy landscape as compared with the independently known groundtruth distribution. Using a complex synthetic model, ManifoldEM, accurately recovers the energy landscape.
In view of the importance of inferring function from structure, we anticipate an increasing spectrum of techniques for mapping energy landscapes. The benchmarking data we have used will facilitates rigorous assessments of the efficacy and reliability of current and future tools for determining biomolecular structure and function. Our approach thus offers a rigorous means for measuring and improving the performance of dataanalytical approaches capable of extracting energy landscapes from cryoEM datasets.
The study design, including the synthetic ribosomelike object, was chosen to mimic conditions typically encountered in conformational motions in biological macromolecules. Of course, any benchmarking exercise pertains to a particular time point. Consistent with the rapid progress in cryoEM singleparticle imaging, dataanalytical software tools are evolving apace^{14,15,16,17,18}. Our approach offers a quantitative method for assessing and guiding this rapidly progressing field.
The recall and accuracy metric are powerful tools for comparing the different algorithms, as it allows the study of energy regions without prior assumptions about the shape and distribution of the energy landscapes. Despite differences in the formulation of the different algorithms, the accuracybased score can be consistently applied to present and emerging algorithms. Currently, ManifoldEM represents the most reliable means for extracting energy landscapes and conformational coordinates from singleparticle cryoEM images. We offer the synthetic data used in the present study as a canonical test vehicle for developing powerful algorithms for extracting reliable structural and functional information from cryoEM data.
Methods
Accuracy metric
Finding an appropriate metric for extracting conformational landscapes from cryoEM data is a challenging task, if only because the metric must quantify the distortion caused by the algorithmic approach. We tackle this problem by using the quantitative metrics Recall and Accuracy^{11}, in order to quantify the ability of each of the four algorithms to correctly assign each snapshot to one of 12 classes. The Recall metric was implemented as follows:

1.
The center of each energy minimum was obtained by calculating the ‘mean’ of all the particles corresponding to that minimum region (for all 12 regions).

2.
Compute the distance matrix (squared Euclidean) using the 12 centers

3.
Using the distance of each particle from the corresponding center (Fig. 4a), bin the particles into each minimum based on the shortest distance to the center.

4.
This procedure assigns particles to a label from 1 to 12.
The Accuracy^{11} of the benchmarked method is given by:
where n is the number of energy minima (in our case, 12) and the numerator is the sum of the Recall metric calculated for each energy minima.
Recall ranges from 0 to 1 (as evident from the definition), where a value of 0 implies no snapshot is correctly assigned to that energy minima and a value of 1 when all snapshots belonging to the energy minima are correctly allocated. The accuracy of each algorithm is the average recall metric for assignment into all 12 minima.
Data availability
The datasets used and/or analyzed, and the code developed in the course of the current study are available from the corresponding author on reasonable request.
References
Ourmazd, A. CryoEM, XFELs and the structure conundrum in structural biology. Nat. Methods 16(10), 941 (2019).
Frank, J. & Ourmazd, A. Continuous changes in structure mapped by manifold embedding of singleparticle data in cryoEM. Methods 100, 61–67 (2016).
Frauenfelder, H., Parak, F. & Young, R. D. Conformational substates in proteins. Annu. Rev. Biophys. Biophys. Chem. 17(1), 451–479 (1988).
Wales, D. Energy Landscapes: Applications to Clusters, Biomolecules and Glasses (Cambridge University Press, 2003).
Thomas, P. D. & Dill, K. A. An iterative method for extracting energylike quantities from protein structures. Proc. Natl. Acad. Sci. 93(21), 11628–11633 (1996).
Nakane, T. et al. Characterisation of molecular motions in cryoEM singleparticle data by multibody refinement in RELION. Elife https://doi.org/10.7554/eLife.36861 (2018).
Punjani, A. & Fleet, D. J. 3D variability analysis: Resolving continuous flexibility and discrete heterogeneity from single particle cryoEM. J. Struct. Biol. 213(2), 107702 (2021).
Dashti, A. et al. Trajectories of the ribosome as a Brownian nanomachine. Proc. Natl. Acad. Sci. U. S. A. 111(49), 17492–17497 (2014).
Zhong, E. D. et al. CryoDRGN: Reconstruction of heterogeneous cryoEM structures using neural networks. Nat. Methods 18(2), 176–185 (2021).
Fawcett, T. An introduction to ROC analysis. Pattern Recogn. Lett. 27(8), 861–874 (2006).
Grandini, M., E. Bagli, and G. Visani, Metrics for multiclass classification: an overview. Preprint at http://arXiv.org/2008.05756 (2020).
Dashti, A. et al. Retrieving functional pathways of biomolecules from singleparticle snapshots. Nat. Commun 11(1), 4734 (2020).
Ikeda, R. & Widom, J. Data Lineage: A Survey (Stanford InfoLab, 2009).
Chen, M. and Ludtke, S. Deep learning based mixeddimensional GMM for characterizing variability in CryoEM. Preprint at http://arXiv.org/2101.10356 (2021).
Nashed, Y.S., et al. EndtoEnd Simultaneous Learning of Singleparticle Orientation and 3D Map Reconstruction from Cryoelectron Microscopy Data. Preprint at http://arXiv.org/2107.02958 (2021).
Gupta, H., et al. CryoGAN: A New Reconstruction Paradigm for Singleparticle CryoEM Via Deep Adversarial Learning. bioRxiv, 2020: p. 2020.03.20.001016.
Gupta, H. et al. MultiCryoGAN: Reconstruction of continuous conformations in CryoEM using generative adversarial networks. In European Conference on Computer Vision (eds Bartoli, A. & Fusiello, A.) (Springer, 2020).
Seitz, E. et al. Recovery of conformational continuum from singleparticle CryoEM images: Optimization of ManifoldEM informed by ground truth. IEEE Trans. Comput. Imaging 8, 462–478 (2022).
Acknowledgements
We acknowledge valuable discussions with Dr. Ahmad Hosseinizadeh, Dr. Russell Fung, and Dr. Eduardo CruzChú. The research was conducted at the University of Wisconsin Milwaukee.
Funding
The development of underlying techniques was supported by the US Department of Energy, Office of Science, Basic Energy Sciences under award DESC0002164 (underlying dynamical techniques), and by the US National Science Foundation under awards STC1231306 (underlying data analytical techniques) and DBI2029533 (underlying analytical models).
Author information
Authors and Affiliations
Contributions
R.D. cowrote the paper, codesigned the benchmarking study, and performed the calculations using the synthetic data. G.M. and R.E. performed the calculations for the synthetic data using ManifoldEM. R.E. cowrote the SI. P.S. codesigned the study, helped analyze the results and generated the synthetic data for the study. A.O. suggested the study, codesigned the study, helped analyze results, and cowrote the paper. R.D., G.M., R.E., P.S. and A.O. analyzed and interpreted the results.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher's note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Dsouza, R., Mashayekhi, G., Etemadpour, R. et al. Energy landscapes from cryoEM snapshots: a benchmarking study. Sci Rep 13, 1372 (2023). https://doi.org/10.1038/s4159802328401w
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s4159802328401w
Comments
By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.