The ability to quantify structural changes of the endoplasmic reticulum (ER) is crucial for understanding the structure and function of this organelle. However, the rapid movement and complex topology of ER networks make this challenging. Here, we construct a state-of-the-art semantic segmentation method that we call ERnet for the automatic classification of sheet and tubular ER domains inside individual cells. Data are skeletonized and represented by connectivity graphs, enabling precise and efficient quantification of network connectivity. ERnet generates metrics on topology and integrity of ER structures and quantifies structural change in response to genetic or metabolic manipulation. We validate ERnet using data obtained by various ER-imaging methods from different cell types as well as ground truth images of synthetic ER structures. ERnet can be deployed in an automatic high-throughput and unbiased fashion and identifies subtle changes in ER phenotypes that may inform on disease progression and response to therapy.
This is a preview of subscription content, access via your institution
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 print issues and online access
$209.00 per year
only $17.42 per issue
Rent or buy this article
Get just this article for as long as you need it
Prices may be subject to local taxes which are calculated during checkout
All data needed to evaluate the conclusions in the paper are present in the Source data. All the datasets used to train and test the model are publicly accessible at the figshare repository: https://figshare.com/articles/dataset/ERnet_datasets/21975878/1. Source data are provided with this paper.
The ERnet model is written in Python. The software and Colab versions of ERnet are also freely available online through GitHub at https://github.com/charlesnchr/ERnet-v2.
Schwarz, D. S. & Blower, M. D. The endoplasmic reticulum: structure, function and response to cellular signaling. Cell. Mol. Life Sci. 73, 79–94 (2016).
Schönthal, A. H. Endoplasmic reticulum stress: its role in disease and novel prospects for therapy. Scientifica 2012, 857516 (2012).
Lu, M. et al. The structure and global distribution of the endoplasmic reticulum network are actively regulated by lysosomes. Sci. Adv. 6, eabc7209 (2020).
Obara, C. J., Moore, A. S. & Lippincott-Schwartz, J. Structural diversity within the endoplasmic reticulum—from the microscale to the nanoscale. Cold Spring Harb. Perspect. Biol. 19, a041259 (2022).
Nixon-Abell, J. et al. Increased spatiotemporal resolution reveals highly dynamic dense tubular matrices in the peripheral ER. Science 354, aaf3928 (2016).
Guo, Y. et al. Visualizing intracellular organelle and cytoskeletal interactions at nanoscale resolution on millisecond timescales. Cell 175, 1430–1442 (2018).
Zhao, W. et al. Sparse deconvolution improves the resolution of live-cell super-resolution fluorescence microscopy. Nat. Biotechnol. 40, 606–617 (2022).
Stringer, C., Wang, T., Michaelos, M. & Pachitariu, M. Cellpose: a generalist algorithm for cellular segmentation. Nat. Methods 18, 100–106 (2021).
Fischer, C. A. et al. MitoSegNet: easy-to-use deep learning segmentation for analyzing mitochondrial morphology. iScience 23, 101601 (2020).
Lefebvre, A. E., Ma, D., Kessenbrock, K., Lawson, D. A. & Digman, M. A. Automated segmentation and tracking of mitochondria in live-cell time-lapse images. Nat. Methods 18, 1091–1102 (2021).
Hollandi, R. et al. nucleAIzer: a parameter-free deep learning framework for nucleus segmentation using image style transfer. Cell Syst. 10, 453–458 (2020).
English, A. R. & Voeltz, G. K. Endoplasmic reticulum structure and interconnections with other organelles. Cold Spring Harb. Perspect. Biol. 5, a013227 (2013).
Pain, C., Kriechbaumer, V., Kittelmann, M., Hawes, C. & Fricker, M. Quantitative analysis of plant ER architecture and dynamics. Nat. Commun. 10, 984 (2019).
Garcia-Pardo, M. E., Simpson, J. C. & O’Sullivan, N. C. A novel automated image analysis pipeline for quantifying morphological changes to the endoplasmic reticulum in cultured human cells. BMC Bioinformatics 22, 427 (2021).
Arganda-Carreras, I. et al. Trainable Weka Segmentation: a machine learning tool for microscopy pixel classification. Bioinformatics 33, 2424–2426 (2017).
Belthangady, C. & Royer, L. A. Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction. Nat. Methods 16, 1215–1225 (2019).
Lee, T. C., Kashyap, R. L. & Chu, C. N. Building skeleton models via 3-D medial surface axis thinning algorithms. CVGIP Graph. Models Image Process. 56, 462–478 (1994).
Peixoto, P. T. The graph-tool Python library (figshare, 2014).
Dosovitskiy, A. et al. An image is worth 16 × 16 words: transformers for image recognition at scale. Preprint at arXiv https://doi.org/10.48550/arXiv.2010.11929 (2020).
Liu, Z. et al. Swin Transformer V2: scaling up capacity and resolution. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 12009–12019 (IEEE/CVF, 2022).
Mateus, D., Marini, E. S., Progida, C. & Bakke, O. Rab7a modulates ER stress and ER morphology. Biochim. Biophys. Acta Mol. Cell Res. 1865, 781–793 (2018).
Chang, J., Lee, S. & Blackstone, C. Protrudin binds atlastins and endoplasmic reticulum-shaping proteins and regulates network formation. Proc. Natl Acad. Sci. USA 110, 14954–14959 (2013).
Qin, J. et al. ER–mitochondria contacts promote mtDNA nucleoids active transportation via mitochondrial dynamic tubulation. Nat. Commun. 11, 4471 (2020).
Qiao, C. et al. Evaluation and development of deep neural networks for image super-resolution in optical microscopy. Nat. Methods 18, 194–202 (2021).
Newman, M. E. J. Assortative mixing in networks. Phys. Rev. Lett. 89, 208701 (2002).
Zhao, X. et al. Mutations in a newly identified GTPase gene cause autosomal dominant hereditary spastic paraplegia. Nat. Genet. 29, 326–331 (2001).
Mannan, A. U. et al. ZFYVE27 (SPG33), a novel spastin-binding protein, is mutated in hereditary spastic paraplegia. Am. J. Hum. Genet. 79, 351–357 (2006).
Sun, S. et al. Identification of endoplasmic reticulum-shaping proteins in Plasmodium parasites. Protein Cell 7, 615–620 (2016).
Ko, D. C., Gordon, M. D., Jin, J. Y. & Scott, M. P. Dynamic movements of organelles containing Niemann–Pick C1 protein: NPC1 involvement in late endocytic events. Mol. Biol. Cell 12, 601–614 (2001).
Merritt, J. E. et al. SK&F 96365, a novel inhibitor of receptor-mediated calcium entry. Biochem. J. 271, 515–522 (1990).
McAbee, D. D. & Weigel, P. H. ATP depletion causes a reversible redistribution and inactivation of a subpopulation of galactosyl receptors in isolated rat hepatocytes. J. Biol. Chem. 262, 1942–1945 (1987).
Paul, S., & Chen, P. Y. Vision transformers are robust learners. In Proc. AAAI Conference on Artificial Intelligence Vol. 36, 2071–2081 (AAAI, 2022).
Vaswani, A. et al. Attention is all you need. Advances in Neural Information Processing Systems 30 (NIPS) (2017).
Christensen, C. N., Lu, M., Ward, E. N., Lio, P. & Kaminski, C. F. Spatio-temporal Vision Transformer for super-resolution microscopy. Preprint at arXiv https://doi.org/10.48550/arXiv.2203.00030 (2022).
Qiao, C. et al. Rationalized deep learning super-resolution microscopy for sustained live imaging of rapid subcellular processes. Nat. Biotechnol. 41, 367–377 (2023).
Liu, J. et al. Automatic reconstruction of mitochondria and endoplasmic reticulum in electron microscopy volumes by deep learning. Front. Neurosci. 14, 599 (2020).
Guo, Y. et al. Deep learning-based morphological classification of endoplasmic reticulum under stress. Front. Cell Dev. Biol. 9, 767866 (2022).
Schroeder, L. K. et al. Dynamic nanoscale morphology of the ER surveyed by STED microscopy. J. Cell Biol. 218, 83–96 (2019).
Michel, C. H. et al. Extracellular monomeric tau protein is sufficient to initiate the spread of tau protein pathology. J. Biol. Chem. 289, 956–967 (2014).
Avezov, E. et al. Retarded PDI diffusion and a reductive shift in poise of the calcium depleted endoplasmic reticulum. BMC Biol. 13, 2 (2015).
Middya, S. et al. Microelectrode arrays for simultaneous electrophysiology and advanced optical microscopy. Adv. Sci. 8, 2004434 (2021).
Lu, M. et al. Structural progression of amyloid-β Arctic mutant aggregation in cells revealed by multiparametric imaging. J. Biol. Chem. 294, 1478–1487 (2019).
Young, L. J., Ströhl, F. & Kaminski, C. F. A guide to structured illumination TIRF microscopy at high speed with multiple colors. J. Vis. Exp. https://doi.org/10.3791/53988 e53988 (2016).
Müller, M., Mönkemöller, V., Hennig, S., Hübner, W. & Huser, T. Open-source image reconstruction of super-resolution structured illumination microscopy data in ImageJ. Nat. Commun. 7, 10980 (2016).
Culley, S. et al. Quantitative mapping and minimization of super-resolution optical imaging artifacts. Nat. Methods 15, 263–266 (2018).
Liang, J. et al. SwinIR: image restoration using Swin Transformer. In Proc. IEEE/CVF International Conference on Computer Vision 1833–1844 (ICCV, 2021).
Zhang, Y. et al. Image super-resolution using very deep residual channel attention networks. In Proc. European Conference on Computer Vision 286–301 (ECCV, 2018).
Boccaletti, S., Latora, V., Moreno, Y., Chavez, M. & Hwang, D. U. Complex networks: structure and dynamics. Phys. Rep. 424, 175–308 (2006).
Costa, L. et al. Characterization of complex networks: a survey of measurements. Am. J. Enol. Vitic. 38, 293–297 (1987).
Hagberg, A., Swart, P. & Chult, S. D. Exploring Network Structure, Dynamics, and Function Using NetworkX. Report No. LA-UR-08-05495 (USDOE, 2008).
Newman, M. E. J. The structure and function of complex networks. SIAM Rev. 45, 167–256 (2003).
Cimini, G. et al. The statistical physics of real-world networks. Nat. Rev. Phys. 1, 58–71 (2019).
Albert, R. Scale-free networks in cell biology. J. Cell Sci. 118, 4947–4957 (2005).
Strogatz, S. H. Exploring complex networks. Nature 410, 268–276 (2001).
We thank A.I. Fernández Villegas and Y. Feng for helping with cell culture. We thank E. Ward for helping with image processing. We thank J. Hu (Chinese Academy of Sciences, China) for giving us the ATL KO cell line. Funding: this research was funded by Infinitus (China) (supporting M.L., C.F.K. and G.S.K.S.); a Wellcome Trust Programme Grant (085314/Z/08/Z to G.S.K.S. and C.F.K); a Swiss National Science Foundation Career Grant (P2EZP2_199843 to N.F.L.); a research fellowship from the Deutsche Forschungsgemeinschaft (DFG; SCHE 1672/2-1 to K.M.S.) and pump-prime funding from the Integrated Biological Imaging Network (G106925 to K.M.S.); the UK Dementia Research Institute, which receives its funding from the UK DRI, funded by the UK Medical Research Council, Alzheimer’s Society and Alzheimer’s Research UK (supporting T.K., E.A. and C.F.K.) and Alzheimer’s Society 525 (AS-PhD-19a-015) supporting E.A. J.M.W.’s PhD scholarship was funded by the Department of Chemical Engineering and Biotechnology, University of Cambridge.
The authors declare no competing interests.
Peer review information
Nature Methods thanks Niamh O’Sullivan and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Primary Handling Editor: Rita Strack, in collaboration with the Nature Methods team. Peer reviewer reports are available.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Extended Data Fig. 1 A test of Weka trainable segmentation with different input data.
Top left: An input image was used to train a classifier of Weka Trainable Segmentation. Top right: The tubules (cyan) and sheet (yellow) can be clearly classified after segmentation. Bottom left: a new image was applied to the trained classifier shown above. Bottom right: segmentation result of the new input data. This test was independently repeated five times with similar results. Scale bars: 5 μm.
Extended Data Fig. 2 ERnet graphical user interface.
Left part of the interface shows the path of input and output images. Bottom left: options of the analysis provided by ERnet. Right part of the interface shows the input images (magenta) and segmented results.
Extended Data Fig. 3 Validation of 2D analysis of 3D ER structure.
(a) 3D projection view of a U2OS cell expressing sec61-mCherry. The peripherical ER is mainly distributed as a flat, single layer while the perinuclear ER is more 3 dimensional in character (see Supplementary Video 12). (b) 3D projection view of a COS-7 cell expressing mEmerald-Sec61b-C1. i: (left) 3D reconstruction of image sections. ii: projection of the volumetric view of corresponding the sections shown in (i). Peripherical ER features are flat and singly layered (Supplementary Video 13). In the perinuclear region, multiple layers of ER are visible. iii: each section from the 3D image stack shown in (i) is segmented individually by ERnet. iv: The resulting segmented frames are then combined into a 3D volume to show the network topology (Supplementary Video 14). Cyan regions (tubules) show flat, 2D regions with little 3D character. Perinuclear regions appear in yellow (sheets), where the topology is 3 dimensional in character.
Extended Data Fig. 4 Performance validation of ERnet as a function of signal-to noise-ratio prevailing in ER image data.
(a) Pipeline to develop synthetic ground truth data to mimic ER structure as it would appear when imaged under the microscope including resolution loss through PSF blurring, and addition of image noise affecting the Signal to Noise ratio (SNR). A network was generated from points distributed randomly over a given field of view. Triangulation and tessellation were then used before cubic spine interpolation to obtain a network mimicking features of a fully connected tubular ER system. We added Gaussian noise and blurred images with a PSF kernel to produce ground truth data as obtained by our microscope setup. (b) Representative synthetic ER networks were generated at different noise levels. Ground truth images (left column) were processed, and Gaussian noise superimposed (middle column). Images were then analysed by ERnet to produce the skeleton map displaying the connectivity (right column), including nodes and edges. The output could then be directly compared with the results from ground truth data. (c) Degradation of network metrics as a function of decreasing image signal to noise ratio. The top panel compares the number of nodes obtained from the noisy data with that of the ground truth data (1 = fully matched, 0 = no match) as a function of noise level. The bottom panel shows how the number of three-way junctions (node of degree three) identified by ERnet decreases as noise increases. In the ground truth image (Noise level 0) ca 90% of all nodes constitute three-way junctions. For image signal to noise-ratios exceeding 5, ERnet reproduced the network topology to within 90% of the ground truth. N = 5 synthetic images per noise level. Data are presented as mean ± SEM. The algorithms and full instructions to generate synthetic ER networks mimicking microscopy data are available on GitHub and see Source Data Extended Data Fig. 4c. (https://github.com/charlesnchr/ERnet-v2).
Extended Data Fig. 5 Validation of ERnet on publicly available datasets of ER images.
ERnet recovers network topologies from ER recorded with different imaging techniques and cell types, including mammalian and plant cells. (a)–(g) Raw images (grayscale images, left hand side) and corresponding results from ERnet image segmentation (colour images, right hand side). Tubules (cyan), sheets (yellow), and sheet-based tubules, SBTs (magenta) are clearly recovered in all cases studied. (a-b) are from the dataset provided by Pain et al.13 and represent ER present in plant cells recorded with AiryScan confocal imaging; 33 different images from this dataset were reanalysed by ERnet and compared with the results from AnalyzER; (c-f) are from the dataset provided by Qiao et al., 2021 (COS-7 cells, recorded with SIM); 18 different images from this dataset were tested by ERnet; (g) is from the dataset provided by Qin et al., 2020 (COS-7 cells, recorded with SIM); 2 different images from this dataset were tested by ERnet; (h) ER morphology metrics for all structures identified by ERnet. See Source Data Extended Data Fig. 5.
Extended Data Fig. 6 Graphical explanation of graph theory concepts.
(a) In this example of an assortative network, nodes show a preference of attaching to others that are similar. Here, nodes with the same degree tend to attach to each other (a node’s degree is represented by its colour and number in the node). (b) In this example of disassortative network, nodes show a preference of attaching to others that are not similar. Here, nodes with different degrees tend to attach to each other. (c) In this example, a network is shown characterised by a high clustering coefficient: here nodes tend to connect also to the neighbours of their neighbours, leading to the formation of network triangles. (d) Example of a network with a low clustering coefficient: nodes do not tend to connect to the neighbours of their neighbours. None, or only few, network triangles are formed as a result.
Extended Data Fig. 7 Quantitative comparison between the 2D and 3D analysis.
(a) Top panel: 3D SIM image sections were overlayed to produce a flattened image of the ER; left: central section of the ER image stack; right: colour coded flattened ER structures. Bottom: comparison of ERnet results for both types of images reveal nearly identical connectivity, supporting the notion that a 2D analysis suffices to capture ER topology in the cell types studied. Red spots: nodes; green lines: edges. Lower bottom: comparison of the connectivity metrics. To do this, we obtained volumetric sections of ER structures with SIM and reconstructed the data in 3D. We then applied ERnet on flattened image stacks or individual 2D sections and compared the two results. For phenotyping the effect of drugs in cell lines such as COS-7 or U2OS, which are morphologically flat, the advantages in speed and applicability of a 2D method greatly outweigh the disadvantage of not capturing the occasionally occurring 3-dimensional network features. N = 3 sectioning SIM images. Data are presented as mean ± SEM. See Source Data Extended Data Fig. 7a. (b) Same analysis to (a) performed on tubular domains only. N = 3 sectioning SIM images. Data are presented as mean ± SEM. See Source Data Extended Data Fig. 7b.
Extended Data Fig. 8 Western blot validation of Protrudin depletion.
Validation of siRNA depletion of Protrudin by Western blot. Two independent experiments were repeated with similar results. See source data file for uncropped scans of blots.
Supplementary Videos 1–14
Supplementary Video 1. Live cell imaging by SIM demonstrates rapid reshaping of the ER. A COS-7 cell expressing mEmerald-Sec61b was imaged over 43.5 s at 1.5 s per frame (Fig. 2a). Supplementary Video 2. Live cell imaging by SIM demonstrates rapid reshaping of SBTs in the ER. A COS-7 cell expressing mEmerald-Sec61b was imaged over 43.5 s at 1.5 s per frame (Fig. 2d). Supplementary Video 3. ERnet segmentation of the sequential images in Supplementary Video 2. All three domains including tubules (cyan), sheets (yellow) and SBTs (magenta) were precisely classified by ERnet (Fig. 2d). Supplementary Video 4. A 3D rendering of ERnet segmented structures demonstrates attachment of tubules to sheets. A 3D reconstruction of SIM image sections validated that SBTs are directly attached to sheets and are not the result of a projection view artifact. All three domains including tubules (cyan), sheets (yellow) and SBTs (magenta) were precisely classified by ERnet (Fig. 2e). Supplementary Video 5. A control (healthy) cell treated with the siRNA vehicle water, imaged by SIM for the comparison of ER phenotypes. Sequential images (43.5 s at 1.5 s per frame) demonstrated rapid reshaping and complex connectivity in a control (healthy) cell sample (Fig. 6a). Supplementary Video 6. A representative sample of ATL KO cells imaged by SIM for the comparison of ER phenotypes. Sequential images (43.5 s at 1.5 s per frame) demonstrated compromised reshaping and connectivity defects in an ATL KO cell sample (Fig. 6a). Supplementary Video 7. A representative sample of protrudin-depleted cells imaged by SIM for the comparison of ER phenotypes. Sequential images (43.5 s at 1.5 s per frame) demonstrated compromised reshaping and connectivity defects in a cell sample treated with siRNA to deplete protrudin (Fig. 6a). Supplementary Video 8. A representative sample of cells treated with U18666A and imaged by SIM for the comparison of ER phenotypes. Sequential images (43.5 s at 1.5 s per frame) demonstrated reduced tubular network and fragmented ER structure in a cell sample treated with U18666A (Fig. 6a). Supplementary Video 9. A representative sample of cells treated with SKF96365 and imaged by SIM for the comparison of ER phenotypes. Sequential images (43.5 s at 1.5 s per frame) demonstrated that the ER was largely fragmented and featured as a disassortative network in cell samples treated with SKF9635 (Fig. 6a). Supplementary Video 10. A representative sample of cells treated with NaN3 and imaged by SIM for the comparison of ER phenotypes. Sequential images (43.5 s at 1.5 s per frame) demonstrated fragmentation of the ER tubular network, which resulted from the lack of ATP (Fig. 6a). Supplementary Video 11. A control (healthy) cell treated with the chemical vehicle DMSO, imaged by SIM for the comparison of ER phenotypes. Sequential images (43.5 s at 1.5 s per frame) demonstrated rapid reshaping and complex connectivity in a control (healthy) cell sample (Fig. 6a). Supplementary Video 12. A 3D projection view of a U2OS cell demonstrates that the tubular ER is largely flat and singly layered. U2OS cells expressing sec61-mCherry were imaged by Airyscan microscopy. Supplementary Video 13. A 3D projection view of a COS-7 cell demonstrates that the tubular ER is largely flat and singly layered. COS-7 cells expressing mEmerald-Sec61b were imaged by SIM. Supplementary Video 14. A 3D projection view of the segmented ER structure of the sample in Supplementary Video 13.
Source Data Fig. 2
Statistical source data.
Source Data Fig. 3
Statistical source data.
Source Data Fig. 4
Statistical source data.
Source Data Fig. 5
Statistical source data.
Source Data Fig. 6
Statistical source data.
Source Data Extended Data Fig. 4
Statistical source data.
Source Data Extended Data Fig. 5
Statistical source data.
Source Data Extended Data Fig. 7
Statistical source data.
Source Data Extended Data Fig. 8
Unprocessed western blots.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Lu, M., Christensen, C.N., Weber, J.M. et al. ERnet: a tool for the semantic segmentation and quantitative analysis of endoplasmic reticulum topology. Nat Methods 20, 569–579 (2023). https://doi.org/10.1038/s41592-023-01815-0