Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Deep learning-inferred multiplex immunofluorescence for immunohistochemical image quantification

Nature Machine Intelligence volume 4, pages 401–412 (2022)Cite this article

Subjects

A preprint version of the article is available at bioRxiv.

Abstract

Reporting biomarkers assessed by routine immunohistochemical (IHC) staining of tissue is broadly used in diagnostic pathology laboratories for patient care. So far, however, clinical reporting is predominantly qualitative or semi-quantitative. By creating a multitask deep learning framework, DeepLIIF, we present a single-step solution to stain deconvolution/separation, cell segmentation and quantitative single-cell IHC scoring. Leveraging a unique de novo dataset of co-registered IHC and multiplex immunofluorescence (mpIF) staining of the same slides, we segment and translate low-cost and prevalent IHC slides to more informative, but also more expensive, mpIF images, while simultaneously providing the essential ground truth for the superimposed brightfield IHC channels. A new nuclear-envelope stain, LAP2beta, with high (>95%) cell coverage is also introduced to improve cell delineation/segmentation and protein expression quantification on IHC slides. We show that DeepLIIF trained on clean IHC Ki67 data can generalize to noisy images as well as other nuclear and non-nuclear markers.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Overview of the DeepLIIF pipeline and sample input IHCs (different brown/DAB markers—BCL2, BCL6, CD10, CD3/CD8, Ki67) with corresponding DeepLIIF-generated haematoxylin/mpIF modalities and classified segmentation masks.
Fig. 2: Qualitative and quantitative analysis of DeepLIIF against state-of-the-art semantic segmentation models tested on the BC dataset.
Fig. 3: Qualitative and quantitative analysis of DeepLIIF against state-of-the-art semantic segmentation models tested on the NuClick dataset and four sample images from the LYON19 challenge dataset.
Fig. 4: IHC quantification of four cancer type images taken from the Protein Atlas IHC Ki67 dataset.
Fig. 5: A t-SNE plot of tested IHC markers on DeepLIIF.

Similar content being viewed by others

Data availability

The complete IHC Ki67 BC Dataset with manual annotations is available at https://sites.google.com/view/bcdataset. The complete lymphocytes detection IHC CD3/CD8 (LYON challenge) dataset is available at https://zenodo.org/record/3385420#.XW-6JygzYuW. The NuClick IHC annotations for crops from the LYON19 dataset can be found at https://warwick.ac.uk/fac/sci/dcs/research/tia/data/nuclick/ihc_nuclick.zip. The DLBCL-Morph dataset with BCL2, BCL6, MUM1, MYC and CD10 IHCs is accessible at https://stanfordmedicine.box.com/s/ub8e0wlhsdenyhdsuuzp6zhj0i82xrb1. The high-resolution tiff images for TP53 IHCs can be downloaded from https://www.proteinatlas.org/ENSG00000141510-TP53. All our internal training and testing data (acquired under IRB protocol approval no. 16-1683) and source data underlying the figures (in excel files), along with the pretrained models, are available at https://zenodo.org/record/4751737#.YV379XVKhH4. Source data are provided with this paper.

Code availability

All code was implemented in Python using PyTorch as the primary deep learning package. All code and scripts to reproduce the experiments of this paper are available at https://github.com/nadeemlab/DeepLIIF and releases are available at https://doi.org/10.5281/zenodo.5553268. For convenience, we have also included docker file as well as Google CoLab Demo project (in case someone does not have access to a GPU and wants to run their images directly via the CoLab project). The Google CoLab project can be accessed at https://colab.research.google.com/drive/12zFfL7rDAtXfzBwArh9hb0jvA38L_ODK?usp=sharing. We have also provided multi-GPU training code as well as highly optimized inference modules implemented in Torchserve as well as Dask and Torchscript. A cloud-native platform with a user-friendly web interface is available at https://deepliif.org for users to upload input images, visualize and download IHC quantification results. The interactive deep learning module for performing multiplex immunofluorescence cell segmentation is available at https://github.com/nadeemlab/impartial.

References

  1. Vahadane, A. et al. Structure-preserving color normalization and sparse stain separation for histological images. IEEE Trans. Med. Imag. 35, 1962–1971 (2016).

    Article  Google Scholar 

  2. Abousamra, S. et al. Weakly-supervised deep stain decomposition for multiplex IHC images. In Proc. 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI) 481–485 (IEEE, 2020).

  3. Fassler, D. J. et al. Deep learning-based image analysis methods for brightfield-acquired multiplex immunohistochemistry images. Diagn. Pathol. 15, 100 (2020).

    Article  Google Scholar 

  4. Chang Colin Tan, W. et al. Overview of multiplex immunohistochemistry/immunofluorescence techniques in the era of cancer immunotherapy. Cancer Commun. 40, 135–153 (2020).

    Article  Google Scholar 

  5. Yeong, J. et al. Multiplex immunohistochemistry/immunofluorescence (mIHC/IF) for PD-L1 testing in triple-negative breast cancer: a translational assay compared with conventional IHC. J. Clin. Pathol. 73, 557–562 (2022).

    Article  Google Scholar 

  6. Lu, S. et al. Comparison of biomarker modalities for predicting response to PD-1/PD-L1 checkpoint blockade: a systematic review and meta-analysis. JAMA Oncol. 5, 1195–1204 (2019).

    Article  Google Scholar 

  7. Caruana, R. Multitask learning. Mach. Learn. 28, 41–75 (1997).

    Article  Google Scholar 

  8. Kumar, N. et al. A dataset and a technique for generalized nuclear segmentation for computational pathology. IEEE Trans. Med. Imag. 36, 1550–1560 (2017).

    Article  Google Scholar 

  9. Huang, Z. et al. BCData: A large-scale dataset and benchmark for cell detection and counting. In Proc. Medical Image Computing and Computer Assisted Intervention—MICCAI 2020 (eds Martel, A. L. et al.) 289–298 (Springer, 2020).

  10. Koohbanani, N. A., Jahanifar, M., Tajadin, N. Z. & Rajpoot, N. NuClick: a deep learning framework for interactive segmentation of microscopic images. Med. Image Anal. 65, 101771 (2020).

    Article  Google Scholar 

  11. Swiderska-Chadaj, Z. et al. Learning to detect lymphocytes in immunohistochemistry with deep learning. Med. Image Anal. 58, 101547 (2019).

    Article  Google Scholar 

  12. Kirillov, A., He, K., Girshick, R. & Dollár, P. A unified architecture for instance and semantic segmentation. https://presentations.cocodataset.org/COCO17-Stuff-FAIR.pdf (2017).

  13. Chaurasia, A. & Culurciello, E. LinkNet: exploiting encoder representations for efficient semantic segmentation. In Proc. 2017 IEEE Visual Communications and Image Processing (VCIP) 1–4 (IEEE, 2017).

  14. He, K., Gkioxari, G., Dollár, P. & Girshick, R. Mask R-CNN. In Proc. IEEE International Conference on Computer Vision 2961–2969 (IEEE, 2017).

  15. Zhou, Z., Rahman Siddiquee, M. M., Tajbakhsh, N. & Liang, J. UNet++: a nested U-Net architecture for medical image segmentation. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support 3–11 (Springer, 2018).

  16. Isensee, F., Jaeger, P. F., Kohl, S. A. A., Petersen, J. & Maier-Hein, K. H. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat. Methods 18, 203–211 (2021).

    Article  Google Scholar 

  17. Xie, W., Alison Noble, J. & Zisserman, A. Microscopy cell counting and detection with fully convolutional regression networks. Comput. Methods Biomech. Biomed. Eng. Imag. Vis. 6, 283–292 (2018).

    Article  Google Scholar 

  18. Chen, L.-C., Papandreou, G., Schroff, F. & Adam, H. Rethinking atrous convolution for semantic image segmentation. Preprint at https://arxiv.org/abs/1706.05587 (2017).

  19. Ram, S. & Rodríguez, J. J. Size-invariant detection of cell nuclei in microscopy images. IEEE Trans. Med. Imag. 35, 1753–1764 (2016).

    Article  Google Scholar 

  20. Sirinukunwattana, K. et al. Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images. IEEE Trans. Med. Imag. 35, 1196–1206 (2016).

    Article  Google Scholar 

  21. Alemi Koohbanani, N., Jahanifar, M., Zamani Tajadin, N. & Rajpoot, N. NuClick: a deep learning framework for interactive segmentation of microscopic images. Med. Image Anal. 65, 101771 (2020).

    Article  Google Scholar 

  22. Negahbani, F. et al. PathoNet introduced as a deep neural network backend for evaluation of Ki-67 and tumor-infiltrating lymphocytes in breast cancer. Sci. Rep. 11, 8489 (2021).

    Article  Google Scholar 

  23. Digre, A. & Lindskog, C. The Human Protein Atlas—spatial localization of the human proteome in health and disease. Protein Sci. 30, 218–233 (2021).

    Article  Google Scholar 

  24. Vrabac, D. et al. DLBCL-Morph: morphological features computed using deep learning for an annotated digital DLBCL image set. Sci. Data 8, 135 (2021).

    Article  Google Scholar 

  25. Tschuchnig, M. E., Oostingh, G. J. & Gadermayr, M. Generative adversarial networks in digital pathology: a survey on trends and future potential. Pattern 1, 100089 (2020).

    Article  Google Scholar 

  26. Rivenson, Y., de Haan, K., Wallace, W. D. & Ozcan, A. Emerging advances to transform histopathology using virtual staining. BME Frontiers 2020, 9647163 (2020).

    Article  Google Scholar 

  27. Liu, D. et al. Unsupervised instance segmentation in microscopy images via panoptic domain adaptation and task re-weighting. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 4243–4252 (IEEE, 2020).

  28. Cohen, J. P., Luck, M. & Honari, S. Distribution matching losses can hallucinate features in medical image translation. In Proc. International Conference on Medical Image Computing and Computer-Assisted Intervention—MICCAI 2018 529–536 (Springer, 2018).

  29. Burlingame, E. A. et al. SHIFT: speedy histological-to-immunofluorescent translation of a tumor signature enabled by deep learning. Sci. Rep. 10, 17507 (2020).

    Article  Google Scholar 

  30. Mercan, C. et al. Virtual staining for mitosis detection in breast histopathology. In Proc. IEEE International Symposium on Biomedical Imaging (ISBI) 1770–1774 (IEEE, 2020).

  31. de Haan, K. et al. Deep learning-based transformation of H&E stained tissues into special stains. Nat. Commun. 12, 4884 (2021).

    Article  Google Scholar 

  32. Borovec, J. et al. ANHIR: automatic non-rigid histological image registration challenge. IEEE Trans. Med. Imag. 39, 3042–3052 (2020).

    Article  Google Scholar 

  33. Martinez, N., Sapiro, G., Tannenbaum, A., Hollmann, T. J. & Nadeem, S. ImPartial: partial annotations for cell instance segmentation. Preprint at bioRxiv https://doi.org/10.1101/2021.01.20.427458 (2021).

  34. Girshick, R. Fast R-CNN. In Proc. IEEE International Conference on Computer Vision (ICCV) 1440–1448 (IEEE, 2015).

  35. Isola, P., Zhu, J.-Y., Zhou, T. & Efros, A. A. Image-to-image translation with conditional adversarial networks. In Proc. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 5967–5976 (IEEE, 2017).

  36. Ronneberger, O., Fischer, P. & Brox, T. U-Net: convolutional networks for biomedical image segmentation. In Proc. Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015 234–241 (Springer, 2015).

  37. Goodfellow, I. et al. Generative adversarial nets. In Advances in Neural Information Processing Systems 2672–2680 (NIPS, 2014).

  38. Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. Preprint at https://arxiv.org/abs/1412.6980 (2014).

  39. Miyato, T., Kataoka, T., Koyama, M. & Yoshida, Y. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations (2018).

Download references

Acknowledgements

This project was supported by an MSK Cancer Center Support Grant/Core Grant (P30 CA008748) and in part by MSK DigITs Hybrid Research Initiative and NSF grants nos. CNS1650499, OAC1919752 and ICER1940302.

Author information

Authors and Affiliations

  1. Department of Computer Science, Stony Brook University, Stony Brook, NY, USA

    Parmida Ghahremani & Arie Kaufman

  2. Department of Pathology, Memorial Sloan Kettering Cancer Center, New York, NY, USA

    Yanyun Li, Rami Vanguri & Travis J. Hollmann

  3. Department of Pathology, Stanford University, Stanford, CA, USA

    Noah Greenwald & Michael Angelo

  4. Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, NY, USA

    Saad Nadeem

Authors
  1. Parmida Ghahremani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yanyun Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Arie Kaufman
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rami Vanguri
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Noah Greenwald
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Michael Angelo
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Travis J. Hollmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Saad Nadeem
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

S.N., T.J.H. and P.G. conceived the study and designed the experiments. S.N. and P.G. wrote the computer codes and performed the experimental analysis. Y.L. and T.J.H. performed the IHC and multiplex staining. M.A., T.J.H. and N.G. conceived the Lap2BETA idea for nuclear envelope staining. P.G., S.N., A.K. and R.V. analysed the results. S.N., T.J.H. and P.G. prepared the manuscript with input from all co-authors. S.N. supervised the research.

Corresponding authors

Correspondence to Travis J. Hollmann or Saad Nadeem.

Ethics declarations

Competing interests

The authors declare no competing interests

Peer review

Peer review information

Nature Machine Intelligence thanks Phedias Diamandis and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 DeepLIIF architecture diagram.

Overview of DeepLIIF. The network consists of a generator and a discriminator component. It uses ResNet-9block generator for generating the modalities including Hematoxylin, mpIF DAPI, mpIF Lap2, and mpIF Ki67 and UNet512 generator for generating the segmentation mask. In the segmentation component, the generated masks from IHC, Hematoxylin, mpIF DAPI, and mpIF Lap2 representations are averaged with predefined weights to create the final segmentation mask. The discriminator component consists of the modalities discriminator module and segmentation discriminator module.

Extended Data Fig. 2 Synthetic IHC generation pipeline.

Overview of synthetic IHC image generation. (a) A training sample of the IHC-generator model. (b) Some samples of synthesized IHC images using the trained IHC-Generator model. The Neg-to-Pos shows the percentage of the negative cells in the segmentation mask converted to positive cells.

Extended Data Fig. 3 Qualitative and quantitative analysis of DeepLIIF against detection-only deep learning models.

Qualitative and quantitative analysis of DeepLIIF against detection models on the testing set of the BC Data 9. (a) An example IHC image from the BC Data testing set, the generated modalities, segmentation mask overlaid on the IHC image, and the detection mask generated by DeepLIIF. (b) The detection masks generated by the detection models. In the detection mask, the center of a detected positive cell is shown with red dot and the center of a detected negative cell is shown with blue dot. We show the missing positive cells in cyan bounding boxes, the missing negative cells in yellow bounding boxes, the wrongly detected positive cells in blue bounding boxes, the wrongly detected negative cells in pink bounding boxes. (c) The detection accuracy is measured by getting average of precision (\(\frac{TP}{TP+FP}\)), recall (\(\frac{TP}{TP+FN}\)), and f1-score (\(\frac{2\times precision\times recall}{precision+recall}\)) between the predicted detection mask of each class and the ground-truth mask of the corresponding class. A predicted point is regarded as true positive if it is within the region of a ground-truth point with a predefined radius (we set it to 10 pixels in our experiment which is similar to the predefined radius in9). Centers that have been detected more than once are considered as false positive. Evaluation of all scores show that DeepLIIF outperforms all state-of-the-art models.

Extended Data Fig. 4 Quantitative and qualitative analysis of DeepLIIF for modality inference.

Quantitative and qualitative analysis of DeepLIIF for modality inference. (a) The Quantitative analysis of the synthetic data against the real data using MSE, SSIM, Inception Score, and FID. The low value of MSE (close to 0) and the high value of SSIM (close to 1) shows that the model generates high quality synthetic images similar to real images. (b) Visualization of first two components of PCA applied to synthetic and real images. We first, calculated a feature vector for each image using VGG16 model and then we applied PCA on the calculated feature vectors and visualized the first two components. As shown in the figure, the synthetic image data points have the same distribution as the real image data points, showing that the generated images by the model have the same characteristics as the real images. (c) The original/real and model-inferred modalities of two samples taken from Bladder and Lung tissues are shown side-by-side.

Extended Data Fig. 5 DeepLIIF results on microscope snapshots.

Microscopic snapshots of IHC images stained with two different markers along with inferred modalities and generated classified segmentation mask (top: Microscope Snapshot for IHC Ki67 with inferred modalities and generated classified segmentation mask. bottom: Microscopic snapshots for IHC PDL1 with inferred modalities and generated classified segmentation mask).

Extended Data Fig. 6 DeepLIIF results on public IHC CD3/CD8 dataset.

Some examples from LYON19 Challenge Dataset 11. The generated modalities and classified segmentation mask for each sample are in a separate row.

Extended Data Fig. 7 DeepLIIF results on a different IHC Ki67 dataset with annotations based on consensus of multiple pathologists.

Samples taken from the PathoNet IHC Ki67 breast cancer dataset 22 along with the inferred modalities and classified segmentation mask marked by manual centroid annotations created from consensus of multiple pathologists. The IHC images were acquired in low-resource settings with microscope camera. In each row, the sample IHC image along with the inferred modalities are shown. The overlaid classified segmentation mask generated by DeepLIIF with manual annotations are shown in the furthest right column. The blue and red boundaries represent the negative and positive cells predicted by the model, while the pink and yellow dots show the manual annotations of the negative and positive cells, respectively.

Extended Data Fig. 8 DeepLIIF results on DLBCL IHC markers.

Examples of tissues stained with various markers. The top box shows sample tissues stained with BCL2, BCL6, CD10, MYC, and MUM1 from DLBCL-morph dataset 24. The bottom box shows sample images stained with TP53 marker from the Human Protein Atlas 23. In each row, the first image on the left shows the original tissue stained with a specific marker. The quantification score computed by the classified segmentation mask generated by DeepLIIF is shown on the top of the whole tissue image, and the predicted score by pathologists is shown on the bottom. In the following images of each row, the modalities and the classified segmentation mask of a chosen crop from the original tissue are shown.

Extended Data Fig. 9 Analysis of LAP2Beta effectiveness in DeepLIIF model.

Analysis of Lap2beta effectiveness. (a) LAP2beta coverage for normal tissues. LAP2beta immunohistochemistry reveals nuclear envelope-specific staining in the majority of cells in spleen (99.98%), colon (99.41%), pancreas (99.50%), placenta (76.47%), testis (95.59%), skin (96.74%), lung (98.57%), liver (98.70%), kidney (95.92%) and lymph node (99.86%). (b) A qualitative comparison of DeepLIIF against noLap2 model. (c) Some example IHC images. The first image in each row shows the input IHC image. In the second image, the generated mpIF Lap2 image is overlaid on the classified/segmented IHC image. The third and fourth images show the segmentation mask, respectively, generated by DeepLIIF and noLap2.

Extended Data Fig. 10 DeepLIIF generalizes out-of-the-box to H&E images.

Application of DeepLIIF on some H&E sample images taken from MonuSeg Dataset [8]. We tested DeepLIIF, trained solely on IHC images stained with Ki67 marker, on H&E images. In each row, the inferred modalities and the segmentation mask overlaid on the original H&E sample are shown.

Supplementary information

Supplementary Information

Protocol for de novo IHC and multiplex immunofluorescence staining.

Reporting Summary

Source data

Source Data Fig. 2

Statistical source data.

Source Data Fig. 3

Statistical source data.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ghahremani, P., Li, Y., Kaufman, A. et al. Deep learning-inferred multiplex immunofluorescence for immunohistochemical image quantification. Nat Mach Intell 4, 401–412 (2022). https://doi.org/10.1038/s42256-022-00471-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-022-00471-x

This article is cited by

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics