As editors at Nature Methods, we see trends in research methods rise and fall over time, with each trend carrying its own momentum and excitement. In microscopy, we would be hard pressed to identify a trend that has so captured the current zeitgeist as deep learning. Indeed, not since the introduction of super-resolution microscopy methods over a decade ago has a class of tools had such potential to disrupt microscopy as we know it.

The power of conventional machine learning, especially for image analysis, is well-established in the microscopy community. In contrast, the benefits of deep learning have emerged more recently, although the advantages have been known in medical imaging for some time. Generally speaking, applications of deep learning in microscopy can be divided into two broad classes — methods used for image analysis, which include classification, phenotyping, segmentation, tracking and more; and those used for image formation or reconstruction. The papers included in this focus issue highlight advances in both classes of techniques and provide means for users to implement these tools.

On the image analysis side, the focus includes a Review by David van Valen and colleagues that introduces deep learning, summarizes major applications and includes specific use-cases for interested readers. Two Analyses describe large-image analysis competitions. The first, from Anne Carpenter and colleagues, presents the 2018 Data Science Bowl, an image analysis competition for automated segmentation of cell nuclei from diverse types of 2D images. The second, from Emma Lundberg and colleagues, describes the results of the Human Protein Atlas Image Classification Competition for automated classification of subcellular protein localizations from fluorescence images. In both cases, the top performers surpassed the previous field standard by using deep learning, and the winning strategies could be mined for insight into how to further improve these types of tasks.

On the image reconstruction side, a Perspective from Loic Royer and colleagues covers recent innovations that deep learning has brought to generating fluorescence microscopy images, including approaches for generating super-resolution images from low resolution or sparse images, methods for creating fluorescence images from phase-contrast images, going from low to high signal-to-noise images and more, and looks toward the future of these tools. In addition, an Article from Aydogan Ozcan and colleagues describes DeepZ, a tool for generating a 3D fluorescence image directly from a 2D snapshot.

Also included in this focus issue is a Correspondence from Christophe Zimmer and colleagues describing ImJoy, an open-source browser-based platform designed to facilitate widespread reuse of deep learning solutions for image analysis and beyond. And to round things out, a Perspective from Fred Hamprecht, Anna Kreshuk and colleagues presents ilastik, the popular and powerful software tool that has brought machine-learning-based image analysis to biologists. Although ilastik stands out for being the only tool featured in this focus issue that does not yet incorporate deep learning, we think it fits in as a testament to the power of conventional machine learning.

Although the general focus of our pieces is on fluorescence microscopy in cell biology, deep learning is also applied much more broadly than what is presented in these papers, for example in neuroscience where many deep-learning-based techniques have been developed for such applications as segmentation and tracing of neurons in fluorescence and electron microscopy images for connectomics, or functional studies tracking animals and their body parts for behavioral analyses, and much more.

While the pieces in this focus promote the use of deep learning in many aspects of microscopy, the authors also provide balanced and nuanced descriptions of the limitations of these approaches. As with all new approaches, it is crucial to separate performance from hype and regard these methodologies with the proper amount of caution.

Deep learning methods have been criticized for being a ‘black box’, where an input is entered and an output is delivered with no insight into the process. However, the black box issue is not true in all cases, and what information can be mined from within neural networks is an active area of research in computer science. Another concern is the huge amount of data, which often require manual annotation, typically needed to properly train deep learning models. This is also an area of active research, where the limits of training data needed for high performance, as well as augmentation strategies and transfer learning, are being explored.

Separately, the use of deep learning in image formation and reconstruction has been questioned for the possibility of introducing artifacts, especially when generating images of structures that were never seen by the model during training. This is an important concern, particularly since rare events are often those that are of the most interest to biologists, and must be addressed within the context of a given experiment. Proper training is also essential, as overtraining and undertraining can negatively impact model output and potentially affect biological interpretation of results. Again, as with all new methods, we suggest that new users do their homework, enlist experts when possible, include proper controls and validate findings with alternative methods when possible. Over time, with careful use on diverse tasks, the true strengths and weaknesses of these approaches on biological applications will be further revealed.

In handling many papers in this area, and with the help of peer reviewers, we have come to identify criteria that we think are important for publishing advances in deep learning in imaging. These include a detailed explanation of the artificial neural network used in the paper, along with justification as to why it was chosen. We also expect a clear description of model training, data on how training affects performance and tests for overtraining. We ask researchers to test the extent to which a trained model can be transferred from data taken on one microscope or sample type to that taken on another microscope or different sample, as well as include quantitative tests of accuracy, ideally in comparison to ground truth. Testing the above on a diversity of datasets is also recommended. Our standard policies on data and software sharing must also be met, and user-friendly tools to access trained models are desirable. We welcome feedback on these requirements, and retain the flexibility to update them over time.

To end, we invite you to explore our curated collection of manuscripts published in Nature Research journals that highlight recent applications of deep learning in microscopy (www.nature.com/collections/deeplearninginmicroscopy).