Artificial intelligence (AI) and machine learning are poised to revolutionize the way biologists acquire and interact with experimental data. In biomedical imaging, such approaches have largely been focused on improving and automating the analysis of acquired images. For example, machine learning has been used to address the challenging problems of image phenotyping, object segmentation and tracking, and even disease classification and diagnosis.

Matthew Rosen, a physicist at the Massachusetts General Hospital Martinos Center for Biomedical Imaging, and Bo Zhu, a postdoctoral fellow in Rosen's laboratory, sought to use machine learning to tackle a different problem in the imaging field: image reconstruction. Image reconstruction is necessary for many types of imaging, including magnetic resonance imaging (MRI), positron emission tomography (PET), and other tomographic imaging modalities. As such images are acquired, they are encoded in an intermediate representation of the imaged object in what is known as the sensor domain. In order for this intermediate representation to lead to an image, the function used to encode the image into the sensor domain, or the encoding scheme, must be inverted in a process called reconstruction.

Conventional approaches for image reconstruction are imperfect: because knowledge of the exact inverse transform is not always possible, methods require the use of approximations by chains of highly tuned signal-processing modules, which can be error prone, especially for realistic, noisy data. This handcrafted solution to the problem of image reconstruction inspired Rosen's team to tackle the problem with AI, and Zhu's background in speech recognition made him uniquely suited for the task. Zhu “was startled to realize that many of the core elements of acoustic modeling that previously required domain-expert design were being completely overturned by more flexible neural network models” after he left the field of machine learning in speech recognition to study the physics of MRI. “I wondered if this paradigm could be similarly advantageous in the field of medical imaging, and in particular image reconstruction.”

To this end, the team developed automated transform by manifold approximation (AUTOMAP), a deep learning approach to solving the image reconstruction problem. Deep learning is a form of machine learning that uses a synthetic neural network architecture composed of interconnected nodes in multiple layers that can be trained on input data to perform a task. In this case, the authors generated their training data by taking a large set of images from natural scenes and reverse-encoding them into the sensor domain via a desired encoding function, making paired data sets. The AUTOMAP network was then trained through this supervised learning approach, which allowed the optimal computational strategy for image reconstruction to emerge. They then applied the trained neural networks to MRI images of human brains. Amazingly, they found that training on images of objects such as animals and plants (and not MRI images of brains) allowed accurate reconstructions of brain MRI images for three out of the four commonly used encoding schemes they tested, highlighting the robustness of the approach to the type of training data. Furthermore, the images reconstructed by AUTOMAP were more accurate than those reconstructed by conventional methods, when tested on simulated data with a known ground truth.

An artistic representation of deep learning for image reconstruction. Credit: B. Zhu and M. Rosen

Although AUTOMAP is a success, its development and implementation were not without challenges. Zhu notes that “neural network training is a computationally intensive task, especially in the context of research, as we need to experimentally iterate over multiple network architectures and hyperparameters,” and says that computational resources received from the Massachusetts General Hospital and Brigham & Women's Hospital Center for Clinical Data Sciences and from NVIDIA were critical for their success. He also notes that the other authors of the work represent a highly interdisciplinary team, with expertise in areas ranging from statistics to imaging physics, and that this was necessary for the project to succeed.

AUTOMAP is poised to be the basis for exciting future research, including the development of new image acquisition modes that allow faster imaging without compromising image quality. Rosen also notes that some of the future advances will address technical issues, such as the effects of patient motion on data acquisition, or data-handling issues such as those encountered with large, 3D data sets.

“We're of course interested in making it easy for people to apply AUTOMAP to all kinds of problems!” said Rosen when asked whether user-friendly software was available. He notes that they are currently developing “a complete workable distribution that is much more turn-key.” Taken together, this work represents a great step forward for both image reconstruction and the use of artificial intelligence in bioimaging.