For most people, a picture is worth a thousand words, but in my world, it can be worth a patient’s life. When I took over care of the cardiology ward one Monday morning a few years ago, I couldn’t understand why one patient with pneumonia was still having fevers after a week on antibiotics. My team and I trooped down to the radiology reading room to have another look at a chest scan the patient had had earlier. After consulting with the radiologists, we determined that what originally appeared to be pneumonia was actually a blood clot in the lung. I had a sinking feeling when I realized that the patient had been getting antibiotics instead of blood thinners. We quickly remedied the situation and the patient did fine, but an untreated clot could have killed him. That experience reminded me just how much medical imaging impacts diagnosis and management.

Credit: Tom Seawell Productions, LLC

In radiology, cardiology and other fields, the best tools we currently have to interpret clinical images are dedicated, highly trained physicians. As a cardiovascular imaging subspecialist, I’m one of them. We do a pretty good job, but to err is human. It is incredibly challenging to interpret image after image accurately and reliably for millions of patients. The quest to decrease diagnostic error is one of the main reasons I have built a research program to bring machine learning—where computers help us acquire and interpret data—to medical imaging.

Reducing medical error is not my sole motivator, however. I spent my early research years studying cardiovascular genetics in the zebrafish model. The work involved countless hours behind a microscope. This was also when our ability to sequence genomes scaled exponentially. However, trying to obtain useful images of zebrafish hearts with different genetic mutations taught me that the genetic information we can gather is only as good as the physical characteristic we can trace it to. Sequencing has allowed us to fine tune the origins of disease and even which treatments to administer, but there hasn’t been a similar leap forward for those of us in the imaging field. The quality of images has vastly improved, but we will only deliver on the promise of precision medicine when we can make sense of biomedical imaging data with the same accuracy, precision and scale with which we now mine the human genome.

As an undergrad at the Massachusetts Institute of Technology, I studied biology but also bioengineering, math and coding. I was especially interested in a type of machine learning called deep neural networks because this method had been getting good at finding patterns in imaging. When a powerful software for building and training neural networks called TensorFlow became freely available in 2015, I realized I could use it to start solving some of the imaging problems that I had encountered.

I started with cardiac ultrasound. I’ve used it in my research and I use it to care for patients every day, so I could see where deep neural networks could help improve its interpretation. Although other scientists were beginning to use neural networks on still photographs in medicine, the noisy ultrasound movies were considered too challenging for nascent machine learning tools. So, I combined my research experience, programming knowledge and clinical expertise to design and train deep neural networks for cardiac ultrasound. I strove to make our models clinically relevant by using images from actual patients, and getting the models to work fast enough so they could someday become staples in the clinic.

My work so far has answered some questions and led to new ones. Should we teach deep learning models to read medical imaging studies in the same way that we train human experts? Might machine learning algorithms detect patterns in images that a human eye never could? What happens to our clinical practice when computers become better than their human counterparts?

I’m often asked whether I’m ‘putting myself out of a job’ by helping develop computational imaging tools. I’m sure our jobs will change when new tools come along, as always, but they won’t disappear. Also, physicians are honor bound to heal, so if new technologies can improve diagnosis and drive new treatment discoveries, we are obligated to use them. And we physicians have to be the ones to use them—by helping set the research agenda, curating data and validating results. I’m now leading a field of translational research that didn’t exist at the onset of my career. I hope to continue combining medicine and data science to create a more picture-perfect version of healthcare.