Last author

In the quest to understand how the brain processes images, neuroscientists are developing something that sounds like science fiction — a tool for decoding brain activity. Until now, scientists could crudely decode the brain activity that resulted from viewing simple predetermined images, such as faces or places. But on page 352, Jack Gallant, a neuroscientist at the University of California, Berkeley, and his colleagues describe a computational model of the early visual system that can be used to identify natural images being viewed by a person for the first time. Gallant tells Nature that it may soon be possible to decode a person's visual experience from brain activity alone.

Is your ultimate goal to build a “brain decoder”?

No, the tool was not the ultimate goal. We want to build a quantitative and predictive model of the brain's visual system, and a brain decoder provides a useful way to test the model.

How were you able to decode novel images?

Our work builds on knowledge gained from more than 50 years of research in many labs. We used a large sample of natural images as stimuli and then constructed a model that links these stimuli to brain responses obtained by functional magnetic resonance imaging — an indicator of brain activity based on blood flow — in the early visual areas of the brain. The model allows us to predict the brain activity in these areas that would be elicited by any arbitrary image. We tested the quality of the model by evaluating the predictions using a separate set of images.

Do you think a model of the entire visual processing circuit will be in place by the time you retire?

I hope so. We know of 30–40 visual areas in the brain, but we currently have good models of how they work for only two of these. It's too early to do meaningful deductive research on visual areas that are essentially unknowns. That's why we use a 'black box' approach — linking random stimuli to brain activity — rather than testing individual hypotheses one at a time.

Do the privacy and ethical issues of 'brain decoding' concern you?

Yes. In science, whenever you learn more about a biological system, you often have to ask if the knowledge gained is worth its potential misuse. A functional model of the brain, our goal, would be a valuable contribution to neuroscience. However, once we have a model, anyone can use it to build a decoder. We're still very far from any potential application, but down the road, ethical and privacy issues must be dealt with.