Editor's Summary

4 September 2008

Big data: science in the petabyte era


In Nature this week, features and opinion pieces on one of the most daunting challenges facing modern science: how to cope with the flood of data now being generated. A petabyte is a lot of memory, however you say it — a quadrillion, 1015, or tens of thousands of trillions of bytes. But that is the currency of 'big data'. We visited the Sanger Institute's supercomputing centre, and its petabyte of capacity. Wikipedia's success shows how well the 'wiki' concept of open-access editing can work. It could work too as a way of coping with the data flows of modern biology. The world's leading search engine is ten this month. Eleven years ago few would have predicted Google's domination: undaunted we ask scientists and business people to try to predict the next big thing, a Google for the petabyte era. Digital data are easily shared, and just as easily wiped or lost. The problem of keeping on-line data accessible is especially difficult for the smaller lab. In Books & Arts, Felice Frankel and Rosalind Reid champion the cause of data visualization as a way of finding meaning in an otherwise daunting data stream. From the 1700s to the mid 1950s, most 'computers' were human. Best known were the 'Harvard computers', a group of women working from the 1880s until the 1940s, at the Harvard College Observatory. Employed to classify stars captured on millions of photographic plates, some of the 'computers' made significant contributions to science. Online databases are a vital outlet for publishing the data being produced by biological research. But the data need to be properly organized. This is the role of the biocurator, but as a team of authors from 15 of the world's major online research resources explains, biocuration is now sadly neglected. An aspect of the data boom with a political dimension is the environment: how much data to collect, how much money to spend. For 'Big data' online, go to http://www.nature.com/news/specials/bigdata/ and to http://www.nature.com/podcast.

EditorialCommunity cleverness required

Researchers need to adapt their institutions and practices in response to torrents of new data — and need to complement smart science with smart searching.

doi:10.1038/455001a

NewsBig data: The next Google

Ten years ago this month, Google's first employee turned up at the garage where the search engine was originally housed. What technology at a similar early stage today will have changed our world as much by 2018? Nature asked some researchers and business people to speculate - or lay out their wares. Their responses are wide ranging, but one common theme emerges: the integration of the worlds of matter and information, whether it be by the blurring of boundaries between online and real environments, touchy-feely feedback from a phone or chromosomes tucked away on databases.

Bill Buxton

doi:10.1038/455008a

ColumnBig data: Data wrangling

Collecting and releasing environmental data have stirred up controversy in Washington, says David Goldston, and will continue to do so.

David Goldston

doi:10.1038/455015a

News FeatureBig data: Welcome to the petacentre

What does it take to store bytes by the tens of thousands of trillions? Cory Doctorow meets the people and machines for which it's all in a day's work.

Cory Doctorow

doi:10.1038/455016a

News FeatureBig data: Wikiomics

Pioneering biologists are trying to use wiki-type web pages to manage and interpret data, reports Mitch Waldrop. But will the wider research community go along with the experiment?

Mitch Waldrop

doi:10.1038/455022a

CommentaryBig data: How do your data grow?

Scientists need to ensure that their results will be managed for the long haul. Maintaining data takes big organization, says Clifford Lynch.

doi:10.1038/455028a

Books and ArtsBig data: Distilling meaning from data

Buried in vast streams of data are clues to new science. But we may need to craft new lenses to see them, explain Felice Frankel and Rosalind Reid.

doi:10.1038/455030a

EssayBig data: The Harvard computers

The first mass data crunchers were people, not machines. Sue Nelson looks at the discoveries and legacy of the remarkable women of Harvard's Observatory.

doi:10.1038/455036a

FeatureBig Data: The future of biocuration

To thrive, the field that links biologists and their data urgently needs structure, recognition and support.

doi:10.1038/455047a