Access to phone records is all over the news at the moment, following media revelations of massive snooping by US intelligence agencies. Collecting phone records is in itself nothing new, and is legitimate — scientists have long done so for research. What is unprecedented is the government weaving together multiple huge data sets for secret state surveillance.

When researchers obtain electronic records of millions of people’s calls for social science — including data on who called whom, when and from where, but not actual conversations — they must first make a strong case for why they need that information, and must comply with multiple layers of oversight and safeguards such as anonymization of numbers. The same goes for research use of health records or any other private personal data. And there is a good reason: to protect privacy.

Privacy concerns are at the heart of the uproar over how the US National Security Agency (NSA) has secretly required telephone companies to hand over similar phone records on almost every US resident. The US government is also vacuuming up billions of e-mails and other Internet communications from traffic outside the United States — all in the name of law enforcement and the war on terror.

What is perhaps most concerning, apart from the mind-boggling scale of the snooping, is that until last week, the very existence of these programmes was secret. Since the revelations, US President Barack Obama has defended this secrecy, on the grounds that if terrorists knew that the government was monitoring phones and the Internet, they would seek ways around the surveillance. But most terrorists probably take that as a given and — unlike most ordinary citizens — already use encryption and other techniques to secure and obfuscate communication. It is a poor excuse for a lack of transparency and public oversight of such snooping. Obama asked Americans to trust the government, but history shows that ‘trust us’ is not good enough.

The revelations seem to vindicate many of the conclusions and recommendations of a 2008 report by the US National Research Council (NRC) — Protecting Individual Privacy in the Struggle Against Terrorists: A Framework for Program Assessment (go.nature.com/bsooux). That report addressed privacy issues raised by the Total Information Awareness programme, a research effort launched by the US Defense Advanced Research Projects Agency in 2002 to develop data mining and other technologies to link and search disparate databases, for example to try to identify suspicious patterns to detect and track terrorists.

Obama asked Americans to trust the government, but history shows that ‘trust us’ is not good enough.

After much controversy, that programme had its funding removed by Congress in 2003. But as the NRC report noted, this was probably a pyrrhic victory for civil liberties. It removed a focused programme subject to congressional oversight and public debate that would determine appropriate uses and safeguards. Instead, much the same work has continued in agencies across government, including the NSA, with less oversight. The report warned that this was “likely to result in little security and, ultimately, brittle privacy protection”. How right it was.

Privacy matters. Yet last week, many defenders of snooping on private individuals sought to play down its significance. Several, including UK foreign secretary William Hague, trotted out tired fallacies, including that people who have nothing to hide have nothing to fear. That has long been debunked by academics; the idea is based on a misconception of what privacy is about.

Privacy is a human right, and is essential if people are to develop autonomy. It is central to freedom of expression and association, and to preventing abuse of personal information. There are numerous examples of misuse of private data by agencies and law enforcement, including intimidation, selective character assassination, repression of dissent and wrongful arrest. Privacy is a cornerstone of a free and creative society, and is an essential defence against unwarranted social control.

Government officials in the United States and elsewhere should find the NRC report and read it carefully. It calls for “robust, independent oversight” of government data mining and surveillance to “mine the miners and track the trackers”. Some data could help security efforts, the report says, but it notes that many security experts have misgivings. They question the feasibility and reliability of data mining to look for and track terrorists in massive data sets, and they raise concerns about the risk of law-abiding individuals and companies being falsely targeted.

Such surveillance is not unique to the United States. In April, a report by the United Nations’ Human Rights Council warned that many countries worldwide, including democracies, are increasingly allowing intelligence and law-enforcement agencies to deploy indiscriminate and extensive surveillance of communications. That weakens or removes safeguards such as justification of individual cases of surveillance, and oversight by a neutral judicial body.

As the World View on page 139 shows, privacy and what it means in the digital age is an increasingly crucial question in the era of big data. A grown-up and open debate is needed, with trust on all sides. It has not started well.