The terrorist attacks of 11 September have thrown into sharp focus a set of problems that have been afflicting the research sections of US spy agencies for years, close observers say.

Agencies such as the Central Intelligence Agency (CIA) and the National Security Agency (NSA), which once led the world in their use of such technologies as satellite surveillance and supercomputing, have seen their technical edge dissipate since the end of the Cold War, according to the observers.

James Hirsch, who was deputy director of the CIA's directorate of science and technology from 1990 to 1995, says that intelligence-agency researchers “are in an identity crisis. They don't know who or what they are or what they want to do because there's no clear sense of mission.”

For half a century, spy technology was geared towards the Soviet Union and its client states, entities that had vast military complexes and fixed lines of communication to spy on. The terrorists who conducted the 11 September attacks had neither of these. And despite signs during the past decade that terrorism was becoming more of a threat to US interests, US intelligence failed to identify and concentrate on terrorism — or anything else — as its principal target.

CIA director George Tenet and President Bush urgently need results from the troubled spy agency. Credit: NEWSMAKERS

Instead, US spies and spycraft have simply become involved in several different problems in turn: crises in North Korea, Somalia, Haiti and the former Yugoslavia, for example. Lacking a clear mission focus, the agencies have been unsure what their research programmes are needed to prepare for, and these programmes have been shrinking as a result.

Although detailed figures are classified, the intelligence agencies are known to have an annual budget of approximately $27 billion, of which an estimated $3.3 billion goes to the CIA. Jeffrey Richelson, author of a recently published book, The Wizards of Langley: Inside the CIA's Directorate of Science & Technology, says that about two or three per cent of the agency's operating budget — $80 million — was devoted to research and development. That's a low figure, by the standard of either corporations or government agencies with strong technical requirements. “There were plans to push it to 5%,” he says, “but it was one of those things that eventually requirements [elsewhere in the agency] ate up.”

Last October the CIA, at least partially aware of the problem, overhauled the directorate and established a new Office of Advanced Technologies and Programs, according to agency documents. In 1999 it had set up In-Q-Tel, a non-profit organization designed to improve agency ties with start-up companies, where expertise in information technology is increasingly concentrated.

“The idea was to try and identify new, innovative information technologies that could be incubated with seed money from this organization,” explains Hirsch.

In-Q-Tel was part of an effort by the CIA, which has no research laboratories of its own, to break its technical reliance on a small circle of contractors who know its way of working and conform with its secrecy requirements. The early results of the concept were “quite impressive”, Richelson claims.

Far more rapid progress will now be expected, to get the CIA and other spy agencies abreast of terrorists' means of communication. Hirsch says that one typical requirement is for something similar to a web-browser, that would scour the US government's vast but fragmented and closely-held intelligence databases.

Security agencies have been tight-lipped about their ability to monitor the Internet, but outside experts say that blanket monitoring of all the traffic on it is out of the question. There are “just way too much data to handle,” says Richard Clayton, a computer scientist with the Computer Security Group at Cambridge University.

The monitoring of traffic to and from certain e-mail addresses and geographical locations is more feasible, and is already undertaken by systems such as the FBI's Carnivore programme, which picks up messages by physically tapping Internet nodes and surveys them for keywords or phrases of interest.

One of the most powerful approaches to monitoring e-mail, however, is to locate one of the computers that sent or received it. Documents and even decrypted and deleted e-mail messages can then be retrieved fairly easily, says Jim Bates, managing director of UK-based Computer Forensics.

Bates says that even supposed computer experts would be unable to cover their cyber-tracks completely. He recalls one case when 98% of the e-mail messages relating to a banking fraud case were found on the computers that had sent or received them, even though the hard drives had been reformatted.

Additional reporting by David Adam