A major, but flawed, study of identity parades, or line-ups, has set science and the police at odds. Laura Spinney investigates.
In 1981, 22-year-old Jerry Miller was arrested and charged with the kidnap, rape and robbery of a woman in downtown Chicago. After two eyewitnesses to the crime picked him out of a line-up, and the victim identified him at his trial, he was convicted and sentenced to 45 years in prison. In March 2007, however, semen on the victim's clothes was subjected to DNA testing and found not to belong to him. His conviction was quashed a month later. He had spent more than 24 years in jail.
Miller's story is not unique: his was the 200th conviction to be overturned in the United States on the basis of DNA evidence. According to the Innocence Project, an advocacy group based in New York that campaigns to overturn wrongful convictions, and that took on Miller's case in 2005, the number today stands at 216, of whom 16 spent time on death row. Mistaken eyewitness testimony was a factor in more than three-quarters of those cases.
Those exonerations have highlighted the fact that police procedures for eyewitness identification in the United States are out of step with almost three decades of psychological research, and have triggered a row over whether the procedures should be reformed. As the situation is similar, or worse, in most other developed countries, the rest of the world is watching closely.
The traditional US procedure is familiar to any fan of television cop shows. Witnesses are presented with a line-up that includes both the suspect and a number of innocent people, or 'foils', and are asked to identify the perpetrator. In the early 1990s, however, when the confidence of the justice system had been badly shaken by the first wave of DNA exonerations, the then attorney-general, Janet Reno, invited experts to form a working group to address how this method could be improved.
The group immediately homed in on the fact that most line-ups are overseen by the case's investigating officer, who knows the suspect's identity. For scientists, this is a major error: even something as seemingly objective as a clinical trial can be affected if the nurse who administers the injection knows whether the syringe contains a drug or a placebo. It is all but impossible for an experimenter — or an investigating officer — to avoid giving away the 'right' answer through body language, tone of voice or other such unconscious hints. “I have argued for years that the more important reform is for line-ups to be conducted double blind,” says Gary Wells, a psychologist at Iowa State University in Ames and a member of Reno's original working party. Witnesses should also be told that the perpetrator may not be in the line-up so that they do not feel obliged to identify someone. In every one of the DNA exonerations that involved mistaken identity, says Wells, the witness had picked the suspect: “It's just that the suspect was innocent.” Although the real perpetrator was not in the line-up, the witness somehow ended up picking the person the detective had in mind.
“The majority of jurisdictions are simply sticking with what they have always done. Stephen Saloom ”
The working group's most important recommendation was that line-ups should be conducted in a double-blind fashion, so that neither the witness nor the official overseeing the procedure would know who the suspect was. The group also recommended that the suspect and foils be presented sequentially rather than simultaneously, and that the witness be asked to make a decision after each one rather than waiting until the end. This would encourage witnesses to compare each individual to their memory of the offender, rather than to one another — a method the scientific data suggested was more likely to produce a correct identification.
In light of these recommendations, several US states reformed their eyewitness identification procedures. However, according to James Doyle of the John Jay College of Criminal Justice in New York, the recommendations encountered pockets of resistance, one of which was in Illinois. Illinois also happens to have one of the highest DNA exoneration counts in the United States, 28, which in 2000 led the state to impose a moratorium on the death penalty until the causes of its wrongful convictions had been identified and eliminated. That moratorium is still in place. Nonetheless, Illinois police tended to view the working group's guidelines not as safeguards against wrongful convictions, but as hopelessly academic and impractical. So in 2003, the Illinois State Police commissioned its own study1 to test line-ups under real-world, field conditions, and put it under the direction of lawyer Sheri Mecklenburg, general counsel to the Superintendent of the Chicago Police Department.
The inherent problem with field studies in this area — and the reason that so few have been done — is that there is no way of knowing for sure whether the suspect is guilty. In a mock crime staged before witnesses in a lab, experimenters know that the suspect is the perpetrator. But in the real world, all that can be said for certain is that the foils did not do it. Hence, the measure that experimenters pay most attention to is how often witnesses identified a foil as the perpetrator.
With this in mind, and with the cooperation of two psychologists and three of the state's police departments — Chicago, Evanston and Joliet — Mecklenburg's Illinois Pilot Program spent a year conducting some 700 eyewitness identifications. Some of the procedures were non-blind and simultaneous; the rest were double-blind and sequential. Both conditions were a mix of live line-ups and photo arrays. The team found that the double-blind, sequential technique produced higher rates of foil picks — that is, clear errors — and lower rates of suspect picks than the traditional, non-blind line-up. Mecklenburg concluded that the existing system should not be changed.
That conclusion, announced in 2006, immediately sowed confusion in the dozen US states that had been considering updating their procedures. One state, New Jersey, and several smaller jurisdictions had already passed reforms, which remain in place. But some states put their reforms on hold, or even threw them out. Horrified, psychologists pointed to deep methodological flaws in the study, which was never submitted to peer review. According to the psychologists, the decision to change two variables at the same time — blind/non-blind and simultaneous/sequential — made it impossible for Mecklenburg and her colleagues to draw any conclusions from their data. “If one of my undergraduates came to me with that experimental design, I would say 'Go away and do it correctly',” says Daniel Wright, psychologist at the University of Sussex in Brighton, UK.
Writing in the journal Law and Human Behaviour2, psychologists Daniel Schachter of Harvard University in Cambridge, Massachusetts, and Nobel laureate Daniel Kahneman of Princeton University in New Jersey and their colleagues were unanimous that the design flaw “has devastating consequences for assessing the real-world implications of this particular study”. After that critique was published, a handful of US jurisdictions did reform their procedures, and others set up task forces to consider the options. But for Stephen Saloom, policy director at the Innocence Project, that wasn't enough. “Given that the vast majority of jurisdictions nationwide have yet to adopt the reforms, and that eyewitness identification has a role in a set of crimes far greater than those in which DNA can prove innocence or guilt, it is of great concern that the majority of jurisdictions are simply sticking with what they have always done,” he says.
Last year, the National Association of Criminal Defense Lawyers in Washington DC and the MacArthur Justice Center at Northwestern University in Chicago, Illinois, both of which back reform, sued the police departments that took part in the pilot programme to release their data so that they could have them reanalysed. So far the departments have refused, and Mecklenburg calls the lawsuit “a waste of energy”. She says it has only alienated the police, making it less likely that they will cooperate with field studies in future.
Regarding the study itself, Mecklenburg has stood her ground. She argues that pristine lab studies capture none of the tension and gravity of an actual line-up. Moreover, although admitting that the study's design was not ideal, she argues that she and her collaborators were doing their best given the operational realities of police work. One problem is manpower. If another person replaces the investigating officer to coordinate a line-up, then that extra person has to be taken off their other duties. Time and complexity are also issues. For the 40% of crimes in the Illinois Pilot Program that involved more than one perpetrator, the police found the sequential technique too cumbersome and abandoned it halfway through the study. Furthermore, real-world witnesses are often reluctant to cooperate. Their willingness to participate can hinge on the investigating officer's ability to build trust — a relationship that could be broken if the witness suddenly had to deal with a stranger conducting the line-up1. That's one reason that modern 'line-ups' are often taken to the witnesses and conducted in the field, by showing them arrays of photos.
Despite these difficulties, however, Mecklenburg claims that her results are still valid — even though another field study3, conducted in Minnesota at around the same time, supported the lab data. And so the dispute rumbles on.
Everyone agrees that more field studies are needed, and several are under way in the United States, including one commissioned by the Department of Justice. In the meantime, some support for the beleaguered Illinois study has come from an unexpected quarter — Britain, which prides itself on its eyewitness identification procedures. These are now routinely done in specialized suites, in the form of video parades. Video images of the suspect and foils are presented to the witness sequentially, but witnesses are asked not to give a decision until they have seen the whole sequence twice.
In a study published last year3, psychologist Tim Valentine of Goldsmith's College at the University of London, UK, and his colleagues applied strict double-blind sequential rules to British-style video parades. This was a lab study, but with what psychologists call high ecological validity — a measure of how well the experimental conditions match those in the real world. Witnesses saw a live, staged theft and attempted to identify the perpetrators a week later from a video parade constructed by police from their database.
Valentine's most striking finding was that the number of correct identifications dropped from 65% under British rules, to 36% under the strict rules. “So there is a big cost in terms of sensitivity,” he says. Under strict rules, “people are inhibited when it comes to choosing”. The Illinois study also found lower sensitivity, but it was overlooked in all the furore about the experimental design. If the effect is real, the implication is that the proposed reforms might reduce the wrongful conviction rate — the number of false positives, or type I errors — but at the cost of increasing false negatives, or type II errors. In other words, more criminals might be let off the hook.
Valentine wholly supports the introduction of double-blind line-ups, but he has serious reservations about the sequential technique. Although his group found a reduction in false positives with sequential line-ups, it was not statistically significant, at least in Britain. “In other words, we are taking a cost for a benefit we can't really be sure about,” he says. Saloom disagrees, arguing that a slightly different picture has emerged from US research: for every three wrongful identifications of innocent people avoided with the sequential technique, you might let one guilty party go free. The Innocence Project understands the reluctance of law enforcers to lose that one correct identification. However, Saloom says, “Ultimately we believe that it's smarter, and a better way to protect the public, to make that larger trade-off.”
Identity parades of one form or another are used in many countries. But Britain has benefited from centralized coordination of police practices through the government's Home Office, and is regarded by many countries as advanced in the way it extracts identifications from eyewitnesses. This is in large part due to its history. In 1976, a government inquiry chaired by high-court judge Lord Devlin combed British case law, searching for wrongful convictions based on mistaken identity4. DNA evidence wasn't available then, but the committee highlighted cases in which other evidence undermined witnesses' identification. The recommendations made by the committee were strongly influenced by the scientific data. For example, they picked up on the need to tell witnesses that the perpetrator might not be in the line-up.
Perhaps the most important outcome of the Devlin report, however, was to encourage a climate of cooperation between police and scientists. “I go and talk to cops and they are very interested in what I have to say,” says Wright. This spirit, in turn, has led to an ongoing process of consultation and reform — the most significant in recent years being the shift from live line-ups to video parades in 2004. Among the benefits this reform has brought is that the parade can take place sooner after the crime, because there is no need to marshal a suspect or foils.
The use of DNA in a forensic context was a British invention, and it is used in a wider range of crimes in Britain than in the United States — not just serious crimes such as rape and murder. Perhaps surprisingly, therefore, no one has systematically collected data on DNA exonerations in Britain. But if they had, Wright believes, those data would show that the British system, too, is far from perfect. Now that parades take place in specialized facilities with trained officers, they are sometimes done in a blinded fashion. But the practice is still not compulsory in the United Kingdom. A spokesman from the Home Office says that before blind parades would be incorporated into the Home Office's codes of practice, they “would require evidence to support the need for change at operational level”.
Indeed, perhaps the one clear lesson from this controversy is that no system is perfect. As Mecklenburg has pointed out, any reforms that psychologists propose must be workable in the real world — and they won't be unless police are given the resources and incentives required to make them work. But as Doyle points out, operational difficulties are no excuse for rejecting a fairer system. Nor are they an excuse for letting policy be swayed by a single, flawed study such as the Illinois Pilot Program. Moreover, he says, this tension urgently needs to be resolved, because the real problem isn't the study. It is the deeper clash of scientific and law-enforcement cultures. “This is really the first of the social science by-products of the DNA exoneration cases,” he says. “So there is a battle being fought now about whether science is going to be allowed to affect practices.” Although mistaken eyewitness testimony is a major cause of wrongful convictions, it isn't the only one. Others include false confessions, the use of convicted informers and flawed or fraudulent scientific evidence. If the scientists back down now, Doyle warns, they will be setting a dangerous precedent.
Mecklenburg, S. H. Report to the Legislature of the State of Illinois: The Illinois Pilot Program on Sequential Double-Blind Identification Procedures (Illinois State Police, Illinois, 2006).
Schachter, D. L. et al. Law Hum. Behav. 32, 3–5 (2008).
Valentine, T., Darling, S. & Memon, A. Appl. Cogn. Psychol. 21, 933–949 (2007).
Devlin, P. Report to the Secretary of State for the Home Department on the Departmental Committee on Evidence of Identification in Criminal Case (HMSO, London, 1976).
Laura Spinney is a freelance writer based in London and Paris.
About this article
Forensic Science International: Genetics (2015)
Improving the analysis of Y-SNP haplogroups by a single highly informative 16 SNP multiplex PCR-minisequencing assay
First all-in-one diagnostic tool for DNA intelligence: genome-wide inference of biogeographic ancestry, appearance, relatedness, and sex with the Identitas v1 Forensic Chip
International Journal of Legal Medicine (2013)
Law and Human Behavior (2012)
Reply to “Bracketing off population does not advance ethical reflection on EVCs: A reply to Kayser and Schneider” by A. M’charek, V. Toom, and B. Prainsack
Forensic Science International: Genetics (2012)