Sir

The GM Nation? report released in the United Kingdom last month (see Nature 425, 331; 200310.1038/425331b) concluded that the general public is overwhelmingly against genetic-modification (GM) technology, with feelings ranging from “suspicion and scepticism, to hostility and rejection”.

The study cost £500,000 (US$830,000). Unfortunately, this was not money well spent. The methodology was so badly flawed that the data not only failed to support the authors' conclusions, but undermined them.

The main fault with the study, as the authors concede, is the self-selected nature of the main sample. About 36,000 people took part in an “open debate”: half of these responses came by mail and half from the GM Nation? website (http://www.gmnation.org.uk). This sample is certainly large, but it is not random. It is in fact most likely to attract those who have strong opinions about GM. One might think that the sheer size of the sample swamps any problems with its self-selected nature, but for that to be the case you would need millions of participants.

Although the authors of the report were aware of this criticism, they offered only two countermeasures. First, they checked a random sample of responses to see if there were any standardized ones being sent in by activist groups, which there weren't. But people with strong views on GM are capable of expressing their own opinions, and this measure does nothing to prevent the sample being biased in favour of them.

Second, they commissioned a 'narrow-but-deep' study from another company, “as a control on the self-selecting participants in the open debate”, to see if there was a “silent majority” with different views. This meant asking 78 people 13 questions from the open debate. This sample was randomly chosen — although the report is short on specific details — and stratified so that it roughly matched the general population. (This group was also re-tested after 2 weeks of group discussion and personal research to see if their attitudes to GM changed.)

The authors of the narrow-but-deep section conceded that their results were not statistically robust, because of the small numbers involved. Nevertheless, they said: “We believe it is an accurate reflection of the general public.” The initial responses of the random group were, however, noticeably different from the results of the open debate. (Even after 2 weeks, the differences, although not as large, remained significant.) Yet the GM Nation? report claimed that, apart from some minor differences, the two groups agreed. The general public, said the authors, is not “a completely different audience with different values and attitudes from an unrepresentative activist minority”.

The actual results from the two groups were buried within the supporting documents, far apart from each other. Once these results are compared side-by-side, startling differences emerge for more than half of the questions used (visit http://www.nottingham.ac.uk/philosophy/staff/Campbell/Table1.htm for a full comparison).

For example, to the question “I would be happy to eat GM food”, only 8% of the open-debate respondents agreed, compared with 35% for the random group. On the topic of whether GM was unnatural, 84% thought so in the open debate, but only 37% did in the random group.

We find it astonishing that the obvious mismatch between the random group and the open-debate group was not discussed anywhere in the report, and that it did not prevent this report being released and becoming headline news.

With £500,000, a larger version of the narrow-but-deep study could have been conducted, avoiding the problem of self-selection. As well as using a 'topic blind' recruitment strategy, questions about GM food would ideally be embedded among questions about other current concerns, so that the participants would be unaware that GM food was the focus of the research. Also, the sort of vague and leading questions used by GM Nation? should be avoided. Only then could we be confident that the findings are reliable and realistic.