Does Measurement Measure Up? How Numbers Reveal and Conceal the Truth

John M. Henshaw Johns Hopkins University Press: 2006. 248 pp. $26.95, £14.63 080188375X

Now, more than ever, we are besieged by numbers, targets, league tables and non-stop auditing. We are inundated with management gobbledygook. Most of these activities produce numbers, but what do all these numbers mean? That is the question asked in John Henshaw's book Does Measurement Measure Up, and it is undoubtedly a timely question

I am not a number: evaluating people by numbers is often meaningless, but we still do it. Credit: IMAGE 100/ALAMY

The book starts by quoting Lord Kelvin: “When you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind.” What scientist could doubt that? Realism soon creeps in though: “The darker side of measurement ... manifests itself in a variety of ways, perhaps nowhere more insidiously than in 'measurements of the mind'.” Henshaw proceeds to take apart the ludicrous idea that a person's worth can be measured by a single number. This is a book that is intended for a wide audience, so the discussion is not very technical. Interesting stuff, such as a good discussion of the nature–nurture problem, is missing almost entirely (perhaps because the author is an engineer not a geneticist).

Henshaw's bête noire is the weighted mean. Nothing wrong with it, of course, when the weights have some proper basis. The problem arises when the weights are just produced from a hat to combine separate measurements into a single number. That means that you can get almost any number you want by adjusting the weights: a classical garbage-in, garbage-out problem. Henshaw illustrates this beautifully by his evaluation of the US system used to rank universities. Why give 25% weight to peer assessment and 10% to expenditure per student? Using other weights would produce different outcomes. His account of the man who was hired by the University of Tulsa, Oklahoma, to improve its ranking is chilling, but many universities now waste money in that way.

In some ways UK university rankings are even worse. Here the rankings involve measures of teaching quality. That sounds entirely reasonable until you look at where the numbers come from. They come from the Quality Assurance Agency (QAA), an organization that costs £11.5 million (US$21 million) annually, but which seems to think that the quality of teaching can be measured numerically without bothering about what is being taught. This leads to absurd results, like university courses in homeopathy (yes, there are some) being given near-perfect scores when they should really be referred to the Office of Fair Trading for describing as science a subject that is more akin to magic. The antics of the press in their attempts to rank universities are a more-or-less honest attempt to make money. The reification imposed by the QAA is an intellectual disgrace. As so often, it is a case of 'Quis custodiet ipsos custodes'? Or possibly, a case of, 'those who can, do; those who can't, assess'.

Now back now to the book. The best bits are the discussions of IQ and of rankings. They deal, if incompletely, with the absurdities, and the real social harm, that can come from trying to measure what is not measurable (at least by a single number). Other chapters discuss ranking in sports, and the sorts of measurement that can be made in business management, magnetic resonance imaging, the genome and global warming. These are subjects of great interest to many people, but I found their treatment disappointing. The chapter about ranking in sports will not mean much to anyone who does not avidly follow baseball results, and to follow one of the author's examples of climate measurement you have to be familiar with summer in Las Vegas. More seriously, though, there is no critical analysis of whether the measures used in business management, genetics and climate studies actually measure what they purport to. I would really like to know how much management-speak is just pretentious gobbledygook, but this book didn't tell me. Systems biology, an area in which clever computations are often based on dubious, or even nonexistent, numbers, deserved a mention but doesn't get it.

The style of writing is easy, but it lacks passion. IQ advocates are gently taken to task, and journalists mildly castigated. I suspect that Henshaw lets off too easily the main villains of the piece, namely academics themselves. They (some of them, anyway) provide the box-ticking and paper-collecting for agencies such as the QAA. It is not unknown for some of us to exaggerate the significance of our findings, partly, of course, because of fear of the next Research Assessment (yet another case of attaching dubious numbers to complex activities). And as they get older, too many academics seem happy to take refuge behind a façade of management-speak in the style so beautifully described by Michael O'Donnell in his A Sceptic's Medical Dictionary (BMJ Books, 1997) as “decorated municipal gothic”.

Numbers, as Henshaw's book says, are often abused, but it is academics who must shoulder much of the blame for that.