The maths of quantum theory might be useful in understanding how we form and use concepts.

When Roger Penrose published his book Shadows of the Mind, 16 years ago, it elicited from many readers what seemed to me an unjustified hostility. Penrose suggested that the puzzle of human consciousness might ultimately have something to do with quantum theory. The very idea seemed to irritate many physicists, biologists and philosophers, who thought it patently absurd and responded, in many cases, with ridicule rather than reasoned counter-argument.

Penrose's idea, crudely, ran as follows. In mathematics, the famous incompleteness theorem of Kurt Gödel implies that the truth of some mathematical propositions cannot be decided in a purely computational or algorithmic way. Human mathematicians, however, have reasoned their way to proofs about some propositions of this kind, implying that the human mind itself must be in some aspects more than algorithmic. Surveying what we know about physical law and biological processes, Penrose suggested that because all known physical laws are algorithmic (they can be simulated computationally), the non-computational element playing a role in the brain, whatever it is, must involve new physics. As promising territory for discovering it, Penrose suggested looking at the unsolved 'measurement problem' in quantum theory, as well as that theory's links to gravity.

Penrose may be totally wrong, but there's nothing essentially illogical in his argument. Still, most scientists dismissed his ideas as ludicrous, immediately and, it seemed, without much thought. I was reminded of this recently when hearing a physicist dismiss as equally ludicrous recent work suggesting a link between quantum theory and human psychology. Sure, it sounds loopy. But look into the details, and you find it really isn't.

A remarkable feature of human thinking is our ability to combine concepts to create wholly new ideas. Our concepts, as expressed most frequently in language, have a strongly contextual character in that meanings depend on complex relationships between words. Perhaps you have a pet, a chihuahua, and he's a rather big one; he's a tall chihuahua. All chihuahuas are, of course, dogs, so if your chihuahua is furry, he's also a furry dog. Likewise, a hungry chihuahua would be a hungry dog. Yet a tall chihuahua is not a tall dog. The word tall doesn't modify meaning in an absolute way, but does so depending on the conceptual category of the word to which is applies. Similarly, consider that the word 'lion' brings with it a host of associations — furry, ferocious, dangerous, wild — and yet, oddly, add one word, stone, and 'stone lion' loses all these associations. Weirdly, a 'guppy' is not a 'typical pet', nor is it a 'typical fish', and yet it is a 'typical pet fish'. (For a discussion of such matters, see P. D. Bruza et al., http://arxiv.org/abs/quant-ph/0612051; 2006.)

Understanding this contextuality of human concepts is a principal requirement for progress in artificial intelligence, and also for the automated analysis of textual information on the web. This information was created and organized by human minds, yet we lack an understanding of how those minds organized it. Present methods for web-based text analysis and information retrieval generally fall into the category of so-called vector-based semantic analyses (VBSA), which treat a text passage as a 'bag of words' in which order is irrelevant. These cannot distinguish 'Mary hit John' from 'John hit Mary' because they are insensitive to contextuality.

Which brings us to quantum theory — the one theory of natural processes we have that is attuned to contextuality. In quantum theory, we're used to the idea that quantum systems have specific properties only in the context of the particular experiments performed on them; which properties are definite change with the experimental context. (A formal expression of contextuality in quantum theory is the well known Kochen–Specker theorem.)

Consequently, a number of scientists have recently suggested that the mathematical formalism of quantum theory might well be useful in describing how people form and use concepts. The idea, roughly, is to replace the real vector spaces in which VBSA represents concepts with a Hilbert space. The relatedness of concepts, represented as vectors, is linked to an inner product between them, and one can form projection operators and so on to calculate how much one concept links up with others.

It sounds a little crazy, yet remarkably this quantum formalism immediately provides an improved capacity to represent the way real people classify concepts in experiments, where they often violate the norms of classical logic. For example, suppose you give individuals a list of things — squirrel, donkey, spider, and so on — and ask them to say how much each belongs within three categories: 1, pets; 2, farmyard animals; and 3, the union, pets or farmyard animals. Many experiments of this kind find that most of us will sometimes (and probably not consciously) put things into category 1 or 2, but not in 3, even though this final category is just the union of the first two (see, for example, D. Aerts et al., http://arxiv.org/abs/1004.2529; 2010).

We're not as logical as we like to think. But this empirical pattern of human classifying behaviour can be handled fairly easily with the formalism of quantum theory, which captures a more flexibly quantum logic. This empirical resonance also translates into practical capabilities. For example, several years ago computer scientists found that they could reduce the number of pages a search engine produces in response to a query — effectively improving the accuracy of a web search — by replacing certain logical commands in the search with their counterparts in quantum logic (D. Widdows and S. Peters in Proc. Math. Lang. 8, 141–154; 2003). The quantum operation led the search to ignore not only pages ruled out explicitly by classical logic, but also other undesirable pages with closely associated meanings.

Why people think this way, and why the mathematics of quantum theory should be so useful, no one knows. But it is clear that much of our thinking isn't the conscious thinking that conforms to classical logic, and so we should not perhaps be so surprised. Our conscious mind actually handles only a tiny fraction of the information we need to survive: neuroscientists estimate that unconsciously we handle of the order of 10 million bits of information each second, while the conscious mind can handle only about 10 bits.

It may seem strange that quantum theory could possibly apply to human thinking, but it is the mathematical structure of the quantum formalism that is relevant, not its physical interpretation as a theory of the microworld. The mathematics of Hilbert space is useful for quantum theory, as well as for many other things. Whether our brains use quantum physics or not, they may still conform to the elements of its mathematical logic.