Forecasting uncertain events with small groups

Focus group predictions improve when money is at stake.

Price ups prediction performance. Credit: © Photodisc

Business, election and weather forecasts are distilled from a blend of insight and ignorance. Now a rubric that makes the most of aggregate knowledge such as this could help analysts anticipate the future, say Bernardo Huberman and colleagues at Hewlett-Packard in Palo Alto, California¹.

Their work should open up a new approach to information markets, in which forecasts use the principles of economics rather than mere opinion polls. One such principle is economists' long-standing belief that market forces automatically distribute goods efficiently, to maximum social benefit, without an omniscient overseer's wisdom.

Information markets collate reliable information and channel it effectively, at a fair price, to appropriate users. One of the fastest-growing opportunities created by electronic information networks, information markets are model economies, with market forces that lead to the efficient extraction and use of diffuse information.

Large information markets can generate better predictions than surveys and opinion polls. The Hollywood Stock Exchange, where anyone can trade fantasy shares (using 'Hollywood dollars') in their favourite actors and movies, accurately predicts box-office revenues and Oscar awards. Stocks soar with a number-one film at the box office and plummet with a stay at the Betty Ford clinic.

Information markets with fewer participants are poorer predictive tools, just as surveys become less accurate as they get smaller. Small groups are susceptible to information traps - fixation on false outcomes - or to manipulation. Companies with small forecasting teams face these and other problems.

Huberman and his colleagues' two-stage method alleviates these difficulties. First, participants run a purely artificial information market among themselves, to calibrate their predictive skill and attitudes to risk.

Some participants might be prone to awarding high probabilities to bad predictions; others might show the same amount of risk-taking, but with better judgement. The participants trade securities - they engage in several rounds of bidding for 'shares' in a particular outcome.

In the second stage, participants make predictions about the event of real interest, such as the next quarter's business forecast. These individual opinions are then combined into an aggregate forecast, each weighted according to a mathematical scheme using the information gathered in the first step. Participants are rewarded for accuracy, ensuring that they are not indifferent.

To test their method, Huberman's team asked students to trade predictions about the outcome of a biased lottery determined by balls drawn from an urn. The technique produced better predictions than a normal information market or the best individual in the group.

Future focus groups, involving small numbers of participants, might be run along these lines, the group suggest, so that "each member has a financial stake in the information coming out".