Nature | News Feature

Crowd-sourcing: Strength in numbers

Researchers are finding that online, crowd-sourced collaboration can speed up their work — if they choose the right problem.

Article tools

Illustration by Denis Carrier; Equation from Ref. 4

At the end of January 2009, Timothy Gowers embarked on what he later called “one of the most exciting six weeks of my mathematical life”. Inspired by the online citizen-science movement, Gowers, a mathematician at the University of Cambridge, UK, posted an esoteric theorem on his blog and challenged his readers to prove it — together.

Crowd-source your expertise, he urged them: “If a large group of mathematicians could connect their brains efficiently, they could perhaps solve problems very efficiently as well.”

They could. Within hours of the problem being posted, Gowers' blog was abuzz with back-and-forth brainstorming, as mathematicians chimed in with ideas and possible avenues of attack. Gowers had hoped for new insights, but even he was surprised that by March, after nearly 1,000 comments, he was able to declare the theorem proved. “The quite unexpected result — an actual solution to the problem — added an extra layer of excitement to the whole thing,” he says. The proof was published1 under the collective pseudonym D. H. J. Polymath.

Gowers' online challenge was a radical suggestion for mathematics — a field that is often viewed as the domain of lonely, secretive figures who work for years in isolation. And it went against the grain of the wider academic culture, which tends to encourage researchers to share their ideas only by publishing them.

Yet this open approach has taken root as an ongoing crowd-sourcing project called Polymath. Today, just past its fifth anniversary, Polymath has a dedicated website where people can post and debate suggestions for new challenges — and, if they agree that the challenge is worthwhile, circulate ideas for its solution.

Not every challenge has worked as well as the first, but other mathematicians offer cautious praise for the project, both for providing solutions to problems and for helping to spark a much-needed shift towards greater openness in mathematical research. “The impact on the community is larger than the net scientific impact,” says Gil Kalai, a mathematician at the Hebrew University of Jerusalem, who coordinated one of the challenges.

Many of the Internet's most popular crowd-sourced science projects require no expertise from their participants. One called Galaxy Zoo, for example, engaged more than 150,000 lay users during its first year, 2007, to sort images of galaxies by shape (see Nature 466, 685687; 2010). Problems on Polymath, by contrast, attract just a few dozen participants, but those users have real expertise. In that way, the project parallels commercial ventures that pose technological and data-analysis problems online, looking for responses from highly skilled people. Companies such as InnoCentive in Waltham, Massachusetts, offer cash rewards to encourage participation, but most of the entries come from individuals motivated by a deep love for solving problems — and the chance to win recognition.

As with Polymath, these companies are finding that some challenges work well, but others go nowhere.

“We're still learning what works and what doesn't,” says Terence Tao, a mathematician at the University of California, Los Angeles, and the coordinator of several Polymath challenges.

A winning formula

One key strength, says Tao, is Polymath's responsiveness. “We can react rather quickly to hot events in mathematics,” he says.

Last April, for example, Yitang Zhang, a previously obscure mathematician at the University of New Hampshire in Durham, announced that he had taken a giant step towards answering one of the great open questions in the theory of prime numbers: is there an infinite number of 'twin primes' that differ by 2, as in (11, 13) and (41, 43)?

Zhang had not given the complete answer: he was able to show only that there is an infinite number of 'near-twin' primes that differ by no more than 70 million (ref. 2). But it was the first time that anyone had put any limit on such pairings. Polymath seemed like an ideal tool for whittling down that limit.

The project, dubbed Polymath 8, got under way last June with Tao as coordinator. Within months, mathematicians all around the world had pitched in with refinements to Zhang's proof, using the Polymath website to discuss and answer one another's questions. They swiftly reduced the separation between the primes from the millions to the thousands3. And by November, James Maynard, a mathematician at the University of Montreal in Canada, had drawn on that impetus to reduce the limit to 600 (ref. 4).

Polymath 8 was a triumph for the collaborative approach, says Tao. If mathematicians had been attacking the problem in the standard way, with what he describes as “a flood of mini-papers”, it might have taken years to get the bound down that far.

Polymath has not always worked so well, however: some of the challenges simply never got off the ground. But after five years of experience with it, users have begun to home in on the features that determine success. For example, says Tao, “It helps if the problem is broadly accessible and of interest to a large number of mathematicians”. This tends to draw a wide range of participants with a rich mix of skills, but it works only if the problem can easily accommodate what they have to offer.

That was one of the virtues of the twin-primes challenge, says Maynard. “The proof can be split into separate sections, with each section more-or-less independent of the others,” he says.

Perhaps the most important lesson is that setting up and sustaining a Polymath project is a big commitment. So far, Tao and Gowers have initiated all but two of the Polymath projects. “It's quite difficult to get people interested,” Gowers admits. “It needs an active leader who is willing to spend a fair amount of effort to organize the discussion and keep it moving in productive directions,” says Tao. “Otherwise, the initial burst of activity can dissipate fairly quickly.”

Keep them coming

Incentives can help with that. “In academia, people are willing to spend a lot of time for 'kudos' or for the sake of science — but only up to a point,” says Thomas Kitching, a cosmologist at University College London who has been involved in crowd-sourcing ventures. Beyond that point, he says, “monetary incentives or prizes seem to be required”.

That is the idea behind prize-based crowd-sourcing initiatives, which offer financial rewards to experts who provide solutions. Some of these initiatives are government-led, such as the NASA Tournament Lab and the US cross-agency Challenge.gov, which offers cash prizes for solutions to a whole range of engineering and technological problems.

Other efforts are completely commercial, and charge clients to post a problem online. Among the most prominent is InnoCentive, which will host any scientific or technological challenge. These range from the mundane but important — developing economical forms of “latrine lighting in emergencies”, for example, or “keeping hair clean for longer without washing” — to the esoteric: “seeking 4-hydroxy-1H-pyridin-2-one analogues”, or “stabilizing foamed emulsions”. It has more than 300,000 registered 'solvers', who stand to gain rewards of between US$5,000 and $1 million if their solution works.

Another commercial venture, Kaggle in San Francisco, California, specializes in data analysis, with applications ranging from oil and gas recovery to predicting drug targets. In 2012, a team of astronomers led by Kitching launched a Kaggle competition called Observing Dark Worlds. It offered $20,000 — donated by London-based financial firm Winton Capital Management — for the three best algorithms to map the distribution of dark matter in galaxies using the matter's gravitational-lensing effects on background objects. The competition was a success for all parties: the winning entries offered the astronomers about a 30% improvement over existing algorithms, and Winton recruited new analysts from the contestant list for a fraction of the usual advertising and interviewing costs.

The need for such projects might well increase, says David Harvey, an astronomer at the University of Edinburgh, UK, and a co-author of a study5 that resulted from the competition. “With new telescopes such as the Square Kilometre Array, the Large Synoptic Survey Telescope and Euclid on the horizon, astronomers will be facing real problems of data processing, handling and analysing.”

But Harvey stresses that Observing Dark Worlds was not an unalloyed success. As interesting as the resulting algorithms were, none of them had been tested and developed to a point at which they could routinely be used on real data. “It's vital that the winners of the competition work in collaboration — post-competition — on the problem and develop the initial idea all the way through to a final package,” he says. That process will probably require a lot more time and compensation than the initial prize money.

The love of the chase

On the surface, at least, Polymath differs from commercial ventures in several ways. Most importantly, its challenges seem to be genuinely collaborative, rather than competitive. People make possibly small contributions that others build on, or they each solve part of the puzzle, rather than vying to be the victor.

But Polymath and the commercial ventures also have some essential elements in common — starting with people's fundamental reasons for participating. “Winning solvers rarely list the cash among their top motivations,” says InnoCentive founder Alph Bingham. “Their motivations are frequently more intrinsic, such as intellectual stimulation or curiosity to explore where an idea might lead.” InnoCentive aims to encourage this through non-cash incentives, such as prospects for further collaboration or joint press releases publicizing the winner. Kaggle invites participants to “compete as a data scientist for fortune, fame and fun”.

“Competition, if well posed, can help in science,” says Kitching. “But a poorly posed problem may just increase noise.”

Journal name:
Nature
Volume:
506,
Pages:
422–423
Date published:
()
DOI:
doi:10.1038/506422a

References

  1. Polymath, D. H. J. Preprint at http://arxiv.org/abs/0910.3926 (2009).

  2. Zhang, Y. Ann. Math. (in the press).

  3. Polymath, D. H. J. Preprint at http://arxiv.org/abs/1402.0811 (2014).

  4. Maynard, J. Preprint at http://arxiv.org/abs/1311.4600 (2013).

  5. Harvey, D., Kitching, T. D., Noah-Vanhoucke, J., Hamner, B. & Salimans, T. Preprint at http://arxiv.org/abs/1311.0704 (2013).

For the best commenting experience, please login or register as a user and agree to our Community Guidelines. You will be re-directed back to this page where you will see comments updating in real-time and have the ability to recommend comments to other users.

Comments

Commenting is currently unavailable.

sign up to Nature briefing

What matters in science — and why — free in your inbox every weekday.

Sign up

Listen

new-pod-red

Nature Podcast

Our award-winning show features highlights from the week's edition of Nature, interviews with the people behind the science, and in-depth commentary and analysis from journalists around the world.