Published online 19 June 2008 | Nature | doi:10.1038/news.2008.902


Is Hawk-Eye's decision final?

Row erupts about accuracy of ball-tracking technology.

tennis ball on lineIn or out? Some experts have taken issue with the way Hawk-Eye is used to adjudicate contentious line calls.JUPITERIMAGES/ Polka Dot / Alamy

A fierce rally of words is being played out ahead of the Wimbledon tennis fortnight in London. The inventor of Hawk-Eye — a high-speed camera-based device used to help tennis umpires make decisions — is arguing with academics at Cardiff University, who claim that the workings of the technology could be represented more transparently to the public.

“We don’t want people to think that there is a technological fix for difficult decisions,” says Harry Collins, one of the academics and the author of a paper on the topic that has been published online in the journal Public Understanding of Science1.

Hawk-Eye, developed by British inventors Paul Hawkins and David Sherry, is a suite of high-speed cameras and software used to track a ball’s path. Already used to enhance television coverage of sports events, the technology was officially adopted by the International Tennis Federation (ITF) in 2006 to help umpires adjudicate on difficult line calls.

Hawk-Eye's involvement has already caused controversy. In last year’s Wimbledon final between Roger Federer and Rafael Nadal, a ball played by Nadal seemed to be out to the umpires, the spectators and Federer, according to the report. But Hawk-Eye called it 'in' by a single millimetre, and the decision went in Nadal’s favour.

Eye on the ball

Collins and his co-author Robert Evans wanted to work out the chances of Hawk-Eye getting it wrong, given that its manufacturers say its average accuracy is 3.6 millimetres. This falls well within the ITF’s criteria, which require any automated line-calling system to be accurate to within 5 millimetres when compared with tests they run with high-speed cameras on-court before tournaments.

So Collins and Evans turned to statistics. Any average error carries with it a standard deviation – a measure of how widely the data points are spread in a set. Collins and Evans say that they assumed the error was 'normally' distributed, and thereby calculated a standard deviation of 4.5 millimetres. This meant that Hawk-Eye might have been mistaken in its ruling on the line call.

A written statement was posted by Hawkins on the Hawk-Eye website on 17 June in response to the media coverage that the paper had received. The statement questions the use of the normal distribution. “This kind of methodology of making assumptions with no grounds for doing so, is not something which Cardiff should be particularly proud of, and does not reflect well on their general approach to science,” the statement read.

“We used a normal distribution because we didn’t know what the true distribution is,” says Collins. “It’s perfectly good scientific practice.” Collins has since published an open letter to Hawkins, offering to update the paper if Hawkins provides more technical details of the system.

But Hawkins disagrees, and says that his system has performed with 100% accuracy in tests. “We never got any wrong,” he says. He claims that a normal distribution is not appropriate in this case. “Our experimental data showed that it doesn’t fit a normal distribution.” Hawkins says that the 3.6-millimetre error arises when balls that land a long way from the line are taken into account, and that perspective is difficult to account for in the system.

Collins suggests that devices such as Hawk-Eye be calibrated differently, to take into account systematic errors that are part of the normal game. “As Hawk-Eye is currently calibrated it seems to make decisions that are different to those that would be made traditionally,” he says. “If the ball clips the line humans see that as 'out', whereas Hawk-Eye calls it 'in'.”

To help get a clearer picture of Hawk-Eye’s fallibility, Collins also suggests that error bars be represented graphically on television for the benefit of viewers.

Hawkins scoffs at that idea. “My grandmother would have no idea what that meant,” he says.

'Inviting trouble'

Recalibrating the system to take into account the kind of error humans make is “inviting trouble” says Stuart Miller, head of the ITF’s scientific and technical division. He is only interested in showing the true position of where the ball landed, within the accuracy limits set by the ITF.

He is also uncertain about showing the extent of possible errors. “I’m not sure that the role of Hawk-Eye is public education,” says Miller.

Miller points out that Hawk-Eye is only called on occasionally, in instances when a player challenges a call. “We’re supplementing what is already a very good performance by on-court officials,” he says.

Both Hawkins and the Cardiff researchers agree that Hawk-Eye has a bright future in sport. “It is a valuable aid to correct random errors,” says Collins, “but those judgements have to be associated with a statement of uncertainty.”

“Yes, it has changed the game,” says Hawkins. “The vast majority of people think it’s for the best.” 

  • References

    1. Collins, H. & Evans, R. Public Underst. Sci. 17, 283-308 (2008). | Article |
Commenting is now closed.