Published online 13 October 2010 | Nature | doi:10.1038/news.2010.538


Counting collaboration

Can metrics be designed to measure researchers' collegiality?

Scientists working in greenhouse laboratoryHow productive is your collaboration?Noel Hendrickson/Blend Images/Corbis

With scientists increasingly working together, assessing their performance is becoming ever harder. Now researchers at an American translational-medicine institute have produced a method for measuring the nebulous concept of 'collaboration'.

Translational medicine, which attempts to link basic research and clinical medicine, has become a huge buzzword, attracting serious funding from governments and industry. Whereas the number of analyses focusing on the performance of individual scientists have grown in recent years, methods for assessing organizations have lagged behind, says John Hogenesch, a pharmacologist at the University of Pennsylvania's Institute for Translational Medicine and Therapeutics (ITMAT) in Philadelphia. Too often, Hogenesch says, "science is not looking at its productivity scientifically".

"In our own institution, we have institutional resources and they get directed to these centres, but we don't measure their output," he explains. "We're making investment decisions without really having a formal description about how we measure success."

He hopes that his team's new paper in Science Translational Medicine1 may eventually be developed into a way of assessing collaboration. By quantifying the number of papers published and grants obtained by ITMAT researchers, the team produced a 'network analysis' of the relationships between them. Hogenesch likens it to a cocktail party.

You might choose to measure how good a party is by looking at how many people talk to each other, he says. For example, if two people speak to each other for longer than five minutes, they would have what the researchers call an "edge" — a connection between them on a network analysis. More edges mean a better party.

"The actual edges in our graph are papers and grant applications," says Hogenesch. And the edges or connections between researchers represent their collaborations.

As well as establishing that the number of collaborations among ITMAT researchers had doubled over the past five years, the team found that non-ITMAT researchers at the university saw no such increase.

The team also produced graphics representing the collaborations and their growth (see image). This method, it hopes, may eventually enable collaboration and research performance to be improved.

Loners and sharers

Hogenesch acknowledges that some researchers may be happy in productive but insular laboratories.

But having more of this type of information could help institutions in assigning their resources as finances become tighter. "If one person has a great insular lab and the other one a great collaborative lab … the value of the insular lab ends at the edge of their lab," he says.

Network analysis of scietnific collaborationsClick for a larger version of this image.PNAS

Hogenesch is far from the only person trying to measure collaboration. Earlier this year, Francesco Giuliani, a physicist at the Casa Sollievo della Sofferenza Research Hospital in San Giovanni Rotondo, Italy, and his team published their suggestion for "collaboration potential" in Scientometrics2.

As with the Science Translational Medicine paper, they use co-authorship as equivalent to collaboration. Giuliani and his co-authors created an index of potential collaboration using this factor and keywords in papers.

Both groups found that, perhaps unsurprisingly, researchers are more likely to collaborate with colleagues within their department than with those from other departments. This is something that could be addressed, for example, by making the involvement of multiple departments in translational funding awards a requirement.

Moot metric

Eliezer Geisler, director of the Center for the Management of Medical Technology at the Stuart School of Business in Chicago, Illinois, and author of The Metrics of Science and Technology3, notes that the method used by the Hogenesch group is widely used in evaluating social networks and has recently been extended to assessing collaborations.

"However, the road from mapping to assessing performance of an organization on the basis of networking is arduous and doubtful," he cautions. "The method of mapping network dynamics is not a sufficiently good metric that allows us to evaluate the performance of the organization, in this case ITMAT, or any other scientific department, university, institute or industrial research and development division."


Geisler has worked on basic models of collaborations involving industrial research and development since the 1970s. "In our model, network dynamics is only one of the sub-processes leading to other processes which lead to ultimate outputs and performance," he says.

Giuliani also cautions that these new metrics are "only indirectly" linked to the actual quality of research.

Nevertheless, such metrics are useful for big research institutions, he adds. They help research managers to understand "where the research of their laboratories is going", Giuliani explains, and the relationships between scientists and research groups.

"The increasing need for such metrics is dictated by the fact that collaboration is an important aspect of science due essentially to the growing complexity of scientific problems," he says. 

  • References

    1. Hughes, M. E., Peeler, J. & Hogenesch, J. B. Science Transl. Med. 2, 53ps49 (2010).
    2. Giuliani, F., De Petris, M. P. & Nico, G. Scientometrics 85, 13-28 (2010).
    3. Geisler, E. The Metrics of Science and Technology (Praeger, 2000).


If you find something abusive or inappropriate or which does not otherwise comply with our Terms or Community Guidelines, please select the relevant 'Report this comment' link.

Comments on this thread are vetted after posting.

Commenting is now closed.