When a US Senate investigation in 2008 revealed that psychiatrist Charles Nemeroff of Emory University in Atlanta, Georgia, had not disclosed at least US$1.2 million in income from drug companies, Senator Charles Grassley decided to do something about it. The Iowa Republican led a charge to push the National Institutes of Health (NIH), which funded Nemeroff’s research, to change how it evaluates researchers who accept money from industry.

The resulting reforms, which took effect in 2012, require scientists to report industry connections in greater detail than before, and charge institutions with determining which ties are problematic. But three years later, it is not clear what the costly, cumbersome rules have accomplished. A Nature analysis suggests that institutions have vastly different standards for what constitutes a conflict — and that they classify relatively few relationships between researchers and industry as troublesome.

“There’s a lot more financial conflict of interest in my view than the NIH is getting from the reports of universities,” says Sheldon Krimsky, who studies conflict-of-interest issues at Tufts University in Medford, Massachusetts. “We’re just seeing the tip of the iceberg.”

The reforms, enacted by the NIH’s parent agency, the Department of Health and Human Services (HHS), do seem to have increased the number of financial relationships that researchers report to their universities — by 45% overall, according to data from 56 universities in a survey released in April by the Association of American Medical Colleges (AAMC) in Washington DC (see go.nature.com/hc5r2b). But the number of conflicts that institutions reported to the NIH has increased only slightly, according to NIH data obtained by Nature through a freedom-of-information request (see ‘Under the microscope’).

Credit: Source: NIH

The agency’s original conflict-of-interest regulations, implemented in 1995, required institutions to report when an HHS-funded researcher received more than $10,000 from an outside source. The revised rule lowered that threshold to $5,000 and directed researchers to disclose a wider variety of potential conflicts, such as sponsored travel and relationships with non-profit organizations.

Institutions, which receive conflict-of-interest reports from their researchers annually, must then convene an internal panel to determine whether a particular relationship could affect a researcher’s work. If so, the panel designs a ‘management plan’ that may require the researcher to disclose the conflict in publications or, in some cases involving human subjects, to stand down as the study’s primary investigator. Institutions then send these plans to the NIH.

Universities have spent millions of dollars and hired extra staff to comply with these reforms, and most administrators are furious about the burden. “We already had an annual disclosure process for all the faculty,” says Andrew Rudczynski, associate vice-president for research administration at Yale University in New Haven, Connecticut. “I can’t see a single benefit to it.”

Yale spent $500,000 to implement the revised NIH rules. In the year after they took effect, the number of disclosures by the university’s researchers doubled — but Yale identified just one new conflict, Rudczynski adds. Other universities report similar experiences.

And whereas the HHS had estimated that the roughly 2,000 institutions that it funds would spend $23.2 million a year to comply with the regulations, the AAMC survey suggests that the true cost has been much higher. Just 71 institutions spent a total of $23 million in the year after the reforms took effect, although their costs going forward may be lower.

Paul Thacker, who led the 2008 Senate investigation as a member of Grassley’s staff, admits that it is difficult to know how well the reforms are working. That is largely because the potential benefits of greater disclosure of financial ties, such as peer reviewers giving closer scrutiny to studies by researchers with conflicts, are tough to measure.

Still, Thacker says, there is a clear need for closer scrutiny. This is backed up by evidence showing that studies funded by private sources, such as drug firms, more often produce results that benefit the funder than do publicly funded studies (A. Lundh et al. Cochrane Database Syst. Rev. 12, MR000033; 2012). And Thacker has little sympathy for universities’ complaints. “It just shows that they still don’t get what the problem is,” he says. “They’re in this place today because they’ve failed to create confidence for the public in the past.”

Others worry that the HHS policy is still not strict enough. Krimsky says that the current rules may give institutions too much power to assess conflicts, without accounting for ways that universities themselves can be compromised by ties to government or industry. This could be one reason why the HHS reforms did not significantly increase the number of reported conflicts, Krimsky adds.

We are still at the mercy of what’s disclosed to us.

Those pushing for greater transparency are also frustrated that the NIH does not require institutions to publish information about researchers’ conflicts and management plans online. Instead, members of the public must ask a university for information on a researcher’s conflicts; the institution has five days to disclose dollar amounts and sources. Nonetheless, the NIH Office of Extramural Research says that about 50% of institutions that submit conflict-of-interest reports have voluntarily created online databases, although these vary in usability and completeness.

Requesting such information from universities directly also produces mixed results. Nature contacted 20 public and private institutions that had reported individual researchers with conflicts of interest involving more than $1 million, seeking details on these relationships. The majority of these institutions responded immediately, but some took as long as two weeks to respond, directed Nature’s reporter to the media office, or instructed her to submit a freedom-of-information request. Most declined to share information about conflicts that occurred before the current calendar year, which is not required by the HHS.

Nor does the department require the release of management plans, which troubles Tobin Smith, vice-president for policy at the Association of American Universities in Washington DC. “If you disclose that there is a conflict but don’t disclose how the university is managing it — which is not part of the regulations — the public doesn’t understand the relationship,” he says.

The NIH also struggles to defend its own regulations. “One could debate whether or not we needed to promulgate a new rule,” says Sally Rockey, director of the NIH Office of Extramural Research. “At the time, there was a lot of scrutiny in the press and Congress got involved.” She concedes that the reforms were mostly in response to this outside pressure. (Grassley declined to comment on the regulations.)

And it is unclear whether the revised regulations would have identified Nemeroff, who did not tell Emory about his industry relationships. “Science and research are built on trust, and we are still at the mercy of what’s disclosed to us,” says Eric Mah, senior director of research compliance at the University of California, San Francisco.

The NIH plans to review the conflict-of-interest reforms later this year, to develop best practices for compliance. The agency will examine data on the type and number of reported conflicts, as well as institutions’ experiences of complying with the requirements. But Rockey says that the HHS is unlikely to make significant changes to the rules, given that they took four years to develop.

In the meantime, research institutions are caught in a bind. The 1980 law that allows US universities to patent inventions encourages relationships with industry, and tight federal research budgets are driving more scientists to seek support from private funders. “There are no easy answers,” Thacker says. “Universities are being pushed into greater reliance on industry funding and until that reverses, these problems just become more and more complicated.”