In physics, the value of a theory is measured by how well it agrees with experimental data. But how should the physics community gauge the value of an emerging theory that cannot yet be tested experimentally? With no reality check, a less than rigorous hypothesis such as string theory may linger on, even though physicists have been unable to work out its actual value in describing nature.

Credit: PALTO

This sort of uncertainty has implications not only for the gathering of knowledge for the scientific enterprise, but also for fledgling physicists. The investment of research time in strong intellectual assets is crucial for graduate students who want to establish their careers on a good foundation. But not all young researchers are aware of the history that accompanies every research area. They often have to rely on word of mouth from their PhD adviser or colleagues for this information.

What if the physicists could call on a ratings agency, not unlike a lender would do before deciding whether to offer credit? I am advocating the creation of a website that is operated by graduate students and that will use various measures of publicly available data (such as the number of newly funded experiments, research grants, publications and faculty jobs) to gauge the future returns of various research frontiers.

Theory bubbles

The study of the cosmic microwave background provides an example of how theory and data can generate opportunities for young scientists. As soon as NASA's Cosmic Background Explorer satellite reported conclusive evidence for the cosmic microwave background temperature fluctuations across the sky in 1992, the subsequent experimental work generated many opportunities for young theorists and observers who joined this field. By contrast, a hypothesis such as string theory, which attempts to unify quantum mechanics with Albert Einstein's general theory of relativity, has so far not been tested critically by experimental data, even over a time span equivalent to a physicist's career.

Senior scientists might seem the people best suited to rate the promise of research frontiers. But too many of these physicists are already invested in evaluating the promise of these speculative theories, implying that they could have a conflict of interest or be wishful thinkers. Having these senior scientists rate future promise would be akin to the 'AAA' rating that financial agencies gave to the very debt securities from which they benefited. This unseemly situation contributed to the last recession, and a long-lived bias of this type in the physics world could lead to similarly devastating consequences — such as an extended period of intellectual stagnation and a community of talented physicists investing time in research ventures unlikely to elucidate our understanding of nature — a theory 'bubble', to borrow from the financial world.

Of course, graduate students are busy. But they could serve a limited term of service for maintaining the site and be government-funded. For example, students supported by US National Science Foundation fellowships maintain astrobites.com, a website that summarizes new astrophysics papers.

Credit rating

The physics 'credit-rating' website would use evaluation metrics to factor in, with the correct weighting, all the ingredients that would ultimately make scientific research successful.

For physics, this might include the existence of an underlying self-contained theory from first principles, the potential for experimental tests of this theory and a track record of related research programmes. Clearly, factors such as intellectual excitement cannot be quantified, but as long as funding agencies are supporting projects and the information provided is accurate, the data about the growth of a field should echo this 'excitement' factor.

The evaluation metric would have to be predetermined and supported by numbers that are based on archival data gathered through automated searches for keywords in electronic data archives (see arxiv.org or nsf.org). Aside from automated searches, practitioners from fields that are being evaluated could submit supplementary data that would be incorporated into the analysis.

The entire data set would include the level of funding allocated to experiments and research grants, the status of the underlying theory and the number of publications and faculty jobs within the particular field of research. The simplest model relates the change in these parameters to a linear combination of their values. For example, the publication rate is expected to relate to a linear combination of the number of experiments, faculty jobs and available research funds. With the right mix of time spent on theory, experimental work and grant support, a research frontier would show exponential growth in this linear model. The next step would be to calibrate this model using historical data about the growth of successful research frontiers.

The website could be helpful to institutions and governments, not just to individual scientists. A balanced assessment of the level of risk and potential benefits from emerging research frontiers can increase the efficiency of the workforce, leading to stronger growth. And it could help funding agencies to optimize their allocation of money to promote progress in research. In fact, it would be in the interests of funding agencies to support the website and help the students to take part (for example, through special grants or fellowships).

The website might also convince senior researchers to shift their focus to new research areas, perhaps as a result of the influence that the rating procedure may have on funding agencies. But maintaining balance and ensuring diversity among subfields, taking some risks and avoiding funnelling resources into a small number of successful but conservative programmes are important considerations for funding agencies (A. Loeb Nature 467, 358; 2010).

Nearly every worthwhile endeavour involves some risk. But mitigating that risk, and helping young scientists to make informed decisions about the field in which they should invest their time and intellect, would yield a more efficient scientific enterprise.