What constitutes appropriate peer review for interdisciplinary research?

How can interdisciplinary research proposals be more effectively assessed through peer review? A key issue is to characterize what constitutes appropriate peer review for interdisciplinary research. This is approached by considering four key elements on which evaluations of funding proposals are based: (1) the significance of the topic, (2) the importance and tractability of the research question, (3) the appropriateness of the methods and (4) the competence of the applicants, based on track record. Two major differences between disciplinary and interdisciplinary research emerge: (1) the unknowns that form the basis of research questions and (2) the methods employed. For peer review of interdisciplinary research proposals to become more effective, agreed criteria for assessment by peer review are required and this will only occur if interdisciplinarity is “organized” through colleges of peers and professional associations. This article is published as part of a collection on interdisciplinarity.

H ow do interdisciplinary research proposals fare in the peer-review system? Despite considerable grumbling from interdisciplinary researchers about the inappropriateness of reviews, there is little evidence about the comparative success rates of disciplinary and interdisciplinary project proposals. Nevertheless, it is worth examining the process, especially to characterize what would constitute appropriate peer review for interdisciplinary research proposals.
Peer-review assessment of research proposals essentially concentrates on four elements, and here each is considered from a disciplinary and an interdisciplinary perspective: The significance of the topic. The importance and tractability of the research question. The appropriateness of the methods. The competence of the applicants, based on track record.
A fifth element, which is more tacit than explicit, is what I call "zing". This is an innovative edge that lifts the proposal above the pack of solid research worthy of funding. The challenge for funders and peer reviewers is that the majority of submitted proposals deserve support, so that "zing" can make the difference in actually getting funded. This element will not be discussed further here.
Before considering the first four elements in detail, let us review some general differences between discipline-based and interdisciplinary proposals. For proposals in disciplines such as immunology, physics or sociology, there are both a strong college of peers and professional associations that concern themselves with developing a general level of agreement within the discipline about research areas of importance, suitable methods and what constitutes a good track record. As a consequence, peer review is reasonably straightforward.
The situation for interdisciplinary research is very different. There is no college of peers and professional associations are small and relatively powerless. Unlike disciplines, interdisciplinarity is unorganized. Even the term "interdisciplinarity" is used loosely and covers a very diverse set of research practices (Bammer, 2012(Bammer, , 2013, for example: one person bringing together concepts or methods from two or more disciplines; two disciplines developing an area of overlap, as when psychology and economics developed behavioural economics; investigations in a field such as women's studies or defence studies, where the field comprises contributions from many disciplines; a team of researchers from closely related disciplines developing a new technology; a team of researchers from diverse disciplines working with non-research stakeholders to tackle a multi-faceted complex problem such as obesity or organized crime. Clearly different colleges of peer reviewers and peer-review panels are likely to be necessary to adequately assess these different kinds of interdisciplinarity. It is also noteworthy that, in practice, the terms multi-, inter-and trans-disciplinarity are often used interchangeably, with some arguing that attempting to draw rigid distinctions among them amounts to "faintly theological hair-splitting" (Rylance, 2015: 314). For the purposes of this commentary, the focus is on the widest possible range of practices to which the term "interdisciplinarity" is commonly applied and when a specific type of interdisciplinarity is referred to, it is specified.
As a final introductory remark, it is worth noting that the peer-review system is, of course, far from perfect. It can become excessively self-referential, and rivalries can interfere with fair assessment, but by and large, to appropriate Winston Churchill's assessment of democracy: [n]o one pretends that peer-review is perfect or all-wise. Indeed … peer-review is the worst form of assessment except all those other forms that have been tried from time to time.
Let us now return to the four elements of peer review to focus specifically on what they mean for assessment of interdisciplinary research proposals.
The significance of the topic Assessing significance involves evaluating the importance of the issue being addressed, against some measure such as government research priorities, the prevalence of the problem or its likely impact. Governments are increasingly directing funding to some areas in preference to others. The Australian government, for instance, currently has nine priority areas: soil and water, transport, cybersecurity, energy, resources, advanced manufacturing, environmental change and health (Australian Government Science, 2016). Within broad priority areas such as these, researchers usually strengthen the case for their proposal by, for example, highlighting the number of people affected by a disease in a public health proposal or developing an argument about cost savings for a transport proposal.
For this element of peer review, the same assessment criteria are relevant for both discipline-based and interdisciplinary research. In both cases, research with a case for significance is more likely to be funded than arcane research.

The importance and tractability of the research question
Assessing the research question requires examining the unknowns being tackled, especially whether they are critical unknowns likely to yield productive outcomes. This assessment, of course, takes into account what is already known. Disciplines tend to have established norms about the kinds of unknowns that are the "business" of that discipline. Becoming a disciplinary expert involves learning which unknowns to tackle and which to ignore. Becoming good at one's discipline involves cultivating the ability to pick productive unknowns.
To the best of my knowledge, assessing interdisciplinary research from the perspective of the unknowns addressed has not been considered previously, but it may well be fruitful, as interdisciplinary research embodies a range of different approaches to unknowns. In laying these out, I focus on one kind of interdisciplinary research-a team of researchers from diverse disciplines working with non-research stakeholders to tackle a multi-faceted complex problem-and draw on illustrations from a large project that I directed which examined the feasibility of prescribing pharmaceutical heroin as a treatment for heroin dependence (Bammer, 1997;Bammer et al., 1999). This case study illustrates that interdisciplinary research can embody at least six different approaches to unknowns.
First, interdisciplinary research can involve combining separate disciplinary considerations of unknowns, which are firmly within the respective disciplines' purviews. The foundation for assessing the feasibility of pharmaceutical heroin prescription involved combining the investigation of unknowns from a number of disciplinary perspectives including: demography to estimate the number of dependent heroin users (Larson and Bammer, 1996), anthropology to investigate the likely impact on ceasing heroin use (Bammer and Weekes, 1994), philosophy to examine the ethics of a trial of heroin prescription (Ostini et al., 1993), epidemiology and biostatistics to review the range of possible trial COMMENT PALGRAVE COMMUNICATIONS | DOI: 10.1057DOI: 10. /palcomms.2016 designs (National Centre for Epidemiology and Population Health and Australian Institute of Criminology, 1994), and political science to assess the likely political context for a heroin prescription trial (Hartland et al., 1992). Some work led to new disciplinary insights, as in the economic comparison of heroin purity versus price as the equilibrating mechanism (Butler and Neil, 1994). Most of the research was a more routine application of disciplinary methods to a new problem.
Second, interdisciplinary research can involve addressing unknowns that are outside the business of any discipline, but are of major concern to stakeholders. In the heroin prescription feasibility research, for example, police were very concerned that the city hosting the trial would become a "honeypot" for drug users from around Australia and possibly beyond . This was not an unknown that any of the disciplines involved brought to the table and it was not addressed using solely discipline-based research.
Third, interdisciplinary research can tackle unknowns that are marginalized by power imbalances. These unknowns are not seen as important in mainstream research and practice, often because they concern relatively powerless stakeholders. An example from the heroin prescription feasibility research concerned how to make treatment more attuned to and respectful of illicit drug users, which was addressed by a group of clinicians, illicit drug users and drug treatment researchers combining forces (McDonald et al., 1994).
Fourth, interdisciplinary research can investigate unknowns that occur in the overlap between disciplines. Such disciplines are often traditionally closely aligned, for example, chemistry and biology. There may also be an overlap with stakeholder concerns. The point is that these unknowns are more effectively addressed by the disciplines combining forces and melding methods than by proceeding separately. An example from the heroin prescription feasibility investigation was background research on drug use, binge drinking and attempted suicide among homeless youth, which involved blending methods from anthropology and sociology, along with insights from youth workers and affected young people (Sibthorpe et al., 1995).
Fifth, interdisciplinary research is characterized by a focus on the problem and this can lead to identification of unknowns that are critical to the problem, but which have received little or insufficient disciplinary consideration. In the heroin prescription feasibility research example, it was realized that very little was known about illicit drug markets and that this was not a major area of research. It led to criminologists and police combining forces to flesh out and investigate key questions about the likely effects of heroin prescription on illicit drug markets (Bammer, 1993).
Finally, the heroin prescription feasibility research illustrated that not all the unknowns involved in assessing feasibility were tractable. Whereas discipline-based research would tend to ignore questions that could not be "solved", this makes less sense in interdisciplinary research, as such unknowns can lead to adverse unintended consequences or unpleasant surprises. In the heroin prescription feasibility research, a major focus was on identifying and addressing potential risks (Bammer, 1999;Bammer et al., 2003). Some possible risks could be addressed through targeted research, for example, concerns about the effects of heroin on driving skills or that a system where users had to inject in a clinic was unworkable (Swiss trials showed that this was not the case). For other risks there were no definitive answers, for example, that heroin prescription could lead to more permissive attitudes to illicit drug use or that it could lead to the "honeypot effect" referred to earlier. Certainly steps could be taken to potentially reduce these risks, but the interconnectedness and complexity of the issues meant they could not be "solved".
What does this mean for peer review? If interdisciplinary proposals are assessed on the criteria used for discipline-based proposals, they would likely come up short. Many of the unknowns that feature in interdisciplinary research would not be considered relevant in discipline-based research. Even when unknowns from different discipline-based studies are combined, these studies may be more pedestrian and less compelling than would be required for research in any one of the component disciplines to be funded. While it is feasible to embed outstanding discipline-based research within an interdisciplinary investigation, what the latter requires to address the problem of concern is often much more straightforward.
It can be argued therefore that interdisciplinary research needs to be assessed by peers who have themselves undertaken similar kinds of interdisciplinary research and therefore understand the importance of these different kinds of unknowns. There are two challenges in finding such reviewers. First is finding interdisciplinary peers in the same problem area, as the range of issues addressed by interdisciplinary research is vast. Nevertheless, for some problems, such as responses to climate change, obesity and terrorism, there may be a large enough group to allow competent peers, without conflicts of interest, to be found. The second challenge is the unorganized nature of interdisciplinarity described earlier, which means there are no obvious mechanisms for finding such peers even when they exist.
The appropriateness of the methods For assessment of the proposal's methods, there are two essential strands. First, are the methods appropriate for the unknowns being tackled? Second, are they currently the best available methods? Again these questions are relatively easy to address in discipline-based research. And again this is not the case for interdisciplinary research.
The methods relevant to interdisciplinary research concern, for example, how disciplinary insights are integrated, how intractable unknowns are dealt with, and even how disciplines and stakeholders are chosen. Unlike the disciplines, there is currently no established set of methods for conducting interdisciplinary research and therefore no easily available set of criteria for assessing the methods in interdisciplinary proposals.
I have previously addressed this problem, specifically in relation to interdisciplinary research involving a team of investigators from diverse disciplines working with non-research stakeholders to tackle a multi-faceted complex problem (Bammer, 2013). That book developed the parameters of a repository of methods for interdisciplinary research. Three main domains were identified: (1) synthesizing disciplinary and stakeholder knowledge, (2) understanding and managing diverse unknowns, and (3) providing integrated research support (combining knowledge and unknowns) for policy and practice change. The book also laid out five key questions for each domain that could be used to organize the various relevant methods, including for undertaking (1) knowledge synthesis, (2) scoping and boundary setting to identify potential disciplines and stakeholders and to choose those to be included, as well as (3) examination of the context of the problem.
Such a repository has yet to be established and to gain widespread acceptance. Unless and until there is a repository of methods for interdisciplinary research, similar to those available in the disciplines, peer review of interdisciplinary research will continue to be hampered. Right now, even reviewers with experience in interdisciplinary research have access only to a small proportion of the methods that have been developed, because knowledge about them is highly fragmented and widely dispersed in the published and grey literatures, with many insights not published at all. As a consequence, while interdisciplinary reviewers may be able to assess whether the methods are appropriate for the unknowns being tackled, they will not be able to evaluate whether they are currently the best available methods.
The competence of the applicants, based on track record Past success, especially in the recent past, is seen to be a guide to the likelihood of future success. Reviewers therefore look for: The number of peer-reviewed papers and the quality of journals in which they are published (or number of books and quality of the publisher). Past success in attracting competitive funding. Evidence that at least some of the team members have worked together successfully in the past, when the disciplinary or interdisciplinary research is team-based (which now characterizes the bulk of research).
There is growing interest in adding research impact on policy and practice change (including commercialization) to this mix.
Some argue that it is harder to publish the outcomes of interdisciplinary research and to do so in high-quality journals. I am not aware of any substantial evidence to support this claim. However, there seem to be few high-quality journals in which the methods used in interdisciplinary research can be published (Bammer, 2016). Furthermore, disciplines vary, for example, in the expected number of papers per year, which has ramifications for interdisciplinary research combining highly uneven disciplines, meaning that an appropriate level of productivity for the interdisciplinary research can be difficult to establish. Finally, there are strong suggestions that interdisciplinary research takes a long time (for example, see Currie et al., 2016), so that productivity relative to time spent may be low.
As highlighted in the introduction, for funding calls open to any kind of research, there is no evidence about differences in funding success between disciplinary and interdisciplinary research proposals or whether interdisciplinary researchers have weaker track records on competitively funded grants than their disciplinary counterparts.
In terms of team work, effective collaboration involves harnessing the differences that provide the rationale for working together and managing differences that can stymie effective partnership (Bammer, 2008). Personality differences are a common example of the latter. It can be argued that interdisciplinary research, especially when it involves diverse disciplines and also stakeholders, encompasses a much wider range of both "good" differences to be harnessed and "bad" differences to be managed, than is the case for discipline-based research. There is, therefore, more scope for things to go wrong in interdisciplinary research, which may make it harder to establish a record of effective working together. But again there seems to be no empirical evidence to support this claim. Finally, if impact becomes a significant assessment factor, interdisciplinary teams may do better than discipline-based teams, although that too is yet to be properly evaluated.
These issues further illustrate the importance of review by peers with similar experience and expertise. Just as peer groups and professional associations of immunologists and sociologists establish the, very different, norms for track record in their disciplines, the challenge for interdisciplinarians is to establish their own norms, which may differ for different kinds of interdisciplinarity.

Conclusion
Interdisciplinarity faces two big challenges. First is establishing appropriate criteria for peer review, not only of grant applications as described in this commentary, but also of publications, as well as performance for hiring and career advancement. As discussed above, interdisciplinarity has significant characteristics that differentiate it from the disciplines and these must be recognized and agreed by the bulk of the interdisciplinary community before they can become the basis for effective review processes.
The second, and arguably much larger, challenge is the unorganized nature of the interdisciplinary community. On the one hand, there are different kinds of interdisciplinarity, as described briefly above, including overlaps between multi-, interand trans-disciplinarity. On the other hand, there are multiple communities-especially of those working across diverse disciplines and with non-research stakeholders to tackle multifaceted complex problems-which are currently separate and which may be able to be effectively combined, especially when it comes to peer review. These include communities that identify as action research, systems thinking, integrated assessment, postnormal science and implementation science (Bammer, 2013). There are also multiple professional associations and networks, all with small memberships and none with significant power or organizational heft. This fragmentation also limits the ability to effectively organize colleges of interdisciplinary peers and hampers evaluation of this way of doing research.
Effective peer review of interdisciplinary research and researchers requires advances on these two fronts at the same time, namely, developing the criteria and the colleges of peers. To date there has been little appetite in either the interdisciplinary community or among funders to acknowledge, let alone address, these issues. There are signs, however, that the situation is changing among funders, with, for example, the Global Research Council, which is a federation of more than 50 national research funders, choosing "interdisciplinarity" as one of two annual themes for an in-depth report, debate and statement for its 2016 meeting (Rylance, 2015). Let us seize this opportunity!