A consensus-based transparency checklist

We present a consensus-based checklist to improve and document the transparency of research reports in social and behavioural research. An accompanying online application allows users to complete the form and generate a report that they can submit with their manuscript or post to a public repository.


This PDF file includes:
Supplementary Methods

Overview
To ensure that the transparency checklist contains most elements that the scientific community considers relevant to the transparency and accountability of behavioural research, we adopted a 'reactive-Delphi' expert consensus process 10; see Fig. S1 for a schematic representation . The items were evaluated and rated by social and behavioural science journal editors and open science advocates. The Transparency Checklist was modified (items were added, abandoned or reworded, answer options were restructured) until a sufficiently high level of convergence was reached. The expert consensus procedure consisted of two stages. In the first stage, an expert panel was recruited and consulted about the checklist items. Then, journal editors were contacted and asked to rate and comment on the items in two rounds. In stage two, the checklist was finalized using those items that reached consensual support (Supplementary Figure 1). The procedure is described in detail below. Figure 1: Overview of the expert consensus process.

Preparation
Preregistering the project Before the start of the project, on 2018-04-18, a research plan was compiled and uploaded to a time-stamped repository at https://osf.io/v5p2r/registrations/.

Creating the initial checklist items
Before the expert consensus process, the project proposers (the first four authors and the last author; henceforth: proposers) created an initial list of items for the Transparency Checklist after brainstorming and reviewing existing checklists and guidelines with a similar purpose. The initial 35 checklist items are available at https://osf.io/x3g2t/.

Recruiting panel members
Following this, the proposers contacted 33 open-science advocates from the field of behavioural and social sciences. The selection of these experts was based on our knowledge of the publications in which they propagated openness and transparency of science. These researchers were invited to participate in the further development of the initial list of items and, at the end of the project, the final revision of the checklist. Upon joining this panel, the experts received an online survey (available at https://osf.io/y29a5/).

Consulting the panel members
Following our inquiry, 18 experts joined our panel (henceforth panel members) and provided comments to the initial checklist items (their names are available at https://osf.io/zj6dr/). The proposers discussed each comment and compiled a new version of the checklist on a rolling basis, that is, the panel members who joined later in time reviewed an improved version of the checklist. Based on the panel comments, a revised version of the checklist was created and its 47 items were assessed in the Consensus Design survey, Round 1 (see below).

Recruiting editors for the expert consensus procedure
In order to recruit members for the expert consensus process, we collected email addresses from journal editors using the following procedure. First, we selected the journals with 'Psychology' subject in the Scimago (scimagoojr.com) portal (1,075 journals). Psychology label in Scimago seemed to be a convenient searchword for finding journals publishing human behavioural and social empirical studies. We aimed to work with the editors of the high ranked journals, assuming that these journals hold high methodological standards, which makes their editors competent to address questions related to transparency. To this aim, we selected the top 10% of these journals, using the Scimago Journal Rank (SJR) indicator of 2018 (107 journals). As the checklist was aimed to assist studies reporting primary empirical data, we excluded all journals with 'Review', 'Trends' and 'Perspectives' in their title. Then, we read the policy of the remaining journals and omitted those that do not publish empirical research, leaving a total of 63 journals. The list of the remaining journals is available at https://osf.io/54pzb/. Beside 'Psychology', these journals were associated with the following 79 labels: Accounting, Aging, Anthropology, Applied Mathematics, Arts and Humanities ( We retrieved the contact information of the chief editors and two associate editors from each journal (189 contacts). When there were more than two associate editors in an editorial board, we picked two names without any systematic method. Finally, we extended the contact list by adding 18 experts from other behavioral sciences (e.g., management science, political science). We selected these experts after a non-systematic literature review suggesting these individuals to be particularly engaged in transparency and open science issues. The resulting list contained the contact details of 207 journal editors and experts. Although the resulting list contained 6 noneditor experts, for the sake of simplicity, we call this group 'editors'.

Consulting the editors
We emailed 207 editors and invited them to participate in the consensus process. Although we had questions in the survey referring to their role at their journal, we did not ask them explicitly to answer the survey as a journal editor. (The text of our invitation is available at https://osf.io/8b52k/). The consensus design survey was sent out individually to everyone on our contact list between 2018-12-14 and 2018-12-18. The participants received the question in a Google Forms Survey.
The survey contained the proposed checklist items along with their answer options. The items covered four components of a study: Preregistration; Methods; Results and Discussion; Data, Code, and Materials Availability. Noted that the boundaries of these labels can vary among research fields and for some researchers certain items might better fit under another label.
The respondents could rate each item on a 9-point Likert-type 'recommendation scale' ranging "Definitely do not include this item" to "Definitely include this item". Following each section, the respondents could leave comments regarding the items. The survey is available at https://osf.io/ptjcz/.

Evaluating the survey results
We elicited responses from 45 editors (one of them was a member of the panel as well) between 2018-12-14 and 2019-02-02 (their list is available at https://osf.io/zj6dr/). Next, the proposers of the project amended and modified the checklist using the collected comments and recommendations. The survey-answers are available at https://osf.io/43hxk/. As described in the preregistration plan, if the interquartile range of the ratings of an item was 2 or smaller on the recommendation scale, we regarded that particular item as having reached consensus. We concluded that support was obtained on a consensus item if the median of its rating was 6 or higher. In this round, 34 out of the 47 items reached consensus. Following our preregistered plan, the proposers of the project discussed the comments and recommendations collected in this round and decided which items to select for the next survey. As a result, out of the 13 items not reaching consensus, 8 items were abandoned. For an in-depth description of all exclusion criteria, we refer the interested reader to our online summary of results. The remaining 5 items were selected for the second round, 1 of which was merged with a consensual item. No other alteration or rephrasing took place in the content of the checklist. The explanations for each excluded item are provided at https://osf.io/zyed8/ and the summary Prior to analyzing the complete data set, a time-stamped preregistration was posted in an independent, thirdparty registry for the data analysis plan.
(2) The manuscript includes a URL to all preregistrations that concern the present study. (

Consulting the editors
For this round, a new rating survey was created with the 5 remaining items along with reflections to the received comments and the aggregated ratings that each item received in Round 1. Only those 43 editors who responded to the first round and indicated that we can get in contact again received the survey. Two editors did not wish to be contacted with new versions of the checklist. The survey is available at https://osf.io/5xh7d/. We received 34 completed surveys between 25/02/2018 and 11/03/2019. The survey-answers are available at https://osf.io/y6457/. After analysing the collected data, 2 items were dropped from the final survey as the ratings failed to show consensus as the interquartile range of the recommendation ratings was greater than 2. As a result of Round 2, 36 items remained in the checklist. A summary of the round two is available at https://osf.io/zyed8/.
Furthermore, we also explored the differences in the item ratings between subdisciplines. We used the following categories: Applied Psychology, Clinical and Personality Psychology, Developmental and Educational Psychology, Experimental and Cognitive Psychology, Neuropsychology and Physiological Psychology, Social Psychology, Other Psychology, Other Behavioral Science. To do this, we calculated the median ratings for each subdiscipline and calculated the interquartile range based on these values across subdisciplines. We found that the interquartile range was greater than 2 for only two items (item 17 and 36) (results are available at https://osf.io/gsd3u/). Therefore, we did not create discipline-specific checklists or checklistsections.
As a final step, we created a user-friendly ShinyApp by which the authors can easily answer the checklist questions, leave comments, and generate a report that contains their checklist answers along with the metadata of their manuscript. In addition, the proposers created a shortened, 12item long version of the checklist as well, containing the most essential items (http://www.shinyapps.org/apps/TransparencyChecklist/ and http://www.shinyapps.org/apps/ShortTransparencyChecklist/ for the shortened, 12-item version). For this admittedly subjective selection, the proposers tried to choose those items that appeared most important for transparency. Their judgment was based on the item ratings and the comments collected during the consensus procedure. As a pilot test, we asked 6 researchers to fill out the 36-item checklist regarding their most recent empirical study. On average, it took them 6 minutes (range: 3-15) to respond to the items.

Stage 2
In Stage 2, the proposers wrote the manuscript, including the present supplement on the basis of descriptive data and free text responses gathered in Stage 1 along with a suggested final list of checklist items. All texts and materials were sent to the panel members and all contributing editors. Each contributor was encouraged to provide feedback on the manuscript, the report, and the suggested final checklists. After all discussions, minor wording changes were implemented for item 1, 9, 11, 13, 23, and 36. No contributor objected to the content and form of the submitted materials and all approved the final item list.

Notable deviations from the preregistration
The project was preregistered on: https://osf.io/v5p2r/registrations. One deviation from our original plan was that we have involved the panel members in the discussion of the items from the very start of the project and not just in the final stage. We slightly changed the wording of the survey questions to increase comprehension. Also, we excluded some low rated items from the consensus process after Round 1. Justifications for our exclusions are added to the summary table of the results, available at https://osf.io/zyed8/. Furthermore, we did not take panel meeting minutes that we planned to share.