Full publication of preprint articles in prevention research: an analysis of publication proportions and results consistency

There is concern that preprint articles will lead to an increase in the amount of scientifically invalid work available. The objectives of this study were to determine the proportion of prevention preprints published within 12 months, the consistency of the effect estimates and conclusions between preprint and published articles, and the reasons for the nonpublication of preprints. Of the 329 prevention preprints that met our eligibility criteria, almost half (48.9%) were published in a peer-reviewed journal within 12 months of being posted. While 16.8% published preprints showed some change in the magnitude of the primary outcome effect estimate, 4.4% were classified as having a major change. The style or wording of the conclusion changed in 42.2%, the content in 3.1%. Preprints on chemoprevention, with a cross-sectional design, and with public and noncommercial funding had the highest probabilities of publication. The main reasons for the nonpublication of preprints were journal rejection or lack of time. The reliability of preprint articles for evidence-based decision-making is questionable. Less than half of the preprint articles on prevention research are published in a peer-reviewed journal within 12 months, and significant changes in effect sizes and/or conclusions are still possible during the peer-review process.


Materials and methods
Our study protocol was registered in the Open Science Framework (OSF) 24 .The overall project was a mixedmethods study and consisted of three parts: a text analysis, a qualitative interview study, and a survey study.Here, we report the results of the text analysis (i.e., objectives 1 and 2 of the larger overall project) as well as the results of the survey study, which we registered as an update to the original protocol in OSF 25 .

Data source and search strategy
We sampled studies from medRxiv (www.medrx iv.org).We developed a Python-based web crawler by analyzing the medRxiv website's http responses using the Python packages requests (handling http) 26 , BeautifulSoup (xml/ html parsing) 27 , and re (for extracting information from character strings related to prevention using regular expressions) 28 (see Table S1 for search string).We ran the web crawler on December 15, 2020 and let it search and extract information from prevention articles that had been posted on the medRxiv website (first run) from January 1, 2020 to September 30, 2020.It downloaded basic data about each identified preprint article: title, abstract, authors, version submission date, version history, download statistics, withdrawal information, funder, first author's institutional affiliation, and information on the publication status (if published, new DOI, and the journal in which it appeared).

Inclusion and exclusion criteria for the study selection
We exported the information provided by the web crawler to Excel® for the abstract review.We dually screened all records identified by the web crawler against our eligibility criteria (see Table 1).Within the project, we used the working definition for prevention research established by the National Institute of Health Prevention Research (NIHR) Coordinating Committee 29 .Using that definition, we included primary and secondary prevention studies that (1) identified and assessed risk and protective factors, (2) screened and identified individuals and groups at risk, (3) developed and evaluated interventions to reduce risk, (4) translated, implemented, and disseminated effective preventive interventions into practice, or (5) developed methods to support prevention research.We pilot-tested the abstract review with 50 records in the first web crawler round and amended the eligibility criteria where necessary.In case of uncertainty, we looked at the preprint's full text and solved disagreements through discussion.We dually categorized each record according to the prevention categories (i.e., chemoprevention, counseling, immunization, screening, other primary prevention, other secondary prevention), whether COVID-19-related (i.e., yes or no), funding source (i.e., any funding vs. no funding, public or noncommercial funding [only one type of funding source involved], public and noncommercial funding [both types of funding sources involved, usually several sources], industrial funding, no funding), and study design (i.e., randomized controlled trial [RCT], cohort study, cross-sectional study, diagnostic study, ecological study, descriptive study, time series, before-after study, case control study, case series).

Search for published preprint articles
To give every preprint a 12-month span to get published, we ran the web crawler again on October 5, 2021 and updated the information on publication status (second run).Previous research has found a median submissionto-publication time of 224 days (range 24 to 1034) for general medical journals and given that some manuscripts have to be submitted more than once, 12months seemed a realistic time for an article to get published 30 .Because we did not want to rely entirely on the information provided by medRxiv regarding publication status, we manually searched Google® and GoogleScholar® for a published version of each unpublished preprint.If we still failed to identify a published version, we contacted the corresponding author of the unpublished preprint by email.For preprints that were published in a peer-reviewed journal, we downloaded the full-texts of the preprint and the published article and performed further data extractions into a structured form using Excel®.One researcher extracted the following data, which were checked by a second researcher: the primary outcome effect estimate and conclusions regarding the primary outcome for both the preprint and the peer-reviewed article, journal name, and publication date.When we detected differences in the effect estimates or conclusions between the preprint and peer-reviewed article, two investigators independently classified these changes.We used the typology developed by Gartlehner et al. 31 to classify these changes but had to simplify it because of the range of effect estimates we identified.Gartlehner et al. 31 used a relative risk increase or reduction of less than 25 percentage points for dichotomous outcomes as one of the thresholds for determining similarity of treatment effects.We assessed the magnitude of change in the effect estimate by applying the following categories to both dichotomous and continuous outcomes: no change, minor change (a relative change of up to 25 percentage points), and major change (a relative change of more than 25 percentage points).We considered the statistical significance of the primary outcome between the preprint and peer-reviewed article as having changed when at least one of the two effect estimates had a P-value that was deemed statistically significant in either the preprint or publication, and not statistically significant in the other.
For changes in the conclusion, we followed the categories suggested by Silagy et al. 32 : no change, minor change (changes in style or wording that do not alter the substance or meaning of a section), and major change (changes that alter the substance or meaning of a section or alter the interpretation).The classifications of changes in the effect estimates and conclusions were done by one person and verified by a second person.
We retrieved the impact factor for each peer-reviewed journal from the Impact Factor List of 2019 provided by the Journal Citation Report (JCR) 33 and calculated the time until publication from the first appearance on medRxiv.
We used descriptive statistics and compared differences in publication characteristics and publication proportion between the different types of prevention articles.We used Bayesian methods to model the effect of a set of predictors on the proportion of peer-reviewed journal preprint publications as an outcome.The predictors included prevention type, whether COVID-19-related (yes/no), study design, and funding sources.We chose to use multiple individual models instead of one joint model for exploratory reasons, due to the lack of an underlying theory and to avoid issues associated with overadjustment and collider bias [34][35][36] .The Bayesian modeling was conducted with Marcov Chain Monte Carlo methods via the brms package 37 and using restrictive priors.The intercept was suppressed.The statistical models were as follows: We conducted all analyses within the R environment (version 4.2.1).Additionally, we used the tidyverse 38 package, readxl 39 , and tidybayes 40,41 packages for data wrangling and creating the plots.

Survey development, participants, and procedure
We designed a questionnaire to explore the reasons for nonpublication in peer-reviewed journals and attitudes toward preprints in general.The questionnaire was developed in English and consisted of 11 items asking about the rationale behind deciding not publish the preprint or the reason(s) for and number of rejections as well as the estimated credibility of preprints, attitudes toward preprints and demographic characteristics.The questionnaire was developed based on the results from the text analysis.The face vailidity was confirmed by the research team.Using the correspondence email address provided in the preprint, we sent the questionnaire to all corresponding authors of the prevention preprints identified in the text analysis that were not published at the start of the survey (n = 152) between September 14 and November 14, 2022.We sent out several reminder emails to increase the response rate.

Ethics approval and compliance
The ethics committee of the University for Continuing Education, Krems, approved the survey study (EK GZ 28/2021-2024).We obtained informed consent from all survey participants.All research was performed in accordance with the regulations laid out by the ethics committe, the European Union data protection law (EU Regulation 2016/679), and the Declaration of Helsinki.Participants were not reimbursed for their participation.

Characteristics of the included preprints
In the first run, the web crawler identified 2238 preprints on medRxiv, of which we identified 594 as prevention research studies (26.5%).Among those, 329 were clinical, epidemiological, or diagnostic studies and met the inclusion criteria for our study selection (Fig. 1).

Proportion of published preprints
Of the 329 prevention preprints we identified, 161 (48.9%) were published in a peer-reviewed journal within 12 months of being uploaded to medRxiv (see Table 3).For published preprints, the median time from upload to journal publication was 5.3 months (range − 0.1-11.3).COVID-19 studies were published more quickly than non-COVID studies (4.7 months; range − 0.1-11.3 vs. 7.0 months; range 1. 3-11.3).The journals those preprints were published in had a median impact factor of 3.2 (range 0.2-74.7).

Proportions of published preprints according to prevention type, study design, and funding source
The proportions of published preprints differed between the prevention types, with preprints on screening having the lowest publication proportion (38.2%, 58/152) and those on chemoprevention having the highest proportion (87.5%, 14/16) (see Table 3).While the time from preprint to peer-reviewed journal publication showed little

Consistency of the conclusions
The conclusions changed in 42.2% (68/161) of the articles after being published in a peer-reviewed journal compared to the preprint, mainly in terms of style or wording (39.1%, 63/161) (i.e., minor change).The content or meaning of the conclusion (i.e., major change) changed in 5 articles (3.1%).Articles with a major change in conclusion were cross-sectional (4/5) or ecological studies (1/5), received no funding (2/5), public or non-commercial funding (2/5), or public and non-commercial funding (1/5).Four of these articles focused on COVID-19.

Survey of authors not publishing their preprint
We received a valid answer from 12 out of the 152 authors of preprints not published in a peer-reviewed journal within 12 months and with a valid email address (7.9% response rate).Eleven respondents were male, five were 50 years or older, and eight had more than 10 years of experience in research (Table S6).The reason most often given for nonpublication of the preprint in a peer-reviewed journal was rejection by at least one journal (58.3%, 7/12), followed by lack of time (25%, 3/12).Other reasons mentioned by one respondent each were that the preprint had received its attention, and that they had never intended to publish the preprint.Among those preprints that were submitted, 57.1% (4/7) got rejected 3-4 times.The official reasons given by journals for rejecting the preprints were manyfold and included lack of novelty (n = 3), too few figures/tables (n = 2), and not www.nature.com/scientificreports/meeting the journal's scope (n = 2), among others.Two-thirds of the preprints did not receive external funding (66.7%, 8/12).The reasons most often indicated for uploading the preprint were sharing the results with the community (n = 11), immediate/fast publication (n = 6), and increased visibility of work (n = 6).All the results are presented in Table S7.

Discussion
To the best of our knowledge, this study is the first to provide a thorough analysis of the publication proportion and consistency of the effect estimates and conclusions of preprints and their subsequent publications in prevention research.Almost half of the prevention preprints (48.9%) were published in a peer-reviewed journal after 12 months of being uploaded to medRxiv, with the median time from upload to publication being 5.3 months (range -0.1*-11.3months).About half of the prevention preprints were on screening (46.2%), a quarter on other primary prevention topics (27.4%), and one-fifth on immunization (20.1%).The results from the modeling analysis indicate that preprints on chemoprevention and cross-sectional studies or those with public and noncommercial funding had the highest probability of publication within their categories.Preprint authors who did not publish their results in peer-reviewed journals mentioned journal rejections followed by lack of time as the main reasons for nonpublication.
We detected a change in the magnitude of the effect estimates in 27 out of the 161 preprints that were published in a peer-reviewed journal (16.8%), but most were minor changes.Although changes in the magnitude of the effect estimate were predominately minor and did not appear very often, they still warrant caution for the use of preprints in decision-making in the prevention field.If 7 out of 161 articles had a major change in the magnitude of the effect estimate, every 23rd article is affected.In addition, it must be considered that as yet we have no knowledge of the quality of unpublished results.We found changes in the conclusions in 42.2% of the preprints that were published within 12 months, but mostly in terms of style or wording, and only in 5 out of 161 articles was the content of the conclusion changed (3.1%).It therefore seems very sensible that medRxiv has issued a warning on the main page of their website that preprints should not be relied on to guide clinical practice or health-related behavior and should not be reported as established information by news media 42 .A definite assessment of the credibility of preprints will be possible when the reasons for not publishing them are fully understood.
Several studies have centered their work on the publication proportions of preprint articles.Recently, studies focusing on COVID-19 preprints reported proportions of 5.7% to 55.3% preprints published within the study periods (5 to 18 months), with a median time to publication between 2.3 and 5.9 months 13,[18][19][20] .A third of the preprint articles uploaded to bioRxiv, a preprint server for biology research (https:// www.biorx iv.org/), before 2017 did not get published as peer-reviewed articles 21 .The proportion of preprints published in our study (48.9%) is close to that reported in the studies by Ostridge et al. 18 and Zeraatkar et al. 19 , but these are not directly comparable given the different study periods (12 months vs. 16 and 18 months).Although we did not find that whether an article was COVID-19-related predicted the publication proportion, our findings demonstrate that such articles are published more quickly.An analysis of COVID-19 articles from January to June 2020 showed that peer review was accelerated for COVID-19 articles but decelerated for non-COVID-19 articles because all resources were pushed toward COVID-19 43 .Other studies found more COVID-19 studies published within their study period than non-COVID-19 studies 22,44 .
Like in our study, Zeraatkar et al. 19 investigated predictors of preprint publication but came up with different results.They found that preprints were more likely to be published if they received government funding.Our study identified public and noncommercial funding as the strongest predictor for publishing but used different categories for funding source.We further found that chemoprevention and cross-sectional studies had the highest publication probabilities within their categories.To fully understand which factors predict preprint publication, it is important to undertake a larger, more detailed analysis of preprints.
As for the changes in effect estimates and conclusions, our study's findings largely mirror those of other studies reporting on the consistency between preprints and subsequently published articles [14][15][16]19 . Forexample, Bero and colleagues 14 did not find large discrepancies in results reporting or the presence of spin between COVID-19 interventional and observational preprints and publications, but small changes were frequent.Another study using a stricter classification scheme (important change in any effect estimate by ≥ 10% and/or change in significance level) classified 21% of COVID-19 intervention trials as having an important change from preprint to peer-reviewed article 45 , which is much higher than in our study, as we found that only 4.4% of studies had a major change in the effect estimates.
Despite our focus on prevention preprints, 73.6% of all preprints we identified were on a topic related to COVID-19, a result of the time period from which we selected the articles from (January 2020 to September 2020).However, other studies have used different eligibility criteria and sources to retrieve their COVID-19 preprint samples than this study.While we focused on epidemiological prevention studies uploaded to medRxiv, Zeratkaar et al. 19 only included COVID-19 trials from the World Health Organization COVID-19 database and the Epistemonikos L*OVE COVID-19 platform, Bero et al. 14 were interested in both interventional and observational COVID-19 treatment and prevention studies from the Cochrane COVID-19 Study Register, Anazco et al. 20 selected all COVID-19-related preprints, regardless of the study design, posted on bioRxiv, medRxiv, and Research Square, and Otridge et al. 18 analyzed all preprints included in the first 100 editions of the CDC COVID-19 Science Update.We believe that both eligibility criteria and sources made a difference in which preprints were analyzed.
The strengths of this study include the use of a web crawler, which allowed us to automatically screen the medRxiv server, identify preprints in prevention research, and extract relevant information.The web crawler also made it easy to track the publication status of these preprints after 12 months.Another strength is the dual screening, dual data extraction, and dual categorization of the prevention preprints.Finally, we performed a thorough assessment of the included preprint prevention articles, ranging from publication proportion to overall changes in effect size and conclusion and the identification of publication predictors.We are very confident not to have missed any preprint publications within the study period, as we contacted all preprint authors for assurance.Another strength is the additional survey among the preprint authors whose preprints were not published after 12 months to gain further insight on the reasons for nonpublication.A limitation, however, is the low response rate (7.9%), which precluded us from making generalized conclusions.
The limitations of the study include the small sample size to analyze differences across the prevention categories, funding sources, and study designs.We further made decisions on the categorization of prevention research, study design, and funding source based only on the information provided in the abstracts; therefore, it is possible that we miscategorized some of them.While the correct classification was important for identifying the predictors, it did not have an influence on the assessment of the changes in the effect estimates and conclusions between the preprint and peer-reviewed article.

Conclusions
This study expands our knowledge that preprints on prevention research topics have few major changes in the effect estimates and conclusions after undergoing the peer-review process.Although, at first sight, these changes appear in a small number of preprints, still, 3-4% of articles are affected.Given that only about half of preprints are published within a reasonable time, and these are likely to be of higher quality, the numbers could be much higher.We therefore warrant caution in using preprints of prevention research in decision-making.

Figure 1 .
Figure 1.Flow chart of preprint selection process.

Table 1 .
Inclusion and exclusion criteria for the selection of preprints to be included in the text analysis.

Table 2 .
Characteristics of the included preprints in total and according to prevention type.difference between prevention categories, the median impact factors of the journal preprints were highest in the other secondary prevention category, at 7.8 (range 5.4-10.2).For the study design, only about one-third of the diagnostic and ecological studies were published within 12 months (32.0%, 32/100 and 31.5%,17/54),while71.4% of the cross-sectional studies (70/98) were published within 12 months (see TableS4).The publication process took the least time for cohort studies (median: 4.8; range 0.3-11.3)uploaded as preprints as compared to > 6 months for diagnostic studies.According to the funding source, publicly and noncommercially funded studies had the highest proportions of published preprints (76.2%, 32/42) and impact factors (median 5.3, range 0.7-59.1).However, the publication of these studies also took the longest (median 6.2 months; range − 0.1-9.8).

Table 3 .
Proportions of published preprints in total and according to prevention type.*Date of publication before first appearance on medRxiv.