Patents in the life sciences are a critical metric of innovation and a cornerstone for the commercialization of new life-science- and healthcare-related technologies. Patent landscaping has emerged as a methodology for analyzing multiple patent documents to uncover technological trends1, geographic distributions of patents2, patenting trends and scope3, highly cited patents4 and a number of other uses5. Many such analyses are published in high-impact journals6,7,8, potentially allowing them to gain high visibility among academic, industry and government stakeholders. Such analyses may be used to inform decision-making processes, such as prioritization of funding areas, identification of commercial competition (and therefore strategy development), or implementation of policy to encourage innovation or to ensure responsible licensing of technologies. Patent landscaping may also provide a means for answering fundamental questions regarding the benefits and drawbacks of patenting in the life sciences, a subject on which there remains considerable debate9,10,11 but limited empirical evidence.

The aim of the patent-landscaping process is to capture a set of pre-defined patent documents and to analyze them in some manner (Box 1). In general, the process comprises three main stages: (i) designing and conducting searches, (ii) data cleaning and curation, and (iii) data analysis and interpretation5. This process is analogous to the process for conducting systematic reviews of academic literature; however, in patent landscapes, the patent documents replace academic articles. Similar to systematic reviews, patent landscapes are important in allowing higher-level insights to be drawn that could not be achieved by analysis of the documents in isolation. Unlike systematic reviews, however, patent landscapes are often published under different guises than those for original research articles, possibly leading to a lack of emphasis on the disclosure of the key pieces of information that underpin the analyses and conclusions.

As in any type of research, to allow for reproducibility and effective evaluation of accuracy and quality, it is essential that studies are reported adequately. The specific methodologies used, and the results generated, must be reported; clear justification and the aims of the investigations need to be provided to assess the validity of any conclusions, and any funding or conflicts of interest should be disclosed to assess potential biases. The importance of adequate reporting has been heavily reinforced and policed in healthcare-related research12,13,14, including in systematic reviews15; however, outside of healthcare-related research, discussion has been more limited. Despite the important potential implications of patent landscapes from a scientific, commercial and political perspective, criticism has emerged over a lack of standardization, transparency, clear justification of the methodology, and reporting of the ad hoc investigations being conducted16,17.

Here, we provide the first systematic assessment of the quality of reporting in patent landscapes in the life sciences that have been published in academic journals. Given the heterogeneous nature of the methodologies used for, and the questions addressed by, patent landscaping16, consistent methodological practice may not be practical or useful, and this could indeed stifle innovative methodologies from emerging. Clarity of reporting, however, is essential regardless of the approach taken and the hypotheses tested, and assessment of reporting clarity should represent a significant component of the peer review process and should be expected in scientific articles. Patent landscapes are published in a variety of journal types of different impact factors, and as various different article types (e.g., review articles and original research articles). The relationship between these indicators and reporting quality is also explored.

Results

Search results, study inclusion and study characteristics. Searches and screening resulted in a total of 81 full-text studies for inclusion in this systematic review (Fig. 1; Supplementary Methods; Supplementary Tables 1 and 2). After the removal of duplicate records, we screened the abstracts of 3,348 articles for relevance and found 577 records that could not be excluded based purely on reviewing the abstracts and titles and that were assessed as full texts, where available. We excluded 380 articles because, after deeper examination, they were not patent landscapes or reviews (n = 201), there was no mention of a search for patents (n = 78), they were not life science focused (n = 100), or they were not journal articles (n = 1). An additional potentially relevant 116 articles were identified, which we were unable to exclude based purely on abstract or title, but for which the full text was not available.

Figure 1: PRISMA flow diagram39 detailing the number of studies included at each stage and the reasons for removal.
figure 1

See Supplementary Methods for methodology.

Full size image

Full-text papers that were included for analysis included a broad range of analyses (Fig. 2a), were published in different article types (Fig. 2b) and examined a number of different research areas (Fig. 2c). The most common forms of analyses were temporal, assignee and geographical, with each type of analysis appearing in 75% or more of the papers. Nearly half of the papers included here were research papers (49%), and the remaining papers were either reviews (29%) or other article types (22%). The majority (77%) of papers were published in scientific journals, and the rest (23%) were published in other categories of journals, such as legal or business journals (Fig. 2d).

Figure 2: Characteristics of included patent landscape papers.
figure 2

(a) Types of analysis—percentage of patent landscaping articles conducting different types of analysis in studies included in this systematic review (n = 81 for all). Each article could be assigned to more than one type of analysis (definitions in Table 1). (b) Article type—article types in which patent landscapes included in this systematic review appear. (c) Technology area—the technology areas focused on in the articles included in this systematic review. Each article was assigned to a single technology area. (d) Journal type—type of journal in which patent landscapes were published, scientific or 'other', which included journals that are primarily focused on legal issues, business or other topics.

Full size image

Quality of reporting

In 81 articles assessed for compliance with respect to 20 items that were considered together to represent an adequately reported study, none of the articles reported all of the items included in our checklist that were relevant to them. Mean compliance across all articles was 64% (s.d. ± 15%). Table 2 lists the number and percentage of articles reporting each item.

Table 2 Number and percentage of articles that have reported items in the reporting-quality checklist (n = 81; n = 70 for items with asterisks (*))
Full size table

The percentage compliance was not normally distributed (Shapiro–Wilk test of normality, W = 0.93392, P ≤ 0.001), and normality could not be achieved through transformation of the data. Therefore, Kruskal–Wallis tests were used for comparisons of compliance against to article and journal type, and a Spearman's rank correlation coefficient was calculated for compliance with respect to the SCImago journal rank (SJR; http://www.scimagojr.com/journalrank.php). No correlation between SJR and compliance was seen (Fig. 3a). Mean compliance in research, review and 'other' articles were 70% (s.d. ± 8%), 55% (s.d. ± 16%) and 60% (s.d. ± 17%), respectively (Fig. 3b). A significant effect of article type on compliance was observed. Mean compliance in scientific and 'other' journals was 64% (s.d. ± 16%) or 63% (s.d. ± 11%), respectively, which did not differ significantly (Fig. 3c).

Figure 3: Compliance relationships.
figure 3

(a) SCImago Journal Ranking: relationship between SJR and percentage compliance to the reporting item checklist (n = 81; Spearman's Rho, rs = 0.085, P = 0.475). (b) Article type: relationship between article type and percentage compliance to the reporting item checklist (error bars ± 95% CI; n = 81; Kruskal–Wallis test, H = 20.5, P < 0.001). (c) Journal type: relationship between article type (scientific journals and 'other' journal types, such as legal or business journals), and percentage compliance to reporting item checklist (error bars ± 95% CI; n = 81; Kruskal–Wallis test, H = 0.114, P = 0.736).

Full size image

Discussion

The final step in conducting academic research is not the analysis of data but the communication and dissemination of the research in a manner that allows the approach taken to be determined and reproduced, and the validity of the findings to be assessed18. The primary mechanism of this dissemination and communication within academia is via journal publications, which represent the means by which the quality of a study is assessed. Our analysis provides evidence that the quality of reporting in patent landscapes in the life sciences in such publications is generally inadequate.

Reporting of methods and results. Many articles do not report critical methodological items. Without full reporting of the following methodological items—eligibility criteria of patents to be included; search strategy; databases, dates, patent offices and components of patents searched; software used; patent selection process; and details of analysis conducted (Table 2, items 9 to 17)—reproducing and validating a patent landscape methodology is, in our opinion, not possible. Eight articles (9.9%) reported all of the methodological items that were applicable to them. Just one article (1.4%) reported that patent selection was blindly reproduced, and 24% of articles listed the patents included in the study, i.e., the data set on which the conclusions and analyses were based.

The lack of reporting of key methodological items and results challenges the interpretation of the conclusions and renders the ability to reproduce studies (e.g., to update them) impossible in many cases. A patent landscape represents a considerable amount of effort, and without clarity over their methods and results, the impact of this effort can be substantially reduced; greater clarity would allow additional analysis to be conducted by researchers in the future, maximizing the potential benefit of the research19. Improvements in the reporting of the methods and results could allow for meta-analyses of patent-landscape-related papers and for independent researchers to use the results in their own investigations for other purposes.

Potential for conflicts of interest. Conflicts of interest and funding sources are also relatively poorly reported items (45%). Patents are inherently linked to commercial interests; therefore, the declaration of conflicts of interest and any funding sources is important in the evaluation of potential biases. In clinical trials, lower quality of reporting has been associated with increased effect sizes20. Although there is no effect size, as such, to evaluate in patent landscaping articles, incentives for the portrayal of inaccurate information by authors or funders could easily be conceived, and the promotion of transparency for conflicts of interest and funding should therefore be encouraged.

Explaining reporting quality. Our findings are not limited to lower-quality journals. It is generally assumed that the higher the quality of, and therefore the quality of reporting in, an article, the higher the likelihood that it will published in a high-impact journal18,21. However, no statistically significant correlation was observed between the SJR and the quality of reporting in patent landscaping articles (Fig. 3a), suggesting that reporting quality is not currently an important determinant in the publication of patent landscapes in high-quality journals. This is unlike other areas, in which a number of studies conducting similar analyses in different fields have found marked relationships between impact factor and reporting quality22,23, and these may be due to the fact that there are a limited number of well-reported patent landscapes on which to model reporting.

Small but significant differences in reporting quality are associated with different article types. Perhaps expectedly, reviews are less well reported than research articles (Fig. 3), which indicates that structured paper formats might be more conducive to complete reporting. Given that patent landscapes may just be one component of an otherwise narrative review, structured reporting, as in many research articles, is unlikely to be practical for all papers. However, emphasizing that the patent-landscaping component of any paper represents research that others may wish to rely on for further analysis could help to improve reporting, and methodologies could be reported in supplementary materials.

Analysis types. Patent landscaping has previously been described, for the most part, as relatively simplistic5, and much of the analysis recorded and observed in our study supports this. A large proportion of papers presents numbers of patents over time, patents per geographical region, patents per assignee or inventor, or other count data (Fig. 2). There is no inherent problem in the use of such analyses; however, patent numbers and other such measures cannot alone portray all of the information that may be useful or required by the reader. For example, it is possible that within one research area, there are a large number of patents that focus on a very narrow set of inventions, whereas in another research area there may be very few patents with broad scope. Without some consideration of the information within the patent documents, discerning these differences would be challenging. With an appropriate discussion of limitations, these issues could be allayed; however, limitations were discussed in less than half of the included papers (46%).

In some cases, more advanced analytical approaches have been used to interrogate patent documents in more detail, although these still raise some concerns. Software was used to thematically cluster patent documents and present the outcome visually in 'cluster maps' in some papers (28%). From such data, areas of high patent activity are often identified1,3,24 and may be used to identify 'gaps' in the technology or research landscape25. Other papers have mentioned the use of patents to identify gaps, although not through an explicit methodology26,27. Proprietary software is often used to generate cluster maps, but the algorithms underpinning such software are rarely discussed or detailed in the papers using them, and in some cases the algorithms underpinning the software do not appear to be publicly available at all. Without detailed analysis of patent claims by an expert, the identification of gaps in research or technologies is difficult to ascertain with existing methods; however, very few papers conducted any form of claims analysis (2%). As mentioned above, an appropriate discussion of the limitations would be useful in addressing these concerns.

Outlook. The findings of this systematic review are congruent with similar investigations that have been conducted in other fields reporting omissions in methods28,29 (including statistical methods30), incomplete presentation of data thereby preventing analysis in the future31 and inadequate statements for conflicts of interest32. The findings also provide empirical evidence for statements previously made in relation to the quality of patent landscapes5. To address reporting issues in other fields, a great number of reporting guidelines have been developed, including checklists of items that should generally be reported, and which together represent an adequately reported study. Introduction33 and endorsement34 of guidelines have been associated with improved quality in the reporting of clinical trials. Our study provides the empirical justification for the development of a guideline to improve the quality of reporting in patent landscaping articles, which is registered as being under development on the Equator Network website (http://www.equator-network.org). The purpose of the guideline is to improve transparency and standardization of reporting to allow reproducibility, comparability and accurate evaluation of patent landscapes.

In the meantime, researchers involved in patent landscaping should aim to improve reporting quality, motivated by the fact that it will ultimately increase the impact of the research and provide a greater contribution to the scientific community. Authors have a responsibility to ensure that information is clearly presented to readers in all forms of research, and patent landscapes should not be considered an exception. Journals and reviewers, too, are responsible for ensuring that reporting is adequate and must act as the gatekeepers by working with authors to ensure that transparency is achieved. We are not suggesting that overwhelming the reader with methodological details and extensive results in every article is required; supplementary materials can and should be used to ensure comprehensive reporting. With improved transparency and comprehensive reporting, patent landscaping will continue to provide useful insights, critically, in a manner that allows them to be reproduced and fairly evaluated.

Limitations. This systematic review focused solely on the reporting quality of patent landscapes in academic journals. It should be noted that patent landscapes are commonly reported outside of academic journals, such as those conducted by government bodies or industry. The quality of reporting in certain available publications35 in general appears to be quite good, perhaps due to the lack of limitation with regards to document length in comparison to academic papers. However, even seemingly detailed landscapes lack full disclosure of search terms36,37 and contain only superficial explanations of the algorithms employed38. The findings of this study may still, therefore, be useful in the context of reporting studies outside of academia.

Additionally, we were unable to access the full texts of 116 potentially relevant articles that could not be excluded on the basis of the abstract alone. There is no reason to suspect that these articles would have provided evidence contrary to that presented.

Conclusions

Patent documents are an exceptionally rich source of information that can and should be mined and analyzed for a number of purposes. The breadth of possibilities for analysis of patent documents may preclude the development of standardized methodologies, and as such this may not be possible. However, without adequate reporting, the full value of such analyses will not be realized, and even the most rigorous and elegant investigations may be limited in reach because they simply cannot be reproduced and critically evaluated. With transparent reporting and consideration to the other observations made in our paper, the value of patent landscape studies within and beyond the academic community could increase considerably.

Author contributions

J.A.S. and Z.A. contributed equally to this work. A.J.C. and D.A.B. contributed equally to this work.

Table 1 Definitions of types of analysis used in the extraction sheet
Full size table