Skip to main content

Thank you for visiting You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

Reply: Some considerations on digital health validation

We thank van Velthoven and Smith for their thoughtful comments regarding our approach to validating digital health. They introduce several points that could benefit from further clarification and future investigation.

Regarding the issue of scope and setting, we believe our approach is flexible enough to be adapted to address both the range of clinical contexts (e.g. home based and acute care) and the variety of solution types (e.g. mobile apps, wearables, artificial intelligence-based decision support). However, not all domains of evaluation (i.e. clinical, technical, usability, cost) will be as relevant or robustly evaluated depending on the technology capability and the context of the use. The larger question of how to determine which solutions should be evaluated remains open to debate and will likely be a function of who is able to support this type of evaluation—essentially the broader the support, the larger the scope.

Given the current fragmentation in healthcare, a stakeholder-centric approach may provide a practical guide. For example, patient advocacy groups could support evaluation in technologies most relevant or used by their constituency. Physician specialty societies may select emerging technologies relevant to their field. Payors could focus on technologies that they believe have the greatest impact on minimizing harm and costs. Industry participants could seek to differentiate their products by undergoing more rigorous evaluation. Relying on existing digital gatekeepers such as app stores to maintain quality standards as they relate to health content is likely an unrealistic expectation as it would require greater investment and expertise on their part and is at odds with the goal of increasing the variety and availability of products in their marketplaces.

van Velthoven and Smith highlighted that there may be little need for extensive product evaluation since the majority of solutions do not appear to cause significant harm. We believe this is, in part, due to the lack of systematic evaluation to assess product safety and to the relatively low technical capabilities of the technologies that are most widely accessible. To this latter point, as diagnostic sensors and therapeutic components advance, the potential for greater harm will also increase. Even in the current relatively “low tech” environment, however, harm may be occurring, albeit covertly, and in ways that are intangible and difficult to measure. In this context, the historical comparison to health information websites is apt. Clinic patients commonly inquire about medical information read online, which can be a frequent source of misunderstanding and anxiety. While unlikely to be causing physical harm, this misinformation likely has clinical impact. Analogously, even well intentioned “low tech” apps can generate confusion and potentially misguided or incorrect recommendations, which can adversely affect health. For example, we have discovered in our pilot that focuses on oncology apps assessment, some information is inaccurate or patently false. With respect to overall standards in digital health, the National Institute for Health and Care Excellence teamed with the National Health Service published a promising framework (concurrent with the release of this work) for different levels of evidence that is robust.1

Regarding the need for incorporating end user and subject matter expertise early in the product lifecycle, we agree and believe this input should be solicited or integrated within every step including identifying “needs” as suggested by van Velthoven and Smith. In our pilot we have made extensive use of user centered design (UCD) principles to define requirements against which the utility of the candidate apps is evaluated. Both patient and clinician viewpoints have been incorporated in the UCD process to ensure these perspectives are adequately represented in the evaluation.

We also agree it is important to recognize the challenges that other multi-stakeholder efforts have encountered. Additionally, one should incorporate experience from low and middle income countries since uptake of digital tools may be faster and in far greater numbers, given limitations in traditional health infrastructure and resources. Finally, we also agree that achieving the multi-stakeholder vision proposed is challenging and will benefit from tangible demonstration of value which we are currently undertaking. The need for a transparent, comprehensive, and standards-based assessment that is guided by end-user requirements is clear and will only continue to grow. We believe our approach provides one feasible path that provides both structure and flexibility.


  1. 1.

    Evidence Standards Framework for Digital Health Technologies. National Institute for Health and Care Excellence. (2019).

Download references

Author information




All authors were responsible for drafting and revision of this response.

Corresponding author

Correspondence to Simon C. Mathews.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Mathews, S.C., McShea, M.J., Hanley, C.L. et al. Reply: Some considerations on digital health validation. npj Digit. Med. 2, 103 (2019).

Download citation


Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing