Main

High-throughput ‘omics’ technologies hold great promise to provide detailed characterization of diseases to more effectively predict a patient’s clinical course or to select the most beneficial therapies (see Box 1). These technologies have been embraced enthusiastically in oncology, as the heterogeneous character of malignant diseases presents substantial challenges for cancer detection, prognosis and optimal selection of therapy. Many preclinical studies using these technologies to elucidate biological features and mechanisms have been published, and retrospective studies applying omics assays to stored human biospecimens have been conducted to develop mathematical models to predict clinical endpoints such as survival or response to therapy.

Despite numerous publications, however, few omics-based predictors have been translated successfully into clinically useful tests. A factor that contributes to the slow pace of clinical translation is the challenge of assessing whether the body of evidence for an omics-based test is sufficiently comprehensive and reliable that the test is ready for definitive evaluation in a clinical trial in which it could be used to direct patient care. Translation from research-grade omics assays to clinical-grade omics-based tests1 requires a rigorous development and validation process with attention to the complexities of omics assays and their application to clinical specimens, specialized expertise required to appropriately develop and evaluate mathematical predictor models built from high-dimensional data, and multiple ethical, legal and regulatory issues.

Recently there have been some widely publicized cases of premature advancement of omics-based tests to use in trials in which they were used to guide patient treatment decisions. These cases led to calls for examination of the field of translational omics. The Institute of Medicine (IOM) conducted a study1 to review the field and formed the Committee on the Review of Omics-Based Tests for Predicting Patient Outcomes in Clinical Trials. The group’s task statement included recommending an evaluation process for determining when omics tests are fit for use in clinical trials and applying it to several specific cases of premature use of omics-based tests1. The resulting report laid out a three-phase process for the development and evaluation of omics-based tests for use in clinical trials: the discovery phase, the test validation phase, and the evaluation for clinical utility and use stage.

During the IOM committee deliberations, the NCI convened a workshop to bring together scientists and stakeholders who had an interest in this area of research to stimulate community dialogue. Subsequently, a working group was formed to develop a checklist that would operationalize the principles set forth in the IOM report and the NCI workshop discussions.

The results of those efforts are presented in Table 1, which lists 30 criteria that should be addressed to determine the readiness of an omics test for use in a prospective clinical trial. These criteria apply to any clinical trial involving the investigational use of an omics test that will influence the clinical management of patients in the trial; for example, the selection of therapy. These criteria cover not only the strength of evidence in support of an omics test but also the practical issues that must be considered before the test is used in a clinical setting. The criteria can also be helpful in assessing the reliability and credibility of an omics predictor to justify its use on valuable non-renewable archived specimens collected from patients who were prospectively enrolled in previous clinical studies. This paper presents the criteria in checklist form with brief background. Readers are referred to a recently published companion paper2 for a more complete explanation and elaboration of the rationale for each criterion.

Table 1 Criteria for the use of omics-based predictors in NCI-supported clinical trials
Full size table

Specimen issues

Molecular profiles generated by the use of omics technologies can be sensitive to specimen collection, processing and storage conditions3. Investigators should consider the conditions under which specimens used in developmental studies were collected and handled to assess the robustness of an omics test to various specimen conditions. It may be necessary to conduct additional feasibility studies to document that the omics test will perform satisfactorily under the range of conditions in which the specimens will be obtained and stored in typical clinical settings; alternatively, more restrictive requirements for specimen collection, processing and storage should be clearly specified before the test is used in a clinical trial or other clinical validation study.

Criteria for specimen quality, amount (mass or volume), and composition should be clearly specified in order to qualify a specimen or its isolated analytes as suitable for assay by the omics test. Appropriate criteria will depend on the specimen type and the particular omics assay platform to be used. Details of the specification might include per cent purity of the target cells or intact analyte of interest and specific mass or volume of the specimen or analytes isolated from the specimen. It should be established that it is feasible to achieve these criteria in clinical settings.

Assay issues

Variations in assay procedures due to differences in technical protocols, reagents, and scoring and reporting methods can have a substantial impact on the analytical performance of an omics assay and its comparability among laboratories4,5. Many omics tests are developed using data from retrospective studies in which these aspects of the assay were not standardized. This can lead to uncertainties in how the test will perform when based on assay data from a new laboratory, including the laboratory or laboratories that will generate the assay data for a prospective trial. It is important to develop detailed standard operating procedures (SOPs) for the assay underlying the omics test and to establish that studies conducted previously to clinically validate the omics test were based on data expected to be comparable to new data generated under the specified SOPs.

Analytical performance of the omics assay under the proposed SOPs must be documented and found to be acceptable in terms of metrics such as accuracy, precision, coefficient of variation, sensitivity, specificity, linear range, limit of detection, and limit of quantification, as applicable. Calibrators, analytical standards, and controls are essential components of the SOPs and should be described clearly. Quality assurance procedures should include criteria for acceptance or rejection of assay batches and results from individual specimens. When multiple technicians or laboratories will conduct the assays, monitoring procedures should be in place to ensure comparability across technicians and laboratories. Methods for assay scoring and reporting should be clearly specified. Turnaround times for return of test results should be within acceptable limits that will be dependent on the particular clinical situation and should be sufficiently rapid to not impede clinical management timelines. Feasibility studies to assess assay analytical performance, reproducibility and turnaround times may be required in advance of initiating a clinical trial to firmly establish the suitability of the omics test for use in a real-time clinical setting.

Model development and evaluation

Many omics tests are developed using existing omics, clinical and pathology data or using data generated from retrospective specimen collections. These data may be incomplete or unreliable and should be examined for errors, inconsistencies or bias. Omics assays can be sensitive to a variety of ancillary technical influences that result in artefacts in the generated data. Of particular concern is the potential for such artefacts to be confounded with clinical variables or endpoints. Efforts should be made to identify potential confounders, including source of specimens (for example, clinical sites processing specimens differently), laboratory performing the omics assay, and assay batches6.

Examples of flawed applications of statistical approaches for development of omics predictors and for evaluation of their performance are abundant in the literature7,8,9. Model overfitting, which occurs when a statistical model describes random noise instead of capturing the true association between predictor variables and a clinical endpoint, is a common problem in omics research projects, in which the number of analytes measured per specimen exceeds the number of specimens studied. Overfitting can be reduced by the use of model ‘regularization’ approaches that constrain the complexity of the model, but these approaches do not completely eliminate overfitting risk. It is common for researchers without the appropriate expertise to misunderstand and misapply modelling techniques. In addition, if flawed methods for model performance assessment are used, then overfitting may escape detection. A common mistake is failure to maintain strict separation between data used to build a model (‘training set’) and data used to assess model performance (‘testing set’). Numerous published papers have inappropriately reported model performance estimates based on resubstitution of data used to build a model back into that same model. These so-called ‘resubstitution estimates’ are severely (optimistically) biased. Assessment of model performance on the combined training and testing data sets is similarly problematic. Re-use of training data is acceptable only if performed properly using data resampling methods10 that iteratively split the training data to hold out subsets of the data that are not used for model building and can therefore be used to check model performance.

Development of an omics predictor can be an iterative process involving several adjustments to improve performance. With regard to the three phases of the development and evaluation process in the IOM report1 on omics tests, it is noted in the report that preliminary validations may occur in the test validation phase, and the definitive evaluation of clinical utility takes place in the final phase. It is important to be able to discern the point at which the omics test is ‘locked down’, or finalized, in all aspects, including specimen requirements, technical protocol for assay, data preprocessing, the form of mathematical predictor model, and interpretation of the test result. The test is then ready to enter the final evaluation for clinical utility and use stage, at which there are three basic options for clinical utility evaluation: first, a prospective evaluation of the omics test on a retrospective specimen collection from a clinical trial or prospective cohort study; second, a prospective clinical trial in which the test does not direct patient management; and third, a prospective clinical trial in which the omics test is used to direct patient management. Ideally, there should have been a blinded and rigorous preliminary validation of performance of the locked-down model on an external independent specimen set during the test validation phase. If an independent external validation set is not possible because adequate specimen collections do not exist, then existing performance evaluations based on internal validations should be carefully reviewed to ensure that they were rigorous and used appropriate methods. In this situation, it may be necessary to use a clinical trial design that does not allow the test to influence patient care.

When further adjustments are made to the omics test or data after the final validation data have been unblinded, there is a risk of compromising the validation. If the omics test is adjusted, either a new validation must be performed or additional evidence must be obtained; for example, by conducting an assay-bridging study to ensure that the adjustments to the test have not adversely affected its performance.

Investigators should be prepared to supply data and computer code as part of the review process for proposals to use omics tests in clinical trials. It is highly recommended that investigators follow reproducible research practices so that they will be able to supply the needed information quickly and easily for verification of the validation of the test and its locked-down form. Readers are referred to the companion publication2 for further discussion of recommended reproducible research practices.

Clinical trial design

A clinical trial for definitive evaluation of an omics test should be conducted using the same rigorous standards expected for clinical trials evaluating experimental therapies. In some circumstances, high-level evidence can be obtained by use of specimens from an already-completed clinical trial11. Accepted standards for good clinical practice must be followed12,13, including development of a formal protocol with clearly stated objectives and eligibility criteria, an informatics plan for management of clinical and omics data, a pre-specified study design14 and statistical analysis plan, complete specification of the omics test, and justification for equipoise for any treatment randomizations (if the trial is conducted prospectively). The study team must include individuals with appropriate expertise to assume responsibility for the clinical, laboratory, pathology, bioinformatics, data management and statistical aspects of the study.

Ethical, legal and regulatory issues

Numerous ethical, legal and regulatory issues must be addressed in the course of developing an omics test for clinical use. Research involving human subjects, which includes retrospective use of specimens from living subjects, requires that adequate protection is in place to ensure the safety of patients and the privacy and confidentiality of patient information15. Ensuring appropriate protections has become more challenging as omics technologies make it possible to provide detailed genetic characterizations of individuals and much research data are made publicly available. Informed consent documents for a clinical trial using an omics test to guide patient management must accurately describe any potential risks from participation in a study and all potential conflicts of interest on the part of study investigators or sponsoring institutions. Laboratory tests must be conducted in environments that meet Clinical Laboratory Improvement Amendments certification requirements if the results will be reported to the patient or the patient’s physician16. Responsible parties at participating institutions (for example, institutional review boards, protocol review committees), trial sponsors (for example, the NCI, universities, companies), and the US Food and Drug Administration (FDA) (for example, for Investigational Device Exemption (IDE)17 or Investigational New Drug18 applications) must be fully informed of study details and approve the study before it proceeds.

If the omics assay to be used in a clinical trial could be considered a significant-risk assay, including—but not limited to—one used to choose among treatments, investigators must consult with the FDA to determine whether an IDE from the Center for Devices and Radiologic Health, or a similar evaluation carried out through the Investigational New Drug process, is required. The complexities of omics-based tests, together with the FDA’s evolving view of regulatory enforcement discretion for these tests, make it important to have early communications with the FDA. Investigators may find it helpful to discuss the trial formally with the FDA in a pre-submission process if they are not familiar with IDE requirements19.

Intellectual property issues may apply to the use of the specimens, biomarkers, assays, and computer software used for calculation of the predictor. Intellectual property rights should be documented and respected by all parties involved. Potential conflicts of interest of study investigators must be disclosed and managed.

Summary

Evaluation of the readiness of an omics test to be used for clinical care requires careful consideration of the body of evidence supporting the test’s analytical and clinical validity and potential clinical utility, as well as an understanding of ethical, legal and regulatory issues. Funding bodies and journals are encouraged to consider using the checklist as an evaluation guide in their review processes. The NCI plans to use the checklist presented here to evaluate proposals for the use of omics tests in clinical trials where the test will be used to guide patient care. Although it is not expected that exploratory studies using omics assays or studies aiming to develop omics tests will meet all of the checklist criteria, the checklist does provide a convenient framework by which to assess the stage of development of an omics test and the strength and quality of the accumulated evidence. Several of the checklist criteria (those that are not specific to the development of models from high-dimensional data) also apply to studies of single biomarkers, or limited panels of biomarkers, measured by a variety of conventional assay methods. The checklist may, therefore, serve as a useful reference in a variety of review settings.

It is hoped that this 30-point checklist will guide investigators towards the use of best practices in omics test development, help them to more reliably evaluate the quality of evidence in support of omics tests, and assist them in planning appropriately for the clinical use of omics predictors. The ultimate goal is to develop a more efficient, reliable and transparent process to move omics assays from promising research results to clinically useful tests that improve patient care and outcome.