ABSTRACT The US National Cancer Institute (NCI), in collaboration with scientists representing multiple areas of expertise relevant to ‘omics’-based test development, has developed a

checklist of criteria that can be used to determine the readiness of omics-based tests for guiding patient care in clinical trials. The checklist criteria cover issues relating to specimens,

assays, mathematical modelling, clinical trial design, and ethical, legal and regulatory aspects. Funding bodies and journals are encouraged to consider the checklist, which they may find

useful for assessing study quality and evidence strength. The checklist will be used to evaluate proposals for NCI-sponsored clinical trials in which omics tests will be used to guide

REAL-WORLD PATIENTS USING MACHINE LEARNING-BASED TRIAL EMULATIONS Article Open access 03 January 2025 MAIN High-throughput ‘omics’ technologies hold great promise to provide detailed

characterization of diseases to more effectively predict a patient’s clinical course or to select the most beneficial therapies (see Box 1). These technologies have been embraced

enthusiastically in oncology, as the heterogeneous character of malignant diseases presents substantial challenges for cancer detection, prognosis and optimal selection of therapy. Many

preclinical studies using these technologies to elucidate biological features and mechanisms have been published, and retrospective studies applying omics assays to stored human biospecimens

have been conducted to develop mathematical models to predict clinical endpoints such as survival or response to therapy. Despite numerous publications, however, few omics-based predictors

have been translated successfully into clinically useful tests. A factor that contributes to the slow pace of clinical translation is the challenge of assessing whether the body of evidence

for an omics-based test is sufficiently comprehensive and reliable that the test is ready for definitive evaluation in a clinical trial in which it could be used to direct patient care.

Translation from research-grade omics assays to clinical-grade omics-based tests1 requires a rigorous development and validation process with attention to the complexities of omics assays

and their application to clinical specimens, specialized expertise required to appropriately develop and evaluate mathematical predictor models built from high-dimensional data, and multiple

ethical, legal and regulatory issues. Recently there have been some widely publicized cases of premature advancement of omics-based tests to use in trials in which they were used to guide

patient treatment decisions. These cases led to calls for examination of the field of translational omics. The Institute of Medicine (IOM) conducted a study1 to review the field and formed

the Committee on the Review of Omics-Based Tests for Predicting Patient Outcomes in Clinical Trials. The group’s task statement included recommending an evaluation process for determining

when omics tests are fit for use in clinical trials and applying it to several specific cases of premature use of omics-based tests1. The resulting report laid out a three-phase process for

the development and evaluation of omics-based tests for use in clinical trials: the discovery phase, the test validation phase, and the evaluation for clinical utility and use stage. During

the IOM committee deliberations, the NCI convened a workshop to bring together scientists and stakeholders who had an interest in this area of research to stimulate community dialogue.

Subsequently, a working group was formed to develop a checklist that would operationalize the principles set forth in the IOM report and the NCI workshop discussions. The results of those

efforts are presented in Table 1, which lists 30 criteria that should be addressed to determine the readiness of an omics test for use in a prospective clinical trial. These criteria apply

to any clinical trial involving the investigational use of an omics test that will influence the clinical management of patients in the trial; for example, the selection of therapy. These

criteria cover not only the strength of evidence in support of an omics test but also the practical issues that must be considered before the test is used in a clinical setting. The criteria

can also be helpful in assessing the reliability and credibility of an omics predictor to justify its use on valuable non-renewable archived specimens collected from patients who were

prospectively enrolled in previous clinical studies. This paper presents the criteria in checklist form with brief background. Readers are referred to a recently published companion paper2

for a more complete explanation and elaboration of the rationale for each criterion. BOX 1: DEFINITION OF ‘OMICS’ In its report, _Evolution of Translational Omics: Lessons Learned and the

Path Forward_, the Institute of Medicine Committee on the Review of Omics-Based Tests for Predicting Patient Outcomes in Clinical Trials defines ‘omics’ as the study of related sets of

biological molecules in a comprehensive fashion. Examples of omics disciplines include genomics, transcriptomics, proteomics, metabolomics and epigenomics. An omics-based test is defined as

“an assay composed of or derived from multiple molecular measurements and interpreted by a fully specified computational model to produce a clinically actionable result”1. SPECIMEN ISSUES

Molecular profiles generated by the use of omics technologies can be sensitive to specimen collection, processing and storage conditions3. Investigators should consider the conditions under

which specimens used in developmental studies were collected and handled to assess the robustness of an omics test to various specimen conditions. It may be necessary to conduct additional

feasibility studies to document that the omics test will perform satisfactorily under the range of conditions in which the specimens will be obtained and stored in typical clinical settings;

alternatively, more restrictive requirements for specimen collection, processing and storage should be clearly specified before the test is used in a clinical trial or other clinical

validation study. Criteria for specimen quality, amount (mass or volume), and composition should be clearly specified in order to qualify a specimen or its isolated analytes as suitable for

assay by the omics test. Appropriate criteria will depend on the specimen type and the particular omics assay platform to be used. Details of the specification might include per cent purity

of the target cells or intact analyte of interest and specific mass or volume of the specimen or analytes isolated from the specimen. It should be established that it is feasible to achieve

these criteria in clinical settings. ASSAY ISSUES Variations in assay procedures due to differences in technical protocols, reagents, and scoring and reporting methods can have a substantial

impact on the analytical performance of an omics assay and its comparability among laboratories4,5. Many omics tests are developed using data from retrospective studies in which these

aspects of the assay were not standardized. This can lead to uncertainties in how the test will perform when based on assay data from a new laboratory, including the laboratory or

laboratories that will generate the assay data for a prospective trial. It is important to develop detailed standard operating procedures (SOPs) for the assay underlying the omics test and

to establish that studies conducted previously to clinically validate the omics test were based on data expected to be comparable to new data generated under the specified SOPs. Analytical

performance of the omics assay under the proposed SOPs must be documented and found to be acceptable in terms of metrics such as accuracy, precision, coefficient of variation, sensitivity,

specificity, linear range, limit of detection, and limit of quantification, as applicable. Calibrators, analytical standards, and controls are essential components of the SOPs and should be

described clearly. Quality assurance procedures should include criteria for acceptance or rejection of assay batches and results from individual specimens. When multiple technicians or

laboratories will conduct the assays, monitoring procedures should be in place to ensure comparability across technicians and laboratories. Methods for assay scoring and reporting should be

clearly specified. Turnaround times for return of test results should be within acceptable limits that will be dependent on the particular clinical situation and should be sufficiently rapid

to not impede clinical management timelines. Feasibility studies to assess assay analytical performance, reproducibility and turnaround times may be required in advance of initiating a

clinical trial to firmly establish the suitability of the omics test for use in a real-time clinical setting. MODEL DEVELOPMENT AND EVALUATION Many omics tests are developed using existing

omics, clinical and pathology data or using data generated from retrospective specimen collections. These data may be incomplete or unreliable and should be examined for errors,

inconsistencies or bias. Omics assays can be sensitive to a variety of ancillary technical influences that result in artefacts in the generated data. Of particular concern is the potential

for such artefacts to be confounded with clinical variables or endpoints. Efforts should be made to identify potential confounders, including source of specimens (for example, clinical sites

processing specimens differently), laboratory performing the omics assay, and assay batches6. Examples of flawed applications of statistical approaches for development of omics predictors

and for evaluation of their performance are abundant in the literature7,8,9. Model overfitting, which occurs when a statistical model describes random noise instead of capturing the true

association between predictor variables and a clinical endpoint, is a common problem in omics research projects, in which the number of analytes measured per specimen exceeds the number of

specimens studied. Overfitting can be reduced by the use of model ‘regularization’ approaches that constrain the complexity of the model, but these approaches do not completely eliminate

overfitting risk. It is common for researchers without the appropriate expertise to misunderstand and misapply modelling techniques. In addition, if flawed methods for model performance

assessment are used, then overfitting may escape detection. A common mistake is failure to maintain strict separation between data used to build a model (‘training set’) and data used to

assess model performance (‘testing set’). Numerous published papers have inappropriately reported model performance estimates based on resubstitution of data used to build a model back into

that same model. These so-called ‘resubstitution estimates’ are severely (optimistically) biased. Assessment of model performance on the combined training and testing data sets is similarly

problematic. Re-use of training data is acceptable only if performed properly using data resampling methods10 that iteratively split the training data to hold out subsets of the data that

are not used for model building and can therefore be used to check model performance. Development of an omics predictor can be an iterative process involving several adjustments to improve

performance. With regard to the three phases of the development and evaluation process in the IOM report1 on omics tests, it is noted in the report that preliminary validations may occur in

the test validation phase, and the definitive evaluation of clinical utility takes place in the final phase. It is important to be able to discern the point at which the omics test is

‘locked down’, or finalized, in all aspects, including specimen requirements, technical protocol for assay, data preprocessing, the form of mathematical predictor model, and interpretation

of the test result. The test is then ready to enter the final evaluation for clinical utility and use stage, at which there are three basic options for clinical utility evaluation: first, a

prospective evaluation of the omics test on a retrospective specimen collection from a clinical trial or prospective cohort study; second, a prospective clinical trial in which the test does

not direct patient management; and third, a prospective clinical trial in which the omics test is used to direct patient management. Ideally, there should have been a blinded and rigorous

preliminary validation of performance of the locked-down model on an external independent specimen set during the test validation phase. If an independent external validation set is not

possible because adequate specimen collections do not exist, then existing performance evaluations based on internal validations should be carefully reviewed to ensure that they were

rigorous and used appropriate methods. In this situation, it may be necessary to use a clinical trial design that does not allow the test to influence patient care. When further adjustments

are made to the omics test or data after the final validation data have been unblinded, there is a risk of compromising the validation. If the omics test is adjusted, either a new validation

must be performed or additional evidence must be obtained; for example, by conducting an assay-bridging study to ensure that the adjustments to the test have not adversely affected its

performance. Investigators should be prepared to supply data and computer code as part of the review process for proposals to use omics tests in clinical trials. It is highly recommended

that investigators follow reproducible research practices so that they will be able to supply the needed information quickly and easily for verification of the validation of the test and its

locked-down form. Readers are referred to the companion publication2 for further discussion of recommended reproducible research practices. CLINICAL TRIAL DESIGN A clinical trial for

definitive evaluation of an omics test should be conducted using the same rigorous standards expected for clinical trials evaluating experimental therapies. In some circumstances, high-level

evidence can be obtained by use of specimens from an already-completed clinical trial11. Accepted standards for good clinical practice must be followed12,13, including development of a

formal protocol with clearly stated objectives and eligibility criteria, an informatics plan for management of clinical and omics data, a pre-specified study design14 and statistical

analysis plan, complete specification of the omics test, and justification for equipoise for any treatment randomizations (if the trial is conducted prospectively). The study team must

include individuals with appropriate expertise to assume responsibility for the clinical, laboratory, pathology, bioinformatics, data management and statistical aspects of the study.

ETHICAL, LEGAL AND REGULATORY ISSUES Numerous ethical, legal and regulatory issues must be addressed in the course of developing an omics test for clinical use. Research involving human

subjects, which includes retrospective use of specimens from living subjects, requires that adequate protection is in place to ensure the safety of patients and the privacy and

confidentiality of patient information15. Ensuring appropriate protections has become more challenging as omics technologies make it possible to provide detailed genetic characterizations of

individuals and much research data are made publicly available. Informed consent documents for a clinical trial using an omics test to guide patient management must accurately describe any

potential risks from participation in a study and all potential conflicts of interest on the part of study investigators or sponsoring institutions. Laboratory tests must be conducted in

environments that meet Clinical Laboratory Improvement Amendments certification requirements if the results will be reported to the patient or the patient’s physician16. Responsible parties

at participating institutions (for example, institutional review boards, protocol review committees), trial sponsors (for example, the NCI, universities, companies), and the US Food and Drug

Administration (FDA) (for example, for Investigational Device Exemption (IDE)17 or Investigational New Drug18 applications) must be fully informed of study details and approve the study

before it proceeds. If the omics assay to be used in a clinical trial could be considered a significant-risk assay, including—but not limited to—one used to choose among treatments,

investigators must consult with the FDA to determine whether an IDE from the Center for Devices and Radiologic Health, or a similar evaluation carried out through the Investigational New

Drug process, is required. The complexities of omics-based tests, together with the FDA’s evolving view of regulatory enforcement discretion for these tests, make it important to have early

communications with the FDA. Investigators may find it helpful to discuss the trial formally with the FDA in a pre-submission process if they are not familiar with IDE requirements19.

Intellectual property issues may apply to the use of the specimens, biomarkers, assays, and computer software used for calculation of the predictor. Intellectual property rights should be

documented and respected by all parties involved. Potential conflicts of interest of study investigators must be disclosed and managed. SUMMARY Evaluation of the readiness of an omics test

to be used for clinical care requires careful consideration of the body of evidence supporting the test’s analytical and clinical validity and potential clinical utility, as well as an

understanding of ethical, legal and regulatory issues. Funding bodies and journals are encouraged to consider using the checklist as an evaluation guide in their review processes. The NCI

plans to use the checklist presented here to evaluate proposals for the use of omics tests in clinical trials where the test will be used to guide patient care. Although it is not expected

that exploratory studies using omics assays or studies aiming to develop omics tests will meet all of the checklist criteria, the checklist does provide a convenient framework by which to

assess the stage of development of an omics test and the strength and quality of the accumulated evidence. Several of the checklist criteria (those that are not specific to the development

of models from high-dimensional data) also apply to studies of single biomarkers, or limited panels of biomarkers, measured by a variety of conventional assay methods. The checklist may,

therefore, serve as a useful reference in a variety of review settings. It is hoped that this 30-point checklist will guide investigators towards the use of best practices in omics test

development, help them to more reliably evaluate the quality of evidence in support of omics tests, and assist them in planning appropriately for the clinical use of omics predictors. The

ultimate goal is to develop a more efficient, reliable and transparent process to move omics assays from promising research results to clinically useful tests that improve patient care and

outcome. CHANGE HISTORY * _ 16 OCTOBER 2013 The print version inadvertently lacked final corrections to affiliation 5 and Table 1; however, the online PDF and HTML versions are correct. _

REFERENCES * Institute of Medicine. _Evolution of Translational Omics: Lessons Learned and the Path Forward_ (eds Micheel, C. M., Nass, S. & Omenn, G. S. ) (The National Academies Press,

2012)A REPORT PRODUCED BY A COMMITTEE FORMED IN RESPONSE TO AN NCI REQUEST FOR RECOMMENDATIONS TO STRENGTHEN OMICS-BASED TEST DEVELOPMENT AND EVALUATION; THIS IDENTIFIES BEST PRACTICES TO

ENHANCE THE DEVELOPMENT, EVALUATION AND TRANSLATION OF OMICS-BASED TESTS WHILE REINFORCING STEPS TO ENSURE THAT THESE TESTS ARE APPROPRIATELY ASSESSED FOR SCIENTIFIC VALIDITY BEFORE THEY ARE

USED TO GUIDE PATIENT TREATMENT IN CLINICAL TRIALS. * McShane, L. M. et al. Criteria for the use of omics-based predictors in clinical trials: explanation & elaboration. _BMC Med._11,

220 (2013)THIS PAPER IS AN ANNOTATED COMPANION TO THE SHORT VERSION OF THE GUIDELINES PRESENTED HERE, ELUCIDATING THE RATIONALE UNDERLYING THE DEVELOPMENT OF THE CRITERIA IN GREATER DETAIL.

Article  Google Scholar  * Moore, H. M. et al. Biospecimen Reporting for Improved Study Quality (BRISQ). _Cancer Cytopath._119, 92–101 (2011) Article  Google Scholar  * Dobbin, K. K. et al.

Interlaboratory comparability study of cancer gene expression analysis using oligonucleotide microarrays. _Clin. Cancer Res._11, 565–572 (2005) CAS  PubMed  Google Scholar  * Shi, L. et al.

The MicroArray Quality Control (MAQC) project shows inter- and intraplatform reproducibility of gene expression measurements._Nature Biotechnol._24, 1151–1161 (2006) Article  CAS  Google

Scholar  * Leek, J. T. et al. Tackling the widespread and critical impact of batch effects in high-throughput data. _Nature Rev. Genet._11, 733–739 (2010) Article  ADS  CAS  Google Scholar 

* Dupuy, A. & Simon, R. M. Critical review of published microarray studies for cancer outcome and guidelines on statistical analysis and reporting. _J. Natl. Cancer Inst._99, 147–157

(2007) Article  Google Scholar  * Simon, R., Radmacher, M. D., Dobbin, K. & McShane, L. M. Pitfalls in the use of DNA microarray data for diagnostic and prognostic classification. _J.

Natl. Cancer Inst._95, 14–18 (2003) Article  CAS  Google Scholar  * Subramanian, J. & Simon, R. Gene expression-based prognostic signatures in lung cancer: ready for clinical use? _J.

Natl. Cancer Inst._102, 464–474 (2010) Article  CAS  Google Scholar  * Molinaro, A. M., Simon, R. & Pfeiffer, R. M. Prediction error estimation: a comparison of resampling methods.

_Bioinformatics_21, 3301–3307 (2005) Article  CAS  Google Scholar  * Simon, R. M., Paik, S. & Hayes, D. F. Use of archived specimens in evaluation of prognostic and predictive

biomarkers. _J. Natl. Cancer Inst._101, 1446–1452 (2009) Article  Google Scholar  * ICH Expert Working Group. International Conference on Harmonisation of Technical Requirements for

Registration of Pharmaceuticals for Human Use; http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Quality/Q2_R1/Step4/Q2_R1__Guideline.pdf (accessed, 19 February 2013) * US

Food and Drug Administration. _Guidance for Industry: Computerized Systems Used in Clinical Investigations_ (US Department of Health and Human Services, 2007) * Freidlin, B., McShane, L. M.

& Korn, E. L. Randomized clinical trials with biomarkers: design issues. _J. Natl. Cancer Inst._102, 152–160 (2010) Article  CAS  Google Scholar  * US Department of Health & Human

Services. SACHRP Letter to the Secretary: FAQs, Terms and Recommendations on Informed Consent and Research Use of Biospecimens;

http://www.hhs.gov/ohrp/sachrp/20110124attachmentatosecletter.html (accessed, 19 February 2013) * Center for Medicare and Medicaid Services. Clinical Laboratory Improvement Amendments

(CLIA); http://www.cms.gov/Regulations-and-Guidance/Legislation/CLIA/index.html?redirect = /CLIA (accessed, 19 February 2013) * US Food and Drug Administration. Device Advice: Comprehensive

Regulatory Assistance; http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/default.htm. (accessed, 19 February 2013) * US Food and Drug Administration. Investigational New Drug

(IND) Application; http://www.fda.gov/Drugs/DevelopmentApprovalProcess/HowDrugsareDevelopedandApproved/ApprovalApplications/InvestigationalNewDrugINDApplication/default.htm (accessed, 19

February 2013) * US Food and Drug Administration. Draft Guidance for Industry and FDA Staff: Medical Devices: the Pre-submission Program and Meetings with FDA Staff (US Department of Health

& Human Services); http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm310375.htm (accessed, 19 February 2013) Download references AUTHOR INFORMATION

AUTHORS AND AFFILIATIONS * Division of Cancer Treatment and Diagnosis, National Cancer Institute, National Institutes of Health, Bethesda, 20892, Maryland, USA Lisa M. McShane, Margaret M.

Cavenagh, Tracy G. Lively, Mei-Yin C. Polley, Kelly Y. Kim, James V. Tricoli, Deborah J. Shuman, Richard M. Simon, James H. Doroshow & Barbara A. Conley * Department of Pathology and

Lineberger Comprehensive Cancer Center, University of North Carolina, Chapel Hill, 27599, North Carolina, USA David A. Eberhard * Department of Pathology and University of Pittsburgh Cancer

Institute, Hillman Cancer Center, University of Pittsburgh School of Medicine, Pittsburgh, 15213, Pennsylvania, USA William L. Bigbee * Frederick National Laboratory for Cancer Research,

National Cancer Institute, National Institutes of Health, Frederick, 21702, Maryland, USA P. Mickey Williams * Broad Institute of Massachusetts Institute of Technology and Harvard

University, Cambridge, 02142, Massachusetts, USA Jill P. Mesirov * Department of Biostatistics, University of Michigan, Ann Arbor, 48109, Michigan, USA Jeremy M. G. Taylor Authors * Lisa M.

McShane View author publications You can also search for this author inPubMed Google Scholar * Margaret M. Cavenagh View author publications You can also search for this author inPubMed 

Google Scholar * Tracy G. Lively View author publications You can also search for this author inPubMed Google Scholar * David A. Eberhard View author publications You can also search for

this author inPubMed Google Scholar * William L. Bigbee View author publications You can also search for this author inPubMed Google Scholar * P. Mickey Williams View author publications You

can also search for this author inPubMed Google Scholar * Jill P. Mesirov View author publications You can also search for this author inPubMed Google Scholar * Mei-Yin C. Polley View

author publications You can also search for this author inPubMed Google Scholar * Kelly Y. Kim View author publications You can also search for this author inPubMed Google Scholar * James V.

Tricoli View author publications You can also search for this author inPubMed Google Scholar * Jeremy M. G. Taylor View author publications You can also search for this author inPubMed 

Google Scholar * Deborah J. Shuman View author publications You can also search for this author inPubMed Google Scholar * Richard M. Simon View author publications You can also search for

this author inPubMed Google Scholar * James H. Doroshow View author publications You can also search for this author inPubMed Google Scholar * Barbara A. Conley View author publications You

can also search for this author inPubMed Google Scholar CONTRIBUTIONS B.A.C. and L.M.M. conceived the idea for this paper and the checklist. The initial draft of the manuscript was the joint

effort of several authors contributing according to their particular areas of expertise (W.L.B., M.M.C., B.A.C., D.A.E., T.G.L. and L.M.M.). All authors provided comments, suggested edits,

and contributed additional expertise to enhance the initial draft and produce the final version of the manuscript. CORRESPONDING AUTHOR Correspondence to Lisa M. McShane. ETHICS DECLARATIONS

COMPETING INTERESTS The authors declare no competing financial interests. RIGHTS AND PERMISSIONS This work is licensed under a Creative Commons Attribution-Non-Commercial-ShareAlike 3.0

Unported licence. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-sa/3.0/. Reprints and permissions ABOUT THIS ARTICLE CITE THIS ARTICLE McShane, L.,

Cavenagh, M., Lively, T. _et al._ Criteria for the use of omics-based predictors in clinical trials. _Nature_ 502, 317–320 (2013). https://doi.org/10.1038/nature12564 Download citation *

Received: 23 April 2013 * Accepted: 15 August 2013 * Published: 17 October 2013 * Issue Date: 17 October 2013 * DOI: https://doi.org/10.1038/nature12564 SHARE THIS ARTICLE Anyone you share

the following link with will be able to read this content: Get shareable link Sorry, a shareable link is not currently available for this article. Copy to clipboard Provided by the Springer

Nature SharedIt content-sharing initiative