Developing scientific confidence in HTS-derived prediction models: Lessons learned from an endocrine case study

- Fitting & cross validation of HTS-derived endocrine prediction models were compared.
- Cross validation (CV) is a more robust approach for true prediction.
- From guideline studies, CV balanced accuracy (BA) for androgen endpoints was 79%.
- The CVBA for estrogen was 85%, 23% for thyroid, & lower for non-guideline studies.
- A scientific confidence framework for HTS-derived prediction models is presented.

High throughput (HTS) and high content (HCS) screening methods show great promise in changing how hazard and risk assessments are undertaken, but scientific confidence in such methods and associated prediction models needs to be established prior to regulatory use. Using a case study of HTS-derived models for predicting in vivo androgen (A), estrogen (E), thyroid (T) and steroidogenesis (S) endpoints in endocrine screening assays, we compare classification (fitting) models to cross validation (prediction) models. The more robust cross validation models (based on a set of endocrine ToxCast™ assays and guideline in vivo endocrine screening studies) have balanced accuracies from 79% to 85% for A and E, but only 23% to 50% for T and S. Thus, for E and A, HTS results appear promising for initial use in setting priorities for endocrine screening. However, continued research is needed to expand the domain of applicability and to develop more robust HTS/HCS-based prediction models prior to their use in other regulatory applications. Based on the lessons learned, we propose a framework for documenting scientific confidence in HTS assays and the prediction models derived therefrom. The documentation, transparency and the scientific rigor involved in addressing the elements in the proposed Scientific Confidence Framework could aid in discussions and decisions about the prediction accuracy needed for different applications.