How important is to detect systematic error in predictions and understand statistical applicability domain of QSAR models?

•We explain the importance of analysis of prediction errors for QSAR models.•We have used data from both real QSAR model derived predictions and also simulated data.•We demonstrate with examples some methods of detection of bias in prediction errors.•We alert the QSAR users to the importance of “statistical applicability domain” of QSAR models.

One of the important applications of quantitative structure-activity relationship (QSAR) models is to fill data gaps by predicting a given response property or activity from known molecular features or physicochemical properties of new compounds which might not have been tested experimentally. The general QSAR users are now already aware that the new compounds which will be predicted from a given QSAR model should be similar in structural and/or physicochemical property space to the training set compounds so as to be included in the chemical applicability domain of the model. It is also well-established, at least among some groups of QSAR scientists, that performance of a model should be evaluated based on the quality of predictions from the test set and not from the training set to obviate any overfitting problem. It is also important to analyze prediction errors of test set compounds in order to search for presence of any systematic error or bias in model predictions. A simple residual plot can provide sufficient information about the type of error present in model predictions. As the test set compounds are structurally and physico-chemically similar to the training set compounds, the model prediction errors for the test set also should obey, at least approximately, the basic assumptions of the least-squares regression under the best linear unbiased estimator (BLUE) framework, provided that a sufficient number of test set compounds is available allowing an acceptable degree of freedom. The present article explains the importance of analysis of prediction errors to check for the presence of systematic error and/or violation of basic assumptions of the least-squares regression models under the BLUE framework with suitable examples using real QSAR model-derived quantitative predictions for test sets and simulated prediction data. The intention of the present authors is neither to make a comparison of performances of different validation metrics for quantitative predictions (eventually favoring or disfavoring one or the other validation metric) nor to compare quality of different models but to indicate the situations where such comparison should not be made due to inappropriate functional form of a model making it unsuitable for quantitative predictions of a set of compounds and also due to the use of unrealistic and biased prediction pattern that never happens in real QSAR problems, thus making the situation unsuitable for making any generalized conclusion. This article also alerts the QSAR users to the importance of “statistical applicability domain” of QSAR models before their application for quantitative prediction of a response of test set compounds in order to compare performance of different validation approaches.