A comparison of different methods to estimate prediction uncertainty using Partial Least Squares (PLS): A practitioner's perspective

There is an increase in the use of latent variable modeling (LVM) techniques, such as Projection to Latent Structure (PLS), in the pharmaceutical industry. Thus, there exists a practical need to estimate prediction uncertainty for PLS models. Metrics such as standard error of prediction (SEP) and standard error of calibration (SEC) do not truly reflect prediction reliability. Several proposals exist in the literature to tackle the problem. This paper describes a comparison exercise for selected uncertainty estimation algorithms by testing representative pharmaceutical industrial data sets. Algorithms evaluated include linearization-based methods, Ordinary Least Squares (OLS) type method, re-sampling based method and empirical method. Algorithm performance was measured using “coverage probability”. Additionally, different approaches to estimate degrees of freedom consumed by PLS model were evaluated for uncertainty estimation purpose. These methods include the Naïve approach, the pseudo degree of freedom (PDF) approach and the generalized degrees of freedom (GDF) approach. Results from this study suggest that none of these algorithms generates accurate coverage rates for all cases considered. Thus, further development in this area is needed. Among all the evaluated algorithms, the simple Faber 96 method seems to perform slightly better under appropriate handling of the degrees of freedom. Different ways to estimate degrees of freedom were shown to have crucial effect on the performance of uncertainty estimation. The results show that the Naïve approach should be discouraged for use in uncertainty estimation in practice and the GDF approach should be preferred.