The costs of complex model optimization

•An easy computer-driven way to calculate approximate degrees of freedom•Can be used for validation and model discrimination•Can be applied to any type of complex prediction problem

Each data-driven action in data modelling consumes degrees of freedom, whether it concerns estimation of parameters, estimation of meta-parameters or selecting variables. By using a double cross validation approach for degrees of freedom calculation the costs for meta-parameter estimation and variable selection can be determined explicitly. The only assumptions are independent and identically distributed errors, which make the approach applicable to many predictive modelling techniques.