The costs of complex model optimization

• An easy computer-driven way to calculate approximate degrees of freedom
• Can be used for validation and model discrimination
• Can be applied to any type of complex prediction problem

Each data-driven action in data modelling consumes degrees of freedom, whether it concerns estimation of parameters, estimation of meta-parameters or selecting variables. By using a double cross validation approach for degrees of freedom calculation the costs for meta-parameter estimation and variable selection can be determined explicitly. The only assumptions are independent and identically distributed errors, which make the approach applicable to many predictive modelling techniques.