The differogram: Non-parametric noise variance estimation and its use for model selection

Model-free estimates of the noise variance are important in model selection and setting tuning parameters. In this paper a data representation is discussed which leads to such an estimator suitable for multivariate data. Its visual representation-called the differogram cloud here-is based on the 2-norm of the differences of input and output data. The crucial concept of locality in this representation is translated as the increasing variance of the difference, which does not rely explicitly on an extra hyper-parameter. Connections with U-statistics, Taylor series expansions and other related methods are given. Numerical simulations indicate a convergence of the estimator. This paper extends results towards a time-dependent setting and to the case of non-Gaussian noise models or outliers. As an application, this paper focuses on model selection for Least Squares Support Vector Machines. For this purpose, a variant of the LS-SVM regressor is derived based on Morozov's discrepancy principle relating the regularization constant directly with the (observed) noise level.