Statistical properties of support vector machines with forgetting factor

Introducing a forgetting factor allows a support vector machine to solve time-varying problems adaptively. However, the exponential forgetting factor proposed in an earlier work does not ensure convergence of average generalization error even for a simple linearly separable problem. To guarantee convergence, we propose a factorial forgetting factor which decays factorially over time. We approximately derive the average generalization error of the factorial forgetting factor as well as that of the exponential forgetting factor using a simple one-dimensional problem, and confirm our theory by computer simulations. Finally, we show that our theory can be extended to arbitrary types of forgetting factors for simple linearly separable cases.