Fast adaptive LDA using quasi-Newton algorithm

A new adaptive algorithm for linear discriminant analysis (LDA) based on the quasi-Newton optimization technique is presented. The proposed algorithm uses the secant method for adaptive computation of the inverse Hessian matrix and the Newton–Raphson method for optimal estimation of the step size at each iteration. Current adaptive method, based on the Newton–Raphson optimization technique, uses a direct calculation of the inverse Hessian matrix, which can be both laborious to calculate and invert for systems with large number of dimensions. The new algorithm has the advantage of automatic optimal selection of the step size using the current data samples and also adaptive computation of the inverse Hessian matrix that overcomes its sensitivity to data condition. Based on the new adaptive algorithm, we present a self-organizing neural network for adaptive computation of the square root of the inverse covariance matrix and use it in cascaded form with a principal component analysis (PCA) network for LDA. Experimental results demonstrated fast convergence and lower computational cost of the new algorithm compared to the adaptive gradient descent and Newton–Raphson LDA algorithms, respectively and justified its advantages for on-line pattern recognition applications with stationary and non-stationary multidimensional input data.