Compressed state Kalman filter for large systems

• Has computational and storage costs that are linear in the size of the state vector, unlike the “textbook” version that has costs that increase with the square of the size of the state vector.
• Uses only direct calls to forward models and is matrix-free, i.e., there is no need to compute or store Jacobian matrices, like the transition matrix.
• Is easy to program for particular applications, with some complementary software for fast matrix–vector multiplications.

The Kalman filter (KF) is a recursive filter that allows the assimilation of data in real time and has found numerous applications. In earth sciences, the method is applied to systems with very large state vectors obtained from the discretization of functions such as pressure, velocity, solute concentration, and voltage. With state dimension running in the millions, the implementation of the standard or textbook version of KF is very expensive and low-rank approximations have been devised such as EnKF and SEEK. Although widely applied, the error behavior of these methods is not adequately understood. This article focuses on very large linear systems and presents a complete computational method that scales roughly linearly with the dimension of the state vector. The method is suited for problems for which the effective rank of the state covariance matrix is much smaller than its dimension. This method is closest to SEEK but uses a fixed basis that should be selected in accordance with the characteristics of the problem, mainly the transition matrix and the system noise covariance. The method is matrix free, i.e., does not require computation of Jacobian matrices and uses the forward model as a black box. Computational results demonstrate the ability of the method to solve very large, say 106106, state vectors.