Real-time data assimilation for large-scale systems: The spectral Kalman filter

• We present a new fast Kalman filter method that use full covariance matrix.
• The computational cost of our method scales linearly with number of unknowns.
• Cross-covariance matrices are updated instead of larger covariance matrices.
• The full covariance matrix can be recovered in a post-processing step.
• Our method provides similar accuracy with smaller computational cost than the EnKF.

The Kalman Filter (KF) is a data assimilation method that has been widely used for estimating spatially varying unknown states evolving in time. Recently, KF methods have shown potential for tracking CO2 plumes injected in deep geologic formations. Such real-time estimation would serve as an early warning system for leakage incidents. However, for the large number of unknowns of such large systems, KF methods are impractical, because updating the huge state covariance matrix is computationally expensive. Low rank approximation methods have been devised to overcome this problem; these methods assume a low rank of the covariance matrix, which they approximate by smaller matrices. The approximation error is small for smooth functions, but may be larger for more complex physical problems, potentially leading to filter divergence and inaccurate state estimates.We present the Spectral Kalman Filter (SpecKF), a new algorithm that utilizes the exact covariance matrix. Thus it avoids the approximation error and does so in a computationally efficient way, which is specially important for systems with large numbers of unknowns. The computational speed-up of the SpecKF is achieved by updating cross-covariance matrices instead of the larger covariance matrices. The benefit can be considerable, especially in large systems, because the computational complexity of the SpecKF scales with the number of measurements, as opposed to the effective rank of the covariance matrix in low-rank KFs or the number of ensemble members in ensemble methods. We investigate the accuracy and performance of the SpecKF for a diffusion problem with random perturbations, and for a more complex case of CO2 injection in a homogeneous two-dimensional domain. Our results show that the SpecKF reduces greatly the computational cost compared to the original KF algorithm, and that it can provide higher accuracy than the Ensemble Kalman Filter (EnKF) with the same or even smaller computational cost. Finally, we discuss some approaches that can be used with the SpecKF for approximating the uncertainty of the final state estimates.