On-The-Fly Processing of continuous high-dimensional data streams

- The On-The-Fly Processing (OTFP) tool for rational handling of continuous high-dimensional data streams is presented.
- The OTFP discovers and extracts the systematic covariance information from flowing data.
- It is based on an automatic, gently evolving modelling process.
- The OTFP renders overwhelming data streams cognitively interpretable and quantitatively useful in their compressed form.
- It exhibited very satisfactory performance in terms of compression rate, computational time and graphical insight.

A novel method and software system for rational handling of time series of multi-channel measurements is presented. This quantitative learning tool, the On-The-Fly Processing (OTFP), develops reduced-rank bilinear subspace models that summarise massive streams of multivariate responses, capturing the evolving covariation patterns among the many input variables over time and space. Thereby, a considerable data compression can be achieved without significant loss of useful systematic information.The underlying proprietary OTFP methodology is relatively fast and simple - it is linear/bilinear and does not require a lot of raw data or huge cross-correlation matrices to be kept in memory. Unlike conventional compression methods, the approach allows the high-dimensional data stream to be graphically interpreted and quantitatively utilised - in its compressed state. Unlike adaptive moving-window methods, it allows all past and recent time points to be reconstructed and displayed simultaneously.This new approach is applied to four different case-studies: (i) multi-channel Vis-NIR spectroscopy of the Belousov-Zhabotinsky reaction, a complex, ill understood chemical process; (ii) quality control of oranges by hyperspectral imaging; (iii) environmental monitoring by airborne hyperspectral imaging; (iv) multi-sensor process analysis in the petrochemical industry. These examples demonstrate that the OTFP can automatically develop high-fidelity subspace data models, which simplify the storage/transmission and the interpretation of more or less continuous time series of high-dimensional measurements - to the extent there are covariations among the measured variables.