On-The-Fly Processing of continuous high-dimensional data streams

•The On-The-Fly Processing (OTFP) tool for rational handling of continuous high-dimensional data streams is presented.•The OTFP discovers and extracts the systematic covariance information from flowing data.•It is based on an automatic, gently evolving modelling process.•The OTFP renders overwhelming data streams cognitively interpretable and quantitatively useful in their compressed form.•It exhibited very satisfactory performance in terms of compression rate, computational time and graphical insight.

A novel method and software system for rational handling of time series of multi-channel measurements is presented. This quantitative learning tool, the On-The-Fly Processing (OTFP), develops reduced-rank bilinear subspace models that summarise massive streams of multivariate responses, capturing the evolving covariation patterns among the many input variables over time and space. Thereby, a considerable data compression can be achieved without significant loss of useful systematic information.The underlying proprietary OTFP methodology is relatively fast and simple - it is linear/bilinear and does not require a lot of raw data or huge cross-correlation matrices to be kept in memory. Unlike conventional compression methods, the approach allows the high-dimensional data stream to be graphically interpreted and quantitatively utilised - in its compressed state. Unlike adaptive moving-window methods, it allows all past and recent time points to be reconstructed and displayed simultaneously.This new approach is applied to four different case-studies: (i) multi-channel Vis-NIR spectroscopy of the Belousov-Zhabotinsky reaction, a complex, ill understood chemical process; (ii) quality control of oranges by hyperspectral imaging; (iii) environmental monitoring by airborne hyperspectral imaging; (iv) multi-sensor process analysis in the petrochemical industry. These examples demonstrate that the OTFP can automatically develop high-fidelity subspace data models, which simplify the storage/transmission and the interpretation of more or less continuous time series of high-dimensional measurements - to the extent there are covariations among the measured variables.