A noise-driven strategy for background estimation and event detection in data streams

A theory is presented for a discrete, finite-horizon, H∞H∞ filter that estimates background in a data stream. Threshold rejection is introduced into the theory by way of an approximation for the H∞H∞ observation innovation. The threshold is simply related to a basis variance that can either be provided as input or accumulated over the data stream. This framework identifies background as the portion of a data stream that varies within the bulk of the noise in the data. Unexpected events in the data stream are therefore synonymous with statistical outliers—especially successive outliers of the same direction. The resulting methodology is robust and suitable for real-time applications. It can handle types of background variation in which smoothing and band pass filtering are ineffective. There are no adjustable parameters because all such quantities either have universal values or are selected using well-defined principles. The performance of the filter is demonstrated using computer simulated data sets and arbitrary instrumental data. Examples of its application are also presented in the fields of finance and computer security.