Processing high data rate streams in System S

High-performance stream processing is critical in many sense-and-respond application domains—from environmental monitoring to algorithmic trading. In this paper, we focus on language and runtime support for improving the performance of sense-and-respond applications in processing data from high-rate live streams. The central tenets of this work are the programming model, the workload splitting mechanisms, the code generation framework, and the underlying System S middleware and Spade programming model. We demonstrate considerable scalability behavior coupled with low processing latency in a real-world financial trading application.

Research highlights
► The split/aggregate/join architectural pattern is identified as a common template for implementing stream processing applications in different domains.
► Per-group processing is proven as an effective means of optimizing the implementation of the split/aggregate/join pattern.
► System S and SPADE are employed to provide a comprehensive performance evaluation of per-group split/aggregate/join processing on a cluster of machines, using a financial engineering application as a case study.