VAYU: Accelerating stream processing applications through dynamic network-aware topology re-optimization

- Compute and network overheads can reduce performance of stream processing systems.
- Efficient technique for dynamic topology re-optimization is presented.
- Technique for network-aware tuple routing using consistent hashing is presented.
- Presents algorithms for optimizing group communication overlay topologies.

Stream processing applications for online analytics are commonly used in domains ranging from sensor data processing to social networking. To achieve high-throughput, stream processing engines support pipelined execution, low-overhead fault-tolerance, and efficient group communication overlays. The throughput of pipelined application workflows is significantly impacted by dynamic system state. In particular, we show that a single bottleneck in the pipeline (congested link or an overloaded operator) can drastically impact the system throughput. In this paper, we present a number of techniques for addressing bottlenecks in stream engines. Our techniques fall into two major classes - network-aware routing for fine grained control of streams; and dynamic overlay generation for optimizing performance of group communication operations. To enable fast workflow re-optimization, we present a light-weight protocol for consistent modification of pipelines. We present detailed algorithms, their implementation in a real system, and address issues of fault tolerance and performance. We evaluate performance of the proposed techniques in the context of three real applications. We show that our techniques improve performance by 20% to 200%, under various overheads, relative to a baseline representative of current implementations. We demonstrate that our techniques are robust to highly dynamic state, as well as complex congestion patterns. Given the widespread use of streaming systems and the need for dealing with dynamic system state, our techniques represent a significant and practical improvement.