OptionStream: An automated system for tracking derivative effects on equity prices

We present the design and development of a flow-based software system that became necessary for the study of a class of options-based price-estimators in the financial markets. Because a combination of factors-cost, reliability, uniformity and convention-made it virtually impossible to obtain historical options data, we developed a data-flow system to capture, process and analyze streaming data over a period of over a year. The system utilizes distributed processing nodes with checkpointing to process input data streams and reliably compute a variety of estimator updates for studying relationships between equity prices and the functions of corresponding option-related variables. The flow-based architecture is designed to support the high-volume data characteristics of the options market, along with the severely taxing computational requirements of hypothetical option-pricing so that experimental investigation of novel estimators becomes possible with current and historical data. Features such as high-volume stream-processing, intermediate checkpointing and load distribution make the design viable for more general streaming data processing applications.