Elastic Infrastructure for Interactive Data Farming Experiments

With the increasing availability of high performance computing power, new possibilities with respect to simulation and analysis become available and feasible. One of such methodologies is data farming, where large amounts of data are generated through simulation of several configurations from large parameter space and then analyzed for patterns. Unfortunately, the availability of versatile data farming systems is very limited and none of existing solutions allows integration with novel Cloud solutions. In this paper, we present our system, which is a flexible solution for running very large Data Farming experiments on both intranet clusters as well as on remote computational resources, including public Clouds. Another important aspect of the presented system is the support of interactive Data Farming experiments with online analysis of partial experiment results and experiment extending capability. Sample application of our system is present on military mission planning support scenario.