Towards repeatability & verifiability in networking experiments: A stochastic framework

Growing complexity of networking experiments has rendered the accuracy, repetition and reproduction of experimental results indispensable. This research work aims at developing a tool to facilitate repeatability and verifiability of experiments on diverse experimentation platforms. The process involves studying an experiment with respect to its constituent events and creating time-series based and event based characterizations from logs and measurement data. The framework thus developed was applied to 6000 experiment trials comprising of 1500 DETER testbed DNS-cache poisoning attack, 3200 ns2 web-traffic generation, 1000 real-time web-traffic generation and 300 ns3 wireless mobility experiment trials. We observed that these characterizations were sensitive to experiment configuration. We also investigated the impact of variations in topology, traffic, hardware and mobility on them (characterizations). Differences in characterizations were quantified by standard distance measures: KL Divergence, Total Variation Distance and Euclidean Distance. Our results were statistically verified by hypothesis testing and analysis through ANOVA, chi-square and correlation tests. Thus, our framework provides a direct and precise method to compare two executions of a stochastic networking experiment for simulations, emulation-based testbeds and real-time network experiments. These results form the groundwork for generating a validity management framework and will help to achieve verifiability in experiment executions.