Comparing simulation tools and experimental testbeds for wireless mesh networks

Simulators can provide full control when simulating wireless network behaviors, but are often plagued by their insufficiencies in reflecting real-world scenarios. Although previous works point out such a shortage is mainly due to the limitations of radio propagation models in simulators, it is unclear how imperfect modeling affects network behaviors and to what degree. In this study, we investigate differences between simulations and testbed experiments in wireless mesh networks. We compare and analyze the experimental results with NS-2 and QualNet simulations. We find that in the PHY layer, antenna diversity can cause differences in distribution of received signal strength between experiments and simulations. The discrepancy in some cases, however, can be compensated by tweaking path loss, a dominating factor in channel modeling. In the MAC layer, a significant performance degradation is observed in experiments when loaded with heavy traffic, but this is less obvious in simulations. The mismatching is further deteriorated by inadequate interference modeling. As transceiver sensitivity differs, resulting in dramatic unfairness at flow-level goodputs, simulators are immune from such a problem. In the IP layer, route prevalence and persistence are carefully examined. A few routes are found to dominate experimental testbed while routes are spread more evenly and are less stable in simulators. These findings give the wireless research community an improved picture of the differences between simulations and testbed experiments.