DBit: Assessing statistically significant differences in CDN performance

As the volume of content served by content distribution networks (CDNs) grows, these networks evolve to improve performance. Their performance is difficult to characterize because it depends on a number of factors. In this paper, we develop a methodology called DBit that can determine whether one CDN's user-perceived performance is statistically different from another. We validate DBit and demonstrate its usefulness on CDNs used for photo delivery. We use PlanetLab to collect HTTP download data for 14.5 million photo fetches and 5 million video fetches and RIPE Atlas nodes hosted in end-user homes in 1470 ASes worldwide to obtain 470,400 photo fetches respectively, from three Photo CDNs and two Video CDNs. We find that DBit can identify significant performance differences not just between CDNs, but also across time and location.