Testing component compatibility in evolving configurations

Software components are increasingly assembled from other components. Each component may further depend on others, and each may have multiple active versions. The total number of configurations—combinations of components and their versions—in use can be very large. Moreover, components are constantly being enhanced and new versions are being released. Component developers, therefore, spend considerable time and effort doing compatibility testing—determining whether their components can be built correctly for all deployed configurations–both for existing active component versions and new releases. In previous work we developed Rachet, a distributed, cache-aware mechanism to support large-scale compatibility testing of component-based software with a fixed set of component versions.In this paper, we observe that it is too expensive to perform compatibility testing from scratch each time a new version of a component is released. We thus add a new dimension to Rachet: to perform incremental and prioritized compatibility testing. We describe algorithms to compute differences in component compatibilities between current and previous component builds, a formal test adequacy criterion based on covering the differences, and cache-aware configuration sampling and testing methods that attempt to reuse effort from previous testing sessions. Because testers are often interested in focusing test effort on newly released and modified components and their versions, we have developed a prioritization mechanism that enhances compatibility testing by examining the configurations that test new or modified component versions first, while also distributing the work over a cluster of machines. We evaluate our approach using the 5-year evolution history of a scientific middleware component. Our results show that our methods can increase performance significantly over Rachet’s previous retest-all approach and also tests important component compatibilities early in the overall testing process, making the process of compatibility testing practical for evolving components.