Test case prioritization of build acceptance tests for an enterprise cloud application: An industrial case study

The use of cloud computing brings many new opportunities for companies to deliver software in a highly-customizable and dynamic way. One such paradigm, Software as a Service (SaaS), allows users to subscribe and unsubscribe to services as needed. While beneficial to both subscribers and SaaS service providers, failures escaping to the field in these systems can potentially impact an entire customer base. Build Acceptance Testing (BAT) is a black box technique performed to validate the quality of a SaaS system every time a build is generated. In BAT, the same set of test cases is executed simultaneously across many different servers, making this a time consuming test process. Since BAT contains the most critical use cases, it may not be obvious which tests to perform first, given that the time to complete all test cases across different servers in any given day may be insufficient. While all tests must be eventually run, it is critical to run those tests first which are likely to find failures. In this work, we ask if it is possible to prioritize BAT tests for improved time to fault detection and present several different approaches, each based on the services executed when running each BAT. In an empirical study on a production enterprise system, we first analyze the historical data from several months in the field, and then use that data to derive the prioritization order for the current development BATs. We then examine if the orders change significantly when we consider fault severity using a cost-based prioritization metric. We find that the prioritization order in which we run the tests does matter, and that the use of historical information is a good heuristic for this order. Prioritized tests have an increase in the rate of fault detection, with the average percent of faults detected (APFD) increasing from less than 0.30 to as high as 0.77 on a scale of zero to one. Although severity slightly changes which order performs best, we see that there are clusters of orderings, ones which improve time to early fault detection ones which don't.