Test coverage optimization for large code problems

Software developers frequently conduct regression testing on a series of major, minor, or bug-fix software or firmware releases. However, retesting all test cases for each release is time-consuming. For example, it takes about 36 test-bed-days to thoroughly exercise a test suite made up of 2320 test cases for the MPLS testing area that contains 57,758 functions in Cisco IOS. The cost is infeasible for a series of regression testing on the MPLS area. Thus, the test suite needs to be reduced intelligently, not just randomly, and its fault detection capability must be kept as much as possible. The mode of safe regression test selection approach is adopted for seeking a subset of modification-traversing test cases to substitute for fault-revealing test cases. The algorithms, CW-NumMin, CW-CostMin, and CW-CostCov-B, apply the safe-mode approach in selecting test cases for achieving full-modified function coverage. It is assumed that modified functions are fault-prone, and the fault distribution of the testing area is Pareto-like. Moreover, we also assume that once a subject program is getting more mature, its fault concentration will become stronger. Only function coverage criterion is adopted because of the scalability of a software system with large code. The metrics of test’s function reachability and function’s test intensity are defined in this study for algorithms. Both CW-CovMax and CW-CostMin algorithms are not safe-mode, but the approaches they use still attempt to obtain a test suite with a maximal amount of function coverage under certain constraints, i.e. the effective-confidence level and time restriction. We conclude that the most effective algorithm in this study can significantly reduce the cost (time) of regression testing on the MPLS testing area to 1.10%, on the average. Approaches proposed here can be effectively and efficiently applied to the regression testing on bug-fix releases of a software system with large code, especially to the releases having very few modified functions with low test intensities.

► We define function reachability and test intensity for developing algorithms.
► We test an industrial software with large code using function coverage for scalability.
► An safe-mode approach is applied for the effectiveness of fault detection capability.
► The most effective algorithm reduces the cost of regression testing on MPLS to 1.10%.
► Our approaches suit to regression testing on bug-fix releases of software systems.