Higher order mutation testing: A Systematic Literature Review

Mutation testing is the process whereby a fault is deliberately inserted into a software system, in order to assess the quality of test data, in terms of its ability to find this fault. Mutation testing is also used as a way to drive the test data development process. Traditionally, faults were inserted one by one into a software system, but more recently there has been an upsurge of interest by the area of higher-order mutation, in which multiple faults are inserted into the system at once. Originally, this was thought to be too expensive, as there was already a concern that the size of the pool of mutants for traditional mutation was already too large to handle. However, following a seminal publication in 2008, it was realized that the space of higher-order mutants (HOMs) could be searched for useful mutants that drive testing harder, and to reduce the overall test effort, by clever combination of first-order mutants. As a result, many authors examined the way in which HOM testing could find subtle hard to kill faults, capture partial fault masking, reduce equivalent mutants problem, reduce test effort while increasing effectiveness, and capture more realistic faults than those captured by simple insertion of first-order mutants. Because of the upsurge of interest in the previous issues, this paper presents the first Systematic Literature Review research specifically targeted at a higher-order mutation. This Systematic Literature Review analyzes the results of more than one hundred sixty research articles in this area. The current paper presents qualitative results and bibliometric analysis for the surveyed articles. In addition, it augments these results with scientific findings and quantitative results from the primary literature. As a result of this work, this SLR presents an outline for many future work.