A quest for appropriate software fault models: Case studies on fault detection effectiveness of model-based test generation techniques

Model-based test generation (MBTG) is becoming an area of active research. These techniques differ in terms of (1) modeling notations used, and (2) the adequacy criteria used for test generation. This paper (1) reviews different classes of MBTG techniques at a conceptual level, and (2) reports results of three case studies comparing various techniques in terms of their fault detection effectiveness. Our results indicate that MBTG technique which employs mutation and explicitly generates state verification sequences has better fault detection effectiveness than those based on boundary values, and predicate coverage criteria for transitions. Instead of a default adequacy criteria, certain techniques allow the user to specify test objectives in addition to the model. Our experience indicates that the task of defining appropriate test objectives is not intuitive. Furthermore, notations provided to describe such test objectives may have inadequate expressive power. We posit the need for a suitable fault modeling notation which also treats domain invariants as first class entities.