A family of test selection criteria for Timed Input-Output Symbolic Transition System models

• Definition of 18 test selection criteria for Timed Input-Output Symbolic Transition System (TIOSTS) models;
• Formalization of a hierarchy of criteria partially ordered by strict inclusion;
• Experiment compares cost-effectiveness of criteria in test suite reduction of model-based testing of real-time systems;
• Experimental results showed that All-Transition-Pairs was the most cost-effective criterion;
• More cost-effective criteria that explore time-related features of models of real-time systems are still needed.

Test selection criteria can be applied to select, from a test suite, cost-effective subsets that are both cheaper to execute and as effective at detecting faults as the original suite. However, the choice of a test selection criterion is not straightforward for real-time systems, because most criteria presented in literature are for untimed systems and there is a lack of studies that investigate their cost-effectiveness when compared to specific criteria for real-time systems. In this paper, we investigate the cost-effectiveness of test selection criteria in test suite reduction for model-based testing of real-time systems, in particular for Timed Input-Output Symbolic Transition Systems (TIOSTS) models. First, we defined 18 test selection criteria for TIOSTS models and formalized a hierarchy of criteria partially ordered by strict inclusion. The defined criteria include transition-based criteria, data-flow-oriented criteria and real-time systems criteria. Second, we evaluated the cost-effectiveness of the criteria in an experiment. In the empirical study, we used a TIOSTS specification of a burglar alarm system, an implementation that simulated the system, and faults seeded into the implementation by using mutation testing. Results showed that, despite being a criterion for untimed systems, All-Transition-Pairs was the most cost-effective criterion for test case selection in model-based testing of real-time systems. We conclude that more cost-effective criteria that explore time-related features of models of real-time systems are still needed.