Testing Real-Time Embedded Systems using Timed Automata based approaches

Real-Time Embedded Systems (RTESs) have an increasing role in controlling the IT that we use on a day-to-day basis. The behaviour of an RTES is not based solely on the interactions it might have with its surrounding environment, but also on the timing requirements it induces. As a result, ensuring that an RTES behaves correctly is non-trivial, especially after adding time as a new dimension to the complexity of the testing process. We previously introduced the ‘priority-based’ approach which tests the logical and timing behaviour of an RTES modelled formally as UPPAAL automata. The ‘priority-based’ approach was based on producing sets of timed test traces by achieving clock region coverage. In this paper, we empirically validate the ‘priority-based’ approach with comparison to well-known timed testing approaches based on a Timed Automata (TA) formalism using a complete test bed based on an industrial-strength case study (production cell). The validation assessment is based on both fault coverage and structural coverage by a minimal number of generated test traces; the former is achieved using the Mutation Analysis Technique (MAT) by introducing a set of timed and functional mutation operators. The latter is based on clock region coverage as a main timed structural coverage criterion. This study shows that ‘priority-based’ approach can combine a high fault coverage and clock region coverage with a relatively small number of test traces in comparison with other test approaches. A set of experiences and lessons learned are highlighted as result of the real-time test bed.

► Propose a timed coverage criterion; clock region and accommodate the mutation operators proposed for TA to suit C-code.
► Generate timed test cases from three timed testing approaches including ours.
► Propose assessment criteria to compare the performance of the testing approaches by executing the generated timed test cases on a test bed.
► Introduce a set of lessons and experience learned from the timed test bed.