Examining random and designed tests to detect code mistakes in scientific software

Successfully testing computational software to detect code mistakes is impacted by multiple factors. One factor is the tolerance accepted in test output. Other factors are the nature of the code mistake, the characteristics of the code structure, and the choice of test input. We have found that randomly generated test input is a viable approach to testing for code mistakes and that simple structural metrics have little predictive power in the type of testing required. We provide further evidence that reduction of tolerance in expected test output has a much larger impact than running many more tests to discover code mistakes.

Research highlights▶ From our research and that of others, the practice of testing computational software almost always focuses on validation testing. The assumption is that code mistakes will be obvious when the entire software model is executed. There have been glaring examples of code mistakes going undetected and causing grave problems. Yet, scientists seldom test specifically for code mistakes. Part of the reason may be that the software engineering community has not offered up testing approaches specifically aimed at computational software and its particular set of problems. ▶ The work described in this paper is part of a research program aimed at providing scientists and engineers advice on improving their software. Specifically, this work looks at the effectiveness of random testing and the conditions under which it is useful. The goal of this testing is finding code mistakes. We have found that randomly generated test data can be a good “first” approach to testing, followed by designed tests targeting boundary conditions related both to the physical phenomena being modeled and to the structure of condition statements. As part of our general program, we are advising scientists to test for code mistakes first, before doing verification or validation testing. Our work is helping fill the gap both in testing activities for computational software and in software engineering research.