Empirical research methods for technology validation: Scaling up to practice

•When scaling up RE technology to practice, researchers want to empirical validate new technology.•We discuss research methods and techniques that can be used for empirical validation of new RE technology.•We discuss expert opinion, single-case experiments, technical action research and statistical experiments.•We also discuss the statistical, analogic, and abductive inference techniques used in these methods.•We illustrate these research methods and techniques with examples from both empirical SE and empirical RE research.

Before technology is transferred to the market, it must be validated empirically by simulating future practical use of the technology. Technology prototypes are first investigated in simplified contexts, and these simulations are scaled up to conditions of practice step by step as more becomes known about the technology. This paper discusses empirical research methods for scaling up new requirements engineering (RE) technology.When scaling up to practice, researchers want to generalize from validation studies to future practice. An analysis of scaling up technology in drug research reveals two ways to generalize, namely inductive generalization using statistical inference from samples, and analogic generalization using similarity between cases. Both are supported by abductive inference using mechanistic explanations of phenomena observed in the simulations. Illustrations of these inferences both in drug research and empirical RE research are given. Next, four kinds of methods for empirical RE technology validation are given, namely expert opinion, single-case mechanism experiments, technical action research and statistical difference-making experiments. A series of examples from empirical RE will illustrate the use of these methods, and the role of inductive generalization, analogic generalization, and abductive inference in them. Finally, the four kinds of empirical validation methods are compared with lists of validation methods known from empirical software engineering. The lists are combined to give an overview of some of the methods, instruments and data analysis techniques that may be used in empirical RE.