Design of an empirical study for comparing the usability of concurrent programming languages

Context: Developing concurrent software has long been recognized as a difficult and error-prone task. To support developers, a multitude of language proposals exist that promise to make concurrent programming easier. Empirical studies are needed to support the claim that a language is more usable than another.Objective: This paper presents the design of a study to compare concurrent programming languages with respect to comprehending and debugging existing programs and writing correct new programs. The design is applied to a comparison of two object-oriented languages for concurrency, multithreaded Java and SCOOP.Method: A critical challenge for such a study is avoiding the bias that might be introduced during the training phase and when interpreting participants’ solutions. We address these issues by the use of self-study material and an evaluation scheme that exposes any subjective decisions of the corrector, or eliminates them altogether.Results: The study template consisting of the experimental design and the structure of the self-study and evaluation material is demonstrated to work successfully in an academic setting. The concrete instantiation of the study template shows results in favor of SCOOP even though the study participants had previous training in writing multithreaded Java programs.Conclusion: It is concluded that the proposed template of a small but therefore easy-to-implement empirical study with a focus on core language constructs is helpful in characterizing the usability of concurrent programming paradigms. Applying the template to further languages could shed light on which approaches are promising and hence drive language research into the right direction.