Data-Driven Retrospective Interviewing (DDRI): A proposed methodology for formative evaluation of pervasive games

• We propose a new formative evaluation method that can be used to evaluate a pervasive game – called DDRI.
• DDRI uses a cyclical data-driven technique using a quantitative record of user behaviors and interviews to allow researchers to contextualize users’ actions to understand how a game is accepted and integrates to users’ life style.
• We describe DDRI and use it to formatively evaluate a pervasive commercial health game called SpaPlay.
• Findings show how DDRI uncovered design issues that were not possible to uncover using current methods.

Recently, games have gained much momentum as enablers of health behavior change. Current projects on health games, however, face a serious challenge: evaluation methods used to assess and formatively evaluate such games do not adequately gauge acceptability and integration of the game into participants’ lives. These are crucial elements to enable adherence of the participants to the game to induce behavior change. In this paper we present a formative evaluation methodology we developed, called Data-Driven Retrospective Interviewing (DDRI). Used within naturalistic settings, DDRI investigates how participants accept and integrate a game into their lives. The method is comprised of several steps. A game is first instrumented to collect behavioral data, which is then analyzed to inform the construction of interview questions used to retrospectively contextualize and reflect on participants’ behaviors. The contribution of this paper is: the methodology and its application to evaluate a pervasive health game called SpaPlay (Seif El-Nasr et al., 2011). Findings of this investigation helped designers re-think and augment their design as they uncovered how users used their game, which, in some cases, was different from how they initially intended. Such findings would have not been possible using conventional evaluation methods.