An organizational model to distinguish between and integrate research and evaluation activities in a theory based evaluation

Theory-based evaluation (TBE) is an evaluation method that shows how a program will work under certain conditions and has been supported as a viable, evidence-based option in cases where randomized trials or high-quality quasi-experiments are not feasible. Despite the model's widely accepted theoretical appeal there are few examples of its well-implemented use, probably due to time and money limitations necessary for planning and a confusion over the definitions between research and evaluation functions and roles. In this paper, we describe the development of a theory-based evaluation design in a Math and Science Partnership (MSP) research project funded by the U.S. National Science Foundation (NSF). Through this work we developed an organizational model distinguishing between and integrating evaluation and research functions, explicating personnel roles and responsibilities, and highlighting connections between research and evaluation work. Although the research and evaluation components operated on independent budgeting, staffing, and implementation activities, we were able to combine datasets across activities to allow us to assess the integrity of the program theory, not just the hypothesized connections within it. This model has since been used for proposal development and has been invaluable as it creates a research and evaluation plan that is seamless from the beginning.

► It may be advantageous to separate evaluation and research activities budgetarily, procedurally, and operationally, but it is not desirable to separate these two empirical processes completely.
► Logic models have little practical or theoretical utility unless empirical processes are added to operationalize constructs and test strength and directionality of connecting links.
► Large-scale, complex, field-based interventions will typically have many more research and evaluation questions to answer than resources allow.
► An organizational model to clarify the roles of the evaluation and research teams can be beneficial in a theory based evaluation, particularly with complex theories or large programs.