From Practice to Theory and back to Practice: Reflexivity in Measurement and Evaluation for Evidence-based Policy Making in the Information Society

This article presents a comprehensive review of literature and practice on the measurement and evaluation of ICT impact in general and of selected public eServices (eGovernment, eHealth, and eInclusion), within the context of the Information Society paradigm. It calls for a sustained meta-level analysis, in which it is important to have a self-reflexive layer where we evaluate and measure how we evaluate and measure, and what the implications and the trade-offs are. The analysis justifies the call for reflexivity as a necessary step towards advancing our understanding of the impact of ICT within the Information Society domains observed. The article concludes presenting some possible ways of re-coupling the conceptual and technical dimensions of the measurement process and linking the production of measurement data more closely with impact evaluation causal models and hypotheses. It suggests that this could be achieved by refocusing on the micro level and stimulating a gradual micro-macro link through an intermediate (meso) level, which effects need to be described using different measurements and evaluation paradigms. In order to do so it is required to refine and test our theories and models to find out where it would be most feasible to scale up to the global (macro) level and to shape and inform the production of more valid measurement indicators for better informing evidence-based policy making in the Information Society.

► The article presents a review of literature and practice on the measurement of ICT impact.
► The article discusses the case of selected eServices and application domains, namely eGovernment, eHealth and eInclusion.
► The article calls for the need of a self-reflexive layer on evaluation and measurement.
► The article discusses how to re-couple the conceptual and technical dimensions of measuring the Information Society.