Impact of meta-analytic decisions on the conclusions drawn on the business value of information technology

Meta-analysis is a quantitative methodology that allows for summarizing the results of primary research studies in a field to provide new insights in terms of the phenomenon observed or the outcomes reported. This paper attempts to answer the fundamental question, “Do methodological decisions in a meta-analytic study affect the conclusions drawn from that study?” Specifically, this paper examines the effects of meta-analytic decisions when applied to the business value of information technology (BVIT) research stream. A closer examination of the variation in the methodological decisions informs us that, with each different decision alternative, we are examining slightly different phenomenon. The findings reveal that study outcomes do change, depending on the meta-analytic decisions that are tested. In other words, methodological decisions matter. Based on the data from 99 primary studies and 531 effect sizes, we tested seven hypotheses, in the BVIT research stream, using a comprehensive set of different methodological conditions. We find support for two findings that are consistent across all the different conditions. First, investing in information technology (IT) is positively associated with the firm's performance. We also find that large firms get more benefits from IT than small firms. These and the overall findings suggest that researchers should be cognizant of their methodological decisions, as they may be observing the phenomenon under different boundary conditions with different methodologies.

► Methodological decisions in a meta-analysis study change eventual results.
► More theoretically founded hypotheses are highly robust to meta-analytic decisions.
► Researchers should provide justifications when a meta-analytic decision is made.
► The value from IT investments is ‘positive’ in all cases examined in this study.