Quantifying IT estimation risks

A statistical method is proposed for quantifying the impact of factors that influence the quality of the estimation of costs for IT-enabled business projects. We call these factors risk drivers as they influence the risk of the misestimation of project costs. The method can effortlessly be transposed for usage on other important IT key performance indicators (KPIs), such as schedule misestimation or functionality underdelivery. We used logistic regression as a modeling technique to estimate the quantitative impact of risk factors. We did so because logistic regression has been applied successfully in fields including medical science, e.g. in perinatal epidemiology, to answer questions that show a striking resemblance to the questions regarding project risk management. In our study we used data from a large organization in the financial services industry to assess the applicability of logistic modeling in quantifying IT risks. With this real-world example we illustrated how to scrutinize the quality and plausibility of the available data. We explained how to deal with factors that cannot be influenced, also called risk factors, by project management before or in the early stage of a project, but can have an influence on the outcome of the estimation process. We demonstrated how to select the risk drivers using logistic regression. Our research has shown that it is possible to properly quantify these risks, even with the help of crude data. We discussed the interpretation of the models found and showed that the findings are helpful in decision making on measures to be taken to identify potential misestimates and thus mitigate IT risks for individual projects. We proposed increasing the auditing process efficiency by using the found cost misestimation models to classify all projects as either risky projects or non-risky projects. We discovered through our analyses that projects must not be overstaffed and the ratio of external developers must be kept small to obtain better cost estimates. Our research showed that business units that report on financial information tend to be risk mitigating, because they have more cost underruns in comparison with business units without reporting; the latter have more cost overruns. We also discovered a maturity mismatch: an increase from CMM level 1 to 2 did not influence the disparity between a cost estimate and its actual if the maturity of the business is not also increased.