Why software repositories are not used for defect-insertion circumstance analysis more often: A case study

ContextRoot-cause analysis is a data-driven technique for developing software process improvements in mature software organizations. The search for individual process correlates of high defect densities, which we call defect insertion circumstance analysis (DICA), is potentially both effective and cost-efficient as one approach to be used when attempting a general defect root cause analysis. In DICA, data from existing repositories (version archive, bug tracker) is evaluated largely automatically in order to determine conditions (such as the people, roles, components, or time-periods involved) that correlate with higher-than-normal defect insertion frequencies. Nevertheless, no reports of industrial use of DICA have been published.ObjectiveDetermine the reasons why DICA is not used more often by practitioners.MethodWe use a single-case, typical-case, revelatory-type case study to evaluate in parallel the importance of six plausible reasons (R1–R6). The case is based on 11 years of repository data from a small but mature software company building a product in the high-end content management system domain and describes a four person-months effort to make use of these data.ResultsWhile DICA required non-negligible effort (R3) and some degree of inventiveness (R2), the most relevant roadblock was insufficient reliability of the results (R6) combined with the difficulty of assessing this reliability (R5). We identify three difficulties that led to this outcome.ConclusionCurrent repository mining methods are too immature for successful DICA. Gradual improvements are unlikely to help; different principles of operation will be required. Even with such different techniques, issues with input data quality may continue to make good results difficult-to-have.