A multi-dimensional quality assessment of state-of-the-art process discovery algorithms using real-life event logs

Process mining is the research domain that is dedicated to the a posteriori analysis of business process executions. The techniques developed within this research area are specifically designed to provide profound insight by exploiting the untapped reservoir of knowledge that resides within event logs of information systems. Process discovery is one specific subdomain of process mining that entails the discovery of control-flow models from such event logs. Assessing the quality of discovered process models is an essential element, both for conducting process mining research as well as for the use of process mining in practice. In this paper, a multi-dimensional quality assessment is presented in order to comprehensively evaluate process discovery techniques. In contrast to previous studies, the major contribution of this paper is the use of eight real-life event logs. For instance, we show that evaluation based on real-life event logs significantly differs from the traditional approach to assess process discovery techniques using artificial event logs. In addition, we provide an extensive overview of available process discovery techniques and we describe how discovered process models can be assessed regarding both accuracy and comprehensibility. The results of our study indicate that the HeuristicsMiner algorithm is especially suited in a real-life setting. However, it is also shown that, particularly for highly complex event logs, knowledge discovery from such data sets can become a major problem for traditional process discovery techniques.

► We assess the quality of process discovery techniques using eight real-life event logs.
► The assessment is multi-dimensional focussing on accuracy, comprehensibility, and scalability.
► There exists a major difference between evaluation based on artificial vs. real-life event logs.
► HeuristicsMiner appears to be the most appropriate and robust technique in a real-life setting.
► High complex event logs can be an insurmountable challenge for traditional discovery techniques.