Failure prediction based on log files using Random Indexing and Support Vector Machines

Research problemThe impact of failures on software systems can be substantial since the recovery process can require unexpected amounts of time and resources. Accurate failure predictions can help in mitigating the impact of failures. Resources, applications, and services can be scheduled to limit the impact of failures. However, providing accurate predictions sufficiently ahead is challenging. Log files contain messages that represent a change of system state. A sequence or a pattern of messages may be used to predict failures.ContributionWe describe an approach to predict failures based on log files using Random Indexing (RI) and Support Vector Machines (SVMs).MethodRI is applied to represent sequences: each operation is characterized in terms of its context. SVMs associate sequences to a class of failures or non-failures. Weighted SVMs are applied to deal with imbalanced datasets and to improve the true positive rate. We apply our approach to log files collected during approximately three months of work in a large European manufacturing company.ResultsAccording to our results, weighted SVMs sacrifice some specificity to improve sensitivity. Specificity remains higher than 0.80 in four out of six analyzed applications.ConclusionsOverall, our approach is very reliable in predicting both failures and non-failures.

► We describe an approach to predict failures based on log files.
► Random Indexing is applied to represent sequences of operations; each operation is characterized in terms of its context.
► Support Vector Machines associate sequences to a class of failures or non-failures.
► We apply our approach to log files collected during approximately three months of work in a large European manufacturing company.
► Our approach is very reliable in predicting both failures and non failures. Weighted SVMs improve the true positive rate.