Classification Tree Extraction from Trained Artificial Neural Networks

Recent advances in neural networks design and training provoked the 2nd artificial neural networks (ANN) renaissance. In many cases classification decision made by trained fully connected neural nets is better than that acquired by models like C4.5 or C5.01,2. But in contrast to decision trees, ANN models are “black boxes”, i.e., it is impossible to understand how classification decision is made. In many areas it is critical and even obligate to understand how a model performs classification thus rendering ANN usage as obsolete. Recently, some researchers have proposed and described separate steps that would allow extracting knowledge from a trained multi-layered fully connected sigmoidal neural network. This process involves several steps such as trained network training, pruning and knowledge extraction. This paper provides an overview of all the aforementioned steps, as well as describes how a knowledge extraction system can be built. We describe our Neural Network Knowledge eXtraction (NNKX) system and provide experimental results of rule extraction from the trained multi-layered feed-forward sigmoidal artificial neural network in the form of binary classification decision trees. The results obtained suggest that extracted decision trees have good classification accuracy and sizes comparable to C4.5 trees and even overcoming them in some cases. Thus the proposed system can be successfully applied to better understand and validate ANN models. We provide link to source code repository with the implementation of described system.