Countering the concept-drift problems in big data by an incrementally optimized stream mining model

•The paper investigates the performance of incremental decision trees for concept drift, including VFDT, ADWIN, and iOVFDT.•The computer simulation results show that iOVFDT has higher accuracy and less memory consumption than the original VFDT.•The proposed method is useful and significant in fields of big data mining especially when concept-drift problem arises.

Mining the potential value hidden behind big data has been a popular research topic around the world. For an infinite big data scenario, the underlying data distribution of newly arrived data may be appeared differently from the old one in the real world. This phenomenon is so-called the concept-drift problem that exists commonly in the scenario of big data mining. In the past decade, decision tree inductions use multi-tree learning to detect the drift using alternative trees as a solution. However, multi-tree algorithms consume more computing resources than the singletree. This paper proposes a singletree with an optimized node-splitting mechanism to detect the drift in a test-then-training tree-building process. In the experiment, we compare the performance of the new method to some state-of-art singletree and multi-tree algorithms. Result shows that the new algorithm performs with good accuracy while a more compact model size and less use of memory than the others.