Impure Decision Trees for Auc and Log loss optimization

摘要

Decision Tree is one of the most popular supervised Machine Learning algorithms; it is also the easiest to understand. But finding an optimal decision tree for a given data is a harder task and the use of multiple performance metrics adds some complexity to the problem of selecting the most appropriate DT. The most used evaluation metrics are Log loss and Auc, but they are penalized against zero and one predictions. Several techniques have been proposed to deal with this issue among them Laplace Correction which adjusts probability estimates at the leaf nodes by avoiding purity nodes after tree construction. In this paper, we propose a new adjustment of the probability estimate by avoiding the creation of pure nodes during the construction of the tree rather than after its construction. This approach aims at improving the performance of the model for the log loss and auc metrics. We present experiments suggesting that our method leads to find better performance regardless of the used metric.