Stacking Bagged and Boosted Forests for E€ective Automated Classification

摘要

Random Forest (RF) is one of the most successful strategies for automatedrnclassi€cation tasks. Motivated by the RF success, recentlyrnproposed RF-based classi€cation approaches leverage the central RFrnidea of aggregating a large number of low-correlated trees, whichrnare inherently parallelizable and provide exceptional generalizationrncapabilities. In this context, this work brings several new contributionsrnto this line of research. First, we propose a new RF-basedrnstrategy (BERT) that applies the boosting technique in bags of extremelyrnrandomized trees. Second, we empirically demonstraternthat this new strategy, as well as the recently proposed BROOF andrnLazyNN RF classi€ers do complement each other, motivating usrnto stack them to produce an even more e‚ective classi€er. Up tornour knowledge, this is the €rst strategy to e‚ectively combine thernthree main ensemble strategies: stacking, bagging (the cornerstonernof RFs) and boosting. Finally, we exploit the ecient and unbiasedrnstacking strategy based on out-of-bag (OOB) samples to considerablyrnspeedup the very costly training process of the stackingrnprocedure. Our experiments in several datasets covering two highdimensionalrnand noisy domains of topic and sentiment classi€cationrnprovide strong evidence in favor of the bene€ts of our RF-basedrnsolutions. We show that BERT is among the top performers in thernvast majority of analyzed cases, while retaining the unique bene€tsrnof RF classi€ers (explainability, parallelization, easiness of parameterization).rnWe also show that stacking only the recently proposedrnRF-based classi€ers and BERT using our OOB-based strategy is notrnonly signi€cantly faster than recently proposed stacking strategiesrn(up to six times) but also much more e‚ective, with gains up to 21%rnand 17% on MacroF1 and MicroF1, respectively, over the best basernmethod, and of 5% and 6% over a stacking of traditional methods,rnperforming no worse than a complete stacking of methods at arnmuch lower computational e‚ort.