Cost-conscious comparison of supervised learning algorithms over multiple data sets

摘要

In the literature, there exist statistical tests to compare supervised learning algorithms on multiple data sets in terms of accuracy but they do not always generate an ordering. We propose Multi ~2Test, a generalization of our previous work, for ordering multiple learning algorithms on multiple data sets from best to worst where our goodness measure is composed of a prior cost term additional to generalization error. Our simulations show that Multi ~2Test generates orderings using pairwise tests on error and different types of cost using time and space complexity of the learning algorithms.