Optimization of out-of-core data preparation methods identifying runtime determining factors

摘要

In the data preparation phase of a data mining task the raw, fine granulated data has to be transformed according to analytical aims into a more compact form in order to represent data at a higher abstraction level suitable for machine processing and human understanding as well. Vast datasets require sophisticated, out-of-core methods, which are prepared to handle these datasets using external storages during their execution. In this paper we investigate different pre-processing approaches to overcome the limitation of the size of the main memory from theoretical and practical points of view. We propose possible alternatives for different processing scenarios. Both of the proposed out-of-core algorithms are capable of processing datasets which are by orders of magnitude larger than the main memory; all this is done in a fault-tolerant way and even on an average PC.