Method and computer program for statistical and data-mining processing of large data sets

摘要

The method according to the present invention relates to the processing of large data files using statistical and data mining approaches. The method is characterized by that- the input data file having a predefined structure is subdivided into blocks containing an equal number of records (S100),- said blocks are consecutively processed thereby creating a local subresult for each block in the main memory that is built up of records having the same structure but different keys (S200),- the records of the local subresult are sorted according to a predefined principle (S300),- the current local subresult and the current global subresult created from all the previous local subresults are merged by iterating through the records of the local and global subresults once, and the new global subresult result is created on the storage device (S400), and finally- the previous global subresult is deleted from the background storage (S500), and if there are any blocks left for processing, the method returns to step S200.