Optimal Strategy of Web Page Text Allocation for Inverted Index Construction on Heterogeneous Cluster Computing System

摘要

Considering the start time consuming and the time cost on delivering and receiving data in the situation of that each slave node has different computing capability and data transfer rate, and based on the triple sort-merge algorithm where the task is scheduling divisible in Web page inverted indexes construction, we constructed an inverted index parallel algorithm using an optimal strategy of the web page text allocation. Compared with the existing algorithms, the proposed optimal algorithm reduces the computing time by 14%-19%.