FREQUENT PATTERN ANALYSIS FOR DISTRIBUTED SYSTEMS

摘要

Methods, systems, and devices supporting frequent pattern (FP) analysis for distributed systems are described. Some database systems may analyze data sets to determine FPs within the data. However, because FP mining relies on combinatorics, very large data sets incur combinatorial explosion of the memory and processing resources needed to handle the FP analysis. To obtain the resources needed for FP analysis of large data sets, the database system may spin up multiple data processing machines and may distribute the FP mining process across these machines. The database system may distribute the data set according to a tradeoff between commonality and data attribute list length, efficiently utilizing the resources at each data processing machine. This may result in data subsets with either large numbers of data objects or large numbers of data attributes for data objects, but not both, limiting the combinatorial explosion and, correspondingly, limiting the resources required.