JOINS AND AGGREGATIONS ON MASSIVE GRAPHS USING LARGE-SCALE GRAPH PROCESSING

摘要

This disclosure is directed to large-scale graph processing to determine second-degree connections for members of a social network. A social graph is duplicated into two graphs, where each of the two graphs are partitioned into various partitions. The partitions are each sorted according to a predetermined key selected from each of the graphs. The partitions are then assigned logical Work Units, where a first set of Work Units are determined from a first graph and second set of Work Units are determined from a second graph. The Work Units are determined to be asymmetrical such that the partitions of the first set of Work Units are assigned differently than the partitions of the second set of Work Units. One set of Work Units are loaded in-memory and another set of Work Units are streamed to a mapping module process, which determines the second-degree connections from the sets of Work Units.