Virtual memory technique for efficiently solving connected problems in a distributed environment

摘要

Embodiments herein present a virtual memory technique for efficiently solving connected problems in a distributed environment. An exemplary method begins by selectively loading computing task segments into a first node, wherein the number of computing task segments is the maximum number of computing task segments that will fit within the first node. The computing task segments are enclosed within wrappers, wherein the wrappers are streamed into the first node at a maximum rate that a disk can read and write data. Further, the wrappers comprise priority wrappers that must be computed in order to process the computing task segments; and thus, the priority wrappers are loaded into the first node first. In addition, the computing task segments comprise computing task segment block collections, wherein the method creates a list of computing task segment block collection jobs that must be completed in order to process one of the computing task segments.