POPE: A distributed query system for high performance analysis of very large persistent object stores

摘要

Analysis of large physics data sets is a major computing task at Fermilab. One step in such an analysis involves culling ''interesting'' events via the use of complex query criteria. What makes this unusual is the scale required: 100's of gigabytes of event data must be scanned at 10's of megabytes per second for the typical queries that are applied, and data must be extracted from 10's of terabytes based on the result of the query. The Physics Object Persistency Manager (POPM) system is a solution tailored to this scale of problem. A running POPM environment can support multiple queries in progress, each scanning at rates exceeding 10 megabytes per second, all of which are sharing access to a very large persistent address space distributed across multiple disks on multiple hosts. Specifically, POPM employs the following techniques to permit this scale of performance and access: Persistent objects: Experimental data to be scanned is ''populated'' as a data structure into the persistent address space supported by POPM. C++ classes with a few key overloaded operators provide nearly transparent semantics for access to the persistent storage. Distributed and parallel I/O: The persistent address space is automatically distributed across disks of multiple ''I/O nodes'' within the POPM system. A striping unit concept is implemented in POPM, permitting fast parallel I/O across the storage nodes, even for small single queries. Efficient Shared access: POPM implements an efficient mechanism for arbitration and multiplexing of I/O access among multiple queries on the same or separate compute nodes.