Parallel membership queries on very large scientific data sets using bitmap indexes

摘要

Many scientific applications produce very large amounts of data as advances in hardware fuelcomputing and experimental facilities. Managing and analyzing massive quantities of scientificdata is challenging as data are often stored in specific formatted files, such as HDF5 andNetCDF, which do not offer appropriate search capabilities. In this research, we investigated aspecial class of search capability, called membership query, to identify whether queried elementsof a set are members of an attribute. Attributes that naturally have classification values appearfrequently in scientific domains such as category and object type as well as in daily life such aszip code and occupation. Because classification attribute values are discrete and require randomdata access, performing amembership query on a large scientific data set creates challenges.Weapplied bitmap indexing and parallelization tomembership queries to overcome these challenges.Bitmap indexing provides high performance not only for low cardinality attributes but also forhigh cardinality attributes, such as floating-point variables, electric charge, or momentum in aparticle physics data set, due to compression algorithms such as Word-Aligned Hybrid. Weconducted experiments, in a highly parallelized environment, on data obtained from a particleaccelerator model and a synthetic data set.