Czip: A Fast Lossless Compression Algorithm for Climate Data

摘要

Climate data have been dramatically increasing in volume in recent years. This huge volume of climate data poses considerable challenges for data storage, archiving and sharing. In this paper, we propose a lossless compression algorithm for climate data, named czip. We efficiently eliminate data redundancy through several new methods, including adaptive prediction, exclusive OR differencing, multiway compression and static regions. To utilize the multiple cores available on modern computers, czip is implemented in parallel. Experimental results show that czip can achieve outstanding compression ratios as well as deflating and inflating throughputs; czip can achieve 800 MB/s deflating throughputs and over 2600 MB/s inflating throughputs on a server with 16 cores.