MLC: An Efficient Multi-level Log Compression Method for Cloud Backup Systems

摘要

With the rapid development of Internet and cloud services, logs become one of the fastest growing data types in backup storage systems. These massive data always require long-term storage, which incurs high storage overhead. Typical compression algorithms, such as traditional lossless compression and log-specific compression algorithms, are employed to increase storage efficiency. However, these algorithms ignore the inherent semantics of logs, particularly for the well-structured compositions within log records and the striking similarities among them. Therefore, they cannot guarantee a satisfactory compression ratio. To address this problem, we propose a novel Multi-level Log Compression (MLC) method for cloud backup systems. MLC can achieve high compression ratio for various applications and workloads. Different from existing compression algorithms, MLC first explores data redundancy among log records and divides them into different buckets in accordance with their similarities. Then, the log records are condensed by a variation of delta compression. After that, a traditional compression algorithm is employed as the secondary compression to further improve the compression ratio. To demonstrate the effectiveness of MLC, we conduct several experiments under different log workloads. The results show that, MLC improves the compression ratio of 7zip, gzip and bzip2 by up to 30.3%, 26.8% and 16.1%, respectively.