CrashMonkey and ACE: Systematically Testing File-System Crash Consistency

摘要

We present CRASHMONKEY and ACE, a set of tools to systematically find crash-consistency bugs in Linux file systems. CRASHMONKEY is a record-and-replay framework which tests a given workload on the target file system by simulating power-loss crashes while the workload is being executed, and checking if the file system recovers to a correct state after each crash. ACE automatically generates all the workloads to be run on the target file system. We build CRASHMONKEY and ACE based on a new approach to test file-system crash consistency: bounded black-box crash testing (B-3). B-3 tests the file system in a black-box manner using workloads of file-system operations. Since the space of possible workloads is infinite, B-3 bounds this space based on parameters such as the number of file-system operations or which operations to include, and exhaustively generates workloads within this bounded space. B-3 builds upon insights derived from our study of crash-consistency bugs reported in Linux file systems in the last 5 years. We observed that most reported bugs can be reproduced using small workloads of three or fewer file-system operations on a newly created file system, and that all reported bugs result from crashes after fsync()-related system calls. CRASHMONKEY and ACE are able to find 24 out of the 26 crash-consistency bugs reported in the last 5 years. Our tools also revealed 10 new crash-consistency bugs in widely used, mature Linux file systems, 7 of which existed in the kernel since 2014. Additionally, our tools found a crash-consistency bug in a verified file system, FSCQ. The new bugs result in severe consequences like broken rename atomicity, loss of persisted files and directories, and data loss.