Benchhub: Store Database Benchmark Result in Database

摘要

Benchmark is an essential part of evaluating database performance. However, the procedure of setting up the environment and collecting result is time-consuming and not well defined. The uncertainty in benchmark procedure leads to low reproducibility. Furthermore, many benchmark results are highly compressed and only published in unstructured formats like document and graph. Without a structural format and the context of a benchmark, comparing benchmark results across sources is time-consuming and often biased.;In this thesis, BenchHub is presented to remedy those problems. It defines a job specification that covers the life cycle of running database benchmark in a distributed environment. A reference implementation of infrastructure is provided and will be hosted as a public service. Using this service, database developers can focus on analyzing benchmark result instead of getting them. BenchHub stores metrics like latency in time series databases and puts aggregated results along with benchmark context in relational databases. Users can query results directly using SQL and compare results across sources without extra preprocessing.;BenchHub integrates time series workloads like Xephon-B and standard database workloads like TPC-C. Comparisons between open source databases are made to demonstrate its usability. BenchHub is open sourced under MIT license and hosted on GitHub.