Rethinking Benchmarking for Data

摘要

Benchmarking has been critical in making progress in the field of data, as it has provided a crucial mechanism to accelerate the progress in the data community. Early bench-marks have been responsible for spurring innovation and serving as a quantitative way to get past marketing salvos. Prime examples of this observation are the Anon et al. benchmark and the Wisconsin benchmark that spurred rapid advances in database transaction and analytic query processing. These dual technologies are now crucial to running our "digital planet" today. However, data benchmarking has changed considerably over the past four decades. In the early days, pioneers like Jim Gray and David DeWitt, were crucial in creating benchmarks that were genuinely designed to move the community forward. Back then the data industry was in its "Wild West" days. A few good-meaning cowboys is all that it took to set the industry in the right direction. Sadly, those halcyon days are long gone. The digital planet is simply too dependent on data. In fact, as has been noted before, data in the new currency. Thus, there are deeply-vested interests in modern benchmarks that simply do not achieve the goals that benchmarks claim to achieve. To address these issues, this article proposes a radical rethinking of data benchmarks. This article makes three concrete suggestions: First, data benchmarks should have no optional components, forcing the vendors to make "hard" choices when reporting benchmark results (e.g. reporting on energy consumption in TPC benchmarks, and reporting results on newer benchmarks that subsume older ones). Second, benchmarking in the cloud-era implies that each customer will have their own measures that are important to them. Thus, a service that offers automated bench-marking (and associated tuning) of customer workload in the cloud is far more important than actual benchmarks. Finally, we should dramatically rethink how our benchmark councils (including TPC) work. We should reverse the stewardship of these councils by replacing vendors from the council by the actual customers of data products, and let customers directly drive the definition of new benchmarks.