Efficient Synthetic Data Generator for structured Data

摘要

Data have become a torrent flowing into every area of the global economy. Data sizes grow beyond zettabyte barrier. These facts make it obvious that, the applications catering this growing data volume must be well capable to handle large amount of data efficiently. Application performance degrades with increasing data size. Break down of applications due to inadequately handling of huge data beyond its capacity may convert into loss of money, time and business reputation. Application performance testing on large data sizes is very necessary to avoid such catastrophic outcomes. Applications must be tested for growing data size to ensure SLA (Service Level Agreement). Testing such, applications need similar kind of data as of in real life and very large in volume. The solution to this problem is 'efficient' generation of customized test data. Efficient means generating data as fast as possible maximally utilizing underlying resources. In this paper, the novel approach of Data Generator has been proposed, which generates large amount of testing data most efficiently. The efficiency of this generator is comparatively more than any other data generator due to its capability of utilizing all underlying resources (CPU, disk, memory) maximally. Maintaining dynamic queue size of every resource, finding most optimal size of buffer on the fly and generation and spawning of optimal number of threads running in parallel are few of the key techniques materialized in making data generator most efficient. AS case study, we have compared the performance of the proposed data generator with TPC-H benchmark data generator, 'dbgen' and it is shown that the proposed data generator is up to 8 times faster than 'dbgen' on same hardware taken for case study.