Data processing in Hive vs. SQL server: A comparative analysis in the query performance

摘要

Data processing means manipulating the input raw data using application program to get the desired output. The main target behind data processing is to convert unusable data into a usable form. Relational database management system (RDBMS) is playing main role for data processing in most of the organizations. MySQL, SQL Server, Oracle, SQLite are some of the well-known database management systems. Moving forward big data technology is becoming more admired towards many organizations as nature and size of data sets grow rapidly. Big data is particularly apt for extreme large volume where conventional data processing application is inadequate to deal. Generally, large organizations use big data technology for processing large volume of data. However, this paper targets the audience of Small Enterprises (SE) where the database size is relatively small and is not distributed over multiple servers. The attempted study examines the query execution time between traditional data warehouse, grounded on the SQLite, SQL Server and a parallel data warehouse grounded on the Hive built on the top of Hadoop so that SE can decide which system performs better in terms of the time of data processing. The study finds that it is better to use traditional database systems if SE does not have a plan in near future to work with vast amount of data i.e. the data set fits on a single computer.