BDLaaS: Big Data Lab as a Service for Experimenting Big Data Solution

摘要

Big Data technologies are complex. Building a Big Data ecosystem for deploying and running data products needs specialized skillset. Although, an exhaustive number of technologies have been developed over the last decade, complexity remained an issue for the users. Unfortunately, there is no solution which can reduce heavy manual effort requires configuring and manage complex environment for running data products. There are several platforms which promise an easy to use infrastructure, however, in reality, these platforms are not adequately helpful for many users (data scientists including statisticians and machine learning experts) due to lack of skills in managing system level complexities of Big Data technologies. As a matter of fact, it is difficult for such users to exploit the power of Big Data technologies. Also, learning these technologies is time consuming. For some users, learning complex technologies is nearly impossible. Recently, the notion of container image has drawn an attention because it reduces configuration complexity. A few Big Data technologies have already been containerized. However, the reality is somewhat not the same because containerization requires learning new technologies called Docker. They require manual intervention for configuring images and managing them at runtime. Furthermore, these tasks are cumbersome. In this paper, we present the design and development the initial version of a virtual lab as a service called BDLaaS which will ease building Big Data infrastructure for deploying Big Data solution such as Analytics. We explain how BDLaaS can be used by users without having high-level expertise.