Benchmarking for End-to-End QoS Sustainability in Cloud-Hosted Data Processing Pipelines

摘要

The end-to-end QoS of a data processing pipeline (DPP) depends upon the performance of its constituent software services. Their performance, in turn, depends upon several factors including (a) the amount of computing resources allocated to each service, (b) the configuration of each service, (c) the inter-dependencies between adjacent services, (d) the data ingestion rate, and (e) the specific tasks each service is assigned. This makes end-to-end quality-assured cloud resource allocation for DPPs a complex and challenging task. Our overall research objective is to develop mechanisms for dynamic resource allocation for services in a pipeline so as to ensure end-to-end QoS while optimizing the associated costs. To that end, in this paper, we propose a systematic approach for building a sustainable QoS model for constituent services, which can then be used to inform resource allocation decisions for the pipeline in response to fluctuating workloads. We demonstrate the proposed approach using a custom DPP for traffic monitoring deployed on the NeCTAR cloud infrastructure.