Monitoring Resources of Machine Learning Engine In Microservices Architecture

摘要

Microservices architecture facilitates building distributed scalable software products, usually deployed in a cloud environment. Monitoring microservices deployed in a Kubernetes orchestrated distributed advanced analytics machine learning engines is at the heart of many cloud resource management solutions. In addition, measuring resource utilization at more granular level such as per query or sub-query basis in an MPP Machine Learning Engine (MLE) is key to resource planning and is also the focus of our work. In this paper we propose two mechanisms to measure resource utilization in Teradata Machine Learning Engine (MLE). First mechanism is the Cluster Resource Monitoring (CRM). CRM is a high-level resource measuring mechanism for IT administrators and analytics users to visualize, plot, generates alerts and perform live and historical-analytics on overall cluster usage statistics. Second mechanism is the Query Resource Monitoring (QRM). QRM enables IT administrators and MLE users to measure compute resource utilization per individual query and its sub-queries. When query takes long time, QRM provides insights. This is useful to identify expensive phases within a query that tax certain resources more and skew the work distribution. We show the results of proposed mechanisms and highlight use-cases.