Serving Very Large Numbers of Low Latency AutoML Models

摘要

Summary form only given, as follows. The complete presentation was not made available for publication as part of the conference proceedings. ML Serving infrastructure is becoming ubiquitous in the emerging ML industry and public cloud offerings. Existing solutions overwhelmingly rely on serving models as containers where one container hosts a single model with all its required dependencies. Salesforce and Einstein Platform have a unique multi-tenancy approach that heavily relies on AutoML and enforces automated feature engineering, training, and serving separate models per tenant. This approach helps us to scale to serve the hundreds of thousands of models. Based on the applications, and type/size of customer data, the model sizes, initialization time, and popularity/volume can vary widely, introducing the model balancing problem. We present our approach to scaling to a large number of models using multi-level routing and load balancing, sharing hundreds of models within each container, and utilizing sophisticated metric-driven mechanisms for model initialization, warmup, and model balancing. We’ll also present our solution for managing model versions and dependencies in shared container scenarios, and finally, lessons learned on our journey in this nascent space.