Dynamic Batch Sizing for Inferencing of Deep Neural Networks in Resource-Constrained Environments

摘要

Methods, systems, and computer program products for dynamic batch sizing for inferencing of deep neural networks in resource-constrained environments are provided herein. A computer-implemented method includes obtaining, as input for inferencing of one or more deep neural networks, (i) an inferencing model and (ii) one or more resource constraints; computing, based at least in part on the obtained input, a set of statistics pertaining to resource utilization for each of multiple layers in the one or more deep neural networks; determining, based at least in part on (i) the obtained input and (ii) the computed set of statistics, multiple batch sizes to be used for inferencing the multiple layers of the one or more deep neural networks; and outputting, to at least one user, the determined batch sizes to be used for inferencing the multiple layers of the one or more deep neural networks.