Accuracy Measurement of Deep Neural Network Accelerator via Metamorphic Testing

摘要

Deep neural networks (DNNs) have been proved to be popular and successful in our daily life. For mobile devices with the services from DNN, it is needed to equip a DNN accelerator to employ fast and accurate DNN models. Different DNN accelerators may have different accuracy performance, which directly affects the user experience. To measure the accuracy of the DNN accelerators, there only exists indirect approach by comparing to the third-party platform with 32-bit float format. In this paper, metamorphic testing is employed to measure the accuracy of the DNN accelerator for the first time. Specifically, our scheme utilizes the mathematical properties of the operators in deep neural networks to measure the accuracy of the DNN accelerator, getting rid of the dependence on the third-party platform. Besides, we design an objective metric to quantitatively measure the accuracy performance. It can be used not only to facilitate the direct competition between different accelerators but also to detect defects in the implementation of the DNN accelerator. An experiment on two well-known DNN accelerators, HiAI and Snapdragon Neural Processing Engine (SNPE), is conducted, and the results show that the metamorphic distance of the former is smaller than that of the latter in terms of their accuracy performance with 16-bit float format on typical Ops, such as softmax and convolution.