A Security Concern About Deep Learning Models

摘要

This paper mainly studies on the potential safety hazards in the obstacle recognition and processing system (ORPS) of the self-driving cars, which is constructed by deep learning architecture. We perform an attack that embeds a backdoor in the Mask R-CNN in ORPS by poisoning the dataset. Under normal circumstances, the backdoored model can accurately identify obstacles (vehicles). However, under certain circumstances, triggering the backdoor in the backdoored model may lead to change the size (bounding box and mask) and confidence of the detected obstacles, which may cause serious accidents. The experiment result shows that it is possible to embed a backdoor in ORPS. We can see that the backdoored network can obviously change the size of bounding box and corresponding mask of those poisoned instances. But on the other hand, embedding a backdoor in the deep learning based model will only slightly affect the accuracy of detecting objects without backdoor triggers, which is imperceptible for users. Eventually, we hope that our simple work can arouse people's attention to the self-driving technology and even other deep learning based models. It brings motivation about how to judge or detect the existence of the backdoors in these systems.