Study of Recognizing Hand Actions from Video Sequences during Suture Surgeries Based on Temporally-Sectioned SIFT and Sliding Window Based Neural Networks

摘要

Towards the realization of a robotic nurse that can support surgeries autonomously by recognizing surgical situations only using video informations, this paper proposes an improved method by using sectioned-SIFT and sliding window based neural network that can recognize surgeon's hand actions: suture and tying. Hand area is detected by using color information and then the video sequence is partitioned into sections. Sectioned-SIFT descriptors are computed in each section and built a word vocabulary. Histogram feature of the action is spliced by using word's frequency in each section. Finally, sliding window and neural network is used to recognize the significant actions: suture and tying. The proposed method has achieved the 100% recognition rate for manually extracted actions and 90% recognition rate for whole surgery video sequences.