Touchless gesture user interface for 3D visualization using the Kinect platform and open-source frameworks

摘要

In this work we present a natural user interface system using hand gestures to provide touchless control of 3D visualization software using the Microsoft Kinect device. Kinect is a relatively low-cost device developed by Microsoft for the gaming industry to provide controller-free gaming, but it has potential applications in many other areas, such as: data visualization, augmented reality, accessibility and robotics. Although our solution can be used with any 3D software, the present work is primarily focused in one particular application which is the visualization of 3D medical images during a surgical procedure. The operating room is a cleansed and sterilized environment and the contact of the surgeon with traditional computer interfaces (like mouse and keyboard) could lead to contamination increasing the risk of patients infection. A touchless interface is an ideal solution since it does not demand any physical contact and still can provide the necessary control features. For the visualization of 3D medical images, we used the InVesalius software that provides 3D reconstruction of medical images with features that enable the rapid prototyping of medical models and provides high-quality visualization of 3D volumes. This software is open-source, cross-platform, multi-language and is freely available. Using open-source software libraries and image processing techniques, we implemented the hands tracking and gestures recognition from the images provided by the Kinect device and enabled the surgeon to successfully navigate through the image.