Process Behavior Monitoring via API Hooking Using Virtualization

摘要

Malwares appear and spread so quickly due to the development of Internet. Information systems are mostly threatened by information stealing and destroying attacks launched by malwares. Now it is difficult to detect malwares by traditional security tools based on signature due to their metamorphosis. To protect users' data, the behaviors of running processes should be monitored. Traditional host-based security solutions, relying on operating system kernel, are vulnerable to be detected, and manipulated by malwares. The solutions outside of the protected VM based on virtualization technologies are advocated for avoiding the detection and manipulation from malwares. However, they can only monitor system call, but not application-level API. API call tracing is a powerful technique for regulating and monitoring program behaviors. In this paper, we propose a novel approach that can enable security tools to trace API call. We present an architecture which can hook APIs in dynamic libraries without preinstalling tools in the guest VM. We utilize virtual machine monitor (VMM) to enforce isolation between guest VM and security tools. We leverage hardware virtualization and Extended Page Tables (EPT) features to provide memory protection for hooks and keep transparency for guest VM. We implement a prototype on Xen to demonstrate the feasibility of the approach. We also develop representative applications for the Windows OS that monitor API calls. The evaluations show our prototype is effective and bring acceptable overhead.