A visualization of anomaly memory behavior of full-virtualized Windows OS using virtual machine introspection

摘要

Nowadays, a variety of applications of Windows OS has become more sophisticated and the complexity of its behavior has also been increased. Large scale applications generate the vast size of traffic and logs hard to understand for a shot while which impose a great burden on administrators Visualization could be one of the solutions for this problem. In this paper we propose a visualization technique of anomaly memory behavior of full-virtualized Windows OS using virtual machine introspection. Proposed system has been implemented in tree steps: (1 modification of Windows OS by inserting library and filter driver, (2) modification of the debug register handler of virtual machine monitor, and (3) deploying visualization tool on Host OS. We generate the sequence of memory behavior using virtual machine introspection. On guest Windows OS side, we intercept memory API And then, virtual machine monitor generates the sequence and transfer it to host OS. To present the visualization by proposed system, we deal with two popular security issues: BoF (buffer overflow) based shellcode execution and P2P network application. We have successfully detected the incidents of BoF based shellcode detection, and visualize the memory state transition of P2P application. We can conclude that visualization is effective way to discover and detect anomaly memory behavior caused by BoF based shellcode execution and P2P application usage.