Detection of Buffer Overflow Attacks with QEMU Emulator

• Main Authors: Hou-Xiang Kuo, 郭后翔
• Other Authors: Li-Ming Tseng
• Format: Others
• Language: en_US
• Published: 2007
• Online Access: http://ndltd.ncl.edu.tw/handle/wty7kc

碩士 === 國立中央大學 === 資訊工程研究所 === 96 === Buffer overflow has always been a dominant issue of system security. Many computer viruses or worms exploit this vulnerability to damage computer systems. Although numerous researches have been proposed to defend such attack, solutions that were really used as standard were rare. The main reason is that few solutions can be compatible with user binary code. This paper chooses QEMU emulator to emulate a hardware behavior and selects SmashGuard mechanism to test its feasibility. The result showed that it will produce some problems, and the reason was analyzed. Hence, this paper proposed a two layer checking mechanism. In addition to checking the consistency of return address, validity of return address was also checked. The result demonstrates that this mechanism can differentiate and detect typical stack-smashing attack.