Understanding and protecting closed-source systems through dynamic analysis

• Main Author: Dolan-Gavitt, Brendan
• Other Authors: Lee, Wenke
• Format: Others
• Language: en_US
• Published: Georgia Institute of Technology 2015
• Subjects: Virtualization; Security; Reverse engineering
• Online Access: http://hdl.handle.net/1853/53012

In this dissertation, we focus on dynamic analyses that examine the data handled by programs and operating systems in order to divine the undocumented constraints and implementation details that determine their behavior in the field. First, we introduce a novel technique for uncovering the constraints actually used in OS kernels to decide whether a given instance of a kernel data structure is valid. Next, we tackle the semantic gap problem in virtual machine security: we present a pair of systems that allow, on the one hand, automatic extraction of whole-system algorithms for collecting information about a running system, and, on the other, the rapid identification of “hook points” within a system or program where security tools can interpose to be notified of security-relevant events. Finally, we present and evaluate a new dynamic measure of code similarity that examines the content of the data handled by the code, rather than the syntactic structure of the code itself. This problem has implications both for understanding the capabilities of novel malware as well as understanding large binary code bases such as operating system kernels.