Exploring binary analysis techniques for security

In this dissertation our goal is to evaluate how the loss of information at binary-level affects the performance of existing compiler-level techniques in terms of both efficiency and effectiveness. Binary analysis is difficult, as most of semantic and syntactic information available at source-level gets lost during the compilation process. If the binary is stripped and/ or optimized, then it negatively affects the efficacy of binary analysis frameworks. Moreover, handwritten assembly, obfuscation, excessive indirect calls or jumps, etc. further degrade the accuracy of binary analysis. Challenges to precise binary analysis have implications on the effectiveness, accuracy, and performance, of security and program hardening techniques implemented at the binary level. While these challenges are well-known, their respective impacts on the effectiveness and performance of program hardening techniques are less well-studied.

In this dissertation, we employ classes of defense mechanisms to protect software from the most common software attacks, like buffer overflows and control flow attacks, to determine how this loss of program information at the binary-level affects the effectiveness and performance of defense mechanisms. Additionally, we aim to tackle an important problem of type recovery from binary executables that in turn help bolster the software protection mechanisms.