Towards assessing the security of program binaries

Software vulnerabilities are widespread, often resulting from coding weaknesses and poor development practices. These vulnerabilities can be exploited by attackers, posing risks to confidentiality, integrity, and availability. To protect themselves, end-users of software may have an interest in knowing whether the software they purchase, and use is secure from potential attacks. Our work is motivated by this need to automatically assess and rate the security properties of binary software.

While many researchers focus on developing techniques and tools to detect and mitigate vulnerabilities in binaries, our approach is different. We aim to determine whether the software has been developed with proper care. Our hypothesis is that software created with meticulous attention to security is less likely to contain exploitable vulnerabilities. As a first step, we examined the current landscape of binary-level vulnerability detection. We categorized critical coding weaknesses in compiled programming languages and conducted a detailed survey comparing static analysis techniques and tools designed to detect these weaknesses. Additionally, we evaluated the effectiveness of open-source CWE detection tools and analyzed their challenges. To further understand their efficacy, we conducted independent assessments using standard benchmarks.

To determine whether software is carefully and securely developed, we propose several techniques. So far, we have used machine learning and deep learning methods to identify the programming language of a binary at the functional level, enabling us to handle complex cases like mixed-language binaries and we assess whether vulnerable regions in the binary are protected with appropriate security mechanisms. Additionally, we explored the feasibility of detecting secure coding practices by examining adherence to SonarQube’s security-related coding conventions.

Next, we investigate whether compiler warnings generated during binary creation are properly addressed. Furthermore, we also aim to optimize the array bounds detection in the program binary. This enhanced array bounds detection will also increase the effectiveness of detecting secure coding conventions that are related to memory safety and buffer overflow vulnerabilities.

Our ultimate goal is to combine these techniques to rate the overall security quality of a given binary software.