This Is AuburnElectronic Theses and Dissertations

Automatic Detection of Software Security Vulnerabilities in Executable Program Files




Tevis, Jay-Evan

Type of Degree



Computer Science and Software Engineering


Secure programming describes those techniques that software developers use to provide security features in their applications. In addition to these techniques, software practitioners use static code security checkers to parse through and scan the source code, looking for potential security problems. Related to static code checking, runtime checkers have been developed that monitor the software while it is in use. In an effort to counter the hacker threat, software security professionals need better methods and tools than these to analyze executable programs the way hackers do: from the binary data level. This level is where the hackers find the secret doorways and security loopholes that are not evident in high-level source code. A few commercial companies have recently started marketing software products that will scan executable files for software security vulnerabilities; however, these products have unpublished methodologies and unverified test results. Consequently, software practitioners have only a loose collection of homegrown, commercial, and operating system software tools to perform their secure programming work and to do so in primarily a manual approach. To help security analysts, programmers, and users detect security vulnerabilities in executable program files, we have created a methodology that uses information located in the headers, sections, and tables of a Windows NT/XP executable file, along with information derived from the overall contents of the file, as a means to detect specific software security vulnerabilities without having to disassemble the code. In addition, we have instantiated this methodology in a software utility program called findssv that automatically dissects an executable file and detects certain anomalies and software security vulnerabilities before installing and running the software. We tested findssv on seven categories of files: software installation files, software development files, Windows XP operating system files, Microsoft application files, security-centric application files, and miscellaneous application files. We show through the test results on these 2700 files that findssv is able to detect table size anomalies, large zero-filled regions of bytes, unknown regions of bytes, compressed files, sections that are both writable and executable, and the use of functions susceptible to buffer overflow attacks. We also list sixteen key security vulnerability findings about software in the seven categories.