Month: November 2023

We’re nearing the end of the quarter and our team has made serious strides in analyzing our chosen malware samples. With the analysis portion of our capstone project mostly complete, I’d like to share a few of my thoughts on my greatest success from this course.

Reverse Engineering

My biggest breakthrough in our project has been using IDA Free (the free version of IDA Pro) to reverse engineer an executable malware file. The challenge of this particular malware sample stemmed from our initial observations that running the file on its own would result in the malware deleting all traces of itself. Having a bit of experience with reverse engineering, I decided to open up the file in IDA and take a look inside.

One of the first things I noticed upon opening up the file in IDA was that multiple paths lead to a function which seems to run a shell command to delete a file. Presumably this is the culprit for the malware deleting itself when not run correctly – whatever “correctly” means. The next big breakthrough came in noticing that a significant chunk of the file’s functionality is gated by a call to a function which seems to process characters in a string. Upon taking a closer look at that function, it was clear that the malware expects a string ("abcd") to be passed in when executing commands. So now we had found the password required to run the malware! What next?

With a little bit more time and reverse engineering effort, I was able to determine that there are 4 different flags that the malware can be run with (-in, -re, -cc, and -c) which respectively install the malware as a Windows service, remove the malware service, print the current malware configuration, and update the malware’s configuration. Testing these commands along with the reversed password in my Windows Virtual Machine revealed that we were making progress on understanding how the malware installs itself and what some of its capabilities are. Further reversing resulted in understanding what the malware could do once it was installed, how it connects to the Command and Control (C2) server, and the format of the commands that the C2 server sends.

Why It Matters

So why go through all the effort to reverse the malware? Without diving into the assembly code, we would never have been able to find the password necessary to install the malware as a service. And while we had previously found the flags as strings, we gained a much deeper understanding of how to use them and what they controlled because of this reversing process. Ultimately, reversing the malware executable using IDA allowed our team to fully comprehend the capability of this malware in a way that simple basic and dynamic analysis could not.

The more personal reason why this work matters, though, is that reverse engineering is the kind of work I expect to be undertaking in my post-graduation job as a cybersecurity researcher. One of the major reasons that I wanted to work on this project was specifically because of the opportunity to perform reverse engineering, and I was excited to finally get that chance.

What I Learned

While I’ve been able to use multiple analysis tools throughout the course of this project, I’ve enjoyed reverse engineering with IDA the most. I’ve grown more comfortable in using the many, many tools that IDA provides (and that’s accounting for the fact that the Free version is already pared down). And from all of the documentation reading I did, I now also feel more comfortable in how Windows system calls work.

My biggest takeaway, though, is just how much more engrossed in the analysis I was once I started reversing the executable. It felt like an irresistible challenge to delve deeper and deeper into the disassembly to suss out what it was trying to accomplish. While other parts of the project were fun, nothing was as addictive as the reverse engineering part, which makes me even more excited for my future.

Last we left off, I was hoping to analyze some malware for my OSU capstone project. And the good news is that’s exactly the project I’ve been assigned! I’m working with two incredibly talented classmates, Andrea Irwin and Sean Mack, to analyze malware samples from the book Practical Malware Analysis, and so far we’ve been making fantastic progress.

This week’s capstone blog assignment asks us to talk about our experiences using AI for our project. Well, we haven’t planned to use AI tools, such as ChatGPT or Bard, as part of our analysis. But during a previous internship I did have the incredible opportunity to design and build a tool that uses ChatGPT to try and help reverse engineer black-box binaries – a task similar to analyzing malware samples.

The result of this work is called OFRAK AI, and it uses the capability of the OFRAK toolkit to unpack and analyze closed-source binaries and then makes requests to ChatGPT to help analyze the results at the binary and disassembly levels.

So without further ado, here are some of my thoughts on the use of AI to assist in reverse engineering software.

The Good

Right off the bat, getting started with ChatGPT is a breeze. Their API is well-documented, and you can find a considerable amount of examples for establishing and maintaining conversations all over the web. The time savings here is well appreciated, too, because you’ll end up spending the majority of your time fine-tuning your prompts to get the kind of results you’re looking for.

Unsurprisingly, when used for analyzing and manipulating executables, ChatGPT excelled best at tasks involving natural language. The majority of information that it was able to glean about a binary came from strings and symbol information, such as identifying the compiler used by the .comment section of an ELF or common library functions by their symbol names.

One particularly fun use of ChatGPT was to patch a binary’s debugging strings after altering their voice. For example, by extracting strings of a certain length from a file, asking ChatGPT to “sassify” them, and then patching them back into the original binary, we were able to considerably improve the console output of a Cisco router.

The Bad

Alas, ChatGPT certainly isn’t the panacea that some may think it is, and that’s especially true when it comes to reverse engineering binary files. While ChatGPT excels at operating on natural languages such as English, it performs much worse at formal languages like you find in disassembly output. And while it may be able to regurgitate answers to common Leetcode questions – thanks to data it’s seen thousands of times in its training corpus, no doubt – it leaves much to be desired when trying to reason about the purpose of a function given only its assembly instructions. (Perhaps more people should be solving Leetcode problems in Assembly, one wonders…)

For starters, the ChatGPT tokenizer simply isn’t meant for formal languages. Given the unnatural character combinations found in assembly instructions (how many words contain “xmm1”?), you rapidly chew through tokens while parsing input. Combined with the rather limited conversation history in terms of token count, it’s impossible to feed ChatGPT the entire disassembly of any program. This makes it unable to use contextual information to form a larger understanding of what the program does.

What about the analysis that ChatGPT was able to perform on human readable strings? Unfortunately, those results aren’t as promising as you might think, either. For example, it doesn’t take an experienced reverse engineer to spot the GNU version in the .comment section using standard RE tools. Furthermore, symbol tables and their associated symbol names are likely to be stripped in production code, especially if you’re dealing with malware.

The Uncertain

So does that mean there’s no future for AI in reverse engineering and closed-source binary analysis? I wouldn’t be quite so quick to write it off. On the one hand, there is an art to reverse engineering – the forward compilation process is lossy after all, and thus the reversing process is non-deterministic – and so it requires creativity (which Large Language Models like ChatGPT arguably do not possess).

On the other hand, LLMs and other machine learning models trained more specifically on formal languages should be expected to perform better than a general chatbot like ChatGPT. For instance, take Binary Diffing (AKA Binary Similarity or Binary Matching), a technique used in reverse engineering to compare some unknown binary file or function to a known one. For many years, research into binary diffing focused on analytic approaches, such as comparing control flow and call graphs between complex blocks. Indeed, this forms part of the approach used by the industry standard tool BinDiff. Looking at more recent research, though, shows a clear trend towards the use of machine learning. And those research papers claim to show great results at using their machine learning models to identify functions based on their disassembly. Disappointingly, though, if code is published alongside this research at all, it has only been for re-training and fine tuning the models. I have yet to find a binary diffing tool available for widespread use that utilizes one of these machine learning models for its core functionality.

So what’s the verdict, can AI help with reverse engineering? Based on my own experiences using ChatGPT for reverse engineering, there’s still a long way to go before it produces the same kinds of success that it has for forward engineering. Considering the amount of active research being conducted in this problem space, though, I’m excited to see what improvements will be made over the next several years.

Marc Zalik