Buffer overflow exploits emerged in the 1980s and remained a chief scourge of the computing industry through the early parts of the 2000s. While many modern operating systems have protections against the simplest buffer overflow attacks, careless programming errors can still leave modern programs wide open to attack. Buffer overflow exploits have recently been discovered in Skype¹, Android², Google Chrome³, and others.

Here are some notable historic examples of buffer overflow exploits.

To help you understand the mechanism of the buffer overflow attack, we provide an executable of a simple program that enables the user to play a guessing game with the program. Download the secret executable at this link and extract it using the tar command:

In the following, we provide a copy of main.c (main.c), the main file associated with the executable:

This game prompts the user to enter first a secret number and then a secret string to win the guessing game. The header file other.h contains the definition of the getSecretCode and calculateValue functions, but it is unavailable to us. How then can a user beat the program? Brute forcing the solution will take too long. One strategy is to analyze the secret executable in GDB and step through the assembly to reveal the secret number and string. The process of examining assembly code to reveal knowledge of how it works is commonly referred to as reverse engineering. Readers comfortable enough with GDB and reading assembly should be able to use GDB to reverse engineer the secret number and the secret string.

However, there is a different, sneakier way to win.

The program contains a potential buffer overrun vulnerability at the first call to scanf. To understand what is going on, let’s inspect the assembly code of the main function using GDB. Let’s also place a breakpoint at address 0x0000aaaaaaaaa92c, which is the address of the instruction right before the call to scanf (placing the breakpoint at the address for scanf causes program execution to halt inside the call to scanf, not in main) and then use ni to advance forward one instruction:

Figure 1 depicts the stack immediately before the call to scanf.

Prior to the call to scanf, the first two arguments for scanf are preloaded into registers x0 and x1, respectively. The address of array buf is stored at stack location x29+0x18 (see <playGame+28>).

Now, suppose that the user enters 1234567890 at the prompt. Figure 2 illustrates what the stack looks like immediately after the call to scanf completes.

Recall that the hex values for the ASCII encodings of the digits 0 to 9 are 0x30 to 0x39, and that each stack memory location is eight bytes long. The frame pointer for main is 56 bytes away from the stack pointer. Readers tracing along can confirm the value of x29 by using GDB to print its value (p x29). In the example shown, the saved value of x29 is 0xffffffffeef0. The following command allows the reader to inspect the 64 bytes (in hex) below register sp:

This GDB command yields output that looks similar to the following:

Each line represents one 64-bit address, or two 32-bit addresses. So, the value associated with the 32-bit address 0xffffffffeedc is located at the rightmost four bytes of the line showing 0xffffffffeed8.

In this example, the address for buf is located at address 0xffffffffeed8. Therefore, the following two addresses hold the bytes associated with input string 1234567890:

The null termination byte \0 appears in the third byte location at address 0xffffffffeee2. Recall that scanf terminates all strings with a null byte.

Of course, 1234567890 is not the secret number. Here is the output when we try to run secret with input string 1234567890:

The echo $? command prints out the return value of the last executed command in the shell. In this case, the program returned 1, because the secret number we entered is wrong. Recall that by convention, programs return 0 when there are no errors. Our goal going forward is to trick the program to exit with a 0 return value, indicating that we won the game.

Next, let’s try typing in the string 12345678901234567890123456789012345:

Interesting! Now the program crashes with a bus error (another type of memory error), with return code 139. Figure 3 shows what the call stack for main looks like immediately after the call to scanf with this new input.

The input string is so long that it not only overwrote the saved x29 stored at address 0xeed8, but it spilled over into the return address below the stack frame for main. Recall that when a function returns, the program tries to resume execution at the address specified by the return address. In this example, the program tries to resume execution at address 0xffff00353433 after exiting main, which does not appear to exist. So the program crashes with a bus error.

Rerunning the program in GDB (input.txt contains the input string above) reveals this devilry in action:

Notice that our input string blew past the stated limits of the array buf, overwriting all the other values stored on the stack. In other words, our string created a buffer overrun and corrupted the call stack, causing the program to crash. This process is also known as smashing the stack.

Our first example smashed the stack by overwriting the saved x29 register and return address for main with junk, causing the program to crash. An attacker whose goal is to simply crash a program would be satisfied at this point. However, our goal is to trick the guessing game to return 0, indicating that we won the game. We accomplish this by filling the call stack with data more meaningful than junk values. For example, we could overwrite the stack so that the return address is replaced with the address of endGame. Then, when the program attempts to return from main, it will instead execute endGame instead of crashing.

To find out the address of endGame, let’s inspect secret again in GDB:

Observe that endGame starts at address 0x0000aaaaaaaaa8ec. Figure 4 illustrates a sample exploit that forces secret to run the endGame function.

Essentially, there are 32 bytes of junk values followed by the return address. Again, because ARM64 is a little-endian system by default, the bytes in the return address appear to be in reverse order.

The following program illustrates how an attacker could construct the preceding exploit:

The \x before each number indicates that the number is formatted as the hexadecimal representation for a character. After defining ebuff[], the main function simply prints it out, character by character. To get the associated byte string, compile and run this program as follows:

To use exploit as input to scanf, it suffices to run secret with exploit. To get the exploit to work on a Raspberry Pi, type the following set of commands as root (we will explain what is going on following the example):

The sudo su command puts you in root mode on the Raspberry Pi. When prompted for a password, use your password (we assume you have root access to the Raspberry Pi). As soon as the password is entered, the next set of commands are typed in root mode. Note that the command prompt changes when a user is in root mode (it looks something like root@pi#).

The echo command overwrites the contents of the file randomize_va_space with the value 0. Next, the exit command returns the user back to user mode.

Now, type the following command at the prompt:

The program prints out "You are so wrong!" because the string contained in exploit is not the secret number. However, the program also prints out the string "You win!" Recall, though, that our goal is to trick the program to return 0. In a larger system, where the notion of "success" is tracked by an external program, it is often most important what a program returns, not what it prints out.

Checking the return value yields:

Our exploit works! We won the game!

The example we showed changed the control flow of the secret executable, forcing it to return a zero value associated with success. We had to accomplish this in a fairly wonky way due to stack protections that ARM and GCC include to combat this particular type of attack. However, buffer overflow exploits can do real damage on older systems. Some older computer systems also executed bytes from stack memory. If an attacker placed bytes associated with assembly instructions on the call stack, the CPU would interpret the bytes as real instructions, enabling the attacker to force the CPU to execute any arbitrary code of their choosing. Fortunately, there are strategies that modern computer systems employ to make it more difficult for attackers to run buffer overflow exploits:

However, the best line of defense is always the programmer. To prevent buffer overflow attacks on your programs, use C functions with length specifiers whenever possible and add code that performs array bounds checking. It is crucial that any defined arrays match the chosen length specifiers. Table 1 lists some common "bad" C functions that are vulnerable to buffer overflow, and the corresponding "good" function to use (assume that buf is allocated 12 bytes):

The secret2 binary (secret2ARM64.tar.gz) no longer has the buffer overflow vulnerability. The playGame function of this new binary (main2.c) appears below:

Notice that we added a length specifier to all calls of scanf, causing the scanf function to stop reading from input after the first 12 bytes are read. The exploit string no longer breaks the program:

Of course, any reader with basic reverse-engineering skills can still win the guessing game by analyzing the assembly code. If you haven’t tried to beat the program yet with reverse engineering, we encourage you to do so now.