June 18, 2018

Preventing Buffer Overflow Exploits Using the Linux Distributed Security Module, Part 1 - page 3


  • December 12, 2005
  • By Ibrahim Haddad

Before we explain the mechanisms of the buffer overflow exploits, we will recall some basics about the process organization in memory.

Each process is divided into three regions (Figure 1):

  • Text: The text region contains the executable code of the process. It is marked as read-only to prevent its modification. This part of the code can be stored in the read only memory (similar to flash memory) and executed from there.
  • Data: The data region contains initialized and un-initialized data. Since the executing process can modify data, this part must be stored in the read-write memory. In some cases when the executing image is stored in the flash and then booted from there, this part must be moved to the RAM.
  • Stack: The stack is a read-write area of the memory used to hold local variables, to pass parameters to functions, and to return values from the functions. The stack works following the LIFO model (Last In, First Out), and it is supported by two stack operations:
    • PUSH--to store data at the top of the stack, and
    • POP--to remove data from the stack.

    Both PUSH and POP instructions are in fact a two-step process: first, the data is put on (or remove from) the top of the stack, and next the stack pointer register is updated. The stack grows downward when the stack pointer register is decreased for the PUSH instruction, pointing to lower memory address as the amount of data on the stack increases. Moreover, the stack pointer register is increased for the POP operation, pointing to higher memory address as the amount of data on the stack decreases. The stack grows upward when stack pointer is increased for the PUSH operation and is decreased for the POP operation. On the Intel processors, the stack grows down. Therefore, we will follow this model in our examples.

We will illustrate how the data is organised in the stack using the following example (Listing 1):

void myfunction (char *str, int len)
	char buffer[20];

void main()

Listing 1: example1.c

We will compile example1.c with the �S switch to generate assembly code to examine how the parameters to a function (in this case myfunction) and its local variables are organised on the stack.

$ gcc �S example1.s example1.c
The file example1.s contains the following instructions (comments were added after compilation):
	pushl	%ebp		# save base frame pointer		[3]
	movl	%esp, %ebp
	subl	$40,	%esp	# reserve space for local variables	[4]
	ret			# return from call			[5]
	. . .
	. . .
        pushl	$8		# save argument len			[1]
	pushl	$0		# save argument str			[1]
	call	myfunction	# pass control to function and put	[2]
	. . .			# return address on the stack
	. . .

Listing 2: example1.

As illustrated, the C call to myfunction(str,len) in main() (Listing 1) translates to more than just one line of the assembler code as shown in Listing 2. Based on the above example, we will describe the process of the function call and its interaction with the stack as shown in Figure 2.

We can present the process of passing control to a function in the following steps:

  1. The function arguments are stored on the stack (len,str)
  2. The assembler instruction call myfunction, saves the return address (ret) on the stack (address of the instruction just after the call), and jumps to the function
  3. Inside the function, the stack frame pointer (sfp) is saved on the stack and the new stack frame pointer is assigned
  4. The space for the function local variables is reserved on the stack (buffer)

When the function returns from the call [5], the saved (in [2]) return address (ret) will be used to pass control to the instruction just after the call.

In this example, the exploit occurs when the return address is overwritten by some data. This occurs when there is no boundary checking when moving data to the buffer (calling strcpy in Listing 1). Usually, the moved data is random and the result will be the segmentation violation.

Usually, almost with a 100% probability, the segmentation violation is the result of the buffer overflow because the buffer contains the random data not executable code. The operating system traps this situation and stops the process execution, so there will be no damage to the other parts of the system except the affected process.

However, what happens if the buffer contains an executable program, which will start executing without our knowledge? This situation is not desirable. It is almost impossible that the random data will execute for long time without a crash. Therefore, at this stage, we have to do a little bit hacking to get the expected results.

Most Popular LinuxPlanet Stories

We have made updates to our Privacy Policy to reflect the implementation of the General Data Protection Regulation.