June 21, 2018

Preventing Buffer Overflow Exploits Using the Linux Distributed Security Module, Part 2 - page 3

Existing Solutions to Buffer Overflow Exploits

  • January 12, 2006
  • By Ibrahim Haddad

In this section, we demonstrate how the DSM module prevents buffer overflow exploits.

Step 1: First, we extend the security of the system by loading the DSM module as root:

$/sbin/insmod lsm.o	# for historical reason, the name of the module is lsm

Step 2: Next, we update and load the security policy (we explain the directives later on in the text):

$vi policy_file

Add the following line (if not present):

      1 2 128 0
$./UpdatePolicy policy_file

As of DSI 0.3 this has been much improved and instead of using a plain text file to express the security policy, it is now done with XML. Therefore, if you experiment with DSI 0.3 or 0.4 then you will express the policy in XML.

Step 3: Then, we change the security ID of the vulnerable program to 1:

$SetSID vulnerable 1

Step 4: Now, we are ready to repeat the same example of the buffer overflow exploit:

$ ./vulnerable $RET
$Error ./vulnerable: not found

Did we gain root privileges? Let us check:


No! We are still a normal user.

Step 5: Exit from the shell spanned by the buffer overflow exploit:


When executing the second ./vulnerable $RET an error is returned (the error code must be changed to reflect what really happened here) and the system will not allow to start a new shell with root privileges.

To be sure that our module is the source of this error, you can unload the DSM module and run the vulnerable program (Step 4) once again. After the DSM module is unloaded, there should be no error message and the shell will be started with root privileges same as before.

Here is the explanation on what really happened.

In Step 2, we updated the policy by adding the line (1 2 128 0). There are four fields: source security ID (1), target security ID (2), class (128) and permission (0). This means that the subjects with the security ID = 1 cannot perform the operation in class = 128 on the objects with the security ID = 2. We implement the class 128 called DSI_CLASS_TRANSITION in this manner and it covers the ELF executable checking at the load time.

In Step 3, we assigned the security id 1 to the vulnerable program. By default, all the subjects start with the security id 2 when it is not explicitly stated otherwise in the security policy file. The third parameter in the policy record is 128. This value describes the operation (look at the source code). In our case, it means the operation of spanning a process. Therefore, when the vulnerable program with the security id 1 executes the code stored in the vulnerable buffer, at one point the system call (0x80) is performed. The system call passes the execution to the kernel and later to the security hook embedded in the kernel. In the security module (DSM) the security id of the subject (vulnerable) is validated. Based on the policy the operation of spanning will not be allowed.

Now to secure the kernel, we have to find all the vulnerable processes and assign them the security ids that do not allow spanning of other processes. Please note that there is no need to have the source code of the vulnerable programs. Based on the process creation tests, which were presented in a previous article that was published in the Linux Journal, we can safely assume that the overhead of the added operations is minimal and will stay in the range between 1% and 2%.

Most Popular LinuxPlanet Stories

We have made updates to our Privacy Policy to reflect the implementation of the General Data Protection Regulation.