Linux Server Troubleshooting With strace

Interpreting the output

strace is a useful little utility – installed by default on most Linux systems – which allows you to find out what a program is doing under the hood by tracing the system calls it's making. strace is a great basic debugging tool; but it's also fascinating to use even when you're not tracing a problem. It can teach you a lot about how a Linux program works.

A system call is a message from the program to the kernel. User programs on a modern system run in a little sandbox: they're not permitted to interact directly with the computer (so you can't just shove numbers directly into registers to do things, as you could back in the day). Instead, every time the program needs to interact with the rest of the system, it sends a request (a system call) to the kernel. strace tracks these messages. Do remember, then, that if you don't see any strace output for a while, it doesn't necessarily mean that your program is stuck. It might just be doing something within its own sandbox that doesn't require any communication with the rest of the system.

Usage

strace program will do the job, but it outputs everything straight to standard error (i.e. to the screen). As you'll see, there can be quite a lot of output; so it's usually best to use the -o option to set an output file:

strace -o outputfile.txt program

Some editors (e.g. vim) can syntax highlight strace output. This means that different parts of the file, and different parts of each line, are shown in different colours. This is incredibly useful, and I strongly recommend using one of these editors to look at your strace output.

Interpreting the output

Try strace -o strace.out ls -l, then open strace.out in your preferred editor, with syntax highlighting turned on.

Before delving into any of the detail, look at the basic structure of each line. strace records each system call made by the program, and outputs it as a single line. The name of the call is at the start of the line, its arguments are in brackets, and the return value is after the = at the end of the line. The first couple of lines from ls -l should look something like this:

execve("/bin/ls", ["ls", "-l"], [/* 21 vars */])      = 0
brk(0)                                                = 0x619000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b412f2b9000
uname({sys="Linux", node="juliet.example.com", ...})  = 0

The first line shows a system call to execve, whose arguments are:

• the location of the program being executed (/bin/ls)
• an array of the two arguments passed in from the command line (ls and -l)
• a pointer to the 21 variables in the environment passed in to the program.

The return value is 0, which indicates success. This is the basic structure for all system calls.