strace - A very powerful troubleshooting tool for all Linux users

May 05, 2006
This article gives a short introduction in using strace to troubleshoot your Linux system.

Many times I have come across seemingly hopeless situations where a program when compiled and installed in GNU/Linux just fails to run. In such situations after I have tried every trick in the book like searching on the net and posting questions to Linux forums, and still failed to resolve the problem, I turn to the last resort which is trace the output of the misbehaving program. Tracing the output of a program throws up a lot of data which is not usually available when the program is run normally. And in many instances, sifting through this volume of data has proved fruitful in pin pointing the cause of error.

For tracing the system calls of a program, we have a very good tool in strace.

When strace is run in conjunction with a program, it outputs all the calls made to the kernel by the program. In many cases, a program may fail because it is unable to open a file or because of insufficient memory. And tracing the output of the program will clearly show the cause of either problem.

Strace usage


$ strace <name of the program>

For example, I can run a trace on ls as follows :

$ strace ls

The above command will print out a lot of data on to the screen which will be hard to read. So you use an option -o to write the output of strace to a file as follows -

$ strace -o strace_ls_output.txt ls

Now you can open the file (strace_ls_output.txt) in a text editor and analyze the output to get the necessary clues.

Analyzing strace output


It is common to find a lot of system function calls in the strace output. The most common of them being open(),write(),read(),close() and so on.

But the function calls are not limited to these four as you will find many others too.

For example, if you look in the strace output of 'ls', you will find the following line:

open("/lib/libselinux.so.1", O_RDONLY)  = 3

This means that some aspect of 'ls' requires the library module 'libselinux.so.1' to be present in the '/lib' folder. And if the library is missing or in a different path, then that aspect of 'ls' which depends on this library will fail to function. The line of code signifies that the opening of the library 'libselinux.so.1' is successful.

A real life example of using strace


Here I will share my experience in using strace to solve a problem I was facing.

I had installed all the multimedia codecs including the 'libdvdcss' which allowed me to play encrypted DVDs in Ubuntu Linux which I use on a daily basis.

But after installing all the necessary codecs, when I tried playing a DVD movie, Totem movie player gave me an error saying that it was unable to play the movie (see the picture below). But I had already installed libdvdcss on my machine, and I was at a loss what to do.

totem error
Totem showing an error that the library libdvdcss cannot be found.

Then I ran strace on totem as follows :

$ strace -o totem-output.txt totem

It wrote the output in the file totem-output.txt. I opened the file and searched for the string "libdvdcss".

Not surprisingly I came across a line of output as highlighted in the listing below.

# Output of strace on totem
open("/etc/ld.so.cache", O_RDONLY)      = 26
fstat64(26, {st_mode=S_IFREG|0644, st_size=58317, ...}) = 0
old_mmap(NULL, 58317, PROT_READ, MAP_PRIVATE, 26, 0) = 0xb645e000
close(26)
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
...
open("/lib/tls/i686/cmov/libdvdcss.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/i686/cmov", {st_mode=S_IFDIR|0755, st_size=1560, ...}) = 0
...
stat64("/lib/i486-linux-gnu", 0xbfab4770) = -1 ENOENT (No such file or directory)
munmap(0xb645e000, 58317)               = 0
open("/usr/lib/xine/plugins/1.1.1/xineplug_inp_mms.so", O_RDONLY) = 26
read(26, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\320\27"..., 512) = 512
fstat64(26, {st_mode=S_IFREG|0644, st_size=40412, ...}) = 0

In the above listing which I have truncated for clarity, the line in bold clearly shows that Totem is trying to find the library in, among other places, the '/lib/tls/i686/cmov/' directory and the return value of -1 shows that it has failed to find it. So I realized that for Totem to correctly play the encrypted DVD, it has to find the 'libdvdcss.so.2' file in the path it is searching.

Then I used the tool find to locate the library and then copy it to the directory '/lib/tls/i686/cmov/'. Once I accomplished this, I tried playing the DVD again in totem and it started playing without a hitch.

totem movie player
Totem shown playing an encrypted DVD movie - libdvdcss error resolved.

Just to make sure, I took another trace of totem and it showed that the error was rectified as shown by the bold line of output below.

# Output of the second strace on totem
open("/etc/ld.so.cache", O_RDONLY)      = 26
fstat64(26, {st_mode=S_IFREG|0644, st_size=58317, ...}) = 0
old_mmap(NULL, 58317, PROT_READ, MAP_PRIVATE, 26, 0) = 0xb644d000
close(26)                               = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
...
open("/lib/tls/i686/cmov/libdvdcss.so.2", O_RDONLY) = 26
...
stat64("/lib/tls/i686/sse2", 0xbffa4020) = -1 ENOENT (No such file or directory)
munmap(0xb645e000, 58317)               = 0
open("/usr/lib/xine/plugins/1.1.1/xineplug_inp_mms.so", O_RDONLY) = 26
read(26, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\360\20"..., 512) = 512
fstat64(26, {st_mode=S_IFREG|0644, st_size=28736, ...}) = 0

Opening the manpage of 'strace', one will find scores of options. For example, if you use the option -t, then strace will prefix each line of the trace with the time of day. One can even specify the system call functions to trace using the -e option. For example, to trace only open() and close() function system calls, one can use the command as follows:

$ strace -o totem-output.txt -e trace=open,close totem

Difference between strace and dtrace


Strace is just a single tool which takes care of a small part which is tracing a single program. Where as Oracle Solaris DTrace toolkit is much more powerful and consists of a collection of scripts which can track, tune and aid the user in troubleshooting ones system in real time. More over, dtrace is a scripting language with close semblance to C/C++ and awk. Put another way, strace tool in GNU/Linux provides only one of the many functions provided by DTrace in Sun Solaris.

That being said, strace plays an important part in aiding the user to troubleshoot ones programs by providing a view of the system calls that the program makes to the Linux kernel.

PS: If you are wondering which movie I was intent on watching, it is "For a Few Dollars More" - an all time classic western starring Clint Eastwood. I really like this movie.

20 comments:

  • Sam Hassell

    Wow, that was really informative. I'll have to give strace a go next time something fubar's for no apparent reason.

    BTW I've been following your site for a while via RSS, you post lots of useful information!

  • Passion

    i've used truss, the strace equivalent in solaris extensively. both are very useful in debugging

    1. if your program links to 3rd party libraries
    2. and you do not have source code to those libraries
    3. if you wanted to fine tune your application. for eg how apache uses truss to fine tune httpd server
    http://httpd.apache.org/docs/2.0/misc/perf-tuning.html

  • Ben

    This looks very useful, I have definitely had a few situations where strace would have come in handy. Thanks for posting!

  • Great article! I'll keep this in mind for the next time I have a problem.

  • Even before Sun added DTrace to Solaris 10, they have long shipped a similar tool called "truss" with their OS.

  • Ravi

    @ Mundlapathi,
    That is really useful information you have provided regarding in which areas strace could be used. Thanks for the same.

  • Vishal Mistry

    Thank you, you've just fixed my Totem problem, and taught me something new about strace (filtering calls) to boot.

  • Please mention 'ltrace' as well.
    It traces calls made to dynamic libraries. Sometimes even more useful.

  • Frostek

    Awesome! It reminds me of a prgaram on the Amiga that used to be great for fixing issues like this. It was so long ago now I don't remember what it was called now, but this is almost identical in the way it works. Fantastic to find out about it. Definitely a big help for some of those cryptic issues!

  • DarkKnight

    I have to agree that strace is really useful.
    I have used it for personal use and at work to solve why a program won't run, and often it is the case that a library isn't in the right place and the program gave no output.

  • Here's a handy trick: Open two terminal windows and create a fifo like this: mkfifo fifo. Run strace -o fifo command in one terminal and tail -f fifo in the other. This way you can see the command by itself in one terminal and the output of strace in the other.

    A refinement would be to do while true; do tail -f fifo; done so you can rapidly run strace multiple times.

  • Good intro to strace, but you really don't need to run a program in strace to find where it searches the dynamic libraries for.

    You can simply do
    ldd <executable>
    and it will show you where it is searching for each dynamic library it refers.

  • wow, thx. great article. i will try it out, seems very usefull..

  • Rohan Dhruva

    Well, isnt everything supposed to "just work" on ubuntu ? Actually, all encrypted dvd's work fine for me -- by just apt-get install libdvdcss and I use xine too. Did you "break" it for the purpose of demonstration, or you are sure you didnt make any mistake ? ;)

  • Thanx. Really useful article. And Yes John I totally agree with you.

    Damijan

  • A couple of other useful notes.

    I think if the program checks whether file exists before opening it (rather than open and check return code) all you will see is the stat call. So that is a useful one to search for in the output file.

    Also useful is the "-f" flag to trace forked processes as well.

    Good article.

    cm

  • Hi,

    I have a question:
    If the command/application that "strace" is tracing forks a new process and I'd like to strace the new process also, I know that strace is doing that automatically, however sometimes if the new process execution time is really short then strace can go into a infinite loop, since it waits to get the id of the new process while this one already finished execution.
    What I would like to do instead is to first pass the id to strace then move forward with the new process execution.
    Any ideas/suggestions on the above.

    Thanks,
    Michael

  • Gaurav

    This is very helpful! Thanks!!

  • 桜木 吉

    THis is really a good article, we support you and alwasy, cheers. I am from China.

  • shadyabhi

    nice tutorial.. strace demystified...
    thanx..