How to Trace Program Execution Using Linux Strace Command

Last updated: December 5, 2022 | Linuxopsys

More often than not, when programmers and system administrators are trying to debug or troubleshoot issues with their compiled binaries and packages, you will, at some point, encounter errors, program crashes and exit status that will undoubtedly whack your brain out if you can’t find what's causing the problem.

In this tutorial, we learn about strace command in Linux with usage examples.

Linux strace command

Strace is a diagnostics and debugging tool in Linux systems that is used to record and intercept system call names called by a running process and the signals which are received by the said running process. It can become a handy bug-isolation tool as well.

Strace, as a powerful tool,  is mainly used to debug, troubleshoot and analyze how an application, program or binary file interacts with your Linux system. So if you are having trouble with your program or your compiled binary, you can use strace to check where the problem is being detected and which specific system call is being targeted.

The strace command can be used by non-root users as well in its basic usage. This versatility gives you in the user space the option to check non-privileged executables that just need to be examined without affecting system-wide changes.

Syntax

strace [-ACdffhikqqrtttTvVwxxyyzZ] [-I n] [-b execve]

              [-e expr]... [-O overhead] [-S sortby] [-U columns]

              [-a column] [-o file] [-s strsize] [-X format]

              [-P path]... [-p pid]... [--seccomp-bpf] { -p pid | [-DDD]

              [-E var[=val]]... [-u username] command [args] }

Options

Some useful options of strace command:

OptionsDescription
-aAlignment COLUMN for printing syscall results (default 40).
-cCount time, calls, and errors for each syscall and report summary.
-CLike -c but also print regular output.
-dEnable debug output to standard error (stderr).
-DRun tracer process as a detached grandchild, not as parent.
-e expr options:A qualifying expression: option=[!]all or option=[!]val1[,val2]...trace, abbrev, verbose, raw, signal symbol, read, write, fault, inject, kvm, desc (file descriptor).
-fFollow forks.
-ffFollow forks with output into separate files.
-iPrint instruction pointer at time of syscall.
-IInterruptible.
-o <file>Send trace output to FILE instead of standard error  (stderr).
-tPrint timestamp in absolute.
-TPrint time spent in each syscall.
-wSummarize syscall latency (default is system time).

To know more, issue the command strace -h to list down all the possible options.

Installation

In more recent Linux distros, strace is already pre-installed. However, if you find out that it has not yet been installed, you can execute the following commands based on your Linux distros:

Debian / Ubuntu 

sudo apt install strace

Fedora 

sudo dnf install strace

Redhat / CentOS stream

sudo yum install strace

Arch Linux

sudo pacman install strace

How strace command works

Strace relies mostly on ptrace system calls to provide meaningful output in its debugging and troubleshooting sense. ptrace is primarily used to inspect what other programs are performing upon a given set of instructions. In simple terms, strace will call and signal on ptrace and laches itself to the program being analyzed.

How to use the strace command

Demonstrating how to use the strace Linux command, below are some of the examples you can check to evaluate and examine the process calls of certain common Linux commands.

By default, the name of each system call, its return value, parameters and arguments are printed on standard error or can be sent to a file by specifying the -o option.

1. Trace a System Calls

To do a basic tracing of a source code or binary executable’s system call name, let us examine the pwd specified command. Execute strace:

strace pwd

The output will contain the following information:

strace default

As seen above, strace will trace the specified command starting from the location of the pwd command up to the output of pwd which is the printing of the working directory. A value of zero (0) means the code had an exit status without errors.

Common elements are seen in strace trace output:

  • execve: Showing which program is being run along with its arguments and environment variables.
  • close: Involved in opening and closing files.
  • read, write: Used for reading from and writing to file descriptors.
  • stat, fstat: Retrieves information about a file.
  • brk, sbrk: Used for managing the heap memory
  • mmap: map a file or a device into memory.
  • ioctl: input/output control.
  • fork, vfork, clone: Involved in creating new processes
  • exit: Process termination.

One use of strace (Except debugging some problem) is that you can find out which configuration files are read by a program.

For example:

strace php 2>&1 | grep php.ini

2. Trace running application using PID

When a particular program or source code has already started running and the output takes way longer than expected, you can also attach strace to an already running application. Just determine the process id of that process. You can do this by executing the given command:

sudo strace -p <PID>

This command will continuously show system calls made by the process. You can press CTRL+C to stop it.

strace process id

Some processes that are executed by other programs can only be examined by having root user access. As shown above, you can examine the exit code for each successive system calls

3. Follow the forks

You can also follow the forks of a particular process. To do this, issue the given command:

strace -f -p <PID>
strace fork

In the above example, we are following the forks of the htop command with PID 4483. The exit code for every successive system call is also displayed.

Note: Here strace attaches to all threads of the process with the specified PID.

4. Filter Specific System Calls

Filtering specific system calls can also be readily achieved by simply adding the -e variable. You can filter raw, signal read or write system call in accordance with the documentation of strace. Execute strace in the following example:

strace -e read htop
strace filter

In the above example, we’ve filtered the read system call names on htop.

You can combine different values assigned to the -e variable. Another example above shows the read and write system calls from the timedatectl command. 

strace read/write

In any case, the return value should always be zero for no errors

To display files opened by a specific process like SSH, run the following command:

strace -f -e open /usr/sbin/sshd 2>&1 | grep ssh

To trace network-related system calls, run the following command:

strace -e network nc -v -n 127.0.0.1 80

5. Print Time Spent on System Calls

To know the time spent of system calls on a particular process, follow this syntax and type it in the command line

strace -r pwd
strace duration

The first column indicates the time spent of the system calls before proceeding to the next.

6. Display timestamp

You can display the timestamp by following the syntax:

strace -tt <executable>

In the below example, the output display a portion of the timedatectl command, with the first column showing the print timestamp.

strace timestamp

To print the wall clock time of each system call, simply execute the following in the command line:

strace -t pwd
strace wallclock

Lastly, to display the time difference, execute the following in the command line:

strace -T pwd
strace time difference

The time difference will be displayed in the last column of the strace output.

7. Display Instruction Pointer of System Call

To display the instruction pointer of a system call name, indicate in the command line the -i flag. Example:

strace -i w
strace pointer

In this example, the instruction pointer of a system call name is printed.

8. Generate a System Call Report

A useful and straightforward way to summarize strace is just by displaying the system call report. Follow the syntax and run strace:

strace -c <executable>
strace report

Similar to -T option, this is useful for profiling and can help in identifying performance bottlenecks. Example - If you observe frequent open and close calls on the same file, it could indicate inefficient file handling, like opening and closing a file within a loop instead of doing it outside.

Notice Title

Tip: If the total execution time is disproportionately high compared to the time spent on system calls, the issue might lie in the application's code rather than in system-level interactions. This is where ltrace can be particularly useful, as it helps in understanding the library calls.

9. Trace System Calls Based on a Certain Condition

Doing a strace based on a particular system call while suppressing unwanted outputs is also achievable. Include the -q flag while using the -e flag to capture race conditions

strace -q -e memory pwd
strace condition

The return value should always display a zero for no errors.

10. Print Debugging Output of strace

To display and print out the debugging information, affix the -d flag. Execute the following example:

strace -d pwd
strace trace debug

In this example, the debug output is being displayed to standard error or stderr at every stage of the system call.

11. Send Trace to a File

In this last example, you can also preserve the strace outputs to a file for further processing and evaluation. To do this, follow the syntax:

strace -o <filename> <executable>
strace send to such file

In this example, the output has been saved to pwd.log file. Return value and other parameters are recorded in the process.

Conclusion

The Linux system has a ton of useful debugging tools for debugging and troubleshooting codes and compiled binaries. Strace process monitoring tool is just one of many that have demonstrated a lot of cool and engaging features that can make your life as a programmer easier and more worthwhile.

Since system calls and signals are events that happen at the user or Linux kernel interface, strace comes in handy when troubleshooting. You can always visit the official strace manpage to know more about the command.

SHARE

Comments

Please add comments below to provide the author your ideas, appreciation and feedback.

Leave a Reply

Leave a Comment