strace is a Linux diagnostic tool that intercepts and records the system calls a process makes and the signals it receives. In Linux monitoring, it answers a question that resource tools such as top or vmstat cannot: *what exactly is this program asking the kernel to do*, including which files it opens, which network addresses it contacts and where it waits.
Because every interaction between a user-space program and the outside world passes through system calls, strace can diagnose failures without source code, debug symbols or recompilation. It is often the fastest way to explain an error message such as a vague permission failure or a program that silently hangs.
How strace intercepts system calls
strace relies on the kernel’s ptrace interface, the same mechanism debuggers use. It either starts a new program as a traced child or attaches to a running process with -p PID. The kernel then stops the tracee at every system call entry and exit, and strace reads the call number, arguments and return value from the process registers and memory before letting it continue.
Each output line shows the call name, decoded arguments and the return value, followed by a symbolic error such as ENOENT (No such file or directory) when the call fails. strace knows the structure of hundreds of calls, so flags, structures and socket addresses appear in readable form rather than raw numbers. Newer versions can also use seccomp-bpf filtering with --seccomp-bpf so that only the selected calls stop the process.
The cost of this design is overhead. Every traced call involves at least two context switches into strace, so a program that makes many small system calls can run several times slower, or far more, while traced. strace is therefore a diagnostic instrument, not a continuous production monitor.
Essential strace options
A handful of options turns strace from an overwhelming stream of output into a focused tool. The most valuable habit is filtering by call class with -e trace= so that only relevant calls are printed.
| Option | Effect | Typical use |
|---|---|---|
| -p PID | Attach to a running process | Inspect a hung daemon |
| -f | Follow forked children and threads | Trace shell scripts and servers |
| -e trace=file | Show only file-related calls | Find missing configuration files |
| -e trace=network | Show only socket calls | See which hosts a program contacts |
| -c | Print a summary table of calls | Find the dominant system calls |
| -T | Show time spent in each call | Locate slow reads or waits |
| -tt | Add microsecond timestamps | Correlate with logs |
| -s 256 | Print longer string arguments | See full paths and payloads |
| -o file | Write output to a file | Keep the program’s own output readable |
Other filters include trace=process for fork, exec and exit, trace=%memory for mapping calls and -e status=failed in newer releases, which prints only calls that returned an error. The -y option resolves file descriptor numbers to paths, which makes long traces much easier to read.
strace compared with other monitoring tools
strace looks at one process in depth, while most monitoring tools look at the whole system in breadth. Choosing the right tool depends on whether the question is about resource usage, open files or the exact sequence of kernel requests.
| Tool | What it shows | Overhead | Best for |
|---|---|---|---|
| strace | System calls and signals of a process | High | Explaining a specific failure |
| ltrace | Library function calls | High | Calls into shared libraries |
| lsof | Currently open files and sockets | Low | Snapshot of resources in use |
| top and htop | CPU and memory per process | Low | Spotting busy processes |
| perf trace | System calls via kernel events | Lower than strace | Tracing busier workloads |
| bpftrace | Custom eBPF probes | Low | Production tracing at scale |
For production systems under load, eBPF-based tools are usually preferable because they collect data in the kernel without stopping the process for every call. strace remains the most universally available and easiest to read option, and it is often installed where advanced tracing tools are not.
Reading strace output to diagnose problems
Most strace investigations look for a small number of recognizable patterns. The trick is to search the output for failed calls and for calls with long durations, then read the lines just before them for context.
| Pattern in output | Likely meaning | Next step |
|---|---|---|
| openat … ENOENT | A file or library is missing | Check paths and search order |
| EACCES or EPERM | Permission or policy denial | Check ownership, mode and SELinux |
| connect … ECONNREFUSED | Nothing listens on the target port | Verify the service and address |
| Long futex or poll wait | Waiting for a lock or an event | Examine other threads or peers |
| Repeated read returning 0 | Busy loop on end of file | Inspect the program logic |
Note that many ENOENT results are normal noise. Programs routinely probe several locations for libraries and configuration files, so the meaningful error is usually the last failed attempt before the program prints its message or exits.
Pitfalls and safe use
- Attaching to production services slows them significantly; trace briefly and with tight filters.
- ptrace restrictions such as the Yama
ptrace_scopesetting can block attaching to processes that are not children, even for the same user. - Containers may lack the CAP_SYS_PTRACE capability, so tracing must be done from the host or with added privileges.
- Sensitive data such as passwords and tokens can appear in read and write buffers, so trace files must be handled carefully.
- Only one tracer can attach to a process, so strace fails when a debugger is already attached.
strace also cannot see work that never reaches the kernel. A program spinning in a pure computation loop shows no system calls at all, and in that case a profiler such as perf is the correct tool.
Put simply
A program cannot touch files, the network or the screen by itself. It must ask the operating system for each of these things. strace writes down every one of those requests and the answer the program got back.
It works like reading the call log of a busy office. When something goes wrong, the log shows who was called, what was asked and which call failed. That is often enough to find the cause, even without knowing how the program was written.
In practice
A system administrator upgrades a web application on a Linux server, and the service now exits at startup with only the vague message “configuration error”. The logs say nothing more. The administrator runs strace -f -e trace=file -o /tmp/app.trace followed by the start command and searches the trace for failed calls.
Near the end, the trace shows an openat call on /etc/app/conf.d/tls.conf returning EACCES, just before the program exits. The new package had installed that file with mode 600 owned by root, while the service runs as an unprivileged user. Correcting the group ownership and mode solves the problem in minutes. The outcome is a precise diagnosis without reading any source code.