Linux strace — Complete Output Fields & Meanings Guide

strace intercepts, records, and decodes the kernel system calls and signals invoked by user-space processes. This guide provides a comprehensive breakdown of every possible heading, column, and diagnostic indicator in standard and profiled strace output.

---

1. Standard Output Line Anatomy

When you run strace without summary flags, every system call executed by the application produces a structured trace line:

openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 3

With extended diagnostic flags enabled (strace -tt -T -y -f -p 1234):

14:32:01.102345 [pid  1234] read(3</var/log/syslog>, "Oct 12 15:00:01...", 4096) = 1024 <0.000128>

Detailed Breakdown of Line Fields

Component	Example	Technical Meaning
Timestamp (-tt)	14:32:01.102345	Absolute wall-clock time (Hours:Minutes:Seconds.Microseconds) when the kernel entered the system call.
Process ID Tag (-f)	[pid 1234]	The OS Process ID (PID) or Thread ID (LWP) executing the specific system call. Essential for multi-threaded applications.
Syscall Name	openat, read, futex	The exact name of the Linux kernel system call function requested by user space.
Arguments	(AT_FDCWD, "/etc/hosts", ...)	Parameters passed into the kernel. strace automatically decodes pointers, memory flags, and bitmasks into human-readable symbolic constants (e.g., O_RDONLY).
File Descriptor Resolution (-y)	3</var/log/syslog>	Translates raw integer file descriptors into their actual underlying targets (file paths, sockets, pipes, or event polling handles).
Assignment Operator	=	Visual divider separating the input parameters from the kernel's response.
Return Value	3, 0, 1024, -1	The value returned by the kernel into CPU registers upon completion. Positive integers denote success (bytes read, file handles), while -1 indicates an error.
Error Code (errno)	ENOENT, EAGAIN, EACCES	Symbolic kernel error constant explaining why the call failed (populated only when the return value is -1).
Error Description	(No such file or directory)	Plain-English translation of the errno code.
Execution Duration (-T)	<0.000128>	Time elapsed (in seconds) between system call entry and exit inside the kernel. Crucial for spotting slow disk or network waits.

---

2. Summary Profiling Table (strace -c)

When you run strace with the -c flag, execution is aggregated into a post-run statistical summary table:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 45.12    0.012450         498        25           select
 22.18    0.006120          61       100           read
 15.02    0.004144          41       101           write
  9.45    0.002607          13       198         4 openat
  5.10    0.001407          14       102           close
  3.13    0.000864          12        72           mmap
------ ----------- ----------- --------- --------- ----------------
100.00    0.027592                   598         4 total

Breakdown of Every Summary Heading

2.1 % time

• Meaning: The percentage of total kernel-space execution time consumed by this specific system call relative to all recorded syscalls.
• Troubleshooting Value: Instantly highlights performance bottlenecks. If futex dominates this column (>70%), the application is spending most of its time blocked on internal concurrency locks or thread synchronization.

2.2 seconds

• Meaning: The cumulative wall-clock time (in seconds) spent servicing all invocations of that particular system call during the run.

2.3 usecs/call

• Meaning: The average duration of a single invocation of the system call, measured in microseconds ($1\,\mu\text{s} = 10^{-6}\text{ seconds}$).
• Troubleshooting Value: Calculated as:

$$\text{usecs/call} = \frac{\text{seconds} \times 1,000,000}{\text{calls}}$$

High values indicate blocking calls waiting on hardware or network responses (e.g., slow database connections or disk flushing), while low values indicate fast, memory-bound or cached kernel operations.

2.4 calls

• Meaning: The total number of times the process invoked that specific system call during the trace.
• Troubleshooting Value: Excessively high call counts on operations like read() or write() often reveal unbuffered or inefficient application I/O loops (e.g., reading a file 1 byte at a time).

2.5 errors

• Meaning: The number of times that system call returned a failure condition (-1 with an errno).
• Troubleshooting Value: For instance, 4 openat errors indicates that the application attempted four file path lookups that failed (such as probing fallback configuration paths or checking non-existent plugin directories).

2.6 syscall

• Meaning: The name of the Linux kernel system call function.

---

3. Specialized Error & Signal Output Meanings

When an application crashes, receives an interrupt, or encounters low-level faults, strace prints specialized markers:

--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x0} ---
+++ killed by SIGSEGV (core dumped) +++

3.1 Signal Interception (--- SIG... ---)

• Meaning: Indicates that a Unix signal was delivered to the process while running.
• Common Signals:
• SIGPIPE: The process attempted to write to a closed network socket or pipe whose reader had vanished.
• SIGTERM / SIGINT: Graceful termination requests triggered by operators or process managers.
• SIGSEGV: Segmentation fault resulting from invalid memory dereferencing (e.g., null pointer access at si_addr=0x0).

3.2 Termination Indicator (+++ killed by ... +++)

• Meaning: The final status line indicating that the process or thread was forcefully terminated by a signal or exited cleanly with an exit code (+++ exited with 0 +++).

---

4. Quick Summary Reference

Output Column / Field	Where It Appears	Meaning
[pid ...]	Standard Trace Line	Thread or process ID executing the current syscall.
-1 <errno> (<message>)	Standard Trace Line	System call failed; shows the error code and description.
<seconds>	Standard Trace Line (via -T)	Elapsed execution time of the syscall inside the kernel.
% time	Summary Table (-c)	Proportion of total kernel-time consumed by the syscall.
usecs/call	Summary Table (-c)	Average duration per call in microseconds.
errors	Summary Table (-c)	Total failed execution count for that specific syscall.

---