Linux ltrace — Complete Learning Notes & Output Guide

ltrace is a dynamic debugging tool used to intercept, record, and display dynamic library calls (such as functions in libc, libssl, or libcurl) executed by user-space processes, as well as the signals received.

---

1. What is ltrace?

ltrace stands for:

Library Call Tracer

Unlike strace (which intercepts kernel system calls like open(), read(), write(), and socket()), ltrace intercepts calls to shared dynamic libraries (.so files) that go through the Procedure Linkage Table (PLT).

It answers diagnostic questions such as:

• Which external C library functions is an application calling?
• What arguments are passed to functions like malloc(), strcpy(), getenv(), or connect()?
• What do those library functions return?
• Which specific shared library function is consuming the most execution time?
• Is an application failing inside user-space library logic before a system call is ever issued?

---

2. Installation

ltrace is available across standard distribution package repositories:

Debian / Ubuntu

sudo apt update
sudo apt install ltrace

RHEL / Rocky / AlmaLinux / CentOS

sudo dnf install ltrace

Arch Linux

sudo pacman -S ltrace

---

3. Basic Syntax

ltrace [options] command [arguments]

Or attach to an active process:

ltrace [options] -p <PID>

Example (tracing the ls command):

ltrace ls

---

4. Anatomy of a Standard ltrace Output Line

When running ltrace without summary flags, output lines follow a strict format:

strlen("hello world")                            = 11
getenv("PATH")                                   = "/usr/local/bin:/usr/bin"
malloc(1024)                                     = 0x55d78a1b22a0
fopen("/etc/hosts", "r")                         = 0x55d78a1b24d0
fgets("127.0.0.1 localhost\n", 1024, 0x55d78...) = 0x7ffd9b8a
fclose(0x55d78a1b24d0)                           = 0

With extended options enabled (ltrace -tt -T -p 1234):

14:32:01.102345 [pid 1234] getenv("DEBUG") = NULL <0.000042>

Element-by-Element Breakdown

Component	Example	Meaning
Timestamp	14:32:01.102345	Wall-clock time when the call occurred (via -tt).
PID Tag	[pid 1234]	The process ID making the call (via -f).
Function Symbol	getenv	The dynamic library function name resolved from the PLT.
Arguments	("DEBUG")	Parameters passed into the function by the caller.
Assignment (=)	=	Separates the function signature from its return value.
Return Value	NULL or 0x55d7...	The value returned to the caller by the library function.
Call Duration	<0.000042>	Time spent inside the library call in seconds (via -T).

---

5. Breakdown of Summary Mode: ltrace -c

Running ltrace -c aggregates execution metrics into a profiling table upon program exit.

ltrace -c ls /tmp

Example Output:

% time     seconds  usecs/call     calls      errors function
------ ----------- ----------- --------- ----------- --------------------
 42.15    0.002450         490         5             malloc
 18.22    0.001059         211         5             free
 14.10    0.000819          18        45             strlen
 10.05    0.000584          58        10             memcpy
  8.20    0.000476         476         1             fopen
  7.28    0.000423         423         1           1 getenv
------ ----------- ----------- --------- ----------- --------------------
100.00    0.005811                   67           1 total

Explanation of Every Column Heading

5.1 % time

• Meaning: The percentage of total library execution time spent inside this specific function relative to all tracked library calls.
• Troubleshooting Significance: Highlights immediate performance bottlenecks. In the example above, malloc accounts for over 42% of library runtime.

5.2 seconds

• Meaning: The cumulative real (wall-clock) time, expressed in seconds, spent executing all instances of this function.

5.3 usecs/call

• Meaning: The average duration of each individual call to this function, measured in microseconds ($1\,\mu\text{s} = 10^{-6}\text{ seconds}$).
• Troubleshooting Significance: $\text{usecs/call} = \frac{\text{seconds} \times 1,000,000}{\text{calls}}$. A high value indicates slow, heavy operations (e.g., cryptographic key generation, disk-backed I/O functions), whereas a low value indicates lightweight operations (e.g., string lookups).

5.4 calls

• Meaning: The total number of times the target process invoked this library function during the tracing window.

5.5 errors

• Meaning: The number of calls that returned an error status (such as NULL, -1, or a non-zero exit condition depending on function definition).
• Troubleshooting Significance: Notice 1 getenv above. One call to getenv failed or returned NULL, pointing directly to missing environment variables.

5.6 function

• Meaning: The exported symbol name of the shared library routine that was intercepted.

---

6. Essential ltrace Command Options

Flag	Purpose	Example
-c	Count and profile library calls (summary table).	ltrace -c ./app
-T	Show time spent inside each individual call.	ltrace -T ./app
-tt	Print microsecond wall-clock timestamps on each line.	ltrace -tt ./app
-r	Print relative timestamps between calls.	ltrace -r ./app
-p <PID>	Attach to a currently running process.	sudo ltrace -p 3482
-f	Follow child processes created via fork() or clone().	ltrace -f ./app
-S	Trace both system calls and library calls together.	ltrace -S ./app
-e <expr>	Filter for specific functions (supports wildcards).	ltrace -e malloc+free ./app
-l <lib>	Trace only calls made into a specific shared library.	ltrace -l /lib/x86_64-linux-gnu/libm.so.6 ./app
-s <size>	Increase maximum printed string length (default is 32).	ltrace -s 128 ./app
-o <file>	Write raw trace output directly to a file.	ltrace -o trace.log ./app

---

7. ltrace vs. strace — Key Differences

Understanding the operational boundary between dynamic libraries and the Linux kernel:

+-------------------------------------------------------+
|                     USER SPACE                        |
|                                                       |
|   +-----------------------------------------------+   |
|   |         Application Code (Binary)             |   |
|   +-----------------------------------------------+   |
|                          |                            |
|                          | Library Calls (e.g., puts) |
|                          v                            |
|       =========================================       |
|       ===> [ ltrace intercepts here ] <=======       |
|       =========================================       |
|                          |                            |
|   +-----------------------------------------------+   |
|   |         Dynamic Libraries (libc, etc.)        |   |
|   +-----------------------------------------------+   |
|                          |                            |
|                          | System Calls (e.g., write) |
|                          v                            |
|       =========================================       |
|       ===> [ strace intercepts here ] <=======       |
|       =========================================       |
+--------------------------|----------------------------+
                           v
+-------------------------------------------------------+
|                    KERNEL SPACE                       |
|   Syscall Dispatcher -> VFS -> Block/Network Drivers  |
+-------------------------------------------------------+

Dimension	ltrace	strace
Target	Dynamic shared libraries (.so).	Linux kernel system calls.
Mechanism	Breakpoint insertion on PLT entries via ptrace.	Kernel syscall interception via ptrace(PTRACE_SYSCALL).
Static Binaries	Fails / Cannot trace (no PLT present).	Works normally.
Scope	User-space function calls.	Boundary transitions between user and kernel space.
Overhead	High (triggers traps on every PLT jump).	Medium-High (triggers traps on every syscall entry/exit).

---

8. How ltrace Works Internally

1. Binary Inspection: ltrace reads the ELF header of the target executable and locates the .plt (Procedure Linkage Table) and .got.plt (Global Offset Table).
2. Breakpoint Injection: Using the ptrace(2) system call (PTRACE_POKETEXT), ltrace replaces the first machine instruction of each PLT target entry with a software breakpoint (int3 on x86/x86_64).
3. Execution Trap: When the program calls a dynamic function, execution hits the breakpoint. The Linux kernel sends a SIGTRAP to the process, pausing it.
4. Inspection & Emulation: ltrace catches the trap, reads registers and arguments off the stack/registers, prints the function name and inputs, restores the original instruction, and allows execution to proceed until the function returns.

---

9. Practical Diagnostic Scenarios

Scenario A: Inspecting Unencrypted Network Strings

An application uses libssl or standard networking libraries, making packet inspection with tcpdump unreadable due to TLS encryption:

ltrace -s 256 -e SSL_write+SSL_read ./secure_client

Displays raw, pre-encrypted payloads handed to OpenSSL functions directly in user space.

Scenario B: Debugging Configuration and Environment Failures

A binary silently exits without logging why a configuration failed:

ltrace -e getenv+fopen ./app

Example output:

getenv("CONFIG_PATH")                            = NULL
fopen("/etc/defaults/app.conf", "r")             = 0

Immediately reveals that CONFIG_PATH was unset and the fallback configuration file failed to open.

Scenario C: Finding Memory Allocation Bottlenecks

An application is experiencing latency spikes:

ltrace -c ./app

If malloc or realloc shows millions of calls with high cumulative % time, the application suffers from memory allocation churn and could benefit from an alternative allocator (like jemalloc or mimalloc) or buffer pooling.

---

10. Key Limitations & Gotchas

• Statically Linked Binaries: If an application is compiled statically (gcc -static), ltrace outputs nothing because there are no dynamic shared object tables or PLT jumps.
• Compiler Inlining / Static Internal Functions: ltrace cannot trace functions compiled within the binary itself or inlined by compiler optimizations (-O2, -O3). It only sees externally linked shared library symbols.
• Modern Toolchains (-fno-plt): Modern compilers using -fno-plt resolve symbols directly through the Global Offset Table without jumping through the PLT, preventing standard ltrace from intercepting them. (In these cases, use eBPF uprobes or perf).

---

11. Interview Questions & Answers

Q: Why does ltrace show nothing when running against a Go binary?

Answer: The standard Go toolchain compiles binaries statically by default and includes its own runtime without depending on dynamic shared C libraries (libc.so). Since there is no Procedure Linkage Table (PLT), ltrace has no linkage hooks to trap.

Q: What does the errors column in ltrace -c specifically measure?

Answer: It counts the number of library calls that returned a predefined failure signature according to ltrace's internal prototype configuration dictionary (typically NULL for pointer-returning functions or -1 for standard integer-returning library calls).

Q: How can you trace a combined timeline of both library calls and system calls?

Answer: Run ltrace -S <command>. The -S flag tells ltrace to intercept and print kernel system calls (like read, write, futex) interspersed alongside standard dynamic library calls (like malloc, strcmp), exposing how higher-level library functions trigger underlying kernel operations.

---