LTTng (Linux Trace Toolkit: Next Generation) is an ultra-low-overhead, industrial-grade tracing framework for Linux. It provides unified, synchronized tracing across both the Linux kernel (lttng-modules) and user-space applications (lttng-ust), writing structured events in binary Common Trace Format (CTF) via per-CPU lockless ring buffers.

---

1. What is LTTng?

Unlike interactive tools like strace (which incur massive context-switch overhead) or ftrace (which primarily targets kernel space), LTTng is engineered for production environments running high-throughput, latency-critical workloads.

It answers complex performance and reliability questions:

• What was the exact sequence of application function calls, library routines, and kernel system calls leading up to a production crash?
• How much time elapsed between a user-space network request submission and the kernel device driver transmitting the packet?
• Which kernel thread preempted a real-time audio or algorithmic trading worker thread, and for how many nanoseconds?
• How can multi-threaded and multi-process interactions across millions of events per second be captured without skewing application timing (probe effect)?

---

2. Architecture & Internal Mechanisms

LTTng achieves sub-microsecond logging overhead through a decoupled daemon and buffer design:

+-------------------------------------------------------------------------+
|                              USER SPACE                                 |
|                                                                         |
|   +-----------------------------------------------------------------+   |
|   |                  Instrumented Application                       |   |
|   |          (C/C++, Java, Python via liblttng-ust)                 |   |
|   +-----------------------------------------------------------------+   |
|                                    |                                    |
|                                    | Writes directly to shared memory   |
|                                    v                                    |
|   +-----------------------------------------------------------------+   |
|   |         Lockless Per-CPU Ring Buffers (Shared Memory)           |   |
|   +-----------------------------------------------------------------+   |
|                                    ^                                    |
|                                    | Consumes via zero-copy             |
|   +--------------------------------+--------------------------------+   |
|   |  lttng-consumerd (User)        |  lttng-sessiond (Control Plane)|   |
|   +--------------------------------+--------------------------------+   |
+------------------------------------|------------------------------------+
                                     |
+------------------------------------|------------------------------------+
|                         LINUX KERNEL SPACE                              |
|                                    v                                    |
|   +-----------------------------------------------------------------+   |
|   |  lttng-modules (Tracepoints, Syscalls, Kprobes, Perf Counters)  |   |
|   +-----------------------------------------------------------------+   |
|                                    |                                    |
|                                    | Lockless ring buffer writes        |
|                                    v                                    |
|   +-----------------------------------------------------------------+   |
|   |              Kernel Per-CPU Ring Buffers (Relay)                |   |
|   +-----------------------------------------------------------------+   |
|                                    ^                                    |
|                                    | Consumes buffers                   |
|   +--------------------------------+                                    |
|   |  lttng-consumerd (Kernel)                                           |
+---+---------------------------------------------------------------------+
                                     |
                                     v
                  +--------------------------------------+
                  |   CTF Binary Trace Files on Disk     |
                  |     (Parsed via Babeltrace 2)        |
                  +--------------------------------------+

Key Architectural Pillars

• Lockless Ring Buffers: Tracing threads write events into per-CPU circular buffers using atomic operations. They never acquire locks or make blocking system calls during an event write.
• Flight-Recorder Mode: Buffers can overwrite old data continuously until a failure triggers a buffer dump, capturing transient bugs without filling disks.
• Separation of Concerns: lttng-sessiond controls session state; lttng-consumerd reads raw buffers and writes CTF files to disk; the application/kernel only writes into RAM.

---

3. Installation

LTTng consists of the control CLI (lttng), the user-space tracer (liblttng-ust), kernel modules (lttng-modules), and the trace viewer (babeltrace2).

Debian / Ubuntu

sudo apt update
sudo apt install lttng-tools lttng-modules-dkms liblttng-ust-dev babeltrace2

RHEL / Rocky / AlmaLinux / CentOS

sudo dnf install epel-release
sudo dnf config-manager --set-enabled crb # (On Rocky/Alma 9)
sudo dnf install lttng-tools lttng-modules-dkms lttng-ust-devel babeltrace2

Arch Linux

sudo pacman -S lttng-tools lttng-ust babeltrace2
# Install kernel modules via AUR: yay -S lttng-modules

Verify daemons and CLI:

lttng --version
babeltrace2 --version

---

4. Standard Tracing Workflow

LTTng uses a session-based operational model:

# 1. Create a tracing session with an output directory
lttng create my-session --output=/tmp/my-trace

# 2. Enable kernel events (e.g., all system calls and scheduler switches)
lttng enable-event --kernel --syscall --all
lttng enable-event --kernel sched_switch,sched_process_fork

# 3. (Optional) Add contextual hardware/OS metrics to every event
lttng add-context --kernel --type=vpid --type=vtid --type=procname

# 4. Start tracing
lttng start

# 5. Run your workload
sleep 3

# 6. Stop tracing and destroy the session (flushes buffers to disk)
lttng stop
lttng destroy

# 7. Read and view the trace output using Babeltrace 2
babeltrace2 /tmp/my-trace

---

5. Anatomy of the Output (babeltrace2)

Because LTTng records trace events in binary Common Trace Format (CTF), outputs are rendered for human analysis using babeltrace2.

Raw Output Example (Kernel Trace)

[14:32:01.102345678] (+0.000001250) prod-node-01 sched_switch: { cpu_id = 2 }, { vpid = 4512, vtid = 4512, procname = "python3" }, { prev_comm = "python3", prev_tid = 4512, prev_prio = 20, prev_state = 1, next_comm = "kworker/u16:1", next_tid = 8912, next_prio = 20 }
[14:32:01.102347100] (+0.000001422) prod-node-01 sys_openat: { cpu_id = 3 }, { vpid = 4512, vtid = 4515, procname = "python3" }, { dfd = -100, filename = "/etc/hosts", flags = 524288, mode = 0 }
[14:32:01.102348500] (+0.000001400) prod-node-01 exit_syscall: { cpu_id = 3 }, { vpid = 4512, vtid = 4515, procname = "python3" }, { ret = 3 }

Raw Output Example (User-Space UST Trace)

[14:32:01.102450120] (+0.000101620) prod-node-01 my_app:database_query_begin: { cpu_id = 1 }, { vpid = 4512, vtid = 4516, procname = "worker" }, { query_id = 9182, table = "users", sql_cmd = "SELECT * FROM users WHERE id = ?" }
[14:32:01.104250320] (+0.001800200) prod-node-01 my_app:database_query_end: { cpu_id = 1 }, { vpid = 4512, vtid = 4516, procname = "worker" }, { query_id = 9182, status = 0, rows_returned = 1 }

---

6. Detailed Breakdown of Output Sections & Fields

Every line output by babeltrace2 consists of four distinct structural blocks:

[TIMESTAMP] (TIME_DELTA) HOSTNAME EVENT_NAME: { STREAM_CONTEXT }, { EVENT_CONTEXT }, { EVENT_PAYLOAD }

---

6.1 Timestamp & System Timing

[14:32:01.102345678] — Wall-Clock Timestamp

• Meaning: The exact wall-clock time down to nanosecond precision ($10^{-9}\text{ s}$) when the event passed through the tracepoint.
• Clock Source: Derived from the system's POSIX monotonic raw clock (CLOCK_MONOTONIC_RAW) or hardware trace clock (x86 TSC: Time Stamp Counter).

(+0.000001250) — Relative Time Delta

• Meaning: The time elapsed (in fractional seconds) since the immediately preceding event in that trace stream.
• Diagnostic Role: Highlights delays between sequential operations without requiring manual subtraction of timestamps.

---

6.2 Host & Identification

prod-node-01 — Hostname

• Meaning: The network hostname of the node where the event occurred.
• Multi-Node Value: When aggregating traces from distributed nodes into a central Babeltrace session, this isolates the exact origin server.

sched_switch / my_app:database_query_begin — Event Name

• Structure: [provider_name]:[event_name]
• Meaning: The identifier for the event.
• In the kernel: Standard subsystem names (e.g., sched_switch, sys_openat, net_dev_xmit).
• In user space: Custom domain providers declared via UST tracepoint macros (e.g., my_app:database_query_begin).

---

6.3 Stream Context: { cpu_id = 2 }

• cpu_id: The zero-indexed physical or logical CPU core on which the thread was actively executing when it generated the event.
• Diagnostic Role: Crucial for detecting CPU migrations, core affinity issues, and cross-core lock contention.

---

6.4 Event Context (Configured via lttng add-context)

LTTng allows operators to append static or dynamic operational metadata to every event passing through a channel:

Context Field	Example	Technical Meaning
vpid	vpid = 4512	Virtual Process ID (process ID inside the caller's PID namespace).
vtid	vtid = 4515	Virtual Thread ID (identifies the specific thread inside a thread pool).
procname	procname = "python3"	Executable command string (comm) associated with the thread.
prio	prio = 20	Dynamic scheduling priority of the task.
ip	ip = 0x7f8a12b0	Current Instruction Pointer address in memory.
perf:thread:instructions	instructions = 45120	Hardware counter reading for instructions executed by this thread.

---

6.5 Event Payload: The Technical Data

The final curly-brace block { ... } contains the actual event parameters recorded by the tracepoint definition.

Common Kernel Event Payloads

sched_switch Payload

{ prev_comm = "python3", prev_tid = 4512, prev_prio = 20, prev_state = 1, next_comm = "kworker/u16:1", next_tid = 8912, next_prio = 20 }

• prev_comm / next_comm: Command names of the task being swapped out and the task taking over the CPU core.
• prev_tid / next_tid: Thread IDs of the outgoing and incoming tasks.
• prev_state: State of the outgoing task at the moment of context switch:
• 0: Task is TASK_RUNNING (voluntarily yielded or preempted by time slice).
• 1: Task is in TASK_INTERRUPTIBLE sleep (waiting on socket, mutex, or event loop).
• 2: Task is in TASK_UNINTERRUPTIBLE sleep (state D, waiting on storage block I/O).

sys_openat Payload

{ dfd = -100, filename = "/etc/hosts", flags = 524288, mode = 0 }

• dfd: Directory file descriptor (-100 translates to AT_FDCWD—Current Working Directory).
• filename: Path parameter requested.
• flags: Bitmask integer representing flags passed (O_RDONLY, O_CLOEXEC).

exit_syscall Payload

{ ret = 3 }

• ret: The return code delivered back to the calling process. Positive values are file descriptors or byte counters; negative values indicate an error (e.g., -2 for -ENOENT).

---

7. Essential lttng CLI Commands Cheat Sheet

Command	Purpose
lttng list -k	List all available static kernel tracepoints, system calls, and probes.
lttng list -u	List all active user-space tracepoint providers running on the system.
lttng create <name> --snapshot	Create a session configured in flight-recorder (snapshot) mode.
lttng snapshot record	Dump the current contents of the lockless ring buffers to disk.
lttng enable-channel -k --subbuf-size=8M --num-subbuf=4 <ch_name>	Tune kernel ring buffer memory allocation (32 MB per CPU).
lttng enable-event -k --tracepoint block_*	Enable kernel tracepoints matching a wildcard pattern.
lttng track --kernel --pid=1234,4512	Restrict kernel tracing strictly to specific Process IDs.
lttng view	Immediately invoke babeltrace2 on the current session's trace files.

---

8. User-Space Tracing: Instrumenting Applications (lttng-ust)

To trace your own C or C++ services, define a tracepoint provider header:

sample_tracepoint.h

#undef TRACEPOINT_PROVIDER
#define TRACEPOINT_PROVIDER my_provider

#undef TRACEPOINT_INCLUDE
#define TRACEPOINT_INCLUDE "./sample_tracepoint.h"

#if !defined(_SAMPLE_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
#define _SAMPLE_TP_H

#include <lttng/tracepoint.h>

TRACEPOINT_EVENT(
    my_provider,
    request_received,
    TP_ARGS(int, req_id, const char*, client_ip),
    TP_FIELDS(
        ctf_integer(int, request_id, req_id)
        ctf_string(ip_address, client_ip)
    )
)

#endif
#include <lttng/tracepoint-event-reset.h>

In Application Code (main.c)

#define TRACEPOINT_DEFINE
#include "sample_tracepoint.h"

void handle_request(int id, const char *ip) {
    // Tracing call has negligible overhead when session is inactive
    tracepoint(my_provider, request_received, id, ip);
    // ... business logic ...
}

Compile with -llttng-ust. When an LTTng session enables my_provider:request_received, events flow automatically into shared memory ring buffers.

---

9. LTTng vs. ftrace vs. perf vs. eBPF

Dimension	LTTng	ftrace	perf	eBPF (bpftrace)
Primary Domain	Kernel + User Space unified	Kernel Space only	Kernel + Hardware PMUs	Kernel + User Space
Target Output	Structured Binary (CTF)	Text buffer / trace.dat	Binary perf.data	Aggregations / Text maps
Runtime Overhead	Lowest ($< 150\,\text{ns}$/event)	Low	Low to Moderate	Very Low
Buffer Model	Per-CPU lockless sub-buffers	Ring buffer	Perf ring buffer	BPF perf/ring buffers
Custom Code Execution	No (predefined tracepoints)	No (triggers/filters only)	No (event counter driven)	Yes (in-kernel sandbox programs)
Flight Recorder Mode	Native (--snapshot)	Native (tracing_on)	Limited	Via custom maps
Production Fit	High-throughput long traces	Quick interactive kernel debugging	CPU profiling & hardware counters	Dynamic ad-hoc observability

---

10. Real-World Troubleshooting Scenarios

Scenario A: Measuring User-to-Kernel Latency (Correlated Tracing)

A distributed web service experiences random 50ms latency spikes on certain requests, but neither application logs nor database queries account for the lost time.

Execution:

1. Enable the application's user-space tracepoints along with kernel context-switching and socket calls:

lttng create correlated-trace --output=/tmp/correlated
lttng enable-event -u my_app:*
lttng enable-event -k sched_switch,net_dev_queue
lttng add-context -u -t vtid -t vpid
lttng add-context -k -t vtid -t vpid
lttng start

1. Trigger the latency spike, then run:

lttng stop && lttng destroy
babeltrace2 /tmp/correlated | grep -C 5 "request_id = 9912"

Diagnosis: Trace shows my_app:request_begin on thread 4515, followed immediately by a sched_switch with prev_state = 1 and next_comm = "kworker". The thread waited 48ms before the next sched_switch returned it to the CPU. The delay was kernel thread scheduling starvation, not bad code logic.

---

Scenario B: Post-Mortem Flight Recording (Zero-Disk Overhead)

An application crashes once every 3 days due to an edge-case concurrency race condition, but recording all system operations continuously fills storage drives.

Execution: Configure LTTng in Snapshot Mode:

lttng create crash-flight-recorder --snapshot
lttng enable-channel -u --subbuf-size=16M --num-subbuf=8 ust-channel
lttng enable-event -u -c ust-channel "my_app:*"
lttng enable-event -k sched_switch
lttng start

Configure your process manager or crash-handler hook (e.g., on SIGSEGV or exception catch) to trigger a snapshot dump:

lttng snapshot record --action=save

Diagnosis: Buffers write only to RAM and overwrite old cycles. The moment a crash occurs, LTTng dumps only the last 128 MB of data to disk, preserving the exact microsecond-by-microsecond sequence that led to the fault.

---

11. Important Interview Questions & Answers

Q: What is CTF (Common Trace Format) and why does LTTng use it instead of JSON or plain text?

Answer: CTF is an open, binary trace format designed for fast recording and compact storage of execution traces. Writing text (like ASCII strings or JSON) directly within an execution path requires CPU cycles for string formatting, integer-to-string conversions, and extensive memory bandwidth. By writing raw binary payloads matching the struct layouts in memory, LTTng minimizes CPU cache pollution and write overhead to under 150 nanoseconds per event.

Q: What happens when the LTTng ring buffer fills up faster than the consumer daemon can write to disk?

Answer: LTTng channels can be configured in two modes:

1. Discard Mode (Default): New events are dropped and a monotonic counter (events_discarded) increments inside the sub-buffer header. babeltrace2 displays a warning marker detailing how many events were lost.
2. Overwrite Mode (Flight-Recorder): New events overwrite the oldest recorded events in the circular ring buffer, maintaining a rolling window of recent execution history without blocking execution threads.

Q: What makes LTTng-UST safer and faster than standard logging frameworks (like log4j or syslog) in production?

Answer: LTTng-UST bypasses standard POSIX file descriptors, filesystem buffers, and synchronous IPC mechanisms. Tracing calls use atomic instructions to claim space in shared memory ring buffers shared between the target process and lttng-consumerd. If tracing is inactive, the tracepoint reduces to a single conditional branch check (if (__builtin_expect(!!tp_is_enabled, 0))), costing less than 5 CPU clock cycles.

---