tcplife

@amitmund September 11, 2026

Linux tcplife — Complete Learning Notes & Output Guide

tcplife is an eBPF/BCC and libbpf performance-tracing tool that tracks the complete lifecycle of TCP sessions from establishment to closure. It logs a single summary line per connection upon termination, reporting the process identity, endpoints, transferred data volume (TX/RX), and total connection lifespan in milliseconds.

1. What is tcplife?

Developed by Brendan Gregg as part of the BCC (BPF Compiler Collection) and ported to modern libbpf-tools, tcplife provides connection-level session accountability.

Standard tools present fundamental operational trade-offs:

  • tcpdump: Captures every individual packet, causing massive CPU, storage, and I/O overhead on high-throughput nodes.
  • **netstat / ss**: Provide point-in-time snapshots, completely missing short-lived connections that establish and terminate between polling intervals.
  • tcptop: Aggregates throughput per second without recording connection lifespan or individual session termination metrics.

tcplife eliminates these drawbacks by hooking into the Linux kernel's TCP state machine. Instead of inspecting millions of packets across a connection, it records a timestamp when a socket enters TCP_ESTABLISHED and emits a single, high-efficiency event when the socket transitions to TCP_CLOSE.

It answers critical operational and application performance questions:

  • How long are client and database TCP sessions lasting (lifespan in milliseconds)?
  • Are backend services suffering from "connection churn" (repeatedly opening and closing TCP connections instead of reusing a persistent connection pool)?
  • Which client or process is opening short-lived connections and transferring zero bytes (health-check probes or hanging handshakes)?
  • What is the total volume of data uploaded (TX_KB) versus downloaded (RX_KB) over a specific session?

2. Installation & Availability

tcplife is available through BCC packages or standalone libbpf-tools. It requires root privileges (sudo or CAP_BPF) and a kernel built with eBPF support (Linux 4.9+ minimum, 5.4+ recommended).

Debian / Ubuntu

sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)

(Executables reside in /usr/sbin/ with a -bpfcc suffix, e.g., tcplife-bpfcc).

RHEL / Rocky / AlmaLinux / CentOS

sudo dnf install bcc-tools kernel-devel-$(uname -r)

(Executables reside in /usr/share/bcc/tools/tcplife).

Arch Linux

sudo pacman -S bcc-tools
# Or for the C/CO-RE version:
sudo pacman -S libbpf-tools

Verify the binary:

sudo tcplife -h 2>/dev/null || sudo tcplife-bpfcc -h

3. Basic Syntax & Command Flags

sudo tcplife [options]

Essential Command Flags

Flag Description Practical Example
(no flag) Trace all IPv4 and IPv6 TCP sessions system-wide. sudo tcplife
-T Include a human-readable 24-hour timestamp (HH:MM:SS) column. sudo tcplife -T
-t Include a monotonic timestamp (seconds from tool start). sudo tcplife -t
-p <PID> Trace TCP connections owned strictly by a specific Process ID. sudo tcplife -p 5410
-L <port> Filter connections matching a specific local port (comma-separated). sudo tcplife -L 80,443
-D <port> Filter connections matching a specific remote/destination port. sudo tcplife -D 5432
-s Trace only IPv4 connections (exclude IPv6). sudo tcplife -4 or -s
-w Trace only IPv6 connections. sudo tcplife -6 or -w

4. Anatomy of Default Output (tcplife -T)

Running tcplife with timestamps enabled displays session completions in real time:

sudo tcplife -T

Raw Output Example

TIME     PID    COMM             LADDR           LPORT  RADDR           RPORT   TX_KB   RX_KB       MS
07:15:01 5410   postgres         192.168.1.50    5432   10.0.0.12       48912     142    1850  1450.25
07:15:02 8912   curl             192.168.1.50    52104  140.82.121.4    443         1      15    82.10
07:15:05 1120   kubelet          127.0.0.1       10248  127.0.0.1       38910       0       0     0.85
07:15:08 4515   nginx            192.168.1.50    80     203.0.113.15    59102      45       2    12.40
07:15:10 8912   python3          192.168.1.50    41022  10.0.0.8        9092        8       0 60012.80

5. Breakdown of Every Output Heading & Field

+----------+------+----------+---------------+-------+---------------+-------+-------+-------+---------+
| TIME     | PID  | COMM     | LADDR         | LPORT | RADDR         | RPORT | TX_KB | RX_KB | MS      |
+----------+------+----------+---------------+-------+---------------+-------+-------+-------+---------+
| 07:15:01 | 5410 | postgres | 192.168.1.50  | 5432  | 10.0.0.12     | 48912 |   142 |  1850 | 1450.25 |
+----------+------+----------+---------------+-------+---------------+-------+-------+-------+---------+

5.1 TIME

  • Format: HH:MM:SS (wall-clock time).
  • Meaning: The exact moment the TCP connection finished closing and was torn down by the kernel.

5.2 PID

  • Format: Numeric integer (e.g., 5410, 8912).
  • Meaning: The Process ID associated with the task that held the socket file descriptor.
  • Kernel Context: If the socket was closed asynchronously by the kernel network stack during teardown or orphan cleanup, this may attribute to the process that initiated the close() system call.

5.3 COMM

  • Format: String (e.g., postgres, curl, nginx, kubelet).
  • Meaning: The short executable command name of the process (first 16 characters of task_struct->comm).

5.4 LADDR & LPORT (Local Endpoint)

  • LADDR: The local IP address (IPv4 or IPv6) bound to the socket.
  • LPORT: The local TCP port number.
  • Server connections show their listening port (e.g., 5432, 80).
  • Outbound client connections show their assigned ephemeral port (e.g., 52104).

5.5 RADDR & RPORT (Remote Endpoint)

  • RADDR: The remote peer's IP address.
  • RPORT: The remote peer's TCP port number.

5.6 TX_KB (Transmitted Kilobytes)

  • Format: Integer Kilobytes.
  • Meaning: Total volume of data transmitted over the wire by the local host across the entire lifetime of this session.
  • Calculation: Extracted from the kernel's internal TCP socket accounting structures (tp->bytes_acked / 1024).

5.7 RX_KB (Received Kilobytes)

  • Format: Integer Kilobytes.
  • Meaning: Total volume of payload data received from the remote peer across the entire lifetime of this session.
  • Calculation: Extracted from socket receive counters (tp->bytes_received / 1024).

5.8 MS (Connection Duration in Milliseconds)

  • Format: Floating-point decimal (e.g., 1450.25, 0.85, 60012.80).
  • Meaning: The total lifespan of the connection in milliseconds:

$$\text{Lifespan} = \text{Timestamp}_{\text{TCP_CLOSE}} - \text{Timestamp}_{\text{TCP_ESTABLISHED}}$$

  • Diagnostic Value: Instantly reveals session behavior:
  • < 5 ms: Rapid ephemeral connection (health check, micro-query, or un-pooled API call).
  • 1000 – 5000 ms: Short-lived transactional session.
  • > 60000 ms: Long-lived stateful connection (keep-alive, database pool, streaming socket).

6. How tcplife Works Internally

tcplife avoids per-packet inspection by attaching eBPF programs directly to the kernel's internal TCP state engine:

+-------------------------------------------------------------------------+
|                              KERNEL SPACE                               |
|                                                                         |
|   1. TCP State Transition (kprobe:tcp_set_state):                       |
|      - Fires on every state transition (SYN_SENT -> ESTABLISHED, etc.)  |
|                                                                         |
|   2. When state == TCP_ESTABLISHED:                                     |
|      - Stores struct sock *sk as Key in BPF Hash Map.                   |
|      - Value = { start_time: bpf_ktime_get_ns(), pid, comm }            |
|                                                                         |
|   3. Connection Active (Data Transfer Phase):                           |
|      - ZERO eBPF execution during packet transit.                       |
|      - Hardware offloads (TSO, GRO) run at full wire speed.             |
|                                                                         |
|   4. When state == TCP_CLOSE:                                           |
|      - Looks up socket pointer in BPF Hash Map.                         |
|      - Calculates: delta_ms = (current_time - start_time) / 1,000,000    |
|      - Reads struct tcp_sock:                                           |
|          tx_bytes = tp->bytes_acked;                                    |
|          rx_bytes = tp->bytes_received;                                 |
|      - Emits event via BPF Perf/Ring Buffer to user space.              |
|      - Deletes map entry (frees kernel memory).                         |
+-------------------------------------------------------------------------+
                                    |
                                    v (BPF Ring Buffer)
+-------------------------------------------------------------------------+
|                              USER SPACE                                 |
|   Python / C CLI: Formats IP addresses, converts bytes to KB, prints    |
|                   single-line summary to stdout.                        |
+-------------------------------------------------------------------------+

7. tcplife vs. Related TCP Tracing Tools

+-------------------------------------------------------------------------+
|  Tool         | Instrumentation Point      | Primary Output Style       |
+---------------+----------------------------+----------------------------+
|  tcpdump      | AF_PACKET / libpcap        | Every raw packet on wire   |
|  tcpconnect   | sys_enter_connect (Active) | Outbound connection start  |
|  tcpaccept    | sys_enter_accept (Passive) | Inbound connection start   |
|  tcptop       | TCP send/receive calls     | Throughput top-like table  |
|  tcplife      | tcp_set_state()            | Full session summary line  |
+-------------------------------------------------------------------------+
Dimension tcplife tcpdump tcptop tcpconnect / tcpaccept
Overhead Extremely Low Very High ($>10\times$ CPU) Low Very Low
Output Frequency 1 line per closed session 1 line per packet Refreshed every $N$ sec 1 line per new session
Session Duration Yes (MS column) Only via manual stream math No No
Transferred Volume Yes (TX_KB, RX_KB) Yes (computes payloads) Yes (KB/s throughput) No
Identifies PID/COMM Yes No Yes Yes
Safe for 40G/100G Yes No (packet drops/lag) Yes Yes

8. Real-World Troubleshooting Scenarios

Scenario A: Diagnosing Missing Connection Pools (Database Churn)

A backend microservice shows high CPU utilization and latency spikes on an upstream PostgreSQL database.

Filter tcplife to destination port 5432:

sudo tcplife -T -D 5432

Output:

TIME     PID    COMM      LADDR         LPORT  RADDR       RPORT  TX_KB  RX_KB  MS
10:00:01 14201  python3   10.0.0.5      41002  10.0.0.20   5432       2      8  3.40
10:00:01 14201  python3   10.0.0.5      41004  10.0.0.20   5432       2      8  3.25
10:00:01 14201  python3   10.0.0.5      41006  10.0.0.20   5432       2      8  3.12
10:00:01 14201  python3   10.0.0.5      41008  10.0.0.20   5432       2      8  3.50

Diagnosis: python3 is establishing hundreds of connections per second, sending tiny payloads ($2\text{ KB}$), and tearing them down after only $3\text{ ms}$. Root Cause: The application is creating a new TCP connection and performing a full TLS/auth handshake for every individual query rather than using a persistent connection pool (such as PgBouncer or an internal pool).

Scenario B: Detecting Asymmetric Transfers & Hanging Keep-Alives

An edge API gateway reports thread exhaustion.

Run tcplife targeting port 80:

sudo tcplife -T -L 80

Output:

TIME     PID    COMM   LADDR         LPORT  RADDR          RPORT  TX_KB  RX_KB  MS
10:04:12 4515   nginx  192.168.1.50  80     198.51.100.22  51200      0      0  60002.15
10:04:12 4515   nginx  192.168.1.50  80     198.51.100.84  51202      0      0  60001.90

Diagnosis: Connections are remaining open for exactly $60,000\text{ ms}$ (60 seconds) with **0 TX_KB and 0 RX_KB**. Root Cause: External clients are initiating TCP handshakes and then going silent (Slowloris attack or dead intermediate mobile links), occupying worker slots until Nginx's client_header_timeout or keepalive_timeout forcefully triggers TCP_CLOSE.

9. Important Interview Questions & Answers

Q: Why does tcplife hook tcp_set_state() rather than monitoring system calls like connect() and close()?

Answer: Monitoring system calls (sys_enter_connect, sys_enter_close) captures only user-space requests, not the actual network state. A process can call connect(), but the connection might fail due to a timeout or TCP RST, never reaching an established state. Conversely, a connection can close without a local close() call—such as when the remote peer initiates teardown with a FIN, when an unrecoverable network drop triggers a keep-alive timeout, or when the connection resets via RST. Hooking tcp_set_state() inside the kernel tracks the ground truth of the socket lifecycle regardless of how or why the state transition was triggered.

Q: How does tcplife accurately calculate TX_KB and RX_KB without tracking every read and write system call?

Answer: The Linux kernel's internal TCP socket data structure (struct tcp_sock) already maintains native cumulative counters:

  • bytes_acked: Tracks the total number of payload bytes sent and acknowledged by the peer.
  • bytes_received: Tracks the total number of payload bytes delivered to the socket.

When the socket transitions to TCP_CLOSE, tcplife reads these pre-existing kernel counters directly from the socket struct. This offloads all accounting work to the kernel's normal TCP operation, requiring zero per-packet or per-syscall tracking.

Q: Does tcplife capture failed connection attempts?

Answer: By design, tcplife tracks connections that successfully reach the TCP_ESTABLISHED state. If an outbound connection is rejected immediately with an ICMP unreachable or a TCP RST, or if a client abandons a handshake during SYN_SENT, it never reaches TCP_ESTABLISHED and is not logged by default. To capture failed connection attempts, tools like tcpconnect or custom bpftrace scripts monitoring tcp_v4_connect and tcp_finish_connect are used instead.

0 Likes
2 Views
0 Comments

Filters

No filters available for this view.

Reset All