tcpretrans
Linux tcpretrans — Complete Learning Notes & Output Guide
tcpretransis an eBPF/BCC and libbpf performance-tracing tool that intercepts TCP retransmission events inside the Linux kernel. It logs timestamps, process identifiers, and endpoint pairs whenever the kernel resends a TCP segment due to unacknowledged timeouts or packet drops.
1. What is tcpretrans?
Developed by Brendan Gregg as part of the BCC (BPF Compiler Collection) and ported to modern libbpf-tools, tcpretrans provides targeted visibility into network reliability without the overhead of full packet capture tools like tcpdump.
It answers critical network diagnostic questions:
- Which applications or processes are experiencing TCP retransmissions right now?
- Is packet loss isolated to specific remote peers (
RADDR), or is it system-wide? - Are retransmissions driven by local network congestion, faulty switches, or dropped packets across the WAN?
- Are segments being retransmitted during the initial 3-way handshake (
SYN) or during active data transfer (ESTABLISHED)?
2. Installation & Availability
tcpretrans requires root privileges (sudo or CAP_BPF) and a Linux kernel with eBPF enabled (Linux 4.9+ minimum, 5.4+ recommended).
Debian / Ubuntu
sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)
(Executables reside in /usr/sbin/ with a -bpfcc suffix, e.g., tcpretrans-bpfcc).
RHEL / Rocky / AlmaLinux / CentOS
sudo dnf install bcc-tools kernel-devel-$(uname -r)
(Executables reside in /usr/share/bcc/tools/tcpretrans).
Arch Linux
sudo pacman -S bcc-tools
# Or for the C/CO-RE version:
sudo pacman -S libbpf-tools
Verify the binary:
sudo tcpretrans -h 2>/dev/null || sudo tcpretrans-bpfcc -h
3. Basic Syntax & Command Flags
sudo tcpretrans [options]
Essential Command Flags
| Flag | Description | Practical Example |
|---|---|---|
| (no flag) | Trace all IPv4 and IPv6 TCP retransmissions system-wide. | sudo tcpretrans |
-c |
Count retransmissions and print aggregated totals instead of line-by-line events. | sudo tcpretrans -c |
-T |
Include a human-readable 24-hour timestamp (HH:MM:SS) column. |
sudo tcpretrans -T |
-t |
Include a monotonic timestamp (seconds from tool start). | sudo tcpretrans -t |
-4 |
Trace only IPv4 retransmissions. | sudo tcpretrans -4 |
-6 |
Trace only IPv6 retransmissions. | sudo tcpretrans -6 |
4. Anatomy of Default Output (tcpretrans -T)
Running tcpretrans with timestamps enabled outputs a single line for every retransmitted segment:
sudo tcpretrans -T
Raw Output Example
TIME LADDR:LPORT RADDR:RPORT T STATE
07:25:01 192.168.1.50:41200 -> 10.0.0.12:5432 R ESTABLISHED
07:25:03 192.168.1.50:41200 -> 10.0.0.12:5432 R ESTABLISHED
07:25:08 192.168.1.50:52104 -> 140.82.121.4:443 R ESTABLISHED
07:25:15 192.168.1.50:48912 -> 198.51.100.15:80 S SYN-SENT
5. Breakdown of Every Output Heading & Field
+----------+---------------------------+---------------------------+---+-------------+
| TIME | Local Endpoint | Remote Endpoint | T | TCP State |
+----------+---------------------------+---------------------------+---+-------------+
| 07:25:01 | 192.168.1.50:41200 | -> 10.0.0.12:5432 | R | ESTABLISHED |
+----------+---------------------------+---------------------------+---+-------------+
5.1 TIME
- Format:
HH:MM:SS(wall-clock time). - Meaning: The exact moment the kernel network stack retransmitted the TCP segment due to an unacknowledged retransmission timeout (RTO) or fast retransmit trigger.
5.2 Local Endpoint (LADDR:LPORT)
- Format:
<IP_Address>.<Port>(IPv4) or[IPv6].Port - Meaning: The local host IP address and source/ephemeral port initiating the retransmission.
5.3 Remote Endpoint (RADDR:RPORT)
- Format:
-> <Remote_IP>:<Port> - Meaning: The remote peer IP address and port that failed to acknowledge the previous transmission in time.
5.4 T (Transmission Type Flag)
R(Retransmit): Standard data segment or TCP state retransmission.S(SYN Retransmit): The initial connection handshake (SYN) packet was lost or dropped, forcing a handshake retry.
5.5 TCP State
- Meaning: The state of the TCP socket within the Finite State Machine when the retransmission occurred:
ESTABLISHED: Active data transfer retransmission (dropped segments or unacknowledged payload bytes).SYN-SENT: Connection initiation packet dropped (target host offline, firewalls silently discarding SYN packets, or routing failure).- **
FIN-WAIT-1/LAST-ACK**: Teardown sequence acknowledgement missed.
6. Aggregation Mode (tcpretrans -c)
Instead of flooding the terminal with a line per event, using -c aggregates retransmissions by socket connection tuple and prints summary totals:
sudo tcpretrans -c
Raw Output Example
TIME LADDR:LPORT RADDR:RPORT T STATE ICOUNT
07:26:00 192.168.1.50:41200 -> 10.0.0.12:5432 R ESTABLISHED 5
07:26:00 192.168.1.50:52104 -> 140.82.121.4:443 R ESTABLISHED 1
07:26:30 192.168.1.50:41200 -> 10.0.0.12:5432 R ESTABLISHED 12
ICOUNT: The cumulative count of retransmitted segments for that specific socket pair during the aggregation interval.
7. How tcpretrans Works Internally
tcpretrans attaches eBPF kprobes directly to kernel functions responsible for handling TCP retransmissions:
+-------------------------------------------------------------------------+
| KERNEL SPACE |
| |
| 1. Kernel Retransmission Trigger: |
| - Hooks: tcp_retransmit_skb() or tcp_enter_loss() |
| - Intercepts segments scheduled for retransmission. |
| |
| 2. Field Extraction: |
| - Reads socket structure (struct sock *sk). |
| - Extracts: Local IP, Remote IP, Local Port, Remote Port, |
| TCP State, and socket family (IPv4/IPv6). |
| |
| 3. Event Emission: |
| - Writes formatted event struct into BPF Perf/Ring Buffer. |
+-------------------------------------------------------------------------+
|
v (Ring Buffer)
+-------------------------------------------------------------------------+
| USER SPACE |
| Python / C CLI: Reads ring buffer and prints formatted line to stdout.|
+-------------------------------------------------------------------------+
8. Real-World Troubleshooting Scenarios
Scenario A: Diagnosing Upstream Switch or Cloud Router Drops
An application reports intermittent transaction timeouts communicating with an internal database.
Run tcpretrans to catch packet loss:
sudo tcpretrans -T
Output shows sporadic retransmissions (R ESTABLISHED) targeting the database IP every few minutes.
Diagnosis: Confirms that packets are leaving the local host interface successfully, but getting dropped somewhere along the network path (e.g., overloaded switch buffers, faulty patch cables, or strict bandwidth policing on cloud virtual network interfaces).
Scenario B: Detecting Firewall Black Holes (SYN Drops)
A deployment script hangs indefinitely when attempting to connect to an external API endpoint.
Run tcpretrans to inspect connection establishment:
sudo tcpretrans -T -t
Output shows repeated entries with S SYN-SENT targeting the exact same external IP.
Diagnosis: The kernel is continuously retransmitting the initial SYN packet without ever receiving a SYN-ACK. This proves the remote port is blocked by an aggressive firewall, security group, or routing misconfiguration.
9. Important Interview Questions & Answers
Q: What is the primary operational advantage of using tcpretrans over tcpdump to diagnose packet loss?
Answer: tcpdump captures every single packet traversing the interface, generating high storage, CPU, and memory overhead on busy systems. Furthermore, analyzing a PCAP file requires filtering out billions of healthy packets to locate the few dropped retransmissions. tcpretrans uses in-kernel eBPF to filter out healthy traffic completely, emitting a line only when a retransmission actually occurs. This reduces overhead to near-zero and isolates packet loss events instantly.
Q: Why does a TCP retransmission occur during the SYN-SENT state (S flag), and what does it signify?
Answer: A retransmission during SYN-SENT means the initial TCP synchronization packet sent by the local host was never acknowledged with a SYN-ACK by the remote peer before the internal Retransmission Timeout (RTO) expired. This typically signifies that the remote server is offline, a firewall is silently dropping packets (DROP instead of REJECT), or the target port is unreachable due to a routing table misconfiguration.