Linux netstat — Complete Learning Notes & Output Guide

netstat (network statistics) is a classical networking diagnostic utility from the net-tools suite. It inspects active network sockets (TCP, UDP), listening server endpoints, interface packet counters, routing tables, and protocol-level error telemetry by parsing virtual files in /proc/net/.

---

1. What is netstat?

netstat monitors the transport and network layers of the Linux kernel network stack.

It answers fundamental systems and networking questions:

• Which process and PID is listening on a specific port (e.g., :80, :443, :5432)?
• How many sockets are stuck in TIME_WAIT, CLOSE_WAIT, or SYN_RECV states?
• Is an application failing to read incoming data, causing bytes to accumulate in the kernel Recv-Q?
• Are outbound packets stalling in Send-Q due to network path congestion or unacknowledged TCP segments?
• How many TCP segments have been retransmitted system-wide due to packet drops?

---

2. Installation & Package Availability

netstat is part of the legacy net-tools package. While modern Linux systems default to ss (from iproute2), netstat remains widely deployed across legacy scripts, container base images, and system administration workflows.

Debian / Ubuntu

sudo apt update
sudo apt install net-tools

RHEL / Rocky / AlmaLinux / CentOS

sudo dnf install net-tools

Arch Linux

sudo pacman -S net-tools

Verify installation:

netstat --version

---

3. Basic Syntax & Core Option Flags

netstat [options]

Essential Command Flags

Flag	Purpose	Diagnostic Context
-t	Filter to TCP sockets only.	Isolates connection-oriented traffic.
-u	Filter to UDP sockets only.	Isolates connectionless datagram traffic (DNS, DHCP, WireGuard).
-l	Show only listening sockets.	Audits open ports accepting new incoming connections.
-a	Show all sockets (both listening and established).	Full view of all network endpoints.
-n	Display numeric addresses and port numbers.	Bypasses DNS lookups and /etc/services port name resolution (essential for speed).
-p	Display the PID and program name owning the socket.	Requires sudo to view processes owned by other users.
-s	Display summary statistics per protocol (IP, ICMP, TCP, UDP).	Diagnoses packet errors, drops, and retransmissions.
-r	Display the kernel routing table.	Legacy equivalent of route -e or ip route.
-i	Display interface packet counters and MTUs.	Legacy equivalent of ifconfig -s or ip -s link.
-c	Continuously refresh output every second.	Real-time interactive socket monitoring.

The Standard Production Flag Combos

# 1. Audit all active listening services with PIDs and numeric ports (Standard Audit)
sudo netstat -tulnp

# 2. View all active established TCP connections without DNS delay
sudo netstat -antp

# 3. View protocol-level error counters (retransmits, drops, resets)
netstat -s

---

4. Anatomy of Default Output (netstat -tulnp)

Querying listening TCP and UDP sockets with numeric addresses and process owners:

sudo netstat -tulnp

Raw Output Example

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      892/sshd: /usr/sbin 
tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN      1420/postgres       
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      4512/nginx: master  
tcp6       0      0 :::8080                 :::*                    LISTEN      9102/java           
udp        0      0 0.0.0.0:68              0.0.0.0:*                           620/dhclient        
udp        0      0 127.0.0.53:53           0.0.0.0:*                           510/systemd-resolve 

Raw Output Example: Established Connections (netstat -antp)

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 192.168.1.50:22         192.168.1.100:54210     ESTABLISHED 1845/sshd: user [pr 
tcp    14250      0 192.168.1.50:80         203.0.113.15:48912      ESTABLISHED 4515/nginx: worker  
tcp        0  48912 192.168.1.50:5432       10.0.0.5:38910          ESTABLISHED 1422/postgres       
tcp        0      0 192.168.1.50:49152      198.51.100.2:443        TIME_WAIT   -                   
tcp        0      0 192.168.1.50:41200      10.0.0.8:9092           CLOSE_WAIT  8910/python3        

---

5. Breakdown of Every Output Heading & Field

+-------+--------+--------+----------------------+----------------------+-------------+--------------------+
| Proto | Recv-Q | Send-Q | Local Address        | Foreign Address      | State       | PID/Program name   |
+-------+--------+--------+----------------------+----------------------+-------------+--------------------+
| tcp   |  14250 |      0 | 192.168.1.50:80      | 203.0.113.15:48912   | ESTABLISHED | 4515/nginx: worker |
+-------+--------+--------+----------------------+----------------------+-------------+--------------------+

5.1 Proto

• Values: tcp, tcp6, udp, udp6, raw, unix.
• Meaning: The transport protocol utilized by the socket. Suffix 6 denotes IPv6 socket binding.

5.2 Recv-Q (Receive Queue)

• Meaning depends on connection state:
• Established Connections: The count of bytes received from the network driver and held in the kernel socket buffer that have not yet been read by the user application (via recv() or read()).

• Listening Sockets (State = LISTEN): In netstat, this value is usually 0. (Note: In modern ss, Recv-Q on a listening socket displays the current number of established connections waiting to be accept()ed).

• Troubleshooting Significance: A persistent or growing value in Recv-Q indicates that the application process is bottlenecked, stalled, or running out of CPU cycles, failing to drain its incoming buffer fast enough.

5.3 Send-Q (Send Queue)

• Meaning depends on connection state:
• Established Connections: The count of bytes queued in the local kernel send buffer that have been sent across the wire but not yet acknowledged (ACKed) by the remote peer.

• Listening Sockets (State = LISTEN): In netstat, this is typically 0. (Note: In ss, Send-Q on a listening socket represents the maximum listen backlog queue limit, somaxconn).

• Troubleshooting Significance: A large, non-draining Send-Q indicates downstream network congestion, packet loss, or a hung remote receiver whose TCP window is closed (win 0).

5.4 Local Address

• Format: <IP_Address>:<Port>
• Meaning: The local host IP binding and TCP/UDP port number assigned to the socket.
• 0.0.0.0:* / :::*: Bound to all available network interfaces (wildcard).
• 127.0.0.1:*: Bound strictly to the loopback interface; unreachable from external networks.
• 192.168.1.50:*: Bound strictly to that specific interface adapter.

5.5 Foreign Address

• Format: <IP_Address>:<Port>
• Meaning: The remote peer IP address and port to which the socket is connected. On listening sockets, this shows 0.0.0.0:* or :::* (unconnected).

5.6 State

• Meaning: The current operational state of the socket within the TCP Finite State Machine. (UDP is stateless; its state field is left blank).

5.7 PID/Program name

• Format: <Process_ID>/<Executable_Name>
• Meaning: The operating system Process ID and executable command name responsible for creating and owning the file descriptor of the socket.
• Special Cases: Sockets in TIME_WAIT show - because the application has already closed its file descriptor, and the socket is maintained solely by the kernel network stack until the timeout expires.

---

6. Deep-Dive: TCP Connection States

Understanding the TCP state machine is critical when diagnosing connection saturation:

       Client                               Server
         |                                    |
         | -------- SYN (SYN_SENT) ---------> | (SYN_RECV)
         | <------- SYN-ACK ----------------- | 
(ESTABLISHED) | -------- ACK -----------------------> | (ESTABLISHED)
         |                                    |
         |        [ Data Transfer Phase ]     |
         |                                    |
         | -------- FIN (FIN_WAIT1) --------> | (CLOSE_WAIT)
         | <------- ACK (FIN_WAIT2) --------- | 
         | <------- FIN --------------------- | (LAST_ACK)
(TIME_WAIT) | -------- ACK -----------------------> | (CLOSED)

TCP State	Trigger / Mechanism	Diagnostic Significance
LISTEN	The server process called listen().	Endpoint is waiting for incoming client connections.
SYN_SENT	Client initiated a connection (connect()), sent SYN.	If persistent, the target IP is unreachable, a firewall is silently dropping packets, or routing is broken.
SYN_RECV	Server received a SYN, sent SYN-ACK, awaiting client ACK.	High counts point to a SYN Flood DDoS attack or asymmetric routing drops.
ESTABLISHED	3-way handshake completed. Data transfer active.	Normal operational state.
FIN_WAIT1	Local application initiated active close and sent FIN.	Waiting for remote ACK or remote FIN.
FIN_WAIT2	Received ACK for local FIN; waiting for remote FIN.	The local end closed cleanly; waiting for the remote peer to terminate its side.
CLOSE_WAIT	Remote peer closed connection; local kernel ACKed.	Application Bug Indicator. The local application has not called close() on the socket descriptor, leaking connections.
LAST_ACK	Local side sent its final FIN, waiting for last ACK.	Brief transitional state before destruction.
TIME_WAIT	Active close initiator waits $2 \times \text{MSL}$ (60 seconds).	Normal TCP cleanup state. Ensures late arriving duplicate segments expire without corrupting future sockets.

---

7. Protocol Statistics: netstat -s

Running netstat -s parses /proc/net/snmp and /proc/net/netstat, displaying system-wide aggregate counters across all network layers:

netstat -s

Critical Excerpt: TCP Health Counters

Tcp:
    145210 active connection openings
    89120 passive connection openings
    1204 failed connection attempts
    3210 connection resets received
    4512 connections established
    1245012 segments received
    1845120 segments sent out
    1450 segments retransmitted
    12 bad segments received
    85 resets sent
TcpExt:
    42 times the listen queue of a socket overflowed
    42 SYNs to LISTEN sockets dropped
    120 fast retransmits
    14 TCP slow start after idle

Key Statistical Metrics Explained

• active connection openings: Outbound connections initiated by this machine (client mode via connect()).
• passive connection openings: Inbound connections accepted by this machine (server mode via listen()).
• segments retransmitted: Total TCP segments sent again due to missing ACKs. A rapidly climbing ratio of $\frac{\text{retransmitted}}{\text{sent out}}$ points to network packet loss.
• times the listen queue of a socket overflowed: Critical Capacity Signal. Incoming connection requests were discarded because the application's listen backlog queue (configured via backlog in listen() and limited by net.core.somaxconn) was completely full.
• SYNs to LISTEN sockets dropped: SYN packets dropped due to full queues or SYN cookies triggering.

---

8. Kernel Routing Table: netstat -r

netstat -rn inspects the kernel IPv4 route cache:

netstat -rn

Raw Output Example

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window irtt Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG        0 0          0 eth0
10.0.0.0        10.200.1.1      255.255.0.0     UG        0 0          0 eth1
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 eth0

Breakdown of Route Headings

• Destination: Target IP subnet or 0.0.0.0 (default route).
• Gateway: Upstream next-hop router address (0.0.0.0 means directly attached local network).
• Genmask: Subnet mask corresponding to the destination network.
• Flags: Route status codes:
• U: Route is Up (active).
• G: Route uses a Gateway (next-hop router).

• H: Target is a single Host, not a subnet.

• Iface: Physical or virtual network interface used to dispatch matching packets.

---

9. netstat vs. ss: Why netstat is Deprecated

On modern Linux environments, ss (Socket Statistics) is universally recommended over netstat:

+-------------------------------------------------------------+
|                          netstat                            |
|  * Reads /proc/net/tcp, /proc/net/udp sequentially.         |
|  * Formats raw text strings and parses line-by-line.        |
|  * Flaw: Saturated servers with 50k+ sockets lock memory    |
|    and take several seconds to execute.                     |
+-------------------------------------------------------------+
                               vs
+-------------------------------------------------------------+
|                            ss                               |
|  * Communicates directly with kernel via Netlink sockets.   |
|  * Uses the sock_diag kernel subsystem.                     |
|  * Advantage: Zero string conversions in kernel space;      |
|    retrieves thousands of sockets in milliseconds.          |
+-------------------------------------------------------------+

Dimension	netstat	ss
Package	net-tools (Legacy / Unmaintained)	iproute2 (Active standard)
Data Source	Reads /proc/net/tcp text files	Kernel Netlink (sock_diag) API
Speed with 100k Sockets	Slow ($5\text{--}30\text{ seconds}$)	Extremely fast ($<0.2\text{ seconds}$)
TCP Internal Info	Basic state & queue bytes	RTT, Congestion Window (cwnd), MSS, Pacing
Listen Queue Interpretation	Shows 0 in Recv-Q / Send-Q	Recv-Q = current backlog; Send-Q = max limit

---

10. Real-World Troubleshooting Scenarios

Scenario A: Resolving "Address Already in Use" Port Conflicts

A web server fails to restart, logging bind: Address already in use: 80.

Identify the process holding the socket:

sudo netstat -tulnp | grep :80

Output:

tcp   0   0 0.0.0.0:80   0.0.0.0:*   LISTEN   1204/apache2

Diagnosis: An old Apache daemon (PID 1204) is still running in the background and holding the socket open. Terminate it with kill -15 1204 to free the port.

---

Scenario B: Diagnosing Application Leaks via CLOSE_WAIT Sockets

A database client service stops accepting new work, and available file descriptors are exhausted.

Check connection counts by state:

netstat -ant | awk '{print $6}' | sort | uniq -c | sort -rn

Output:

  14200 CLOSE_WAIT
    120 ESTABLISHED
     14 LISTEN

Diagnosis: Over 14,000 sockets sit in CLOSE_WAIT. The remote server closed the connection, the Linux kernel acknowledged it, but the local application code forgot to call socket.close(), causing a catastrophic socket descriptor leak inside the application runtime.

---

Scenario C: Detecting Backpressure and Thread Starvation via Recv-Q

Users report high API latency on an internal microservice, though CPU and memory metrics appear normal.

Inspect queue backlogs:

netstat -antp | grep :8080

Output:

Proto Recv-Q Send-Q Local Address       Foreign Address     State       PID/Program name
tcp   131072      0 10.0.0.5:8080       10.0.0.12:45120    ESTABLISHED 8912/python3
tcp   131072      0 10.0.0.5:8080       10.0.0.14:48102    ESTABLISHED 8912/python3

Diagnosis: Recv-Q is maxed out at 128 KB across multiple sockets. The network is delivering data instantly, but the single-threaded application (python3) is blocked on a slow database query or synchronous lock and is not reading from the OS socket buffer.

---

11. Important Interview Questions & Answers

Q: What does a persistent non-zero value in Recv-Q vs. Send-Q indicate on an established TCP connection?

Answer: On an established connection:

• **Non-zero Recv-Q**: Data has arrived safely from the network and has been acknowledged by the kernel, but the local user-space application has not read it from the socket receive buffer via read() or recv(). This indicates that the local application is CPU-bound, blocked, or experiencing thread starvation.
• **Non-zero Send-Q**: Data has been written to the socket by the local application via write() or send(), but the kernel has not yet received an ACK from the remote peer. This indicates downstream network transit delays, dropped packets causing retransmits, or a slow/unresponsive remote host whose TCP receive window is full (ZeroWindow).

Q: Why does an accumulation of CLOSE_WAIT sockets indicate an application bug, whereas TIME_WAIT is generally an OS-level networking state?

Answer:

• CLOSE_WAIT: Occurs on the passive close side. The remote peer sent a FIN packet, and the local OS kernel automatically responded with an ACK. The socket must now wait for the local application process to explicitly issue a close() system call to send its own FIN. If sockets remain stuck in CLOSE_WAIT, it proves that the **application code failed to invoke close()**, leaking file descriptors.
• TIME_WAIT: Occurs on the active close side (the side that initiates the teardown). The application has already called close(). The OS kernel keeps the socket structure alive for $2 \times \text{MSL}$ (typically 60 seconds) to ensure that any lingering or delayed packets on the wire expire safely without corrupting future connections reusing the same 4-tuple (IPs and ports).

Q: Why should ss be used instead of netstat on systems handling high connection volumes?

Answer: netstat parses human-readable text representations from /proc/net/tcp and /proc/net/udp. When a server handles 50,000 to 100,000 concurrent sockets, formatting and parsing this multi-megabyte text file in kernel and user space causes noticeable CPU overhead and can freeze monitoring scripts. ss communicates directly with the kernel's sock_diag Netlink subsystem, transferring binary structs directly from kernel memory. This is orders of magnitude faster and consumes significantly fewer system resources.

---