nicstat

@amitmund September 11, 2026

Linux nicstat — Complete Learning Notes & Output Guide

nicstat is a command-line performance diagnostic utility that prints network interface traffic throughput, packet rates, error counts, and CPU utilization metrics in a clean, tabulated format. Originally ported from Solaris, it provides a time-interval breakdown of network activity across all physical and virtual interfaces.

1. What is nicstat?

Unlike standard utilities like ifconfig or ip -s link (which show cumulative packet and byte totals since system boot), nicstat computes delta rates per interval (Megabytes per second, packets per second, saturation percentages).

It answers critical network performance questions:

  • Which network interfaces are actively driving throughput right now?
  • What is the ratio of incoming (rKB/s) versus outgoing (wKB/s) bandwidth?
  • Are interfaces approaching line-rate saturation (e.g., nearing 1 Gbps or 10 Gbps caps)?
  • Are packet drops or interface errors (Errs, Coll) occurring during traffic bursts?
  • How much CPU overhead is associated with network interrupt processing (%Util, %Sat)?

2. Installation & Availability

nicstat is available across standard distribution package repositories or can be compiled from source.

Debian / Ubuntu

sudo apt update
sudo apt install nicstat

RHEL / Rocky / AlmaLinux / CentOS

sudo dnf install nicstat

Arch Linux

sudo pacman -S nicstat

Verify installation:

nicstat -v

3. Basic Syntax & Primary Options

nicstat [options] [interval [count]]
Flag Purpose Practical Example
(no flag) Output KB/s read/write and packets per second across all active interfaces at 1-second intervals. nicstat 1
-t Display TCP-specific packet statistics (active/passive connections, segments in/out). nicstat -t 2 5
-u Display UDP-specific packet statistics (UDP datagrams in/out). nicstat -u
-a Display all metrics (combines network interface throughput, TCP, and UDP). nicstat -a 1
-v Print values in megabits (M), gigabits (G), or bytes (b) instead of KB/s. nicstat -M 1
-i <interfaces> Filter output to specific interfaces (comma-separated). nicstat -i eth0,eth1 2
-S Display separate lines for inbound and outbound traffic rows. nicstat -S 1

4. Anatomy of Default Output (nicstat 1)

Running nicstat with an interval argument (e.g., 1 for 1-second updates) generates a live scrolling table:

nicstat 1

Raw Output Example

    Time      Int   rKB/s   wKB/s   rPk/s   wPk/s    Util   Sat 
07:12:10     lo      0.00    0.00    0.00    0.00    0.00    0.00
07:12:10   eth0   4520.10 1204.50 3510.20  840.10    4.52    0.00
07:12:10   eth1      0.00    0.00    0.00    0.00    0.00    0.00
07:12:11   eth0   8910.40 2450.80 7100.10 1620.50    8.91    0.00

5. Breakdown of Every Output Heading & Field

+----------+-------+---------+---------+---------+---------+-------+-------+
| Time     | Int   | rKB/s   | wKB/s   | rPk/s   | wPk/s   | Util  | Sat   |
+----------+-------+---------+---------+---------+---------+-------+-------+
| 07:12:10 | eth0  | 4520.10 | 1204.50 | 3510.20 |  840.10 |  4.52 |  0.00 |
+----------+-------+---------+---------+---------+---------+-------+-------+

5.1 Time

  • Format: HH:MM:SS (wall-clock timestamp).
  • Meaning: The exact time at which the sample interval measurement concluded.

5.2 Int (Interface Name)

  • Format: String (e.g., lo, eth0, bond0, enp3s0).
  • Meaning: The network interface identifier being reported. Loopback (lo) is included by default unless filtered.

5.3 rKB/s (Read Kilobytes per Second)

  • Format: Floating-point number.
  • Meaning: The average volume of data received (in KB/s) by the interface during the interval.
  • Troubleshooting Significance: Primary metric for tracking inbound network load (HTTP requests, data ingestion, downloads).

5.4 wKB/s (Write Kilobytes per Second)

  • Format: Floating-point number.
  • Meaning: The average volume of data transmitted / sent (in KB/s) out of the interface during the interval.
  • Troubleshooting Significance: Tracks outbound throughput (database replication, backups, API responses).

5.5 rPk/s (Read Packets per Second)

  • Format: Floating-point number.
  • Meaning: The total number of incoming packets received per second during the interval.
  • Troubleshooting Significance: High packet rates (rPk/s) with low throughput (rKB/s) indicates a small-packet storm (e.g., DDoS attacks, excessive ARP requests, or chatty microservice heartbeats), which stresses CPU interrupt handling regardless of total bandwidth.

5.6 wPk/s (Write Packets per Second)

  • Format: Floating-point number.
  • Meaning: The total number of outgoing packets transmitted per second during the interval.

5.7 Util (Interface Utilization Percentage)

  • Format: Percentage (0.00 to 100.00+).
  • Meaning: The estimated percentage of bandwidth capacity utilized by the interface.
  • Calculation: Computed based on the interface's maximum rated speed (queried via system drivers, e.g., 1 Gbps vs. 10 Gbps) and combined read/write bit rates.
  • Troubleshooting Significance: When Util approaches 100.00, the interface has reached its physical line rate.

5.8 Sat (Saturation)

  • Format: Floating-point metric.
  • Meaning: An indicator of interface congestion and queue drops. When an interface cannot transmit or receive packets fast enough due to saturated hardware buffers or switch-side flow control, saturation metrics increase.

6. Advanced Monitoring: TCP & UDP Statistics (nicstat -a)

Combining network throughput with protocol-level statistics helps isolate application socket behavior from physical layer traffic.

nicstat -a 1

Raw Output Example

    Time      Int   rKB/s   wKB/s   rPk/s   wPk/s    Util   Sat
07:15:01   eth0   120.00   45.00  110.00   40.00    0.12    0.00
--- TCP Statistics ---
    Time      Int    актив   passv   isocd   retrans
07:15:01   eth0     12.0    45.0     0.0     0.2
--- UDP Statistics ---
    Time      Int   rKB/s   wKB/s   rPk/s   wPk/s
07:15:01   eth0     2.00    1.00   15.00   10.00

Key Protocol Sub-Metrics Explained

  • актив (Active Connections): Number of outgoing TCP connection initiations (connect()) per second.
  • passv (Passive Connections): Number of incoming TCP connections accepted per second (accept()).
  • retrans (TCP Retransmissions): Number of TCP segments retransmitted per second. High retransmission rates indicate packet loss, network congestion, or firewall drops.

7. Real-World Troubleshooting Scenarios

Scenario A: Diagnosing Small-Packet CPU Interrupt Storms

CPU utilization is at 100% (specifically in %sys interrupt handling), but iftop or standard bandwidth monitors show only 50 Mbps of traffic on a 10 Gbps interface.

Run nicstat to inspect packet rates vs. throughput:

nicstat 1

Diagnosis: rKB/s is low, but rPk/s is sitting at 1,200,000 packets per second. The server is being flooded with tiny packets (e.g., a SYN flood DDoS or misconfigured IoT telemetry stream). Each packet triggers a hardware interrupt, exhausting CPU cores despite low overall bandwidth consumption.

Scenario B: Verifying Network Bonding / LACP Load Balancing

You configure an 802.3ad LACP bond (bond0) across four 10G interfaces and want to verify whether traffic is hashing evenly across all slave ports:

nicstat -i bond0,eth0,eth1,eth2,eth3 2

Diagnosis: If bond0 shows 30 Gbps total throughput, but eth0 carries 30 Gbps while eth1, eth2, and eth3 show 0.00, your source/destination hashing algorithm (e.g., Layer 2 vs. Layer 3+4 hash) is misconfigured, hashing all flows onto a single physical link.

8. nicstat vs. iftop vs. sar vs. nload

Feature / Dimension nicstat iftop sar -n DEV nload
Output Style Clean text rows per interval Interactive flow-by-flow TUI Historical & interval system metrics Visual bandwidth graph
Per-Connection Tracking No Yes (shows IP:port flows) No No
Packet Rate (rPk/s) Yes No Yes No
Interface Utilization (Util) Yes (automatic % calculation) No No Visual graph
Scripting / Log Piping Excellent Poor (Interactive) Excellent Poor (Interactive)

9. Important Interview Questions & Answers

Q: How does nicstat calculate the Util (Utilization) percentage, and what happens if it reports over 100%?

Q: Why might nicstat report high packet rates (rPk/s) while bandwidth throughput (rKB/s) remains minimal?

Answer: nicstat computes interface utilization by comparing total bits transferred per second against the interface's maximum operational speed reported by the kernel driver (e.g., 10,000 Mbps for a 10G NIC). If nicstat reports utilization exceeding 100%, it typically indicates that the interface is operating in full-duplex mode where simultaneous bi-directional traffic (e.g., 8 Gbps inbound + 4 Gbps outbound on a 10G link) exceeds the single-direction rated capacity calculation, or that speed negotiation was incorrectly reported by the driver.

Answer: High packet rates with low throughput indicate a small-packet workload. Network overhead (Ethernet, IP, and TCP headers) consumes a fixed amount of packet framing regardless of payload size. When an application processes millions of tiny packets (e.g., DNS queries, UDP heartbeats, or DDoS packets) averaging 64 to 128 bytes each, packet per second rates soar while total byte throughput remains low. This severely taxes CPU interrupt handlers (softirq) without saturating physical link bandwidth.

0 Likes
2 Views
0 Comments

Filters

No filters available for this view.

Reset All