hardirqs
Linux hardirqs — Complete Learning Notes & Output Guide
hardirqsis an eBPF/BCC and libbpf performance-tracing tool that measures the execution time and duration histograms of Hardware Interrupt Service Routines (ISRs). By tracing hardware IRQ handlers across physical and virtual devices, it isolates interrupt storms, driver stalls, and CPU core contention caused by high-rate I/O.
1. What is hardirqs?
When a physical hardware device (such as a network interface card, NVMe controller, disk controller, or USB bus) requires immediate attention from the CPU, it asserts an electrical signal across an interrupt line or sends a Message Signaled Interrupt (MSI/MSI-X).
The CPU suspends its current task, jumps to a registered Hardware Interrupt Service Routine (Hard IRQ), and executes the low-level driver handler to acknowledge the event before scheduling deferred work (softirq).
hardirqs instruments these hardware interrupt handlers using eBPF.
It answers critical low-level kernel and performance engineering questions:
- Which hardware devices are generating the highest volume of interrupts?
- How much CPU time is spent inside hard interrupt handlers versus user space and application threads?
- Are specific IRQ handlers stalling (exhibiting long execution durations), leading to CPU core starvation?
- How are interrupts distributed across logical CPU cores (verifying IRQ affinity / smp_affinity settings)?
2. Installation & Availability
hardirqs is bundled within the BCC (BPF Compiler Collection) toolkit (bpfcc-tools on Debian/Ubuntu, bcc-tools on RHEL/CentOS).
Debian / Ubuntu
sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)
(Executables reside in /usr/sbin/ with a -bpfcc suffix, e.g., hardirqs-bpfcc).
RHEL / Rocky / AlmaLinux / CentOS
sudo dnf install bcc-tools kernel-devel-$(uname -r)
(Executables reside in /usr/share/bcc/tools/hardirqs).
Arch Linux
sudo pacman -S bcc-tools
Verify the binary:
sudo hardirqs -h 2>/dev/null || sudo hardirqs-bpfcc -h
3. Basic Syntax & Command Flags
sudo hardirqs [options] [interval] [count]
Essential Command Flags
| Flag | Description | Practical Example |
|---|---|---|
| (no flag) | Display a cumulative time-duration histogram of interrupt latencies from start until Ctrl-C. |
sudo hardirqs |
interval |
Print summaries continuously at a set interval in seconds. | sudo hardirqs 5 |
count |
Number of interval outputs before terminating. | sudo hardirqs 2 10 |
**-C, --count** |
Display raw interrupt event counts (frequency) instead of execution duration histograms. | sudo hardirqs -C 5 |
**-N, --nanoseconds** |
Display duration histograms in nanoseconds instead of the default microseconds. | sudo hardirqs -N 5 |
**-d, --dest** |
Include total execution time column (sum of all durations). | sudo hardirqs -d |
4. Anatomy of Output: Duration Histogram (Default)
Running hardirqs without flags records hardware interrupt execution durations and outputs power-of-2 logarithmic histograms upon Ctrl-C:
sudo hardirqs
Raw Output Example (Duration Histogram)
Tracing hard interrupt latency... Hit Ctrl-C to end.
^C
harirq = ahci [IRQ: 27]
usecs : count distribution
0 -> 1 : 45210 |****************************************|
2 -> 3 : 1204 |* |
4 -> 7 : 142 | |
8 -> 15 : 12 | |
harirq = enp3s0 [IRQ: 34]
usecs : count distribution
0 -> 1 : 891024 |****************************************|
2 -> 3 : 45210 |** |
4 -> 7 : 3201 |* |
8 -> 15 : 412 | |
16 -> 31 : 14 | |
5. Anatomy of Output: Event Counts Mode (hardirqs -C)
Using -C aggregates interrupt frequency (total interrupts per interval) instead of execution duration:
sudo hardirqs -C 2 3
Raw Output Example (Event Counts)
Tracing hard interrupt counts... Output every 2 secs. Hit Ctrl-C to end.
TIME IRQ# VECTOR CPU NAME
07:50:01 27 0x3b 0 ahci
07:50:01 34 0x42 2 enp3s0-0
07:50:01 35 0x43 3 enp3s0-1
07:50:01 128 0x80 1 nvme0q1
--------------------------------------
TIME IRQ# VECTOR CPU NAME
07:50:03 27 0x3b 0 ahci
07:50:03 34 0x42 2 enp3s0-0
6. Breakdown of Every Output Heading & Field
6.1 Duration Histogram Fields (hardirqs)
harirq = <driver/name> [IRQ: <number>]: Identifies the device driver name and kernel IRQ vector line (e.g.,enp3s0 [IRQ: 34]).- **
usecs/msecs**: Power-of-2 logarithmic buckets measuring the time elapsed from when the hardware interrupt entered the ISR until it returned. count: Total number of hardware interrupts falling into that duration bracket.distribution: Normalized ASCII bar graph representing relative frequency.
6.2 Event Count Fields (hardirqs -C)
TIME: Wall-clock timestamp of the sample interval.IRQ#: The kernel interrupt request line number.VECTOR: The CPU interrupt vector hex offset.CPU: The logical CPU core that handled the interrupt.NAME: The device driver or interrupt handler name (e.g.,nvme0q1,enp3s0-0,xhci_hcd).
7. How hardirqs Works Internally
hardirqs attaches eBPF kprobes or tracepoints to the kernel's low-level interrupt handling pathways:
+-------------------------------------------------------------------------+
| KERNEL SPACE |
| |
| 1. Hardware Interrupt Entry: |
| - Hooks: handle_irq_event_percpu() or irq_handler_entry() |
| - Records entry timestamp: bpf_ktime_get_ns() |
| - Maps starting timestamp to active IRQ descriptor in BPF Map. |
| |
| 2. Hardware Interrupt Exit: |
| - Hooks: irq_handler_exit() |
| - Calculates: duration = current_time - entry_time |
| - Computes power-of-2 bucket and increments BPF Histogram. |
+-------------------------------------------------------------------------+
|
v (Read upon interval / Ctrl-C)
+-------------------------------------------------------------------------+
| USER SPACE |
| CLI Tool: Formats BPF maps into readable ASCII histograms or counts. |
+-------------------------------------------------------------------------+
8. Real-World Troubleshooting Scenarios
Scenario A: Diagnosing Interrupt Storms and CPU Saturation
A server's CPU utilization sits at 100% in system mode (%sys), but top-level process monitors (top, htop) show no user-space processes consuming CPU.
Run hardirqs -C to check interrupt frequency:
sudo hardirqs -C 1
Output: Shows a single network interface interrupt (enp3s0-0) firing over 800,000 times per second.
Diagnosis: The server is experiencing an interrupt storm (caused by a malfunctioning NIC driver, a broadcast packet flood, or a misconfigured packet capture). The CPU spends all its clock cycles servicing hardware interrupt handlers, leaving no cycles for user applications.
Scenario B: Correlating Storage Latency with Slow SATA/NVMe ISRs
A database experiences high storage wait times during heavy write bursts.
Measure hard interrupt durations:
sudo hardirqs 5 1
Diagnosis: If the histogram for the storage driver (nvme0q1 or ahci) reveals a long tail extending past $50\text{--}100\,\mu\text{s}$, the disk controller driver ISR is blocked or stalled, delaying I/O completion signaling back to the block layer.
9. Important Interview Questions & Answers
Q: What is the operational distinction between Hard IRQs (hardirqs) and Soft IRQs (softirqs) in Linux?
Answer:
- Hard IRQs are executed immediately when physical hardware signals the CPU. They run with hardware interrupts temporarily disabled on that core and must execute as quickly as possible to acknowledge the hardware.
- Soft IRQs (Software Interrupts) handle deferred, non-urgent work spawned by Hard IRQs (such as networking packet processing via
NET_RX_SOFTIRQor block device completion). Soft IRQs run with interrupts re-enabled and can be scheduled across multiple CPU cores.
Q: Why can an unmanaged interrupt storm on a single CPU core lock up an entire server?
Answer: By default, hardware interrupts are often routed to CPU 0 during boot if IRQ balancing (irqbalance daemon) is disabled or misconfigured. If an interrupt storm hits that specific device, CPU 0 becomes pegged at 100% servicing hard interrupt handlers and softirqs. Because CPU 0 is overwhelmed, it may fail to respond to kernel inter-processor interrupts (IPIs) or timer ticks, making the system unresponsive to SSH connections or administrative commands.