biosnoop
Linux biosnoop — Complete Learning Notes & Output Guide
biosnoopis an eBPF/BCC performance-tracing tool that traces every block device I/O (bio) request at the device driver layer. It records the issuing process, disk device, request type, sector offset, size, and—most importantly—the exact latency of each individual I/O request.
1. What is biosnoop?
biosnoop belongs to the BCC (BPF Compiler Collection) toolkit.
While tools like iostat provide aggregate averages (like device utilization %util or average wait time await), and biolatency shows statistical histograms, biosnoop operates at the individual request level. Every time a block I/O request is dispatched to a storage device and subsequently completed, biosnoop prints a line detailing its complete lifecycle.
It answers critical performance questions:
- Which specific processes are generating disk I/O requests right now?
- How long did each individual block request take to complete from issue to completion?
- Are storage requests predominantly reads or writes?
- Which sector ranges are being hammered (revealing random vs. sequential access patterns)?
2. Installation
biosnoop requires root privileges (sudo) and kernel development headers:
Debian / Ubuntu
sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)
(On Debian/Ubuntu, BCC tools are often named with a -bpfcc suffix: biosnoop-bpfcc located in /usr/sbin/).
RHEL / Rocky / AlmaLinux / CentOS
sudo dnf install bcc-tools kernel-devel-$(uname -r)
(Executables reside in /usr/share/bcc/tools/biosnoop).
Arch Linux
sudo pacman -S bcc-tools
Verify:
sudo biosnoop -h 2>/dev/null || sudo biosnoop-bpfcc -h
3. Basic Syntax
sudo biosnoop [options]
To run live system-wide:
sudo biosnoop
4. Anatomy of biosnoop Output
When you execute biosnoop, it outputs a real-time stream of block I/O requests:
TIME(s) COMM PID DISK R/W SECTOR BYTES LAT(ms)
0.000000 kworker/0:1H 120 sda W 2457600 4096 2.10
0.102450 postgres 5410 sda W 1048576 8192 14.25
0.251020 python3 8912 sda R 5242880 32768 1.02
0.541021 rsyslogd 1120 sdb W 0 4096 42.10
5. Breakdown of Every Output Heading & Field
+-----------+--------------+------+------+-----+----------+--------+---------+
| TIME(s) | COMM | PID | DISK | R/W | SECTOR | BYTES | LAT(ms) |
+-----------+--------------+------+------+-----+----------+--------+---------+
| 0.102450 | postgres | 5410 | sda | W | 1048576 | 8192 | 14.25 |
+-----------+--------------+------+------+-----+----------+--------+---------+
5.1 TIME(s)
- Meaning: Elapsed time in seconds since the
biosnooptool was started. - Troubleshooting Significance: Provides a relative timeline to track bursts of storage activity.
5.2 COMM
- Meaning: The short executable command name of the task that triggered the block I/O request (derived from
task->comm, truncated to 16 characters).
5.3 PID
- Meaning: The operating system Process ID associated with the task issuing the block I/O. (Note: For background kernel writeback flushing like
jbd2orkswapd, the PID shown belongs to the kernel thread managing the operation).
5.4 DISK
- Meaning: The name of the target block device (e.g.,
sda,nvme0n1,dm-0).
5.5 R/W (Request Type)
Indicates the storage operation type:
R: Read request.W: Write request.WS: Synchronous Write / Forced Unit Access (FUA) or journal commit.D: Discard / TRIM operation for SSDs.
5.6 SECTOR
- Meaning: The starting logical block address (LBA) sector number on the disk where the request was targeted (typically 512 bytes per sector).
- Troubleshooting Significance: Comparing consecutive sector numbers reveals whether the workload is sequential (sectors incrementing smoothly) or random (sectors jumping erratically across the disk).
5.7 BYTES
- Meaning: Total size of the block request in bytes (e.g.,
4096= 4 KB,32768= 32 KB).
5.8 LAT(ms)
- Meaning: The latency duration in milliseconds from the moment the block request was issued (
block_rq_issue) until it was completed by the driver (block_rq_complete). - Troubleshooting Significance: This is your primary diagnostic metric for physical disk performance. Latencies $>30\,\text{ms}$ on SSDs or $>100\,\text{ms}$ on HDDs indicate severe storage saturation.
6. Essential Command Options & Cheat Sheet
| Flag | Description | Practical Example |
|---|---|---|
-Q |
Include queue time in latency measurements (tracks time spent waiting in kernel block queues prior to device dispatch). | sudo biosnoop -Q |
-d <disk> |
Filter tracing to a specific disk device only. | sudo biosnoop -d nvme0n1 |
7. Real-World Troubleshooting Scenarios
Scenario A: Identifying Latency Outliers on Storage Subsystems
An application experiences sporadic transaction lags, but aggregate metrics look normal.
Run biosnoop and watch for high latency spikes:
sudo biosnoop
Diagnosis: If most lines show LAT(ms) between 0.5 and 2.0, but an outlier logs 150.00 ms, you have isolated an intermittent storage subsystem stall or controller reset.
Scenario B: Distinguishing Random vs. Sequential Disk Access
Determine whether a database or logging daemon is fragmenting storage reads:
sudo biosnoop -d sda
Diagnosis: If SECTOR values jump randomly back and forth across wide ranges, the workload is heavily random-access, which degrades mechanical hard drive performance and causes high cache miss rates.
8. Important Interview Questions & Answers
Q: What is the primary operational difference between biosnoop and iostat?
Answer: iostat reports aggregated, system-wide averages over a set polling interval (e.g., average throughput, average queue size, and average device utilization %util). In contrast, biosnoop traces every single block I/O request individually using in-kernel eBPF tracepoints, showing you the exact PID, command name, sector, size, and individual latency of every request as it happens.
Q: What kernel tracepoints does biosnoop hook into?
Answer: biosnoop attaches eBPF programs to the block layer tracepoints:
block:block_rq_issue: Fired when a request is dispatched to the device driver.block:block_rq_complete: Fired when the storage device finishes processing the request.
By recording the timestamp on issue and computing the delta upon completion, it derives exact per-request latencies.