ext4dist
Linux ext4dist — Complete Learning Notes & Output Guide
ext4distis an eBPF/BCC performance-analysis tool that measures the latency distribution of ext4 filesystem operations (reads, writes, opens, and fsyncs). It aggregates measurements into power-of-2 logarithmic histograms directly within kernel space to isolate storage subsystem stalls from CPU-bound application latency.
1. What is ext4dist?
ext4dist belongs to the BCC (BPF Compiler Collection) toolkit.
While generic block tools like biolatency measure time spent on physical disks, ext4dist instruments the filesystem layer (ext4). This distinction is critical because an application does not talk directly to disk sectors—it interacts with the ext4 filesystem interface.
ext4dist answers essential storage performance questions:
- Is my application slow because ext4 operations are stalling, or is the application code itself slow?
- Are reads fast because they hit the page cache, or are they stalling on physical disk fetches?
- What is the latency distribution of
fsync()calls (database transaction commit bottlenecks)? - Are slow operations rare outliers, or does a substantial portion of the workload experience multi-millisecond latency?
2. Installation
ext4dist is part of standard BCC tooling packages and requires root privileges (sudo) and kernel headers:
Debian / Ubuntu
sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)
(On Debian/Ubuntu, BCC tools are often named with a -bpfcc suffix: ext4dist-bpfcc located in /usr/sbin/).
RHEL / Rocky / AlmaLinux / CentOS
sudo dnf install bcc-tools kernel-devel-$(uname -r)
(Executables reside in /usr/share/bcc/tools/ext4dist).
Arch Linux
sudo pacman -S bcc-tools
Verify:
sudo ext4dist -h 2>/dev/null || sudo ext4dist-bpfcc -h
3. Basic Syntax
sudo ext4dist [options] [interval] [count]
Common Flags
| Flag | Description | Example |
|---|---|---|
interval |
Output interval in seconds (prints a histogram per interval). | sudo ext4dist 1 |
count |
Total number of interval reports to generate before exiting. | sudo ext4dist 1 5 |
-m |
Output latency buckets in milliseconds instead of microseconds. | sudo ext4dist -m |
-p <PID> |
Trace operations performed strictly by a single Process ID. | sudo ext4dist -p 4512 |
-T |
Print a timestamp above each interval histogram. | sudo ext4dist -T 1 |
4. Anatomy of ext4dist Output
When you run ext4dist, it captures filesystem operations, categorizes them by operation type, and renders an ASCII histogram upon interval completion or when you press Ctrl-C:
sudo ext4dist -m 5 1
Raw Output Example
operation = 'read'
msecs : count distribution
0 -> 1 : 124502 |****************************************|
2 -> 3 : 1420 | |
4 -> 7 : 210 | |
8 -> 15 : 42 | |
16 -> 31 : 8 | |
32 -> 63 : 2 | |
operation = 'write'
msecs : count distribution
0 -> 1 : 48910 |****************************************|
2 -> 3 : 320 | |
4 -> 7 : 85 | |
operation = 'fsync'
msecs : count distribution
0 -> 1 : 12 |** |
2 -> 3 : 45 |********* |
4 -> 7 : 189 |****************************************|
8 -> 15 : 94 |******************* |
16 -> 31 : 18 |*** |
32 -> 63 : 4 | |
5. Breakdown of Every Output Heading & Field
+-------------------+-----------------------+------------------------------------------+
| Header / Metric | Example | Technical Meaning |
+-------------------+-----------------------+------------------------------------------+
| operation = '...' | operation = 'fsync' | The ext4 filesystem operation tracked |
| msecs / usecs | 4 -> 7 | The logarithmic latency duration bucket |
| count | 189 | Number of operations in that latency bin |
| distribution | |*******************| | ASCII normalized distribution bar |
+-------------------+-----------------------+------------------------------------------+
5.1 operation = '<name>'
Specifies the category of filesystem routine being measured:
read: File read calls passing through ext4 (ext4_file_read_iter). Measures both page cache hits and disk-backed reads.write: File write calls (ext4_file_write_iter). Writes typically return rapidly when buffered into the Linux page cache.open: Path and inode resolution operations (ext4_file_open). Measures directory traversal and metadata lookup duration.fsync: Synchronous disk flushes (ext4_sync_file). Forces dirty memory pages and journal entries to persist to non-volatile storage.
5.2 Latency Range Column (msecs or usecs)
- Format: Power-of-2 logarithmic buckets (e.g.,
0 -> 1,2 -> 3,4 -> 7,8 -> 15,16 -> 31,32 -> 63,64 -> 127). - Meaning: Represents the time taken from the moment the kernel entered the ext4 function until the function returned to the caller.
- Without
-m: Measured in microseconds ($\mu\text{s}$). - With
-m: Measured in milliseconds ($\text{ms}$).
5.3 count
- Format: Unsigned integer (e.g.,
124502,189). - Meaning: The exact number of filesystem operations whose execution duration fell within that specific interval bucket.
5.4 distribution
- Format: ASCII bar chart (
|****...|). - Meaning: A visual distribution bar normalized against the highest count bucket in that section (the peak bucket gets the maximum width of 40 asterisks
*).
6. Diagnosing Storage Patterns with ext4dist
Pattern A: Healthy Page Cache Activity (Memory-Speed Reads)
operation = 'read'
usecs : count distribution
0 -> 1 : 852100 |****************************************|
2 -> 3 : 1240 | |
- Analysis: Latency is concentrated in sub-microsecond and microsecond ranges (
0 -> 3 us). The workload is hitting the Linux page cache in RAM. Physical disk seeks are almost non-existent.
Pattern B: Cache Misses / Physical Disk Seeks
operation = 'read'
msecs : count distribution
0 -> 1 : 4500 |****************************************|
2 -> 3 : 120 |* |
4 -> 7 : 850 |******* |
8 -> 15 : 1400 |************ |
- Analysis: A bimodal distribution. Fast reads (
0 -> 1 ms) hit the page cache, while a secondary peak at8 -> 15 msreflects cache misses forcing physical disk seeks on mechanical HDDs or busy SATA SSDs.
Pattern C: Journal Contention & fsync Bottlenecks
operation = 'fsync'
msecs : count distribution
0 -> 1 : 5 |* |
8 -> 15 : 42 |******** |
16 -> 31 : 180 |****************************************|
32 -> 63 : 95 |********************* |
64 -> 127 : 12 |** |
- Analysis:
fsyncoperations are clustering heavily between 16ms and 63ms. Because relational databases (PostgreSQL, MySQL) executefsyncto guarantee durability on transaction commits (WAL writes), this distribution explains high transaction commit latency and write stalls.
7. ext4dist vs. ext4slower vs. biolatency
+-------------------------------------------------------------+
| ext4dist |
| * Measures ext4 VFS operations in kernel space. |
| * Outputs logarithmic histograms (full distributions). |
| * Best for: Understanding workload latency profiles. |
+-------------------------------------------------------------+
|
v
+-------------------------------------------------------------+
| ext4slower |
| * Measures same ext4 operations as ext4dist. |
| * Streams individual line logs exceeding a threshold (ms). |
| * Best for: Finding specific culprit PIDs and filenames. |
+-------------------------------------------------------------+
|
v
+-------------------------------------------------------------+
| biolatency |
| * Measures block layer / physical device I/O. |
| * Does not see page cache hits or VFS metadata overhead. |
| * Best for: Identifying physical disk/hardware queues. |
+-------------------------------------------------------------+
| Dimension | ext4dist |
ext4slower |
biolatency |
|---|---|---|---|
| Layer | Filesystem (ext4) |
Filesystem (ext4) |
Block device / Driver queue |
| Output Style | Logarithmic Histograms | Per-event text stream | Logarithmic Histograms |
| Captures Page Cache? | Yes (shows microsecond hits) | Yes (if slower than threshold) | No (only disk I/O requests) |
| Shows Filenames? | No | Yes | No |
| Performance Overhead | Extremely low (in-kernel maps) | Low (depends on threshold) | Extremely low |
8. Real-World Troubleshooting Scenarios
Scenario A: Database Transaction Stalls
A database team reports that PostgreSQL queries are intermittently taking hundreds of milliseconds to commit.
Run ext4dist filtered to the PostgreSQL writer process:
sudo ext4dist -m -p $(pgrep -f "postgres: .* writer") 10 1
If fsync shows values above 30ms: The disk write cache, battery-backed cache, or underlying EBS/SAN volume cannot flush journaling writes fast enough.
Scenario B: Verifying Cache Warming
After an application deploy, determine whether file read requests have settled back into RAM:
sudo ext4dist 5 1
Diagnosis:
- If the
readhistogram is concentrated under10 usecs, memory warming was successful. - If significant counts remain in
msecsranges, the working set exceeds available RAM or read operations are unbuffered.
9. Important Interview Questions & Answers
Q: Why can ext4dist show slow read latency while iostat or biolatency reports low disk utilization?
Answer: ext4dist traces operations at the filesystem layer, whereas iostat and biolatency measure block-level requests submitted to physical devices. A read operation in ext4 might stall not on physical media, but on filesystem-level lock contention (such as inode mutex locks or readers-writer locks during file truncations/appends) or memory allocation delays in the page allocator. ext4dist captures this end-to-end software latency before block I/O is ever scheduled.
Q: What kernel functions does ext4dist attach to?
Answer: ext4dist attaches eBPF kprobes and kretprobes to core ext4 file operation vectors:
- Reads:
ext4_file_read_iter - Writes:
ext4_file_write_iter - Opens:
ext4_file_open - Syncs:
ext4_sync_file
It records a high-resolution timestamp at function entry, computes the delta upon function return, and increments the corresponding bucket in a BPF histogram map directly within kernel space.