Linux ext4slower — Complete Learning Notes & Output Guide

ext4slower is an eBPF/BCC performance-tracing tool that traces ext4 file operations (read, write, open, and fsync) exceeding a user-defined latency threshold, streaming individual event details directly to the terminal.

---

1. What is ext4slower?

While tools like ext4dist aggregate latencies into logarithmic histograms, ext4slower targets outliers.

It answers critical diagnostic questions:

• Which specific files are causing application latency spikes on the ext4 filesystem?
• Which processes (PID and COMM) are issuing slow disk or metadata reads?
• Are database writes or configuration reloads stalling on synchronous disk flushes (fsync)?
• Is a storage bottleneck system-wide or isolated to a single poorly optimized file path?

---

2. Installation

ext4slower is packaged as part of the standard BCC toolkit and requires root privileges (sudo) and kernel development headers:

Debian / Ubuntu

sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)

(Binaries install with a -bpfcc suffix in /usr/sbin/, e.g., ext4slower-bpfcc).

RHEL / Rocky / AlmaLinux / CentOS

sudo dnf install bcc-tools kernel-devel-$(uname -r)

(Executables reside in /usr/share/bcc/tools/ext4slower).

Arch Linux

sudo pacman -S bcc-tools

Verify:

sudo ext4slower -h 2>/dev/null || sudo ext4slower-bpfcc -h

---

3. Basic Syntax

The basic syntax requires specifying a latency threshold in milliseconds. Any ext4 operation taking longer than this threshold is printed live.

sudo ext4slower [threshold_ms]

Example (log any ext4 operation taking longer than 10 milliseconds):

sudo ext4slower 10

---

4. Anatomy of ext4slower Output

When executed, ext4slower runs silently until an operation exceeds the threshold:

sudo ext4slower 5

Raw Output Example

Tracing ext4 operations slower than 5 ms... Hit Ctrl-C to end.
TIME     COMM           PID    T D BYTES   OFF_KB   LAT(ms) FILENAME
15:04:12 postgres       5410   W 0 8192    1048576    14.25 base/16384/2610
15:04:15 python3        8912   R 0 4096    0           8.12 config.json
15:04:20 rsyslogd       1120   S 0 0       0          42.10 syslog
15:04:22 nginx          6102   O 0 0       0           6.50 html/index.html

---

5. Breakdown of Every Output Heading & Field

+----------+--------------+------+---+-------+----------+---------+-------------------+
| TIME     | COMM         | PID  | T | D     | BYTES    | LAT(ms) | FILENAME          |
+----------+--------------+------+---+-------+----------+---------+-------------------+
| 15:04:12 | postgres     | 5410 | W | 0     | 8192     |   14.25 | base/16384/2610   |
+----------+--------------+------+---+-------+----------+---------+-------------------+

5.1 TIME

• Format: HH:MM:SS (wall-clock timestamp).
• Meaning: The exact time at which the slow ext4 operation completed and returned control to the calling process.
• Troubleshooting Significance: Correlates file-system stalls directly with application log error timestamps or database timeout alerts.

5.2 COMM

• Format: String (e.g., postgres, python3, nginx).
• Meaning: The short executable command name of the task that initiated the slow file operation.

5.3 PID

• Format: Numeric integer (e.g., 5410, 8912).
• Meaning: The operating system Process ID executing the call.

5.4 T (Operation Type)

Indicates the specific filesystem routine being tracked:

• R: Read (ext4_file_read_iter).
• W: Write (ext4_file_write_iter).
• O: Open (ext4_file_open).
• S: Sync (ext4_sync_file / fsync).

5.5 D (Direct I/O Flag)

• 0: Buffered I/O (operated through the Linux page cache in RAM).
• 1: Direct I/O (O_DIRECT), bypassing the page cache and forcing direct communication with the block device driver.

5.6 BYTES

• Format: Integer byte size.
• Meaning: The volume of data requested in the read or write operation. For open (O) or sync (S) operations, this typically displays 0.

5.7 OFF_KB

• Format: Integer offset in Kilobytes.
• Meaning: The file offset where the read or write operation began.

5.8 LAT(ms)

• Format: Decimal floating-point number in milliseconds.
• Meaning: The elapsed execution time of the filesystem operation. Because this exceeds your threshold argument (e.g., > 5 ms), it prints to the terminal.

5.9 FILENAME

• Format: Path or relative filename string.
• Meaning: The target file path being accessed on the ext4 mount.

---

6. Real-World Troubleshooting Scenarios

Scenario A: Identifying Slow Database Transaction Commits

A PostgreSQL instance is experiencing high transaction latency. You want to see which WAL (Write-Ahead Log) or data files are stalling fsync:

sudo ext4slower 20

Diagnosis: If fsync (S) operations targeting base/ or pg_wal/ consistently log latencies $>50\,\text{ms}$, the storage controller queue or underlying SSD is failing to flush commits in a timely manner.

---

Scenario B: Detecting Unbuffered Configuration File Reads on Startup

A Python application lags during initialization:

sudo ext4slower 5

Diagnosis: Catches slow file open or read calls on large JSON/YAML configuration bundles or unindexed SQLite files.

---

7. Important Interview Questions & Answers

Q: What is the primary operational difference between ext4slower and ext4dist?

Answer: ext4dist measures the statistical distribution of all ext4 operations, rendering power-of-2 logarithmic histograms to show overall performance health. ext4slower acts as an outlier filter, ignoring fast operations and printing only individual events that exceed a user-defined latency threshold (e.g., $>10\,\text{ms}$), complete with PIDs, filenames, and exact latencies.

Q: Why might an ext4 write operation take longer than expected even when the file is small?

Answer: Even for small writes, an ext4 operation can stall due to journaling locks (jbd2), metadata updates (allocating new blocks or extending inode sizes), or lock contention when multiple threads attempt to acquire exclusive mutex locks on the same file inode simultaneously.

---