Linux mdflush — Complete Learning Notes & Output Guide

mdflush is an eBPF/BCC performance-tracing tool that tracks Multiple Device (MD) software RAID driver flushes. It identifies which processes are forcing metadata or data synchronizations across software RAID arrays or Linux LVM volume groups, logging timestamps, process IDs, and device names.

---

1. What is mdflush?

mdflush stands for:

Multiple Device Flush Tracker

Part of the BCC (BPF Compiler Collection), mdflush instruments the Linux kernel's MD driver subsystem (drivers/md/md.c).

When applications or the kernel subsystem issue sync operations (such as fsync(), sync(), or journal commits like jbd2), software RAID arrays (mdadm) or LVM metadata layers must flush write caches and synchronize stripe caches across underlying physical drives. mdflush intercepts these flush requests.

It answers critical storage accountability questions:

• Which processes (PID and COMM) are triggering disk flush storms across our software RAID or LVM volumes?
• How frequently are software RAID metadata writes or barrier flushes occurring?
• Are background maintenance scripts, backup jobs, or database engines causing unnecessary multi-disk synchronization stalls?

---

2. Installation

mdflush requires root privileges (sudo) and a kernel with eBPF and tracing support enabled.

Debian / Ubuntu

sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)

(On Debian/Ubuntu, BCC tools are often named with a -bpfcc suffix: mdflush-bpfcc located in /usr/sbin/).

RHEL / Rocky / AlmaLinux / CentOS

sudo dnf install bcc-tools kernel-devel-$(uname -r)

(Executables reside in /usr/share/bcc/tools/mdflush).

Arch Linux

sudo pacman -S bcc-tools

Verify:

sudo mdflush -h 2>/dev/null || sudo mdflush-bpfcc -h

---

3. Basic Syntax

To run live system-wide:

sudo mdflush

---

4. Anatomy of mdflush Output

When you run mdflush, it sits quietly until an MD device flush event is triggered by the kernel:

sudo mdflush

Raw Output Example

Tracing md flushes... Hit Ctrl-C to end.
TIME     PID    COMM             DEVICE
14:22:01 5410   postgres         md0
14:22:05 1120   rsyslogd         md1
14:22:12 8912   jbd2/md0-8       md0
14:22:18 1521   sync             md0

---

5. Breakdown of Every Output Heading & Field

+----------+-------+------------------+--------+
| TIME     | PID   | COMM             | DEVICE |
+----------+-------+------------------+--------+
| 14:22:01 | 5410  | postgres         | md0    |
+----------+-------+------------------+--------+

5.1 TIME

• Format: HH:MM:SS (wall-clock timestamp).
• Meaning: The exact time at which the kernel MD subsystem processed the device flush request.

5.2 PID

• Format: Numeric integer (e.g., 5410, 1120).
• Meaning: The Process ID of the task that issued the sync command or triggered the writeback flush.

5.3 COMM

• Format: String (e.g., postgres, rsyslogd, sync, jbd2/md0-8).
• Meaning: The short executable command name of the task performing the flush (derived from task->comm, truncated to 16 characters).

5.4 DEVICE

• Format: String (e.g., md0, md1, md2).
• Meaning: The name of the target Linux software RAID device handle (/dev/mdX) where the flush was dispatched.

---

6. Real-World Troubleshooting Scenarios

Scenario A: Pinpointing Processes Causing Software RAID Latency Spikes

An array built with mdadm experiences periodic write latency spikes and dropped IOPS.

Run mdflush to identify the culprits:

sudo mdflush

Diagnosis: If you see high-frequency flushes originating from an unoptimized logging utility, a backup scanner, or a monitoring agent, you can adjust their scheduling priority or redirect their scratch directories away from the primary RAID array.

---

Scenario B: Verifying Database Journal Flush Frequency

A PostgreSQL database running on a software RAID 10 array exhibits high disk wait times.

Monitor mdflush during peak traffic:

sudo mdflush

Diagnosis: Confirms how often database checkpointing or WAL writes (postgres or jbd2) force full array syncs, helping determine if commit_delay or checkpoint intervals need tuning.

---

7. Important Interview Questions & Answers

Q: Why are software RAID (mdadm) flushes more expensive than flushes on a single physical disk?

Answer: When an application issues an fsync() or barrier write on a single drive, only that drive's volatile cache must be flushed. On a software RAID array (especially RAID 1, 5, or 6), a flush command must synchronize parity blocks, update superblock metadata across all member disks, and ensure that all underlying physical drive controllers (/dev/sda, /dev/sdb, etc.) have committed data to non-volatile storage before the kernel returns success to the caller. This introduces multi-disk coordination overhead.

Q: What kernel functions does mdflush attach to?

Answer: mdflush hooks into the MD driver layer function responsible for handling block device requests with flush flags (REQ_OP_FLUSH or legacy barrier requests) within the Linux kernel source code (drivers/md/md.c).

---