fatrace
Linux fatrace — Complete Learning Notes & Output Guide
fatracestands for File Access Trace. It reports file access events (open, read, write, close) across an entire system in real time using the Linux kernel'sfanotify(File Access Notification) API.
1. What is fatrace?
fatrace monitors file access events across entire mounted file systems without requiring you to set thousands of individual directory watches.
Unlike single-process tracers (strace, ltrace) or filesystem watchers that require recursive tree walking (inotifywait), fatrace listens directly at the VFS/mount layer.
It answers fundamental diagnostic questions:
- Which background processes are writing to the disk and preventing hard drives from spinning down?
- What files are being read or modified during a specific system event or service startup?
- Which daemon is generating sudden write activity on a mounted volume?
- Are temporary files being properly closed after write operations?
- What processes are accessing sensitive configuration or secret files in real time?
2. Installation
fatrace is available in standard package repositories:
Debian / Ubuntu
sudo apt update
sudo apt install fatrace
RHEL / Rocky / AlmaLinux / CentOS (via EPEL)
sudo dnf install epel-release
sudo dnf install fatrace
Arch Linux
sudo pacman -S fatrace
Verify:
fatrace --version
3. Basic Syntax
fatrace requires root privileges (CAP_SYS_ADMIN) to interact with the kernel's fanotify API:
sudo fatrace [options]
Run live system-wide:
sudo fatrace
4. Anatomy of fatrace Output
A standard fatrace event line follows a compact, space-delimited structure:
python3(4512): R /etc/ssl/certs/ca-certificates.crt
dockerd(1420): O /var/lib/docker/engine-id
rsyslogd(812): CW /var/log/syslog
touch(9120): CW /tmp/marker.txt
cat(9125): RC /etc/hosts
With timestamps enabled (sudo fatrace -t):
15:04:12.102345 python3(4512): R /etc/ssl/certs/ca-certificates.crt
Component Breakdown
| Component | Example | Technical Meaning |
|---|---|---|
Timestamp (-t) |
15:04:12.102345 |
Time (Hours:Minutes:Seconds.Microseconds) when the kernel reported the event. |
| Process Name | python3, rsyslogd |
The executable name (comm) of the process performing the file access. |
| PID | (4512), (812) |
The Process ID responsible for the file event. |
| Colon Separator | : |
Separator between the process identifier and event metadata. |
| Event Type(s) | O, R, W, C |
The specific file operation flags emitted by fanotify. |
| File Path | /var/log/syslog |
The absolute file system path of the target file being accessed. |
5. Output Headings & Event Flag Combinations
The event field between the colon and the file path indicates the operations that occurred:
| Flag Character | Event Meaning | Technical Description |
|---|---|---|
O |
Open | The process opened a file descriptor to the file (open() or openat()). |
R |
Read | The process read data from the file (read(), pread(), readv()). |
W |
Write | The process wrote or appended data to the file (write(), pwrite()). |
C |
Close | The process closed the file descriptor (close()). |
Common Event Combinations
fatrace often aggregates events occurring within the same notification window:
CW(Close after Write): Indicates a file was modified or created, written to, and then closed. This is the primary signature of an active disk write or log update.CR(Close after Read): Indicates a file was opened for reading and has now been closed.RC(Read & Close): The file was read and closed in rapid succession.Ofollowed byW: A file opened with write access flags (O_WRONLYorO_RDWR).
6. Essential Command Options & Cheat Sheet
| Option Flag | Description | Practical Example |
|---|---|---|
**-t, --timestamp** |
Add microsecond-accurate wall-clock timestamps to each line. | sudo fatrace -t |
**-c, --current-mount** |
Limit monitoring strictly to the current partition/mount point where fatrace is run. |
cd /var && sudo fatrace -c |
**-f <types>, --filter** |
Filter by specific event types: O (Open), R (Read), W (Write), C (Close). |
sudo fatrace -f W |
**-C <cmd>, --command** |
Trace only processes matching a specific command name. | sudo fatrace -C nginx |
**-p <PID>, --ignore-pid** |
Ignore events generated by a specific Process ID (repeatable). | sudo fatrace -p 1243 |
-s <seconds> |
Automatically stop execution after a set number of seconds. | sudo fatrace -s 30 |
**-o <file>, --output** |
Write output directly to a file on disk. | sudo fatrace -o /tmp/fatrace.log |
7. Architecture: fatrace (fanotify) vs. inotify vs. opensnoop
Understanding where fatrace fits among Linux filesystem monitoring tools:
+-------------------------------------------------------------------+
| MONITORING TOOLS |
| |
| inotify / inotifywait |
| * Recursive watch limits (fs.inotify.max_user_watches). |
| * Bound to specific inodes/directories. |
| * Best for: Local folder synchronization (e.g., Live Reloaders).|
| |
| fatrace (fanotify) |
| * Mount-point level monitoring. |
| * Captures Open, Read, Write, and Close. |
| * Zero watch limits; lightweight VFS hooks. |
| * Best for: Catching write thrashing, power-saving spindown. |
| |
| opensnoop (eBPF) |
| * Intercepts open/openat system calls globally via eBPF. |
| * Shows error codes, latency, and open flags. |
| * Does not monitor reads/writes directly. |
| * Best for: Debugging missing files, file permissions, latency. |
+-------------------------------------------------------------------+
| Dimension | fatrace (fanotify) |
inotifywait (inotify) |
opensnoop (eBPF) |
|---|---|---|---|
| Kernel Interface | fanotify(7) |
inotify(7) |
eBPF (kprobe / tracepoint) |
| Scope | Entire mount point | Specific directories | Entire OS (global syscalls) |
| Tracks Read/Write? | Yes (R, W) |
Yes (MODIFY, ACCESS) |
No (only open calls) |
| Watch Limits | None | Limited by sysctl |
None |
| Root Required? | Yes (CAP_SYS_ADMIN) |
No | Yes (CAP_BPF / root) |
8. Real-World Troubleshooting Scenarios
Scenario A: Finding What Keeps Waking Up the Disk (Spindown Blocker)
On laptops or storage servers configured to spin down mechanical drives to conserve power, an unknown background process keeps waking the disk:
sudo fatrace -f W -t -s 60
Output:
15:10:02.102345 updatedb(12890): W /var/lib/mlocate/mlocate.db.tmp
15:10:20.450123 tracker-miner(4512): CW /home/user/.cache/tracker/meta.db
Diagnosis: updatedb and a desktop search indexing daemon (tracker-miner) are performing periodic disk writes, preventing low-power sleep states.
Scenario B: Catching High-Frequency Write Thrashing
Storage monitoring tools (iostat) report high write IOPS, but you need to know which files are continuously being hit:
sudo fatrace -f W | head -n 30
Output:
node(1842): W /var/log/app/debug.log
node(1842): W /var/log/app/debug.log
node(1842): W /var/log/app/debug.log
Diagnosis: A Node.js application is writing unbuffered debug logs on every incoming request, generating disk I/O thrashing.
Scenario C: Auditing Access to Sensitive Secret Files
Verify whether any unexpected process reads an SSL certificate private key or production credential:
sudo fatrace -f R | grep -E "id_rsa|private.key|\.env"
Output:
backup_agent(3210): R /etc/ssl/private/server.key
Diagnosis: Confirms that only the authorized backup agent read the private key.
9. Key Limitations & Gotchas
- Virtual Filesystems Unsupported:
fanotifymonitors block device mount points. It cannot trace synthetic pseudo-filesystems such as/proc,/sys, or/dev. Usestraceor eBPF tools if you need to observe reads to/proc. - Network Filesystems (NFS/CIFS): Support for remote network file systems can be incomplete depending on how client caches handle local VFS reads and writes without notifying the server.
- Overhead from Massive Read Tracing: Running
fatrace -f Ron a heavily loaded server captures every single read operation across all files, which can flood standard output and add noticeable kernel notification overhead. Always filter by event type (e.g.,-f Wor-f C).
10. Important Interview Questions & Answers
Q: What kernel mechanism does fatrace use, and how does it avoid the limitations of inotify?
Answer: fatrace uses the Linux kernel's fanotify (File Access Notification) API. Traditional inotify requires user space to explicitly register a watch descriptor for every individual file or directory; traversing large trees exhausts the kernel watch ceiling (fs.inotify.max_user_watches). In contrast, fanotify registers watches at the entire filesystem mount point level (FAN_MARK_MOUNT), capturing events across all files on the partition with a single system call.
Q: Why doesn't fatrace display read or write events for files inside /proc or /sys?
Answer: /proc and /sys are virtual, memory-based pseudo-filesystems implemented directly via kernel callbacks rather than standard block storage backing layers. The fanotify API hooks into the VFS layer for filesystem mounts holding disk inodes; it does not emit notifications for non-cacheable synthetic pseudo-file handles.
Q: What does an event line showing CW signify?
Answer: CW stands for Close after Write (FAN_CLOSE_WRITE). It indicates that a process opened a file with write access permissions (O_WRONLY or O_RDWR), performed modifications, and subsequently closed the file descriptor, flushing changes to the filesystem.