opensnoop

@amitmund September 10, 2026

Linux `opensnoop` — Complete Learning Notes & Output Guide

opensnoop is an eBPF-based tracing tool that monitors file open() and openat() system calls across the entire Linux operating system in real time. It reveals which processes are opening files, whether those attempts succeed or fail, the access flags used, and the latency of the operation.

1. What is `opensnoop`?

opensnoop stands for:

Open System Call Snooper

Originally created by Brendan Gregg as part of the BCC (BPF Compiler Collection) and modern libbpf-tools, opensnoop runs lightweight eBPF programs inside the Linux kernel.

It answers critical production questions:

Which processes are constantly opening and closing files?
What configuration files is an application attempting to read on startup?
Which file access attempts are failing due to missing files or bad permissions?
Which short-lived background processes are causing unexpected disk or page-cache activity?
How long does the VFS layer and file system take to resolve path lookups and open descriptors?

2. Installation

opensnoop requires root privileges and an eBPF-capable Linux kernel (Linux 4.4+ for basic features, 5.2+ for full libbpf features).

Debian / Ubuntu

sudo apt update
sudo apt install bpfcc-tools libbpf-tools linux-headers-$(uname -r)

(On Debian/Ubuntu, BCC tools are often installed with a -bpfcc suffix, e.g., opensnoop-bpfcc or in /usr/sbin/opensnoop-bpfcc).

RHEL / Rocky / AlmaLinux / CentOS

sudo dnf install bcc-tools

(Binaries reside in /usr/share/bcc/tools/opensnoop).

Arch Linux

sudo pacman -S bcc-tools

Verify:

sudo opensnoop -h || sudo opensnoop-bpfcc -h

3. Basic Syntax

sudo opensnoop [options]

To run with live terminal output:

sudo opensnoop

4. Default Output Breakdown

When running opensnoop without arguments, it captures system-wide file opens:

PID    COMM               FD ERR PATH
1243   systemd-journal     9   0 /var/log/journal/634.../system.journal
4512   python3             3   0 /etc/ssl/certs/ca-certificates.crt
4512   python3            -1   2 /home/app/.env
8910   nginx               7   0 /var/www/html/index.html
9122   cat                -1  13 /etc/shadow

Element-by-Element Explanation

Heading	Example	Technical Meaning
`PID`	`4512`	The operating system Process ID of the thread that invoked the open syscall.
`COMM`	`python3`	The short command/executable name (truncated to 16 characters by the kernel task struct).
`FD`	`3`, `9`, `-1`	The allocated File Descriptor integer returned to user space. A value of `-1` signifies that the open call failed.
`ERR`	`0`, `2`, `13`	The numeric error status (`errno`). `0` indicates success; non-zero values correspond to POSIX standard error codes.
`PATH`	`/etc/shadow`	The target file system path string requested by the process.

5. Extended Headings & Diagnostic Flags

Adding flags provides extra columns such as timestamps, user IDs, open flags, and latency measurements:

sudo opensnoop -Tu -d -e

Example Output:

TIME(s)        UID   PID    COMM             FD ERR FLAGS    LAT(ms) PATH
15:04:12.102  1000  4512   python3           3   0 O_RDONLY    0.04 /etc/app.conf
15:04:12.105  1000  4512   python3          -1   2 O_RDONLY    0.01 /etc/app.override.conf
15:04:12.890     0  1243   systemd-journal   9   0 O_WRONLY    0.12 /var/log/syslog

Detailed Explanation of Extended Columns

5.1 `TIME(s)` or `TIME` (via `-T` or `-t`)

Meaning: Wall-clock time or seconds since trace start when the openat() call completed.
Troubleshooting Significance: Correlates file opens directly with application log entries, API latency spikes, or automated cron triggers.

5.2 `UID` (via `-u`)

Meaning: The numeric Real User ID owning the process executing the call (e.g., 0 for root, 1000 for standard user).
Troubleshooting Significance: Detects privilege leaks or verifies that a worker daemon is correctly dropped to an unprivileged service account.

5.3 `FLAGS` (via `-e`)

Meaning: The symbolic file access modes and intent bitmasks passed into the kernel.
Common Values:
O_RDONLY: Opened for read-only access.
O_WRONLY: Opened for write-only access.
O_RDWR: Opened for bidirectional read and write operations.
O_CREAT: Creates the file if it does not already exist.
O_TRUNC: Truncates existing file content down to 0 bytes upon opening.
O_APPEND: Appends new writes to the end of the file.
O_CLOEXEC: Automatically closes the file descriptor when calling execve() to spawn sub-processes.
O_NONBLOCK: Prevents the open call from blocking on FIFO pipes or device special files.

5.4 `LAT(ms)` or `LAT(us)` (via `-d`)

Meaning: The total latency (in milliseconds or microseconds) between entering the open system call and returning to user space.
Troubleshooting Significance: High open latency indicates slow path resolution in the VFS layer, disk contention on cold directory metadata traversal, or network file system (NFS/Ceph) lag.

6. Decoding the `ERR` Column: Common `errno` Codes

When FD is -1, ERR displays the numeric kernel error code. Knowing these numbers speeds up debugging:

`ERR` Code	Symbolic Constant	Plain-English Meaning	Common Cause
`0`	`SUCCESS`	Call succeeded	File was located and opened properly.
`2`	`ENOENT`	No such file or directory	Missing `.env`, misplaced config file, or binary searching through library paths.
`13`	`EACCES`	Permission denied	Linux DAC permissions or read/write bit flags forbid access to the UID.
`17`	`EEXIST`	File exists	File already present when opened with `O_CREAT
`20`	`ENOTDIR`	Not a directory	A component of the path prefix is not a directory.
`24`	`EMFILE`	Too many open files	Process hit its `ulimit -n` maximum file descriptor limit.
`28`	`ENOSPC`	No space left on device	Storage filesystem or inode table is 100% full.
`30`	`EROFS`	Read-only file system	Process attempted write access (`O_WRONLY` / `O_RDWR`) on a read-only mount.

7. Practical Filtering Flags Cheat Sheet

Flag	Description	Command Example
`-x`	Show only failed open calls (`ERR != 0`).	`sudo opensnoop -x`
`-p <PID>`	Trace only a specific Process ID.	`sudo opensnoop -p 3482`
`-n <name>`	Trace only processes matching a specific name.	`sudo opensnoop -n nginx`
`-u <UID>`	Trace only actions by a specific user.	`sudo opensnoop -u 1001`
`-e`	Decode and display open flags (`O_RDONLY`, etc.).	`sudo opensnoop -e`
`-T`	Print full timestamps for each event.	`sudo opensnoop -T`
`-d`	Measure and display open call latency in ms.	`sudo opensnoop -d`

8. `opensnoop` vs. `strace`

+-------------------------------------------------------------+
|                           strace                            |
|  - Uses ptrace(2)                                           |
|  - Pauses process on syscall enter & exit (2 context switches) |
|  - Heavy overhead (10x-100x slowdown)                      |
|  - Traces single process tree                               |
+-------------------------------------------------------------+
                              vs
+-------------------------------------------------------------+
|                         opensnoop                           |
|  - Uses eBPF kernel probes / tracepoints                    |
|  - Zero context switches back to user space for filtering   |
|  - Negligible overhead (safe in heavy production)           |
|  - Traces entire operating system simultaneously            |
+-------------------------------------------------------------+

Feature	`opensnoop` (eBPF)	`strace` (`ptrace`)
System-wide Visibility	Yes (watches all PIDs at once)	No (must attach to specific PID/tree)
Production Overhead	Negligible (~nanoseconds per event)	Very High (can crash high-throughput servers)
Kernel Requirements	Linux 4.4+ with eBPF enabled	Any Linux kernel
Root Privileges	Always required	Only required for other users' PIDs
Scope of Calls	`open`, `openat`, `openat2`	All 400+ Linux system calls

9. Real-World Troubleshooting Scenarios

Scenario A: Diagnosing Missing Application Configuration

A backend service fails to start, logging a vague "Configuration missing" error:

sudo opensnoop -n my_service -x

Output:

PID    COMM        FD ERR PATH
10245  my_service  -1   2 /home/app/.config/settings.yaml
10245  my_service  -1   2 /etc/my_service/prod.yaml

Conclusion: The service searched its local config path, then fell back to /etc/ and failed because both files are absent (ERR = 2).

Scenario B: Detecting Unauthorized File Access / Security Probing

Monitor attempts by malicious or unprivileged processes to access sensitive files:

sudo opensnoop -x -e

Output:

PID    COMM     FD ERR FLAGS    PATH
14201  script   -1  13 O_RDONLY /etc/shadow
14201  script   -1  13 O_RDONLY /root/.ssh/id_rsa

Conclusion: Process 14201 is actively probing private keys and system password hashes, failing with Permission Denied (ERR = 13).

Scenario C: Identifying Disk Spikes from Micro-Loggers

A server exhibits persistent storage write IOPS, but iotop refreshes too slowly to identify the cause:

sudo opensnoop -e | grep -E "O_WRONLY|O_CREAT|O_APPEND"

Conclusion: Catches short-lived scripts and ephemeral worker tasks that open log or state files for writes and terminate before standard interval pollers can sample them.

10. Important Interview Questions & Answers

Q: What kernel system calls does `opensnoop` attach to?

Answer: opensnoop attaches eBPF tracepoints or kprobes to sys_enter_open, sys_enter_openat, and modern kernels' sys_enter_openat2, along with their corresponding sys_exit return hooks to measure latency and capture the returned file descriptor or error code.

Q: Why is `opensnoop` safe for production while running `strace -e open` on all PIDs is not?

Answer: strace relies on the ptrace(2) interface, forcing the kernel to pause process execution, execute two context switches per call, and send signals to the debugger. In contrast, opensnoop loads JIT-compiled eBPF bytecode directly into the kernel execution path. Filtering logic (e.g., matching by PID or error code) runs in-kernel without pausing application threads or incurring user-kernel context switches.

Q: What does `FD = -1` with `ERR = 24` indicate in `opensnoop`?

Answer: ERR = 24 corresponds to EMFILE ("Too many open files"). It means the calling process has hit its allocated open file descriptor resource ceiling (RLIMIT_NOFILE / ulimit -n), preventing it from opening any further files, sockets, or pipes.

Linux opensnoop — Complete Learning Notes & Output Guide

1. What is opensnoop?