lsof

@amitmund September 10, 2026

Linux lsof — Complete Learning Notes & Output Guide

lsof stands for List Open Files. Because Unix and Linux treat almost everything as a file (regular files, directories, block/character devices, pipes, shared libraries, and network sockets), lsof serves as a comprehensive window into active process resource handles.

1. What is lsof?

lsof inspects kernel data structures to report information about files currently opened by active user-space processes.

It answers fundamental diagnostic questions:

  • Which process is holding a port open (e.g., :80 or :5432)?
  • Why does umount /mnt/data fail with "target is busy"?
  • Why hasn't disk space been freed up after deleting a multi-gigabyte log file?
  • What dynamic shared libraries (.so) or configuration files is a process using?
  • Is an application leaking file descriptors and approaching its ulimit -n limit?

2. Installation

lsof is standard across Linux distributions:

Debian / Ubuntu

sudo apt update
sudo apt install lsof

RHEL / Rocky / AlmaLinux / CentOS

sudo dnf install lsof

Arch Linux

sudo pacman -S lsof

Verify:

lsof -v

3. Basic Syntax

lsof [options] [file_or_path]

Run without arguments (lists every open file on the system; best run with sudo to view files owned by other users):

sudo lsof

4. Default Output Breakdown

A standard lsof invocation produces a 9-column output table:

COMMAND    PID   USER   FD      TYPE             DEVICE SIZE/OFF    NODE NAME
systemd      1   root  cwd       DIR              259,2     4096       2 /
systemd      1   root  rtd       DIR              259,2     4096       2 /
systemd      1   root  txt       REG              259,2  1873992  262150 /usr/lib/systemd/systemd
systemd      1   root  mem       REG              259,2  3215280  262164 /usr/lib/x86_64-linux-gnu/libc.so.6
systemd      1   root    0u      CHR                1,3      0t0       6 /dev/null
dockerd   1420   root    7u     IPv6              41205      0t0     TCP *:2375 (LISTEN)
python3   5812   app     3r      REG              259,2 10485760 1052189 /var/log/app.log (deleted)
nginx     6102  nginx    6u     IPv4              52184      0t0     TCP 192.168.1.50:80->192.168.1.10:54210 (ESTABLISHED)

Quick Overview of Headings

Heading Example Meaning
COMMAND systemd, python3 The command or executable name running the process.
PID 1, 5812 The Process ID that opened the file.
USER root, nginx The operating system username owning the process.
FD cwd, txt, mem, 3r, 7u The File Descriptor number and its access/lock mode.
TYPE REG, DIR, CHR, IPv4 The node type of the file.
DEVICE 259,2 Major and minor device numbers containing the filesystem.
SIZE/OFF 4096, 0t0, 10485760 The size of the file or current read/write byte offset.
NODE 2, 262150, TCP The filesystem inode number or protocol identifier.
NAME /var/log/app.log, *:2375 The target file path, link, or network socket address.

5. Deep-Dive into Output Headings

5.1 COMMAND

  • Meaning: The base name of the executable that initiated the process (truncated to 15 characters by default).
  • Extended Option: Adding +c 0 forces lsof to print the full, untruncated command name.

5.2 PID & TID

  • Meaning: The process ID.
  • Threads: When tracing with the -K flag, lsof adds a TID (Task/Thread ID) column, showing which specific thread inside a multi-threaded application owns the open file descriptor.

5.3 USER

  • Meaning: The login name or User ID (UID) under which the process is executing.

5.4 FD (File Descriptor)

This is the most critical column for system diagnostics. It consists of two components:

  1. Special File Descriptors (System handles not represented by standard numbers):
  • cwd: Current Working Directory of the process.
  • rtd: Root Directory (useful for spotting chroot or containerized paths).
  • txt: Program code/text segment (the executable binary itself mapped into memory).
  • mem: Memory-mapped file (such as shared libraries like libc.so, dynamic plugins, or mmap regions).
  • del: A Linux mapping for a file that has been unlinked/deleted from disk.
  1. Numeric File Descriptors followed by Access Mode Characters:
  • 0, 1, 2: Standard Input (stdin), Standard Output (stdout), and Standard Error (stderr).
  • 3, 4, 5...: Custom file descriptors opened by the program code.

Access Mode Suffixes

  • r: Opened for read access.
  • w: Opened for write access.
  • u: Opened for read and write access.
  • : Mode unknown or not applicable.

File Lock Suffixes (Follows mode)

  • R: Read lock on part of the file.
  • r: Read lock on the entire file.
  • W: Write lock on part of the file.
  • w: Write lock on the entire file.
  • U: Lock of unknown type.

Example: 3uW indicates file descriptor 3 is open for read/write (u) and holds a write lock on the entire file (W).

5.5 TYPE

Specifies the architectural type of the file node:

Type Code Description Example Target
REG Regular disk file Log files, binaries, database tables (.db).
DIR Directory Directories currently open or traversed.
CHR Character special device /dev/null, /dev/urandom, terminal devices (/dev/pts/0).
BLK Block special device Physical storage partitions (/dev/sda1, /dev/nvme0n1p1).
FIFO First-In, First-Out Named pipe IPC mechanisms.
unix UNIX domain socket Inter-process sockets (e.g., /run/systemd/private, /tmp/mysql.sock).
**IPv4 / IPv6** Internet network sockets TCP or UDP connections, listening services.
netlink Netlink socket Kernel-to-user-space routing and network configuration pipes.
a_inode Anonymous inode epoll instances, timerfd, eventfd, signalpipe.

5.6 DEVICE

  • Meaning: The major and minor device numbers identifying the device hosting the file, separated by a comma (e.g., 259,2).
  • Disk Matching: Corresponds to the major:minor numbers visible in ls -l /dev/ or /proc/partitions.

5.7 SIZE/OFF

  • Meaning: Displays either the total file size in bytes or the current file offset pointer within the file.
  • Offset Notation: An offset is typically designated with a 0t prefix followed by decimal digits (e.g., 0t1024 means the file pointer is currently at byte 1024).

5.8 NODE

  • Meaning: Displays the underlying inode number on the filesystem for regular files/directories, or the protocol identifier/port number for sockets.
  • Diagnostics: Inode numbers allow you to confirm whether two processes are accessing the identical physical disk file even if they use different symlinks or paths.

5.9 NAME

  • Meaning: The mount point, absolute file path, pipe identifier, or network endpoint.
  • Network Socket Format: [local_ip]:[port]->[remote_ip]:[port] (STATE)
  • Example: 192.168.1.50:80->10.0.0.4:54321 (ESTABLISHED)

  • Example: *:22 (LISTEN)

  • Deleted Indicator: (deleted) denotes that the file was unlinked from the directory tree by rm, but remains preserved on disk by the kernel because this process still holds an open file handle.

6. Essential Diagnostic Flags & Cheat Sheet

Flag Description Practical Example
-i List all network files (IPv4, IPv6, TCP, UDP). sudo lsof -i
-i :<port> Find what is using a specific port. sudo lsof -i :8080
-i TCP -sTCP:LISTEN List only listening TCP sockets. sudo lsof -i TCP -sTCP:LISTEN
-n Suppress hostname DNS resolution (prevents blocking). sudo lsof -n -i
-P Suppress port number conversion to service names (80 vs http). sudo lsof -nP -i :80
-p <PID> List all files opened by a specific PID. sudo lsof -p 4210
+D <path> Recursively search for open files inside a directory tree. sudo lsof +D /mnt/storage
-u <user> List files opened by a specific user. sudo lsof -u www-data
-c <comm> Filter by command name prefix. sudo lsof -c nginx
-t Terse mode; output only PIDs (ideal for pipe scripts). kill -9 $(sudo lsof -t -i :8080)
+L1 Show open files with 0 link count (unlinked/deleted files). sudo lsof +L1

Performance Rule of Thumb: Always pair -i with **-n and -P** (lsof -nP -i). By default, lsof attempts reverse-DNS lookups on every IP and port translation via /etc/services, which can cause significant delays on busy servers.

7. Real-World Troubleshooting Scenarios

Scenario A: "Target is busy" when unmounting storage

You attempt to unmount a disk partition, but the kernel blocks the operation:

umount /mnt/backup
# umount: /mnt/backup: target is busy.

Find the blocking process:

sudo lsof +D /mnt/backup

Output:

COMMAND   PID USER FD   TYPE DEVICE SIZE/OFF   NODE NAME
bash    18420 root cwd   DIR  259,3     4096      2 /mnt/backup/logs

Diagnosis: Process 18420 (a bash shell) has set its current working directory (cwd) to /mnt/backup/logs. Exiting or moving that shell frees the mount point.

Scenario B: Reclaiming disk space from deleted files

Disk usage (df -h) reports 100% utilization, but directory search (du -sh /*) shows minimal usage. A large log file was deleted using rm, but disk space was not reclaimed:

Identify processes holding deleted file handles:

sudo lsof +L1
# or
sudo lsof -nP | grep "(deleted)"

Output:

COMMAND   PID USER   FD   TYPE DEVICE   SIZE/OFF   NODE NAME
app.bin  4210 root    3w   REG  259,2 5368709120 131075 /var/log/app.log (deleted)

Diagnosis: Process 4210 holds an active write handle (3w) on the 5 GB file. Remediation without restarting the process: Truncate the file via the proc filesystem:

: > /proc/4210/fd/3

This reduces the file size to 0 bytes and instantly reclaims disk blocks.

Scenario C: Resolving a port bind conflict (EADDRINUSE)

A backend server fails to start, returning bind: address already in use [::]:5432:

Identify the process holding the port:

sudo lsof -nP -i TCP:5432 -sTCP:LISTEN

Output:

COMMAND   PID     USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
postgres 1120 postgres    6u  IPv4  28410      0t0  TCP *:5432 (LISTEN)

Diagnosis: A lingering PostgreSQL daemon (PID 1120) is already bound to that port.

Scenario D: Debugging File Descriptor Exhaustion

An application logs EMFILE: Too many open files.

Count open file descriptors for the target process:

sudo lsof -p 3482 | wc -l

Check against the process system limits:

cat /proc/3482/limits | grep "Max open files"

Output:

Max open files            1024                 4096                 files

Diagnosis: If lsof -p 3482 | wc -l approaches 1024, the process has leaked file or socket handles and hit its RLIMIT_NOFILE soft limit.

8. Important Interview Questions & Answers

Q: Why does lsof -i run slowly on production servers, and how do you fix it?

Answer: By default, lsof performs synchronous reverse DNS lookups on every IP address and translates port numbers into service names via /etc/services. On a server handling thousands of network sockets, this introduces significant network and resolution latency. You fix this by using lsof -nP -i (-n disables network IP-to-hostname lookups; -P disables port-to-service name resolution).

Q: What does FD column mem signify, and why can it appear multiple times for one binary?

Answer: mem represents a memory-mapped file (mmap). Whenever a dynamically linked executable loads shared libraries (.so files), dynamic configurations, or localized font/locale files into its virtual memory address space, the kernel tracks these active mappings as open handles. Each mapped library appears on its own line under mem.

Q: What is the operational difference between lsof +d <dir> and lsof +D <dir>?

Answer: +d inspects open files located strictly at top-level inside the specified directory without traversing subdirectories. +D performs a full recursive traversal through all child directories and sub-mounts within that path (computationally slower, but necessary when identifying why an entire filesystem tree cannot be unmounted).

0 Likes
2 Views
0 Comments

Filters

No filters available for this view.

Reset All