Linux vmstat — Complete Learning Notes & Output Guide

vmstat (Virtual Memory Statistics) is a foundational performance monitoring utility from the procps-ng suite. It provides a compact, continuous summary of system throughput: process scheduling runqueues, virtual and physical memory paging, disk block transfers, interrupt/context-switch rates, and CPU utilization.

---

1. What is vmstat?

vmstat samples and computes rates by parsing kernel data files in /proc—primarily /proc/meminfo, /proc/stat, and /proc/vmstat.

Because it summarizes virtually every major hardware subsystem in a single 80-character line, it is typically the very first command run during performance triage to determine whether a performance problem is bound by:

• CPU saturation or scheduling runqueue backlogs (r, us, sy)
• Physical memory exhaustion or disk paging (si, so)
• Storage controller saturation and blocking I/O (b, wa, bi, bo)
• System call and interrupt storms (in, cs)

---

2. Installation & Availability

vmstat is bundled inside the procps-ng (or legacy procps) package, pre-installed on every Linux distribution.

vmstat -V

If missing on minimal container base images:

# Debian / Ubuntu
sudo apt update && sudo apt install procps

# RHEL / Rocky / AlmaLinux / CentOS
sudo dnf install procps-ng

# Arch Linux
sudo pacman -S procps-ng

---

3. Basic Syntax & Core Option Flags

vmstat [options] [delay [count]]

CRITICAL RULE OF vmstat: The very first line of output printed by vmstat displays averages since system boot, NOT the activity during the last second. Always provide an interval delay (e.g., vmstat 1) and ignore the first line when evaluating current performance.

Essential Command Flags

Flag	Description	Practical Example
delay	Sampling interval in seconds.	vmstat 1
count	Number of sample reports before exiting.	vmstat 1 5
**`-S, --unit	K	m	M>`**	Display memory metrics in user-specified units (m = $10^6$ MB, M = $2^{20}$ MiB). Default is KiB.	vmstat -S M 1
-a, --active	Display active vs. inactive memory instead of buffer/cache allocations.	vmstat -a 1
-d, --disk	Display aggregate disk read/write operation statistics per disk device.	vmstat -d
-D, --disk-sum	Summarize global disk activity counters (reads, writes, partitions).	vmstat -D
-s, --stats	Print a one-shot vertical ledger of kernel counters since boot.	vmstat -s
-t, --timestamp	Append a wall-clock timestamp to each line (essential for log files).	vmstat -t 1
-w, --wide	Wide mode; expands column widths for systems with hundreds of gigabytes of RAM.	vmstat -w 1

---

4. Anatomy of Default Output (vmstat 1)

Running vmstat with a 1-second refresh interval:

vmstat -t 1

Raw Output Example

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- -----timestamp-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st                 UTC
 2  0      0 1420512  45120 4891240    0    0    12    45 1204 2450 12  4 84  0  0 2026-09-11 02:10:01
 8  1      0 1419840  45120 4891240    0    0     0   840 8910 14200 68 28  4  0  0 2026-09-11 02:10:02
 0  4 524288  124100  12040  245100  450 1200  4500  8900 4510  5102  4  6 12 78  0 2026-09-11 02:10:03

---

5. Breakdown of Every Output Heading & Field

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0      0 1420512  45120 4891240    0    0    12    45 1204 2450 12  4 84  0  0

vmstat groups its metrics into 6 distinct architectural domains:

5.1 procs (Process Scheduler Queues)

Field	Meaning	Diagnostic Significance
r	Runnable / Runqueue	The number of tasks runnable and either actively running on a CPU core or waiting in scheduler runqueues. If r consistently exceeds the total number of physical CPU cores, the system is CPU-saturated and tasks are experiencing scheduling delays.
b	Blocked	The number of tasks placed in uninterruptible sleep (D state). These tasks are stalled waiting for disk I/O, paging, NFS, or kernel locks. Non-zero b indicates storage or hardware bottlenecks.

5.2 memory (Physical Memory Utilization, in KiB)

Field	Meaning	Diagnostic Significance
swpd	Virtual memory used	Amount of swap space currently allocated. Normal if non-zero, provided si and so remain zero.
free	Free RAM	Completely idle, unallocated memory. A low number is normal in Linux because the kernel uses idle RAM for caches.
buff	Buffer memory	Memory used by the kernel to buffer raw disk blocks (temporary block I/O storage).
cache	Page cache	Filesystem page cache and shared memory (in-memory cached copies of files on disk).

(Note: When running vmstat -a, buff and cache are replaced by inact [inactive memory eligible for reclamation] and active [memory accessed recently]).

5.3 swap (Paging Activity, in KiB/s)

Field	Meaning	Diagnostic Significance
si	Swap In	Amount of memory paged in from disk swap space into physical RAM per second.
so	Swap Out	Amount of memory paged out of physical RAM onto disk swap space per second.

PERFORMANCE WARNING: Having a positive swpd value is harmless. However, persistent non-zero values in si and so indicate active swap thrashing. The system is out of physical RAM, and CPU threads are blocking while waiting for memory pages to be read from slow disk media.

5.4 io (Block Storage Transfers, in Blocks/s)

Linux filesystem blocks are typically 1024 bytes (1 KiB).

Field	Meaning	Diagnostic Significance
bi	Blocks In	Blocks received from a block device (disk reads) per second.
bo	Blocks Out	Blocks sent to a block device (disk writes) per second.

5.5 system (Interrupts & Context Switches)

Field	Meaning	Diagnostic Significance
in	Interrupts	Total number of system interrupts per second, including clock timer ticks and hardware IRQs.
cs	Context Switches	Number of voluntary and involuntary task context switches per second. High cs (>100k/s) indicates thread lock contention, socket polling storms, or hyper-threaded scheduling friction.

5.6 cpu (Percentage of Total CPU Time)

These values represent percentages summing to $100\%$ across all cores:

Field	Meaning	Diagnostic Significance
us	User Time	Time spent executing un-niced user-space application code.
sy	System Time	Time spent executing kernel code and system calls. If sy exceeds $20\text{--}30\%$, check for high system call frequency or lock contention.
id	Idle Time	Time spent with no runnable tasks.
wa	Wait I/O (iowait)	Time CPU cores spent idle while at least one I/O request was in flight. High wa indicates disk/NFS bottlenecks, not slow CPUs.
st	Steal Time	Time stolen by the underlying hypervisor (KVM, Xen, ESXi) to service other virtual machines sharing the physical CPU. High st (>5%) indicates a noisy neighbor on a virtualized host.

---

6. Granular Inspection Modes

---

6.1 Disk Device Statistics: vmstat -d

Displays reads, writes, and active I/O progress per disk partition:

vmstat -d

Raw Output Example

disk- ------------reads------------ -----------writes----------- -----IO------
       total merged sectors      ms   total merged sectors      ms    cur    sec
sda    45120   1200 4521000  145020  184510  42100 8910400  412050      0     42
sdb    12040    450 1204000   45120   89100  12000 4120000  189010      1     12

• reads/writes - total: Successfully completed read/write requests.
• reads/writes - merged: Contiguous requests merged into a single I/O by the block scheduler.
• reads/writes - sectors: Number of sectors read/written ($512\text{ bytes}$ per sector).
• reads/writes - ms: Total time in milliseconds spent servicing reads or writes.
• IO - cur: Number of I/O operations currently in progress (active queue depth).
• IO - sec: Total seconds spent doing I/O.

---

6.2 System Cumulative Counters: vmstat -s

Prints an instant snapshot of cumulative kernel metrics since boot:

vmstat -s

     131820148 K total memory
      95529370 K used memory
      59320600 K active memory
      26300700 K inactive memory
      33351200 K free memory
         45120 K buffer memory
       4891240 K swap cache
     128880570 K total swap
        524288 K used swap
     128356282 K free swap
    1452104521 CPU ticks user
       4120541 CPU ticks nice
     412054120 CPU ticks system
    8910452100 CPU ticks idle
      14205120 CPU ticks IO-wait
         45120 CPU ticks IRQ
        184512 CPU ticks softirq
          1200 CPU ticks stolen
    1245012450 pages paged in
    1845120410 pages paged out
        124500 pages swapped in
        845120 pages swapped out
    1420512400 interrupts
    4891204512 CPU context switches
    1684201452 forks

---

7. Real-World Troubleshooting Scenarios

Scenario A: Identifying CPU Saturation & Runqueue Starvation

An application's response times spike, but individual process metrics are inconclusive.

Run vmstat 1:

 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
18  0      0 124510  45120 4891240    0    0     0    12 4510 8910 88 12  0  0  0
22  0      0 124510  45120 4891240    0    0     0     8 4620 9102 90 10  0  0  0

Diagnosis:

• The server has 4 CPU cores, but the runqueue column **r is sitting at 18 and 22**.
• us is at ~90%, and **id is 0**.
• Conclusion: 18–22 threads are contending for 4 cores. The bottleneck is user-space computational load. Scale out worker processes across more cores or hosts.

---

Scenario B: Diagnosing Active Swap Thrashing

A system becomes sluggish and SSH sessions lag.

Run vmstat 1:

 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  4 2097152  34100   2100  45100 4500 8900  4600  8950 3200 1850  2  8  0 90  0
 0  5 2101248  32500   2100  44800 5100 9200  5150  9300 3410 1920  1  9  0 90  0

Diagnosis:

• si ($4.5\text{--}5.1\text{ MB/s}$) and so ($8.9\text{--}9.2\text{ MB/s}$) are continuously active.
• wa is at 90%, and b (blocked processes) is at 4–5.
• Conclusion: The machine has exhausted physical RAM and is thrashing virtual memory against swap disk space. The CPU is idle (id 0), blocked almost entirely in I/O wait (wa 90) while moving memory pages to and from disk.

---

Scenario C: Detecting Virtual Machine Hypervisor Steal Time

A cloud virtual machine (e.g., AWS EC2, GCP Compute) experiences latency spikes even though total utilization appears low.

Run vmstat 1:

 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0      0 891240  45120 214500    0    0     0     0  450  820 15  5 45  0 35
 3  0      0 891240  45120 214500    0    0     0     0  480  910 18  4 42  0 36

Diagnosis:

• st (Steal Time) is sustained at $35\text{--}36\%$.
• Conclusion: The physical CPU host is oversubscribed. The cloud hypervisor is pausing this virtual machine's execution cycles to allocate CPU time to other tenant VMs sharing the physical socket.

---

8. Important Interview Questions & Answers

Q: Why must you ignore the first row of numbers output by vmstat?

Answer: The first row emitted by vmstat does not reflect current system behavior. Instead, it displays the cumulative and calculated rate averages since system boot (derived from the total uptime counter in /proc/uptime and totals in /proc/stat). To assess active real-time behavior, you must specify a time delay (e.g., vmstat 1) and evaluate rows from the second sample onwards.

Q: What is the operational difference between high iowait (wa) and high system CPU (sy)?

Answer:

• wa (I/O Wait): Represents CPU idle time where the CPU core had no work to do except wait for outstanding disk, storage controller, or network filesystem (NFS) requests to complete. It indicates a storage/disk bottleneck, not a processing bottleneck.
• sy (System Time): Represents active CPU core execution time spent executing kernel-space code, system calls (read, write, fork), software interrupts, or driver routines. A high sy indicates the CPU is heavily loaded executing kernel operations.

Q: What does a high value in the r column mean if CPU idle (id) is also high?

Answer: While rare, a high runqueue count (r > cores) accompanied by high idle percentage (id) typically indicates severe CPU affinity restrictions (threads pinned to a specific core using taskset or cgroups that are queuing up on that specific core, while other cores remain completely idle) or rapid scheduling oscillations where threads wake up, increment the runqueue counter, and immediately yield or sleep before consuming full scheduling quanta.

---