iostat

@amitmund September 10, 2026

Absolutely. Below is a structured learning markdown note on iostat, designed to take you from beginner concepts to practical Linux performance troubleshooting.

# Linux `iostat` — Complete Learning Notes

> `iostat` is a Linux performance-monitoring command used to analyze **CPU utilization** and **I/O performance**, especially disk and block-device activity.

---

# 1. What is `iostat`?

`iostat` stands for:

**I/O Statistics**

It is part of the `sysstat` package and is commonly used by Linux administrators, DevOps engineers, SREs, and performance engineers.

It helps answer questions such as:

- Is my CPU overloaded?
- Is my disk busy?
- Is the system waiting for disk I/O?
- Which disk is receiving the most I/O?
- Is disk latency high?
- Is the disk saturated?
- Are applications waiting because of storage?
- Is the system performing mostly reads or writes?
- Are I/O requests being merged?
- How many I/O operations are occurring per second?

---

# 2. Installing `iostat`

`iostat` is normally provided by the `sysstat` package.

## Debian / Ubuntu

```bash
sudo apt update
sudo apt install sysstat

RHEL / CentOS / Rocky / AlmaLinux

sudo dnf install sysstat

Older CentOS:

sudo yum install sysstat

Arch Linux

sudo pacman -S sysstat

Verify:

iostat --version

or:

which iostat

3. Basic Syntax

The basic syntax is:

iostat [options] [interval] [count]

Example:

iostat

Run every 2 seconds:

iostat 2

Run every 2 seconds, 5 times:

iostat 2 5

4. Understanding the Default Output

Run:

iostat

You may see something similar to:

Linux 6.x.x hostname  09/10/2026  _x86_64_

avg-cpu:  %user   %system  %iowait  %steal   %idle
           2.10      1.20     0.50     0.00    96.20

Device        tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda          25.00       320.00       850.00      123456      456789

There are two major sections:

  1. CPU statistics
  2. Device/I/O statistics

5. CPU Statistics

Example:

avg-cpu:

%user   %system   %iowait   %steal   %idle
 2.10      1.20       0.50      0.00    96.20

Let's understand each field.

5.1 %user

Percentage of CPU time spent executing user-space processes.

Example:

%user = 70%

This means approximately 70% of CPU time was spent running applications.

Examples of user-space workloads:

  • Python programs
  • Java applications
  • databases
  • web servers
  • shell scripts

5.2 %system

Percentage of CPU time spent executing kernel code.

Example:

%system = 10%

High %system may indicate significant kernel activity.

Possible causes:

  • Heavy networking
  • Heavy disk I/O
  • Many system calls
  • Packet processing
  • File-system operations
  • Device-driver activity

5.3 %iowait

One of the most important metrics.

%iowait

It represents CPU time during which the CPU was idle while the system had outstanding I/O.

Example:

%iowait = 20%

This may indicate that storage or another I/O subsystem is causing processes to wait.

Important

Do not interpret %iowait as:

"20% of the CPU is being used by the disk."

That is incorrect.

Instead, think:

The CPU was idle while outstanding I/O existed.

High %iowait can be a clue that I/O is contributing to poor performance, but it should be investigated alongside disk metrics.

6. %steal

Relevant mainly on virtual machines.

%steal

It represents CPU time taken by the hypervisor to run another virtual machine.

Example:

%steal = 15%

This can indicate CPU contention on the virtualization host.

If:

%steal

is consistently high on a VM, investigate host-level CPU contention.

7. %idle

Percentage of CPU time during which the CPU was idle.

Example:

%idle = 95%

This generally indicates that the CPU has plenty of unused capacity.

8. Device Statistics

Modern iostat commonly produces output such as:

Device       tps   kB_read/s   kB_wrtn/s   kB_dscd/s
sda         100.0     5000.0      2000.0        0.0

Depending on the version and options, additional columns may appear.

Important device metrics include:

  • tps
  • rkB/s
  • wkB/s
  • r/s
  • w/s
  • rMB/s
  • wMB/s
  • await
  • r_await
  • w_await
  • aqu-sz
  • %util

9. tps

tps means:

Transfers Per Second

It represents the number of I/O operations per second.

Example:

tps = 500

This means approximately 500 I/O transfers per second.

10. Read Throughput

Depending on the version, you may see:

rkB/s

or:

kB_read/s

or:

rMB/s

These represent read throughput.

Example:

rMB/s = 100

Approximately 100 MB/s of data is being read.

11. Write Throughput

You may see:

wkB/s

or:

kB_wrtn/s

or:

wMB/s

These represent write throughput.

Example:

wMB/s = 50

Approximately 50 MB/s is being written.

12. Read Operations Per Second

With extended statistics:

iostat -x

you may see:

r/s

This means:

read requests per second

Example:

r/s = 200

The device is processing approximately 200 read requests per second.

13. Write Operations Per Second

w/s

means:

write requests per second

Example:

w/s = 100

Approximately 100 write operations are being processed per second.

14. Extended Statistics

One of the most useful commands is:

iostat -x

-x means:

extended statistics

Example:

iostat -x

You may see:

Device      r/s    w/s   rkB/s   wkB/s   await  aqu-sz  %util
sda        50.0   20.0   4000.0  2000.0    8.5     0.6    75.0

This gives much more information than normal iostat.

15. await

One of the most important disk metrics.

await

represents the average time, in milliseconds, for I/O requests to complete.

It generally includes:

  • time waiting in the queue
  • time spent being serviced

Example:

await = 5 ms

The average I/O request took approximately 5 milliseconds.

16. High await

Suppose:

await = 100 ms

That is significantly higher than:

await = 2 ms

Possible causes:

  • Disk saturation
  • Heavy workload
  • Slow storage
  • Storage-network problems
  • Virtualized storage contention
  • Large I/O queue
  • Underlying SAN/NAS issues

Do not diagnose based on await alone.

Always examine:

await
aqu-sz
%util
r/s
w/s
read/write throughput

together.

17. r_await

r_await

Average time for read requests to complete.

Example:

r_await = 4 ms

Read requests are taking approximately 4 ms on average.

18. w_await

w_await

Average time for write requests to complete.

Example:

w_await = 15 ms

Write requests are taking approximately 15 ms.

If:

r_await = 3 ms
w_await = 30 ms

writes are experiencing considerably higher latency than reads.

19. aqu-sz

aqu-sz

means:

Average queue size

It indicates the average number of I/O requests waiting/active in the device queue.

Example:

aqu-sz = 5

A larger queue can indicate that the storage subsystem is struggling to keep up with incoming I/O.

20. %util

Very important.

%util

represents the percentage of elapsed time during which the device was servicing I/O.

Example:

%util = 95%

The device was busy for approximately 95% of the measured interval.

21. Does %util = 100% Always Mean the Disk Is Slow?

No.

This is an important interview concept.

For some devices, especially modern SSD/NVMe devices, high %util does not necessarily mean the device is incapable of handling more I/O.

For example:

%util = 100%
await = 0.8 ms

may be perfectly acceptable for a high-performance device.

But:

%util = 100%
await = 100 ms
aqu-sz = 20

is much more suspicious.

Therefore:

Never diagnose storage saturation using %util alone.

Look at latency and queue depth too.

22. The Most Useful Command

For troubleshooting:

iostat -xz 1

Breakdown:

-x

Extended statistics.

-z

Omit devices with no activity.

1

Refresh every second.

This is an excellent command for real-time disk troubleshooting.

23. Better Monitoring Command

A very useful command is:

iostat -xz 1 10

Meaning:

-x   extended statistics
-z   hide inactive devices
1    update every 1 second
10   collect 10 reports

24. First Report vs Subsequent Reports

This is extremely important.

When you run:

iostat

the first report may represent statistics accumulated since system boot.

When you run:

iostat 1

the first report can represent a longer historical period, while subsequent reports represent approximately one-second intervals.

Therefore, for real-time troubleshooting:

iostat -xz 1

pay close attention to the subsequent interval reports.

25. -c

Show CPU statistics only.

iostat -c

Useful when you only want CPU information.

26. -d

Show device statistics only.

iostat -d

Example:

iostat -d 1

This is useful when you want to focus on storage.

27. -x

Extended statistics.

iostat -x

This is one of the most useful options.

28. -z

Hide devices with zero activity.

iostat -z

Useful on systems with many disks.

For example:

sda
sdb
sdc
sdd
sde
sdf
...

If only sda is active, -z makes the output easier to read.

29. -m

Display throughput in MB/s.

iostat -m

Example:

iostat -xm

This is easier to read than kB/s when dealing with high-throughput storage.

30. -k

Display throughput in KB/s.

iostat -k

31. -t

Display timestamps.

iostat -t

Useful when collecting logs.

Example:

iostat -xt 1

32. -h

Human-readable output.

Depending on the installed sysstat version:

iostat -h

can make output easier to interpret.

Always check:

iostat --help

because available options can vary by version.

33. Important Command Examples

Basic

iostat

Refresh every second

iostat 1

Extended statistics

iostat -x

Extended statistics every second

iostat -x 1

Extended + hide inactive devices

iostat -xz 1

Extended + MB/s

iostat -xm 1

CPU only

iostat -c 1

Disk only

iostat -d 1

Ten samples

iostat -xz 1 10

34. A Practical Example

Suppose:

Device     r/s    w/s   rMB/s   wMB/s   await   aqu-sz   %util

sda       500.0  300.0   80.0    40.0    35.0     25.0    99.0

Let's analyze it.

IOPS

r/s = 500
w/s = 300

Total:

500 + 300 = 800 IOPS

Throughput

80 MB/s reads
40 MB/s writes

Total:

120 MB/s

Latency

await = 35 ms

This is relatively high for many storage workloads.

Queue

aqu-sz = 25

There is a significant amount of queued/active I/O.

Utilization

%util = 99%

The device is continuously busy.

Initial conclusion

The device is likely under heavy I/O pressure.

But we should investigate further before declaring:

"The disk is definitely the bottleneck."

35. Example of a Healthy Fast SSD

Imagine:

Device     r/s     w/s    await   aqu-sz   %util

nvme0n1   5000    3000     0.5      2.0     80%

This may be perfectly healthy.

Why?

await = 0.5 ms

Latency is low.

The device is processing thousands of operations per second.

Therefore:

high IOPS
+
low latency

can indicate a healthy high-performance storage system.

36. Example of a Storage Problem

Device     r/s    w/s    await    aqu-sz   %util

sda        100    50      150 ms     30     100%

This is much more concerning.

We have:

%util = 100%
await = 150 ms
aqu-sz = 30

This suggests significant storage pressure.

37. Read-Heavy Workload

Example:

r/s = 5000
w/s = 100

This is a read-heavy workload.

Possible applications:

  • Search systems
  • Databases
  • File servers
  • Caching systems
  • Analytics

38. Write-Heavy Workload

Example:

r/s = 100
w/s = 5000

This is write-heavy.

Possible workloads:

  • Logging
  • Databases
  • Message queues
  • Backup systems
  • Data ingestion

39. IOPS vs Throughput

A very important concept.

Two storage workloads can have the same throughput but very different IOPS.

Example:

100 MB/s

Workload A:

1000 IOPS

Average I/O size:

100 MB/s / 1000
= 100 KB per I/O

Workload B:

10000 IOPS

Average I/O size:

100 MB/s / 10000
= 10 KB per I/O

Same throughput.

Very different I/O patterns.

40. IOPS

IOPS means:

Input/Output Operations Per Second

Examples:

100 IOPS
1000 IOPS
10000 IOPS
100000 IOPS

IOPS is particularly important for workloads involving many small random operations.

41. Throughput

Throughput measures the amount of data transferred.

Examples:

10 MB/s
100 MB/s
1 GB/s

Throughput is especially important for workloads involving large sequential operations.

42. Latency

Latency measures how long an I/O operation takes.

Example:

await = 2 ms

means average I/O completion time is around 2 milliseconds.

A workload may have:

High IOPS
Low latency

and perform extremely well.

43. Queue Depth

Queue depth represents outstanding I/O waiting to be processed.

Conceptually:

Application
    |
    v
+-----------+
| I/O Queue |
+-----------+
    |
    v
+-----------+
| Storage   |
+-----------+

If incoming I/O exceeds the device's ability to process it:

Incoming I/O
      |
      v
+-------------+
| Large Queue |
+-------------+
      |
      v
 Storage

Latency can increase.

44. Understanding the Relationship

A useful mental model:

IOPS
  +
Throughput
  +
Queue Depth
  +
Latency
  +
Utilization

should be analyzed together.

Do not focus on only one metric.

45. CPU + Disk Troubleshooting

Suppose a server feels slow.

Run:

iostat -xz 1

If you see:

%iowait = 30%

and:

await = 100 ms

and:

%util = 100%

then storage is a strong candidate for the bottleneck.

But if:

%iowait = 1%

and:

%util = 20%

storage may not be the main problem.

You should investigate CPU, memory, networking, or the application.

46. A Simple Troubleshooting Workflow

When a Linux system is slow:

Step 1 — Check CPU

iostat -c 1

Look at:

%user
%system
%iowait
%steal
%idle

Step 2 — Check disk

iostat -xz 1

Look at:

r/s
w/s
rkB/s
wkB/s
await
aqu-sz
%util

Step 3 — Identify the busy device

Example:

sda     95% util
sdb      5% util
sdc      2% util

sda deserves investigation.

Step 4 — Check latency

Look at:

await
r_await
w_await

High latency is often more informative than utilization alone.

Step 5 — Check queue

Look at:

aqu-sz

A growing queue can indicate that storage cannot keep up with incoming requests.

47. iostat and vmstat

Both commands are useful but serve different purposes.

iostat

Primarily useful for:

CPU
Disk I/O
Block devices
Storage performance

vmstat

Useful for a broader system overview:

CPU
Memory
Processes
Paging
I/O
System activity

Example:

vmstat 1

A common troubleshooting combination:

vmstat 1
iostat -xz 1

48. iostat and top

top helps answer:

Which processes are consuming CPU/memory?

top

iostat helps answer:

Is the storage subsystem under pressure?

iostat -xz 1

Use them together.

49. iostat and iotop

iostat shows device-level I/O.

iostat -xz 1

iotop shows process-level I/O.

sudo iotop

Example workflow:

iostat
   |
   | "Disk is very busy"
   v
iotop
   |
   | "Which process is causing it?"
   v
Application

50. iostat and pidstat

pidstat can show per-process statistics.

For I/O:

pidstat -d 1

This can help identify which processes are performing I/O.

51. Logical vs Physical Devices

Be careful when interpreting device statistics.

You may see:

sda
sdb
nvme0n1
dm-0
dm-1

dm-* often represents device-mapper devices.

For example:

Application
     |
     v
Filesystem
     |
     v
Logical Volume
     |
     v
dm-0
     |
     v
Physical Disk
     |
     v
SSD

The same underlying physical I/O can appear at different layers.

52. RAID Considerations

In RAID environments, interpreting I/O statistics requires additional understanding.

For example:

Application
     |
     v
RAID / LVM
     |
     +---- Disk 1
     +---- Disk 2
     +---- Disk 3
     +---- Disk 4

One logical request can generate multiple physical operations depending on:

  • RAID level
  • Read/write operation
  • Stripe size
  • Write policy
  • Cache
  • Controller

Therefore, don't assume:

logical IOPS = physical IOPS

53. NVMe Devices

Modern systems may use:

nvme0n1
nvme1n1

instead of:

sda
sdb

Example:

iostat -xz 1

Output:

Device      r/s      w/s     await    %util

nvme0n1    50000    20000     0.3      95%

This can be perfectly reasonable for a high-performance NVMe device.

Again:

High %util does not automatically mean poor performance.

54. Important Interview Question

Q: What does %iowait mean?

Answer:

%iowait represents the percentage of CPU time during which the CPU was idle while the system had outstanding I/O.

It is an indicator that can point toward I/O-related delays, but it should not be treated as a direct measurement of disk utilization.

55. Important Interview Question

Q: What does %util mean?

Answer:

%util represents the percentage of elapsed time during which the device was busy servicing I/O during the measurement interval.

High %util can indicate a busy device, but it does not alone prove that the storage device is saturated.

56. Important Interview Question

Q: What is await?

Answer:

await is the average time, in milliseconds, for I/O requests to complete, including time spent waiting in the queue and time spent being serviced.

57. Important Interview Question

Q: What is aqu-sz?

Answer:

aqu-sz represents the average number of I/O requests in the device's queue or being serviced.

A high value can indicate significant I/O pressure.

58. Important Interview Question

Q: How do you troubleshoot high disk latency?

Start with:

iostat -xz 1

Check:

await
r_await
w_await
aqu-sz
%util
r/s
w/s

Then identify the process:

sudo iotop

or:

pidstat -d 1

Then investigate:

  • Storage hardware
  • RAID
  • LVM
  • Filesystem
  • Network storage
  • Cloud storage
  • Application workload

59. Important Interview Question

Q: Is 100% disk utilization always bad?

No.

For example:

%util = 100%
await = 0.5 ms

may indicate a high-performance SSD handling a heavy workload efficiently.

Compare that with:

%util = 100%
await = 200 ms
aqu-sz = 50

which is much more concerning.

60. Important Interview Question

Q: What command would you use to monitor disk performance every second?

A strong answer:

iostat -xz 1

For a fixed number of samples:

iostat -xz 1 10

61. Important Interview Question

Q: How do you identify whether a system is I/O-bound?

Start with:

iostat -xz 1

Look for a combination of:

High %iowait
High await
High aqu-sz
High %util
High I/O rate

Then confirm with other tools.

For example:

iotop

and:

pidstat -d 1

62. Quick Metric Cheat Sheet

Metric Meaning
%user CPU time in user space
%system CPU time in kernel
%iowait CPU idle while I/O was outstanding
%steal CPU time taken by hypervisor
%idle CPU idle time
tps Transfers per second
r/s Reads per second
w/s Writes per second
rkB/s Read KB/s
wkB/s Write KB/s
rMB/s Read MB/s
wMB/s Write MB/s
await Average I/O completion time
r_await Average read latency
w_await Average write latency
aqu-sz Average I/O queue size
%util Percentage of time device was busy

63. Most Useful Commands Cheat Sheet

# Basic statistics
iostat

# Update every second
iostat 1

# Extended statistics
iostat -x

# Extended statistics every second
iostat -x 1

# Hide inactive devices
iostat -xz 1

# Show MB/s
iostat -xm 1

# CPU statistics
iostat -c 1

# Device statistics
iostat -d 1

# Timestamped output
iostat -xt 1

# 10 samples
iostat -xz 1 10

64. Recommended Learning Method

Learn iostat in this order:

1. CPU statistics
       |
       v
2. Device statistics
       |
       v
3. IOPS
       |
       v
4. Throughput
       |
       v
5. await
       |
       v
6. Queue depth
       |
       v
7. %util
       |
       v
8. Troubleshooting
       |
       v
9. Real-world workloads
       |
       v
10. Interview questions

65. Hands-On Practice

Run:

iostat -xz 1

Observe the output for at least 30 seconds.

Record:

%iowait
%idle
r/s
w/s
await
aqu-sz
%util

Then generate some disk activity.

For example:

dd if=/dev/zero of=/tmp/testfile bs=1M count=1024 conv=fdatasync

Then watch:

iostat -xm 1

Observe how:

w/s
wkB/s
await
%util

change.

Remove the test file afterward:

rm /tmp/testfile

66. Mini Exercise

Suppose you see:

avg-cpu:

%user  %system  %iowait  %idle
10.0     5.0      30.0    55.0


Device     r/s    w/s    await   aqu-sz   %util

sda        100    200     80 ms    15      100%

Questions

  1. Is the CPU completely saturated?
  2. Is there evidence of I/O pressure?
  3. Is sda busy?
  4. Is latency high?
  5. Is the queue significant?
  6. What tool could identify the process generating the I/O?

Think before looking at the answers.

67. Exercise Answers

1. Is the CPU completely saturated?

No.

%idle = 55%

There is substantial idle CPU time.

2. Is there evidence of I/O pressure?

Yes.

%iowait = 30%

combined with:

await = 80 ms

suggests I/O may be contributing to system latency.

3. Is sda busy?

Yes.

%util = 100%

4. Is latency high?

await = 80 ms

This is relatively high for many workloads and deserves investigation.

5. Is the queue significant?

aqu-sz = 15

There is significant outstanding I/O activity.

6. What tool could identify the process?

Use:

sudo iotop

or:

pidstat -d 1

68. Real-World Troubleshooting Example

Problem:

"The application is very slow."

Start with:

iostat -xz 1

Suppose:

%iowait = 25%

and:

await = 120 ms

and:

aqu-sz = 20

and:

%util = 100%

Initial hypothesis:

Storage may be a bottleneck.

Next:

sudo iotop

Find the process generating I/O.

Then:

pidstat -d 1

Correlate application behavior with I/O.

Finally investigate:

Application
Filesystem
LVM
RAID
Disk
Storage controller
SAN/NAS
Cloud storage

69. Golden Rules

Remember these rules:

Rule 1

Don't look at %util alone.

Rule 2

Don't look at %iowait alone.

Rule 3

Always consider latency:

await
r_await
w_await

Rule 4

Look at queue depth:

aqu-sz

Rule 5

Understand the workload:

random vs sequential
small vs large I/O
read vs write

Rule 6

Identify the process causing I/O.

Use:

iotop

or:

pidstat -d 1

Rule 7

Compare metrics over time.

One snapshot is often insufficient.

Use:

iostat -xz 1

70. Final Mental Model

When analyzing iostat, think:

                 APPLICATION
                      |
                      v
                 I/O REQUESTS
                      |
          +-----------+-----------+
          |                       |
        READ                    WRITE
          |                       |
          +-----------+-----------+
                      |
                      v
                  I/O QUEUE
                      |
                      v
                   DEVICE
                      |
          +-----------+-----------+
          |                       |
        IOPS                  THROUGHPUT
          |                       |
          +-----------+-----------+
                      |
                      v
                   LATENCY
                    await
                      |
                      v
                 PERFORMANCE

The key questions are:

How much I/O?
       ↓
r/s + w/s

How much data?
       ↓
rMB/s + wMB/s

How long does it take?
       ↓
await

How much is waiting?
       ↓
aqu-sz

How busy is the device?
       ↓
%util

Who is causing it?
       ↓
iotop / pidstat

71. One Command to Remember

If you remember only one iostat command:

iostat -xz 1

Use it as your starting point for Linux storage-performance troubleshooting.


### Quick learning challenge

After studying the notes, run:

```bash
iostat -xz 1

and paste the output here. I can then teach you how to read every column in your actual machine's output, including how to decide whether your system is CPU-bound, I/O-bound, or storage-saturated.

0 Likes
2 Views
0 Comments

Filters

No filters available for this view.

Reset All