numastat
Linux numastat — Complete Learning Notes & Output Guide
numastatis a memory diagnostic utility that displays Non-Uniform Memory Access (NUMA) allocation statistics and memory consumption across physical CPU sockets and memory nodes. It parses per-node kernel accounting files in/sys/devices/system/node/to reveal whether processes are allocating local or remote memory, detecting NUMA memory imbalances and remote memory latency penalties.
1. What is numastat & NUMA Architecture?
On modern multi-socket or multi-chip-module (MCM) servers (such as AMD EPYC or Intel Xeon systems), the system memory is partitioned into distinct NUMA nodes, each directly attached to a specific CPU socket or core complex:
+------------------------------+ +------------------------------+
| NUMA Node 0 | | NUMA Node 1 |
| +------------------------+ | | +------------------------+ |
| | Socket 0 | | | | Socket 1 | |
| | (CPU Cores) | | | | (CPU Cores) | |
| +------------------------+ | | +------------------------+ |
| | | | | |
| [Local Bus: Low Latency] | [Local Bus: Low Latency]
| v | Interconnect| v |
| +------------------------+ | (UPI / IF) | +------------------------+ |
| | Local RAM |<=================>| Local RAM | |
| | (64 GB) | High Latency | | (64 GB) | |
| +------------------------+ | | +------------------------+ |
+------------------------------+ +------------------------------+
- Local Memory Access: A CPU core accesses RAM wired directly to its own memory controller (~60–80 ns).
- Remote Memory Access: A CPU core accesses RAM wired to a different socket across an interconnect bus (Intel UPI or AMD Infinity Fabric), introducing a $1.5\times$ to $2.5\times$ latency penalty and consuming cross-socket bandwidth.
numastat answers critical system and database memory questions:
- Are CPU cores fetching data from fast local RAM or stalling on slow remote RAM?
- Is one NUMA node out of memory while other nodes have gigabytes of free RAM?
- How is a specific database process (
postgres,mysqld,java) distributing its memory footprint across nodes? - Are memory allocations balanced evenly, or is memory locality degraded?
2. Installation & Availability
numastat is part of the numactl package.
Debian / Ubuntu
sudo apt update
sudo apt install numactl
RHEL / Rocky / AlmaLinux / CentOS
sudo dnf install numactl
Arch Linux
sudo pacman -S numactl
Verify:
numastat -V 2>/dev/null || numastat
3. Basic Syntax & Primary Options
numastat [options]
Essential Command Flags
| Flag | Description | Practical Example |
|---|---|---|
| (no flag) | Output raw event counters (numa_hit, numa_miss, etc.) for all nodes. |
numastat |
-c |
Compact output table (condenses column widths). | numastat -c |
-m |
Display a detailed /proc/meminfo-style breakdown in Megabytes (MB) per node. |
numastat -m |
-p <PID> |
Display per-node memory allocation footprint for a specific Process ID. | numastat -p 5410 |
-s |
Sort nodes by total memory consumption (descending). | numastat -s |
-z |
Suppress display of lines or processes that have zero memory allocated. | numastat -p 5410 -z |
4. Anatomy of Default Output (numastat)
Executing numastat without options prints kernel allocation counters directly from /sys/devices/system/node/node*/numastat:
numastat
Raw Output Example
node0 node1
numa_hit 1420512401 1389102450
numa_miss 124500 845120
numa_foreign 845120 124500
interleave_hit 45120 45118
local_node 1420102400 1388502400
other_node 410001 600050
5. Breakdown of Every Output Heading & Metric
+-----------------+--------------------+--------------------+
| Metric | node0 | node1 |
+-----------------+--------------------+--------------------+
| numa_hit | 1420512401 | 1389102450 |
| numa_miss | 124500 | 845120 |
| numa_foreign | 845120 | 124500 |
| interleave_hit | 45120 | 45118 |
| local_node | 1420102400 | 1388502400 |
| other_node | 410001 | 600050 |
+-----------------+--------------------+--------------------+
5.1 numa_hit
- Meaning: The total number of memory pages that were successfully allocated on this node as intended by the allocating process.
- Diagnostic Significance: Higher is better. A healthy system exhibits a
numa_hitratio close to 99% of all allocations.
5.2 numa_miss
- Meaning: The process intended to allocate memory on this node, but the allocation fell back to another node because this node was low on free memory.
- Diagnostic Significance: High
numa_missindicates that this node is exhausted or constrained, forcing threads to place memory on remote nodes.
5.3 numa_foreign
- Meaning: Another node intended to allocate memory on itself, but fell back to allocating on this node due to memory exhaustion on the other node.
- Invariant Relationship:
$$\sum \text{numa_miss} = \sum \text{numa_foreign}$$
(For every page that misses on Node A and spills over to Node B, Node A records a numa_miss and Node B records a numa_foreign).
5.4 interleave_hit
- Meaning: Pages allocated successfully on this node as part of an explicit interleave memory policy (e.g., using
numactl --interleaveto stripe allocations round-robin across all NUMA nodes).
5.5 local_node
- Meaning: A process running on a CPU core belonging to this node successfully allocated memory on this node.
- Diagnostic Significance: The primary metric of memory locality. Indicates that threads are accessing memory directly attached to their local socket.
5.6 other_node
- Meaning: A process running on a CPU core belonging to a different node allocated memory on this node.
- Diagnostic Significance: High
other_nodecounts paired with lowlocal_nodecounts indicate poor CPU-memory affinity—threads running on one socket are allocating and accessing memory wired to a different socket.
6. Detailed Memory Breakdown Mode: numastat -m
Running numastat -m formats per-node memory utilization in Megabytes (MB), breaking down where physical RAM is committed:
numastat -m
Raw Output Example
Per-node system memory usage (in MBs):
Node 0 Node 1 Total
--------------- --------------- ---------------
MemTotal 64380.12 64500.45 128880.57
MemFree 1250.40 32100.80 33351.20
MemUsed 63129.72 32399.65 95529.37
Active 45120.10 14200.50 59320.60
Inactive 14200.50 12100.20 26300.70
Active(anon) 38910.00 8400.10 47310.10
Inactive(anon) 1200.10 410.20 1610.30
Active(file) 6210.10 5800.40 12010.50
Inactive(file) 12990.40 11690.00 24680.40
Unevictable 0.00 0.00 0.00
Mlocked 0.00 0.00 0.00
Dirty 4.20 1.10 5.30
Writeback 0.00 0.00 0.00
FilePages 19200.50 17490.40 36690.90
Mapped 4120.80 2100.20 6221.00
AnonPages 40110.10 8810.30 48920.40
Shmem 1200.40 850.10 2050.50
KernelStack 45.10 42.80 87.90
PageTables 420.50 180.20 600.70
NFS_Unstable 0.00 0.00 0.00
Bounce 0.00 0.00 0.00
WritebackTmp 0.00 0.00 0.00
Slab 2100.40 1850.20 3950.60
SReclaimable 1800.20 1600.10 3400.30
SUnreclaim 300.20 250.10 550.30
AnonHugePages 32100.00 4096.00 36196.00
ShmemHugePages 0.00 0.00 0.00
HugePages_Total 0.00 0.00 0.00
HugePages_Free 0.00 0.00 0.00
HugePages_Surp 0.00 0.00 0.00
Critical Imbalance Signals in numastat -m
MemFreeDiscrepancy: In the example above, Node 0 has only 1,250 MB free, while Node 1 has 32,100 MB free. A process scheduled on Socket 0 requiring 2 GB of memory will experience anuma_missand spill into Node 1, or trigger local page reclamation.AnonPages(Heap/Stack Memory): Highlights which node holds active process memory vs. clean filesystem page cache (FilePages).
7. Per-Process Memory Inspection: numastat -p <PID>
To determine how a single process (such as a database server or Java JVM) has allocated its memory across NUMA nodes:
numastat -p 5410
Raw Output Example
Per-node process memory usage (in MBs) for PID 5410 (postgres):
Node 0 Node 1 Total
--------------- --------------- ---------------
Huge 0.00 0.00 0.00
Heap 4.20 1.10 5.30
Stack 0.08 0.04 0.12
Private 24510.40 2100.20 26610.60
Shared 18400.10 18400.10 36800.20
--------------- --------------- ---------------
Total 42914.70 20501.40 63416.10
Breakdown of Fields
Huge: Memory allocated via explicit HugeTLB pages.Heap: Standard process dynamic heap memory.Stack: Memory consumed by thread execution stacks.Private: Anonymous private pages (mmap/ heap) exclusive to this process.Shared: Shared memory segments (e.g., PostgreSQLshared_buffersor POSIX shared memory). Notice how the shared buffer is split evenly (18400.10 MBeach) if an interleave policy was applied.Total: Aggregate physical footprint of the process on each node.
8. Tuning & Optimization Strategies
When numastat reveals high numa_miss or other_node counts, use the following mechanisms:
1. Pinning Workloads to a Single Node (numactl)
For latency-critical databases or cache instances that fit within a single socket's RAM capacity:
# Run PostgreSQL pinned to CPU Socket 0 and Node 0 RAM only
numactl --cpunodebind=0 --membind=0 sudo -u postgres /usr/lib/postgresql/bin/postgres
2. Interleaving Allocations Across All Nodes
For large single-instance databases (e.g., a Redis or PostgreSQL instance utilizing 128 GB on a 128 GB dual-socket server), pinning to one node will cause memory exhaustion. Interleave memory allocations round-robin across all nodes:
numactl --interleave=all /usr/bin/mysqld --basedir=/usr
Verification: In numastat, the interleave_hit counter will increment evenly across all nodes.
3. Automatic NUMA Balancing
The Linux kernel includes an in-kernel balancing engine that periodically scans memory, unmaps pages to trigger minor page faults, and migrates pages to the NUMA node where the accessing CPU thread resides:
# Check if automatic NUMA balancing is enabled (1 = enabled, 0 = disabled)
cat /proc/sys/kernel/numa_balancing
# Enable or disable dynamically
sudo sysctl kernel.numa_balancing=1
(Note: In predictable high-throughput database environments with manual pinning, automatic NUMA balancing is often set to 0 to eliminate page fault migration overhead).
4. Zone Reclaim Mode (vm.zone_reclaim_mode)
Controls whether the kernel reclaims local page cache memory when a node runs out of RAM, or simply allocates from a remote node:
0(Default): Disabled. The kernel allocates memory from a remote node instead of aggressively evicting local cached data.1: Enabled. The kernel aggressively scans and reclaims cached pages on the local node to satisfy memory requests locally.
# Ensure zone reclaim is disabled for databases to avoid stalls
sudo sysctl vm.zone_reclaim_mode=0
9. Real-World Troubleshooting Scenarios
Scenario A: Diagnosing the "50% Free RAM but System is Swapping" Paradox
A multi-socket server with 128 GB RAM begins swapping out processes (si/so in vmstat), even though free -m reports 50 GB of total free RAM.
Step 1: Check per-node free memory:
numastat -m
Output:
Node 0 Node 1 Total
MemFree 120.40 51020.10 51140.50
Diagnosis:
- Node 0 has only 120 MB free, while Node 1 has 51 GB free.
- A process pinned to Node 0 CPUs or running with strict local allocation policies requested memory. Because Node 0 was out of free pages, and memory policies or
zone_reclaim_modeprevented or delayed remote fallback, the kernel was forced to page anonymous memory out to disk swap. - Fix: Enable interleaving (
numactl --interleave=all) for the workload or adjust CPU affinity to allow tasks to run on Node 1 cores.
Scenario B: Diagnosing Cross-Socket Memory Bottlenecks in PostgreSQL
PostgreSQL query latency increases by 40% after migrating to a dual-socket server.
Step 1: Inspect NUMA hits and misses:
numastat
Output shows numa_miss and other_node incrementing at millions of pages per hour.
Step 2: Inspect the Postgres master process footprint:
numastat -p $(pgrep -f "postgres: checkpointer" | head -n 1)
Output shows 90% of Shared memory was allocated on Node 0 during initial database startup, while worker queries are executing on Node 1 CPUs.
Diagnosis: Every time worker threads on Node 1 query table pages from shared_buffers, they must cross the interconnect bus to fetch data from Node 0 RAM.
Fix: Start the PostgreSQL service with numactl --interleave=all so that shared_buffers are distributed uniformly across all physical memory channels.
10. Important Interview Questions & Answers
Q: What is the exact difference between numa_miss and other_node in numastat?
Answer:
numa_missreflects intent failure due to memory capacity: The kernel wanted to allocate memory on this node (because the thread was running here or had an explicit affinity rule), but couldn't because this node was low on free memory, forcing allocation onto a different node.other_nodereflects CPU location at allocation time: Memory was allocated on this node by a process that was executing on a CPU core located on a different node (e.g., a CPU on Node 1 placed its data into Node 0's RAM).
Q: Why should vm.zone_reclaim_mode typically be set to 0 on database servers?
Answer: If zone_reclaim_mode is enabled (1), when a NUMA node runs low on free memory, the kernel will attempt to synchronously reclaim cached file pages or unmap memory within that local node before falling back to allocating free memory available on other NUMA nodes. On database systems, this causes severe latency spikes and micro-freezes because CPU threads block in kernel space waiting for disk syncs and dirty cache flushes, even though hundreds of gigabytes of idle RAM are readily available on an adjacent NUMA node. Setting it to 0 allows immediate remote allocation.
Q: What is the relationship between numa_miss and numa_foreign across the entire system?
Answer: Across the entire system, the total sum of numa_miss must equal the total sum of numa_foreign. When a memory allocation cannot be fulfilled locally on Node A and spills over to Node B, the kernel increments numa_miss on Node A (the intended target that failed) and increments numa_foreign on Node B (the alternate node that satisfied the request on behalf of Node A).