perf
Linux perf — Complete Learning Notes & Output Guide
perf(also known asperf_events) is the official, low-overhead performance-analysis tool built directly into the Linux kernel. It bridges hardware performance counters (PMCs) with software instrumentation to diagnose CPU bottlenecks, cache misses, context switches, and hot code paths.
1. What is perf?
perf interacts with CPU performance-monitoring hardware and kernel instrumentation points.
It answers critical performance questions:
- Which exact functions, source lines, or assembly instructions are burning the most CPU cycles?
- Is our application suffering from hardware cache misses or branch mispredictions?
- How many context switches, page faults, or CPU migrations are occurring per second?
- Which kernel functions or system calls are consuming CPU time during peak load?
2. Installation
perf is packaged as part of the Linux kernel source tree and must match your exact running kernel version.
Debian / Ubuntu
sudo apt update
sudo apt install linux-tools-common linux-tools-$(uname -r)
RHEL / Rocky / AlmaLinux / CentOS
sudo dnf install perf
Arch Linux
sudo pacman -S perf
Verify installation:
perf --version
3. Basic Syntax & Core Subcommands
perf operates through a suite of subcommands:
perf <subcommand> [options] [command]
| Subcommand | Purpose | Practical Example |
|---|---|---|
perf stat |
Count system-wide or process-level performance events. | perf stat -p 1234 |
perf record |
Sample CPU execution profiles and save raw data to perf.data. |
perf record -F 99 -p 1234 |
perf report |
Interactively analyze recorded profiling data from perf.data. |
perf report |
perf top |
Real-time interactive CPU profiling (like top for functions). |
sudo perf top |
perf trace |
Modern, lightweight system call tracer (alternative to strace). |
sudo perf trace -p 1234 |
4. Decoding perf stat: Output & Metrics
perf stat runs a command or attaches to a PID, counting hardware and software events during execution.
perf stat -d sleep 5
Example Output:
Performance counter stats for 'sleep 5':
2.23 msec task-clock # 0.000 CPUs utilized
0 context-switches # 0.000 /sec
0 cpu-migrations # 0.000 /sec
0 page-faults # 0.000 /sec
8,142,512 cycles # 3.651 GHz
4,102,100 instructions # 0.50 insn per cycle
1,024,510 branches # 459.421 M/sec
12,410 branch-misses # 1.21% of all branches
102,450 L1-dcache-loads # 45.942 M/sec
4,120 L1-dcache-load-misses # 4.02% of all L1-dcache hits
Detailed Breakdown of perf stat Fields
| Metric Heading | Meaning | Troubleshooting Significance |
|---|---|---|
task-clock |
Total CPU time (in milliseconds) spent running the target task. | Indicates active execution duration. |
context-switches |
Number of times the OS switched execution away from this task. | High values indicate lock contention, I/O waits, or thread yielding. |
cpu-migrations |
Number of times tasks migrated across different physical CPU cores. | High migrations can degrade CPU cache locality. |
page-faults |
Number of memory page faults (minor and major). | Major faults indicate disk swapping/paging. |
cycles |
Total number of CPU clock cycles elapsed. | Core metric for measuring execution cost. |
instructions |
Total instructions executed by the CPU. | Combined with cycles to compute IPC. |
insn per cycle (IPC) |
Instructions Per Cycle ($\frac{\text{Instructions}}{\text{Cycles}}$). | IPC < 0.5 indicates stall conditions (memory wait or branch misprediction); IPC > 1.5 indicates efficient execution. |
branch-misses |
Percentage of conditional branches incorrectly predicted by the CPU. | High miss rates (>5%) cause CPU pipeline flushes and severe stalls. |
L1-dcache-load-misses |
Percentage of data loads that missed the L1 data cache. | High misses indicate poor data structure locality, causing slow memory stalls. |
5. Profiling with perf record & perf report
When diagnosing where a program spends its time, use sampling recording:
# Record at 99Hz frequency for a running PID
sudo perf record -F 99 -p 4210 --call-graph dwarf sleep 30
5.1 Analyzing the Profile: perf report
Running sudo perf report opens an interactive TUI displaying function hotspots:
Overhead Command Shared Object Symbol
------------------------------------------------------------
38.12% python3 libc.so.6 [.] __GI_memcpy
22.45% python3 python3.11 [.] dict_lookup
10.10% python3 libpython3.11.so [.] PyEval_EvalFrameDefault
5.20% python3 _sqlite3.cpython [.] sqlite3Step
Breakdown of perf report Headings
Overhead: The percentage of CPU sampling hits spent inside this specific function. This is your primary target for optimization.Command: The process name handling the execution.Shared Object: The library or binary file containing the symbol (libc.so, executable binary, kernel).Symbol: The function name (memcpy,dict_lookup). Items prefixed with[.]are user space;[k]designates kernel space routines.
6. Real-World Troubleshooting Scenarios
Scenario A: Identifying CPU Hotspots in a Slow Service
An application is maxing out a CPU core, but standard logs don't show why:
sudo perf top -p 1842
Conclusion: Instantly displays live, updating functions consuming CPU cycles without needing to restart the application or modify code.
Scenario B: Diagnosing Kernel vs. User Time Skew
top reports high %sys (system) utilization, indicating heavy kernel activity:
perf stat -p 3482 sleep 10
Conclusion: If cycles are heavily dominated by kernel symbols (e.g., sys_epoll_wait, vfs_read, futex), the bottleneck lies in kernel-level system call overhead, network socket polling, or thread locking.
7. Important Interview Questions & Answers
Q: What is the difference between counting events (perf stat) and sampling events (perf record)?
Answer: perf stat configures hardware counters to maintain a running tally of events (total cycles, instructions, cache misses) during a process's lifecycle, providing aggregated totals. perf record periodically interrupts the CPU (e.g., 99 times per second via -F 99), capturing the exact instruction pointer and stack trace at each interrupt. This profiling sample data (perf.data) allows perf report to reconstruct call graphs and pinpoint specific code hotspots.
Q: Why is a sampling frequency of 99Hz (-F 99) preferred over 100Hz or 1000Hz in production?
Answer: Frequencies of 100Hz or 1000Hz align cleanly with operating system timer interrupts and periodic background polling routines. If your application executes periodic tasks at identical intervals, sampling at 100Hz can create phase locking, resulting in skewed profiling data that oversamples or completely misses specific code blocks. 99Hz is an odd prime frequency that prevents harmonic synchronization with OS scheduling loops.