Linux tiptop — Complete Learning Notes & Output Guide

tiptop is an interactive, display-mode hardware performance monitoring tool for Linux. Like standard top, it provides a dynamically updating table of running processes; however, instead of reporting simple CPU time percentages, it interfaces with CPU Hardware Performance Counters (PMCs) via the perf_event_open() system call to report hardware-level execution metrics such as Instructions Per Cycle (IPC), cache misses, and memory bus utilization per process.

---

1. What is tiptop?

tiptop was developed by Inria (the French National Institute for Computer Science and Applied Mathematics) as a lightweight, non-intrusive alternative to complex hardware sampling suites.

Standard top can show that a process is consuming 100% of a CPU core, but it cannot tell you whether the CPU is actively crunching calculations or spending 80% of its clock cycles stalled waiting on memory fetches from DRAM. tiptop bridges this gap.

It answers critical performance engineering questions:

• Is a CPU-intensive process compute-bound or memory-stall-bound?
• What is the real-time IPC (Instructions Per Cycle) of individual threads or PIDs?
• Which specific processes are causing hardware L1/L2/LLC cache thrashing?
• How many CPU clock cycles are lost to branch mispredictions per process?

---

2. Installation & Prerequisites

tiptop requires access to CPU hardware performance monitoring units (PMUs), which Linux manages via the perf_event subsystem.

Debian / Ubuntu

sudo apt update
sudo apt install tiptop

RHEL / Rocky / AlmaLinux / CentOS

tiptop is available via EPEL or direct compilation:

sudo dnf install epel-release
sudo dnf install tiptop

Arch Linux

sudo pacman -S tiptop

Verify:

tiptop --version

Kernel Capability Requirements

Accessing uncore or per-process hardware counters requires elevated privileges or adjusting the kernel's perf_event_paranoid level:

# Check current paranoid level (default is often 2, 3, or 4)
cat /proc/sys/kernel/perf_event_paranoid

# Allow non-root users to read hardware counters (0 or 1)
sudo sysctl kernel.perf_event_paranoid=1

---

3. Basic Syntax & Core Operation Modes

tiptop [options]

Essential Command Flags

Flag	Purpose	Practical Example
(no flag)	Runs the default Live Screen mode (cycling through pre-configured metrics).	tiptop
-p <PID>	Attach hardware counter monitoring strictly to a specific Process ID.	tiptop -p 4512
-t	Show individual threads (LWP) instead of aggregating at the process level.	tiptop -t
-d <delay>	Set the refresh interval delay in seconds (default is 2.0 seconds).	tiptop -d 1.0
-n <iterations>	Run for a fixed number of update cycles and then exit (batch mode).	tiptop -n 5
-b	Batch mode: Outputs plain text lines without clearing the screen (ideal for logging/scripting).	tiptop -b -n 3 > run.log
-u <user>	Filter monitored tasks to a specific username.	tiptop -u postgres

---

4. Default Screen Mode (IPC & Core Cycles)

Running tiptop launches an ncurses terminal UI similar to top:

tiptop

Raw Output Example (Default Screen Mode)

[tiptop - Live] Tasks: 182, 1 thr; CPU: Intel(R) Xeon(R) Gold 6248R @ 3.00GHz
  PID [USER   ] %CPU   P    CYCLE      INST      IPC  [PROGRAM         ]
 5410 postgres  98.5   2   2950.4M   4130.5M    1.40  postgres: worker
 8912 python3   99.1   5   2975.1M    892.5M    0.30  python3 ml_train.py
 1120 root       1.2   0     36.0M     43.2M    1.20  systemd-journald
 4515 nginx      0.8   1     24.0M     36.0M    1.50  nginx: worker

---

5. Breakdown of Every Output Heading & Field

+------+------------+------+---+---------+---------+------+--------------------+
| PID  | [USER    ] | %CPU | P | CYCLE   | INST    | IPC  | [PROGRAM         ] |
+------+------------+------+---+---------+---------+------+--------------------+
| 8912 | python3    | 99.1 | 5 | 2975.1M |  892.5M | 0.30 | python3 ml_train   |
+------+------------+------+---+---------+---------+------+--------------------+

5.1 PID

• Meaning: The Linux Process ID (or Thread ID when running with -t).

5.2 [USER]

• Meaning: The owner username of the running process.

5.3 %CPU

• Meaning: Percentage of a single CPU core consumed by the process over the refresh interval (identical to standard top).

5.4 P (Processor / Core ID)

• Meaning: The physical/logical CPU core number on which the task was executing at the time of the sample.

5.5 CYCLE

• Format: Integer with engineering suffix (M for Millions, G for Billions).
• Meaning: The total number of hardware CPU clock cycles elapsed for this process during the refresh window.
• Troubleshooting Significance: Compares against the processor's base/boost clock speed to determine if the task was running unimpeded on the CPU.

5.6 INST

• Format: Integer with engineering suffix (M for Millions, G for Billions).
• Meaning: The total count of retired instructions executed by the CPU pipeline for this process during the window.
• Note on Speculative Execution: This counter tracks retired instructions (architectural instructions that actually committed results), ignoring instructions discarded due to branch mispredictions.

5.7 IPC (Instructions Per Cycle)

• Format: Decimal floating-point ratio:

$$\text{IPC} = \frac{\text{Retired Instructions}}{\text{CPU Cycles}}$$

• Troubleshooting Significance: This is the primary indicator of execution efficiency:
• $\text{IPC} \ge 1.5$ (Compute-Bound / Efficient): The CPU pipeline is executing multiple instructions per clock cycle. Code is instruction-dense, utilizing registers and L1 cache effectively.
• $0.8 \le \text{IPC} < 1.5$ (Nominal): Balanced general-purpose workload.
• $\text{IPC} < 0.5$ (Memory-Stalled / Cache-Starved): The core is spending the vast majority of its clock cycles idling (stalled), waiting for data to arrive across the memory bus from higher-level caches (L3) or main DRAM.

5.8 [PROGRAM]

• Meaning: Command string or process binary name.

---

6. Alternative Screens & Hardware PMU Presets

tiptop includes built-in configurations targeting different hardware subsystems. Pressing specific keys or specifying presets switches metric views:

# Run with cache-miss monitoring preset
tiptop -W

Common Built-in Views

1. Cache Hierarchy Mode (-W / Cache View)

  PID %CPU      L1D_MISS     LLC_MISS     BRANCH_MISS  [PROGRAM]
 8912 99.1       45.2M        12.4M         1.2M       python3 ml_train
 5410 98.5        1.2M         0.05M        0.08M      postgres: worker

• L1D_MISS: Level 1 Data Cache read misses. High values indicate poor temporal or spatial data locality.
• LLC_MISS: Last Level Cache (L3) misses. Every LLC miss forces the CPU to issue an off-chip request to physical DRAM, incurring an ~80–200 CPU cycle stall.
• BRANCH_MISS: Number of branch instructions mispredicted by the CPU branch predictor, causing pipeline flushes.

2. Thread-Level Mode (tiptop -t)

Expands processes into their individual operating system threads, identifying which thread in a worker pool is suffering from cache starvation.

---

7. tiptop vs. top vs. perf top vs. htop

+-------------------------------------------------------------+
|                        top / htop                           |
|  * Reads /proc/[pid]/stat.                                  |
|  * Measures: CPU time %, Memory RSS, State.                 |
|  * Cannot see: Hardware stalls, IPC, or cache efficiency.   |
+-------------------------------------------------------------+
                               |
                               v
+-------------------------------------------------------------+
|                          tiptop                             |
|  * Uses perf_event_open() hardware PMUs per process.        |
|  * Measures: Real-time IPC, Cycle counts, Cache misses.     |
|  * Layout: top-like interactive UI ranked by hardware cost. |
+-------------------------------------------------------------+
                               |
                               v
+-------------------------------------------------------------+
|                         perf top                            |
|  * Kernel sampling engine. Interrupts CPU at frequency (Hz).|
|  * Measures: Specific function symbols and assembly lines.  |
|  * Use case: Finding the hot function inside a slow binary. |
+-------------------------------------------------------------+

Feature / Dimension	top	tiptop	perf top
Primary Metric	CPU Time %	IPC & Hardware PMCs	Symbol Overhead %
Mechanism	/proc polling	perf_event_open()	CPU sampling interrupts
Memory Stall Detection	None	Immediate (via IPC & LLC)	Indirect (via memory events)
System Overhead	Very Low	Low	Moderate
Symbol/Function Resolution	No	No	Yes (down to assembly)

---

8. Real-World Troubleshooting Scenarios

Scenario A: Resolving the "100% CPU Utilization" Dilemma

Two backend microservices (Service A and Service B) both consume 100% of a CPU core according to top. However, Service A processes 10,000 requests/sec while Service B processes only 1,200 requests/sec.

Run tiptop to inspect hardware efficiency:

sudo tiptop

Output:

  PID %CPU      CYCLE       INST     IPC  [PROGRAM]
 4102 99.8    3000.1M    5400.2M    1.80  service_a
 4105 99.9    2998.5M     600.1M    0.20  service_b

Diagnosis:

• service_a has an $\text{IPC} = 1.80$: The pipeline is healthy and executing compute logic effectively.
• service_b has an $\text{IPC} = 0.20$: The CPU is spending ~85–90% of every clock cycle sitting idle waiting for memory. The bottleneck is not compute capacity—it is cache thrashing, random pointer chasing, or unaligned data structures causing continuous DRAM fetches.

---

Scenario B: Diagnosing Lock Contention & Spinlock Waste

A multi-threaded application's CPU usage spikes to 800% (pegging 8 cores) when concurrency increases, but throughput collapses.

Run tiptop -t -p <PID>: Diagnosis:

• If **$\text{IPC} \ge 2.0$ with high INST**: The threads are executing instructions rapidly in a tight loop—a classic busy-waiting / spinlock contention anti-pattern (while(locked) {}).
• If **$\text{IPC} < 0.2$ with low INST**: Threads are contending on cache lines containing shared data, triggering continuous cross-core cache invalidation cycles (false sharing).

---

9. Important Interview Questions & Answers

Q: What does an Instructions Per Cycle (IPC) below 0.5 indicate during performance analysis?

Answer: Modern superscalar x86/ARM processors are capable of retiring 3 to 6 instructions per cycle under optimal conditions. An IPC below 0.5 indicates that the execution pipeline is heavily stalled. This is almost always caused by memory latency—the CPU is waiting for data to arrive from main RAM following Last Level Cache (LLC) misses—or by frequent branch mispredictions, where the processor mispredicts execution paths and flushes its pipeline stages.

Q: How does tiptop collect hardware PMU metrics without patching or restarting target processes?

Answer: tiptop utilizes the Linux kernel's native perf_event_open() system call. When directed to monitor a task, it instructs the kernel to configure the physical CPU's Performance Monitoring Unit (PMU) control registers (MSRs) for that process's software execution context. Whenever the Linux scheduler swaps that process onto a CPU core, the hardware counters automatically track cycles and instructions, and tiptop reads these 64-bit counter values directly through file descriptors without injecting code into the target process.

---