AMD ROCm (GPU)
AMD ROCm Profiling & Diagnostics (rocm-smi & rocprof) — Complete Learning Notes & Output Guide
AMD ROCm (Radeon Open Compute Platform) is the open-source software development platform for GPU computing, AI/ML acceleration, and High-Performance Computing (HPC) on AMD Radeon and Instinct accelerators. Its diagnostic and profiling toolchain centers on
rocm-smi(system and hardware management) and **rocprof/rocprofv2**(kernel and runtime performance profiling).
1. What is the AMD ROCm Stack?
The ROCm platform provides a complete alternative to NVIDIA's CUDA ecosystem. It operates at multiple abstraction layers:
+-------------------------------------------------------------+
| USER SPACE |
| Frameworks: PyTorch, TensorFlow, JAX, vLLM |
+-------------------------------------------------------------+
|
v
+-------------------------------------------------------------+
| HIP (Heterogeneous-Compute Interface) |
| C++ Runtime & Kernel Language (Code portability for CUDA) |
+-------------------------------------------------------------+
|
v
+-------------------------------------------------------------+
| ROCm Runtime & Driver Layers |
| * ROCR (HSA Runtime API) |
| * rocm-smi (Hardware management & telemetry) |
| * rocprof / Omniperf / Omnitrace (Kernel profiling) |
+-------------------------------------------------------------+
|
v
+-------------------------------------------------------------+
| LINUX KERNEL SPACE |
| KFD (Kernel Fusion Driver) <---> amdgpu.ko Driver Module |
+-------------------------------------------------------------+
|
v
+-------------------------------------------------------------+
| HARDWARE LAYER |
| AMD Instinct (MI200/MI300 CDNA) / Radeon (RX 7000 RDNA) |
+-------------------------------------------------------------+
It answers critical operational questions:
- What are the real-time thermals, power draws, and VRAM utilization across GPUs?
- Are compute units (CUs) stalling on memory fetches or experiencing low wavefront occupancy?
- Which HIP API calls or kernel dispatches are consuming the most execution time?
- Is PCIe or Infinity Fabric throughput causing host-to-device bottlenecks?
2. System & Device Verification: rocminfo
Before profiling, confirm that the kernel fusion driver (kfd) recognizes your GPU compute agents:
rocminfo
Key Output Fields to Inspect
Name:GPU product family (e.g.,gfx90afor AMD Instinct MI250X,gfx942for MI300X,gfx1100for Radeon RX 7900 XTX).Compute Unit:Total hardware Compute Units (CUs) available on the die.SIMDs per CU:Number of Single Instruction, Multiple Data execution units per CU (typically 4).Wavefront Size:The native SIMD execution width:- 64 threads on CDNA data-center architectures (MI series).
32 or 64 threads (Wave32 / Wave64) on RDNA consumer architectures.
Pool Info / Size:Total addressable High Bandwidth Memory (HBM) or GDDR VRAM.
3. System Telemetry: rocm-smi
rocm-smi (ROCm System Management Interface) is the AMD equivalent of nvidia-smi. It monitors hardware health, clocks, power, and memory utilization.
rocm-smi [options]
Running Concise Live Monitoring
rocm-smi
Example Default Output
======================= ROCm System Management Interface =======================
================================= Concise Info =================================
GPU Temp (Edge) AvgPwr SCLK MCLK VRAM% GPU% Perf PwrCap VRAM Alloc
0 43.0c 115.0W 1700Mhz 1200Mhz 42% 99% auto 300.0W 27520MB
1 41.0c 45.0W 800Mhz 400Mhz 2% 0% auto 300.0W 1024MB
================================================================================
============================= End of ROCm SMI Log ==============================
4. Detailed Breakdown of rocm-smi Headings
| Heading | Example | Technical Meaning | Troubleshooting Significance |
|---|---|---|---|
GPU |
0, 1 |
Numeric physical index of the GPU device. | Maps physical cards to system PCIe addresses. |
Temp (Edge) |
43.0c |
Edge sensor temperature on the GPU die package. | Primary indicator for chassis airflow issues. |
AvgPwr |
115.0W |
Average real-time socket power consumption in Watts. | Compares against PwrCap to check for power throttling. |
SCLK |
1700Mhz |
System / Engine Core Clock (Graphics/Compute clock). | Drops during thermal throttling or idle pipeline bubbles. |
MCLK |
1200Mhz |
Memory Clock frequency (HBM/GDDR interface speed). | Stays pegged during heavy memory access workloads. |
VRAM% |
42% |
Percentage of physical GPU memory currently reserved. | Warns of impending Out-Of-Memory (OOM) faults. |
GPU% |
99% |
Percentage of time Compute Units (CUs) were active. | Pipeline activity; 0% with high power indicates hangs. |
Perf |
auto |
Dynamic Power Management (DPM) performance level. | Usually auto, low, or high. |
PwrCap |
300.0W |
Maximum configured Total Board Power (TDP) limit. | Hard ceiling before GPU throttles clock frequencies. |
VRAM Alloc |
27520MB |
Total memory capacity explicitly allocated to buffers. | Exact footprint of loaded models and context windows. |
Essential rocm-smi Diagnostic Commands
| Command | Purpose |
|---|---|
rocm-smi --showtopobw |
Displays NUMA nodes and Infinity Fabric / PCIe inter-GPU link bandwidth. |
rocm-smi --showmeminfo vram |
Displays exact breakdown of used, free, and total VRAM in bytes. |
rocm-smi --showpids |
Shows active Process IDs (PIDs) running workloads on each GPU. |
rocm-smi --setperflevel high |
Locks clocks to maximum performance states to prevent DPM latency dips. |
rocm-smi -d 0 --resetclocks |
Resets GPU 0 clocks and power limits back to factory defaults. |
5. Kernel Profiling with rocprof
rocprof (and modern rocprofv2 / rocprofv3) is the official command-line performance profiler for intercepting HIP APIs, HSA runtime calls, and reading hardware architecture counters.
Basic Profiling Workflow
# Profile HIP runtime and kernel execution times
rocprof --hip-trace --stats python3 train.py
This generates three output files:
results.csv: Chronological execution trace of all kernel launches and memory copies.results.stats.csv: Statistical summary table of aggregated kernel runtimes.results.json: Chrome tracing format viewable inchrome://tracingor Perfetto.
6. Breakdown of rocprof Kernel Summary Output (results.stats.csv)
When opening or printing results.stats.csv:
"Name","Calls","TotalDurationNs","AverageNs","Percentage"
"matmul_kernel_fp16",10000,4502010200,450201,62.15
"layernorm_forward",5000,1820102400,364020,25.13
"elementwise_add",5000,924010100,184802,12.72
Explanation of Headings
Name: The demangled kernel symbol name executed on the GPU.Calls: The total number of times this specific kernel was dispatched to GPU command queues.TotalDurationNs: Cumulative time (in nanoseconds) spent executing this kernel across all invocations.AverageNs: The mean execution duration per launch ($\frac{\text{TotalDurationNs}}{\text{Calls}}$).Percentage: Proportion of total GPU execution time consumed by this kernel. This directly prioritizes which algorithm or layer requires optimization.
7. Hardware Performance Counters Mode (rocprof -i)
To capture internal Compute Unit hardware efficiency, define counters in an input file:
cat <<EOF > metrics.txt
pmc: VALUUtilization, SALUUtilization, SQ_WAVES, VFetchInst, VWriteInst
EOF
Run rocprof against the metrics file:
rocprof -i metrics.txt -o kernel_metrics.csv python3 app.py
Breakdown of Key AMD Hardware Counters
| Hardware Counter | Meaning | Performance Diagnostic Role |
|---|---|---|
VALUUtilization |
Vector ALU Utilization (%) | High indicates healthy, vectorized compute efficiency. Low indicates stall bubbles. |
SALUUtilization |
Scalar ALU Utilization (%) | Measures control-flow and pointer calculation overhead performed by scalar units. |
SQ_WAVES |
Sequencer Wavefronts Created | Total wavefronts dispatched. High wave count indicates sufficient parallelism to hide latency. |
VFetchInst |
Vector Memory Read Instructions | Frequency of memory reads issued to L1 cache/HBM. High counts indicate memory-heavy kernels. |
VWriteInst |
Vector Memory Write Instructions | Frequency of global memory write instructions. |
VALUBusy |
Percentage of time VALU is actively computing | Determines if code is compute-bound vs memory-bandwidth bound. |
8. Mapping NVIDIA CUDA Tools to AMD ROCm Stack
| Functionality | NVIDIA Ecosystem | AMD ROCm Ecosystem |
|---|---|---|
| System Monitor | nvidia-smi |
rocm-smi |
| Device Information | nvidia-smi -q |
rocminfo |
| Timeline Profiler | NVIDIA Nsight Systems (nsys) |
rocprof --hip-trace / AMD Omnitrace |
| Kernel Micro-Profiler | NVIDIA Nsight Compute (ncu) |
rocprof -i / AMD Omniperf |
| C++ GPU Language | CUDA C++ | HIP (Heterogeneous-Compute Interface for Portability) |
| Multi-GPU Communication | NCCL (NVIDIA Collective Communications) | RCCL (Radeon Collective Communications) |
| BLAS Acceleration | cuBLAS | rocBLAS / hipBLAS |
| Deep Learning Primitives | cuDNN | MIOpen |
9. Real-World Troubleshooting Scenarios
Scenario A: Identifying VRAM Out-of-Memory (OOM) Spikes
A distributed training run crashes with memory allocation errors, but rocm-smi snapshots look normal:
watch -n 0.1 rocm-smi --showmeminfo vram
Diagnosis: If VRAM Alloc approaches 95%+ of capacity during gradient calculation, dynamic memory allocation or activation checkpointing is spiking. Offload activations using CPU memory paging or reduce the batch size.
Scenario B: Diagnosing GPU Under-Utilization (Kernel Launch Latency)
rocm-smi reports GPU% = 20% while training, but the CPU is near 100%:
rocprof --hip-trace -o trace_out.csv python3 train.py
Diagnosis: Open results.json in Chrome tracing (chrome://tracing). If there are wide gaps between kernel completions and subsequent dispatches, the CPU host is bottlenecked on data loading (DataLoader), tokenization, or Python GIL contention, starving the GPU queue.
10. Important Interview Questions & Answers
Q: What is the architectural difference between an NVIDIA "Warp" and an AMD "Wavefront"?
Answer: Both are the fundamental units of lockstep execution in SIMT (Single Instruction, Multiple Threads). NVIDIA warps are strictly fixed at 32 threads. On AMD architectures, wavefront execution depends on the generation: AMD data-center compute architectures (CDNA: MI100, MI200, MI300) use Wave64 (64 threads) to maximize throughput on dense matrix operations, whereas consumer/graphics architectures (RDNA) support both Wave32 (for lower-latency branch diversion) and Wave64.
Q: How does AMD HIP achieve portability across both NVIDIA and AMD hardware?
Answer: HIP is a C++ runtime API and kernel language syntactically similar to CUDA. When compiling on AMD systems using hipcc, the code translates through Clang/LLVM down to native AMD GPU machine code (AMDGPU ISA). When compiling on an NVIDIA platform, hipcc uses header translation to route HIP calls directly to the native NVIDIA nvcc compiler and CUDA driver, allowing a single codebase to target both architectures with zero runtime translation overhead.
Q: What does low VALUUtilization combined with high VFetchInst mean in a rocprof report?
Answer: This profile indicates that the kernel is memory-bandwidth bound. The Vector ALUs are spending most of their clock cycles stalled, waiting for data to arrive from High Bandwidth Memory (HBM) over the cache hierarchy rather than executing arithmetic calculations. Optimization strategies include improving memory access coalescing, reusing data within local data share (LDS/shared memory), or using lower-precision formats (e.g., FP16 or FP8) to halve data transfer volumes.