nsys (NVIDIA Nsight System)

@amitmund September 10, 2026

NVIDIA Nsight Systems (`nsys`) — Complete Learning Notes & Output Guide

NVIDIA Nsight Systems (nsys) is the flagship system-wide performance analysis tool for applications running on NVIDIA GPUs and CPUs. It provides a holistic timeline visualization of CPU execution, multi-GPU orchestration, CUDA kernel launches, memory transfers over PCIe, and graphics API calls (Vulkan, DirectX, OpenGL).

1. What is Nsight Systems?

While tools like Nsight Compute profile individual CUDA kernels at the instruction level, Nsight Systems profiles the macro timeline of your entire application across both host (CPU) and device (GPU).

It answers critical performance questions:

Why are my GPUs sitting idle while the CPU sits at 100% (CPU bottleneck)?
Are host-to-device (HtoD) and device-to-host (DtoH) memory copies overlapping with computation, or are they serial bottlenecks?
Which CUDA streams are serializing execution when they should be running concurrently?
Where are the latency bubbles between multi-GPU communication loops (NCCL / NVLink)?

2. Installation & Prerequisites

Nsight Systems is packaged alongside the NVIDIA CUDA Toolkit and can also be downloaded independently.

Installation via CUDA Toolkit / APT (Ubuntu)

sudo apt update
sudo apt install nsight-systems-cli

Verify installation:

nsys --version

3. Basic Syntax & Core Subcommands

nsys executes a target program, collects tracing timelines, and saves them to a report file (.nsys-rep) for analysis.

nsys <subcommand> [options] [command] [args]

Subcommand	Purpose	Practical Example
`nsys profile`	Profile a command line application and generate a report.	`nsys profile -o my_report python3 train.py`
`nsys stats`	Generate an ASCII summary table directly from a `.nsys-rep` report file.	`nsys stats my_report.nsys-rep`
`nsys export`	Convert binary reports into alternative formats (e.g., SQLite databases).	`nsys export -t sqlite -o report.sqlite my_report.nsys-rep`

4. Essential Profiling Options

Flag	Description	Practical Example
`-o, --output <name>`	Base name for the generated report file.	`nsys profile -o resnet_run python3 train.py`
`--trace=<list>`	Specify which APIs/subsystems to trace (e.g., `cuda,nvtx,osrt,opengl`).	`nsys profile --trace=cuda,nvtx python3 train.py`
`-f, --force-overwrite`	Overwrite existing report files with the same name.	`nsys profile -f -o run python3 app.py`
`--cpuctxsw=none`	Disable OS context switch tracking to reduce report size.	`nsys profile --cpuctxsw=none python3 app.py`

5. Analyzing Reports via CLI Summary (`nsys stats`)

Instead of opening the GUI visualizer (nsight-sys), you can instantly extract performance tables directly in your terminal.

Command

nsys stats report1.nsys-rep

Raw Output Example (CUDA Kernel Summary)

 -----------------------------------------------------------------------------------
 Time (%)  Total Time (ns)  Instances  Avg (ns)  Med (ns)  Min (ns)  Max (ns)  StdDev (ns)  Name
 --------  ---------------  ---------  --------  --------  --------  --------  -----------  ----
     45.2       1420102450      10000    142010    141900    141000    145200         1250  sgemm_kernel
     32.1       1008450100       5000    201690    201500    200000    210000         2400  relu_activation
     22.7        714501050       5000    142900    142800    141500    148000         1800  layer_norm
 -----------------------------------------------------------------------------------

Detailed Breakdown of `nsys stats` Headings

Heading	Technical Meaning	Troubleshooting Significance
`Time (%)`	Proportion of total GPU execution time consumed by this specific kernel.	Instantly highlights your dominant computation kernels.
`Total Time (ns)`	Cumulative nanoseconds spent across all instances of the kernel.	Measures overall workload weight.
`Instances`	Total number of times the kernel was launched during the profiling window.	High instance counts with tiny durations can reveal kernel launch overhead bottlenecks.
`Avg (ns)`	Average execution duration per launch instance.	Identifies stability and performance consistency.
`Name`	The mangled or demangled CUDA kernel symbol name.	Correlates back to source code functions.

6. Raw Output Example: Memory Transfer Summary (`HtoD` / `DtoH`)

nsys stats --report gpumemsum report1.nsys-rep

Raw Output

 -----------------------------------------------------------------------------------
 Total Time (ns)  Count  Avg Size (B)  Total (MB)  Throughput (GB/s)  Operation
 ---------------  -----  ------------  ----------  -----------------  ---------
      4520102450   2000    104857600       190.73             12.45  [CUDA memcpy HtoD]
      1201024500   2000     52428800        95.36             15.10  [CUDA memcpy DtoH]
 -----------------------------------------------------------------------------------

Breakdown of Memory Metrics

Total (MB): Cumulative data volume transferred over the PCIe bus during the session.
Throughput (GB/s): Effective bandwidth achieved. If this is drastically lower than your PCIe generation limit (e.g., PCIe 4.0 x16 theoretical max ~31.5 GB/s), your application is suffering from uncoalesced memory allocations or small, fragmented transfers.

7. Using NVTX Markers to Custom-Instrument Code

To trace specific Python or C++ loops in the Nsight Systems timeline, use NVTX (NVIDIA Tools Extension) markers.

Python Example

import torch
import torch.cuda.nvtx as nvtx

# Mark a specific training step
nvtx.range_push("Forward_Pass")
output = model(inputs)
nvtx.range_pop()

nvtx.range_push("Backward_Pass")
loss.backward()
nvtx.range_pop()

When profiled with nsys profile --trace=cuda,nvtx python3 train.py, these custom ranges appear as colored collapsible rows on the timeline UI, allowing you to map raw CUDA execution back to high-level application phases.

8. Real-World Troubleshooting Scenarios

Scenario A: Diagnosing CUDA Stream Serialization Bubbles

Your multi-GPU training script shows lower GPU utilization than expected.

Profile with Nsight Systems:

nsys profile --trace=cuda,osrt -o stream_test python3 train.py

Diagnosis: Opening the timeline reveals that multiple operations are executing sequentially on Stream 0 (the default stream) instead of overlapping across asynchronous custom CUDA streams (cudaStream_create), creating idle "bubbles" of time where the GPU waits for host launches.

Scenario B: Finding Host-Side Synchronization Stalls

The GPU stops executing for milliseconds at a time between iterations.

Profile with OS runtime and CUDA tracking:

nsys profile --trace=cuda,osrt -o sync_test python3 app.py

Diagnosis: Identifies that the host thread is blocking on cudaDeviceSynchronize() or excessive CPU-GPU round-trips (such as calling .item() or .numpy() inside a tight training loop in PyTorch, forcing the CPU to halt until the GPU finishes).

9. Important Interview Questions & Answers

Q: What is the fundamental difference in purpose between Nsight Systems (`nsys`) and Nsight Compute (`ncbr` / `ncu`)?

Answer: Nsight Systems is a system-wide macro profiler. It visualizes the entire application timeline across CPUs, multiple GPUs, memory transfers, and operating system runtimes to identify scheduling bottlenecks, CPU-GPU idle bubbles, and concurrency issues. Nsight Compute is a micro-profiler that dives deep into a single specific CUDA kernel, analyzing instruction-level metrics, register pressure, memory cache hit rates, and arithmetic intensity (FLOPs/byte) on the streaming multiprocessors (SMs).

Q: Why does profiling with `nsys profile` sometimes alter application execution behavior?

Answer: nsys intercepts driver APIs, injects instrumentation hooks, and monitors kernel execution queues. This instrumentation adds minor overhead to API call latencies and memory allocations (known as the probe effect). While usually negligible on macro-level timelines, it can occasionally perturb race conditions or fine-grained thread timings in highly sensitive multi-threaded applications.

NVIDIA Nsight Systems (nsys) — Complete Learning Notes & Output Guide