Quick Overview of AI-Centric Data Center networks

AI data centers use multiple separate networks for compute, storage, in-band management, and out-of-band management—to isolate traffic for better performance, fault tolerance, security, and scalability.

NOTE on the network fabric:

                 AI DATA CENTER
                       │
        ┌──────────────┴──────────────┐
        │                             │
   SCALE-UP                       SCALE-OUT
        │                             │
 GPU ↔ GPU ↔ GPU                 Server ↔ Server
        │                             │
  NVLink / UALink                 IB / RoCE / UET
        │                             │
        └──────────────┬──────────────┘
                       │
                 MANAGEMENT /
                  STORAGE /
                  INTERNET
                       │
                    Ethernet

In a large Distributed AI factories there is increasingly another level:

Scale-up
   ↓
Scale-out
   ↓
Scale-across
   ↓
Multiple data centers / sites

NVLink Fabric)

NVLink = GPU-to-GPU scale-up fabric

This is primarily about connecting GPUs very closely together.

        NVLink / NVSwitch
                    │
       ┌────────────┼────────────┐
       │            │            │
      GPU          GPU          GPU
       │            │            │
      GPU          GPU          GPU
       │            │            │
      GPU          GPU          GPU

Instead of thinking, "I have 9 independent GPUs", the system can make communication between GPUs extremely fast so that the group behaves much more like a tightly coupled accelerator domain.

AI training uses collective operations such as:

AllReduce
AllGather
ReduceScatter
Broadcast

These create massive GPU-to-GPU traffic and NVLink is designed specifically for this type of communication.

NVLink as the scale-up fabric, while InfiniBand/Ethernet are used for scale-out.

Think:

NVLink = inside the GPU supercluster

NOTE:

NVLink is primarily designed for communication within a single node (intra-node). It creates a direct, high-speed interconnect between GPUs, CPUs, and other components located within the same server chassis or rack, enabling them to share memory and data without routing through the slower PCIe bus.

Key characteristics of NVLink's single-node operation include:

High Bandwidth: It provides significantly higher bandwidth than PCIe, reaching up to 1.8 TB/s per GPU on Blackwell architectures (NVLink 5.0) and 900 GB/s total in standard 8-GPU H100 nodes.

Low Latency: It achieves sub-microsecond latency, which is critical for tasks like tensor parallelism and KV cache sharing in AI workloads.

IMP: Direct Access: GPUs can directly access each other's memory without CPU involvement, eliminating traditional memory copying overhead.

While standard NVLink is limited to a single node, NVIDIA Multi-Node NVLink technology extends this capability to connect multiple servers, effectively creating a unified, large-scale GPU fabric across a rack or data center. However, for most standard configurations, NVLink serves as the backbone for high-performance computing within a single system.

Note:

you cannot communicate between GPUs on different nodes using only standard NVLink without an external network fabric like InfiniBand or Ethernet.

While standard NVLink is strictly limited to intra-node (within a single server) communication, NVIDIA has introduced Multi-Node NVLink (MNNVL) technology that extends this capability, but it still requires specific hardware infrastructure and does not eliminate the need for a physical interconnect fabric between nodes.

Here's the architecture diagram showing the relationships between CUDA Cores, GPUs, NVLink, and why the CPU/PCIe path is avoided:

Key relationships shown:

CUDA Cores live inside Streaming Multiprocessors (SMs), which are the compute units within each GPU.

Each SM contains 64–128 CUDA cores plus Tensor/RT cores.

NVLink (or NVSwitch) creates a direct high-speed highway between GPUs at ~600–900 GB/s — this is Peer-to-Peer (P2P) communication where data never leaves GPU memory.

PCIe connects the CPU to GPUs at only ~32–64 GB/s — that's 10–20× slower than NVLink and forces data to go through the CPU (extra copy overhead).

Why skip the CPU?

• No CPU copying: NVLink is zero-copy — GPU 1 writes directly to GPU 2's HBM memory.

• PCIe bottleneck: The CPU-mediated path is too slow for massive tensor exchanges needed in LLM training.

• Latency & scale: NVSwitch enables all-to-all communication across up to 256 GPUs, which PCIe simply cannot do.

Feature	NVLink Path (Recommended)	PCIe/CPU Path (Avoided)
Route	GPU Memory ↔ NVLink ↔ GPU Memory	GPU ↔ PCIe ↔ CPU RAM ↔ PCIe ↔ GPU
Bandwidth	~900 GB/s (Very High)	~32–64 GB/s (Low)
Latency	Sub-microsecond	Microseconds (Higher)
CPU Involvement	None (Direct DMA)	High (Data copying & management)
Use Case	AI Training, Large Model Inference	Legacy systems, initialization

NVLink and NVSwitch

The fundamental difference is that NVLink is the high-speed communication protocol and physical link (the "cable" or "lane"), while NVSwitch is the silicon chip (the "traffic controller") that manages these links to create a fully connected network.

NVLink: The "Highway" (Protocol & Link)

NVLink is a point-to-point interconnect technology that replaces the slower PCIe bus for GPU communication.

Function: It provides the physical lanes and protocol for data to travel between two devices (GPU-to-GPU or GPU-to-CPU).

Limitation: Without a switch, NVLink can only connect devices in fixed, limited topologies (e.g., GPU A directly to GPU B). Connecting 8 GPUs directly to each other would require an impractical number of physical links on every GPU.

Analogy: Think of NVLink as the individual lanes of a highway.

NVSwitch: The "Interchange" (Switching Fabric)

NVSwitch is a dedicated silicon chip (ASIC) that acts as a central crossbar switch for NVLink connections.

Function: It aggregates multiple NVLink ports and allows any connected GPU to communicate with any other GPU simultaneously at full bandwidth. It creates a non-blocking, all-to-all fabric.

Capability: It eliminates the need for GPUs to "hop" through other GPUs to reach a distant one. In systems like the DGX H100, NVSwitch chips ensure every GPU has a direct logical path to every other GPU

Advanced Features: Modern NVSwitch generations (3.0+) include hardware acceleration for collective operations (like SHARP), allowing the switch itself to perform math operations (e.g., summing gradients) rather than just moving data.

Analogy: Think of NVSwitch as the massive highway interchange that allows cars from any entrance ramp to exit at any other ramp simultaneously without traffic jams.