Notes on AI Infrastructure and Operations
NCA-AIIO
The NVIDIA Certified Associate AI Infrastructure and Operations (NCA-AIIO) certification is designed for professionals who manage or deploy AI-centric data centers. It focuses on foundational knowledge of AI infrastructure, including NVIDIA technologies like GPUs, networking, and operational tools.
What is Algorithm breakthroughs in AI ?
Algorithm breakthroughs in AI refer to significant advancements in the methods and architectures used to build and train AI models, enabling them to perform complex tasks more efficiently and accurately. Key breakthroughs include the development of neural networks, convolutional neural networks (CNNs), and transformer models.
Transformers, in particular, revolutionised natural language processing by introducing the attention mechanism, which allows models to focus on relevant parts of the input data.
What is Inference?
inference is the process where a trained model is used to make predictions or decisions based on new input data.
After a model has been trained on large datasets (model training), inference applies that learned knowledge to analyse new information and generate results, like recognising objects in images or understanding text.
What is AI?
AI (Artificial Intelligence) is the broad concept where machines simulate human intelligence and decision-making.
What is ML?
Machine Learning (ML) is a subset of AI where machines learn from data without explicit programming.
What is DL?
Deep Learning (DL) is a specialized form of ML that uses artificial neural networks inspired by the human brain to process data.
what is Gen AI?
Generative AI (Gen AI) builds on these by creating new content like text, images, or code based on learned patterns.
What is transformer model?
A transformer model is an AI architecture that helps machines understand the relationships between words in a sentence by focusing on important parts of the input, a concept called "attention."
This model uses parallel computing to handle many calculations at once, making it efficient. It's a key technology behind generative AI, enabling machines to create text, stories, or even novels by predicting and building on words step-by-step.
What is PUE?
PUE, or Power Usage Effectiveness, is a metric used to measure how efficiently a data center uses energy.
It is calculated by dividing the total energy consumed by the entire data center by the energy used specifically by the IT equipment (servers, storage, and network).
A lower PUE means the data center is more efficient, with less energy wasted on cooling, power conversion, and other overhead.
Modern AI-focused data centers aim for a PUE of around 1.2 or less, meaning about 80% of the power goes directly to IT equipment, making operations greener and more cost-effective.
What is the difference between CPU and GPU in the context of an AI-centric data center?
In an AI-centric data center, the main difference between CPU and GPU lies in their design and purpose:
CPU (Central Processing Unit): It's a general-purpose processor designed to handle a wide variety of tasks sequentially. CPUs are great for complex logic and control operations but have fewer cores optimized for single-threaded performance.
GPU (Graphics Processing Unit): Originally built for rendering graphics efficiently by handling many operations in parallel, GPUs have thousands of smaller cores designed for parallel processing. This makes them ideal for AI workloads like training and inference, where many calculations happen simultaneously.
Architecturally, GPUs excel at processing large blocks of data in parallel, which accelerates AI model training and inference, while CPUs manage overall system control and diverse computing tasks.
This distinction is why AI-centric data centers rely heavily on GPUs alongside CPUs to achieve high-performance AI processing.
What is CUDA?
CUDA (Compute Unified Device Architecture) is a parallel computing platform and programming model developed by NVIDIA.
It allows developers to use NVIDIA GPUs for general-purpose computing, not just graphics.
This means you can run complex AI and machine learning tasks much faster by harnessing the power of many GPU cores working in parallel.
CUDA is essential for accelerating AI workloads in AI-centric data centers, making it a key technology covered in this course.
What is DPU (Data Processing Unit)?
A Data Processing Unit (DPU) is a specialized processor in AI-centric data centers designed to handle data-centric tasks like networking, storage acceleration, and security.
DPUs offload work such as packet processing, load balancing, encryption, and firewall operations from CPUs and GPUs, allowing those processors to focus on core computing tasks.
This improves overall efficiency and performance in AI data centers by distributing responsibilities effectively.
Networks inside an AI-centric Data Center:
AI data centers use multiple separate networks for compute, storage, in-band management, and out-of-band management—to isolate traffic for better performance, fault tolerance, security, and scalability.
The compute network handles communication between nodes for application traffic,
while the storage network ensures fast, bottleneck-free data access.
In-band management networks support configuration and monitoring without interfering with application traffic.
Out-of-band management networks allow remote management even if the server OS is down, using specialized hardware like Baseboard Management Controllers (BMCs).
This separation of network fabrics is crucial for maintaining efficient and reliable AI data center operations.
What is Compute Network Fabric?
The compute network fabric in AI-centric data centers is commonly implemented using technologies like InfiniBand, RoCE (RDMA over Converged Ethernet), and NVLink.
These provide the high bandwidth and ultra-low latency needed for fast GPU-to-GPU communication within and across nodes, ensuring scalable and efficient AI training and inference workloads.
Key:
GPU-to-GPU, within the node and across the Nodes.
What is Storage Network Fabric?
Storage Network Fabric in an AI-centric data centerconnects compute nodes(like servers) to storage appliances such asstorage arraysorparallel file systems`.
Its main role is to support file system access, checkpoints, and input/output traffic efficiently.
This fabric is typically implemented using technologies like InfiniBand or Ethernet RoCE, depending on the design.
A key requirement is that it delivers multi-gigabytes per second throughput per node and stays isolated from compute network traffic to avoid performance bottlenecks.
This separation ensures smooth and fast data access for AI workloads without interference from compute operations.
What is In-Band management network fabric?
The In-Band management network fabric in an AI-centric data center handles control plane traffic like cluster management, SSH access, DNS, job scheduling, and downloading patches or dependencies.
It's Ethernet-based and designed with isolation techniques such as Leaf-Spine architecture and VLAN/VXLAN/EVPN to keep management traffic separate from compute and storage traffic.
This network needs to be reliable, secure, and support moderate to high bandwidth to ensure smooth management operations without interfering with AI workloads.
what is out-of-band management network fabric?
The out-of-band management network fabric in an AI-centric data center provides remote management capabilities even if the server's operating system is down or the server is powered off.
It allows tasks like power control and serial console access through specialized hardware such as Baseboard Management Controllers (BMCs).
This network is separate from others, uses low-speed switches, and must always be available with strong access control and security.
It acts as a last-resort way to manage and recover servers when in-band management is not accessible, ensuring reliable and secure data center operations.
NOTE on the network fabric:
AI DATA CENTER
│
┌──────────────┴──────────────┐
│ │
SCALE-UP SCALE-OUT
│ │
GPU ↔ GPU ↔ GPU Server ↔ Server
│ │
NVLink / UALink IB / RoCE / UET
│ │
└──────────────┬──────────────┘
│
MANAGEMENT /
STORAGE /
INTERNET
│
Ethernet
In a large Distributed AI factories there is increasingly another level:
Scale-up
↓
Scale-out
↓
Scale-across
↓
Multiple data centers / sites
NVLink Fabric
NVLink = GPU-to-GPU scale-up fabric
This is primarily about connecting GPUs very closely together.
NVLink / NVSwitch
│
┌────────────┼────────────┐
│ │ │
GPU GPU GPU
│ │ │
GPU GPU GPU
│ │ │
GPU GPU GPU
Instead of thinking, "I have 9 independent GPUs", the system can make communication between GPUs extremely fast so that the group behaves much more like a tightly coupled accelerator domain.
AI training uses collective operations such as:
AllReduce
AllGather
ReduceScatter
Broadcast
These create massive GPU-to-GPU traffic and NVLink is designed specifically for this type of communication.
NVLink as the scale-up fabric, while InfiniBand/Ethernet are used for scale-out.
Think:
NVLink = inside the GPU supercluster
NOTE:
NVLink is primarily designed for communication within a single node (intra-node). It creates a direct, high-speed interconnect between GPUs, CPUs, and other components located within the same server chassis or rack, enabling them to share memory and data without routing through the slower PCIe bus.
Key characteristics of NVLink's single-node operation include:
High Bandwidth: It provides significantly higher bandwidth than PCIe, reaching up to 1.8 TB/s per GPU on Blackwell architectures (NVLink 5.0) and 900 GB/s total in standard 8-GPU H100 nodes.
Low Latency: It achieves sub-microsecond latency, which is critical for tasks like tensor parallelism and KV cache sharing in AI workloads.
IMP: Direct Access: GPUs can directly access each other's memory without CPU involvement, eliminating traditional memory copying overhead.
While standard NVLink is limited to a single node, NVIDIA Multi-Node NVLink technology extends this capability to connect multiple servers, effectively creating a unified, large-scale GPU fabric across a rack or data center. However, for most standard configurations, NVLink serves as the backbone for high-performance computing within a single system.
Note:
you cannot communicate between GPUs on different nodes using only standard NVLink without an external network fabric like InfiniBand or Ethernet.
While standard NVLink is strictly limited to intra-node (within a single server) communication, NVIDIA has introduced Multi-Node NVLink (MNNVL) technology that extends this capability, but it still requires specific hardware infrastructure and does not eliminate the need for a physical interconnect fabric between nodes.
Here's the architecture diagram showing the relationships between CUDA Cores, GPUs, NVLink, and why the CPU/PCIe path is avoided:
Key relationships shown:
CUDA Cores live inside Streaming Multiprocessors (SMs), which are the compute units within each GPU.
Each SM contains 64–128 CUDA cores plus Tensor/RT cores.
NVLink (or NVSwitch) creates a direct high-speed highway between GPUs at ~600–900 GB/s — this is Peer-to-Peer (P2P) communication where data never leaves GPU memory.
PCIe connects the CPU to GPUs at only ~32–64 GB/s — that's 10–20× slower than NVLink and forces data to go through the CPU (extra copy overhead).
Why skip the CPU?
No CPU copying: NVLink is zero-copy — GPU 1 writes directly to GPU 2's HBM memory.
PCIe bottleneck: The CPU-mediated path is too slow for massive tensor exchanges needed in LLM training.
Latency & scale: NVSwitch enables all-to-all communication across up to 256 GPUs, which PCIe simply cannot do.