1. Hardware, Drivers & GPU Foundation
Nothing above this layer works if this layer is wrong.
| Term |
Description |
| NVIDIA Drivers |
Kernel-level driver binding the OS to the physical GPU. Version mismatches against CUDA/container images are the #1 cause of container-startup failures. |
| CUDA Toolkit / cuDNN |
NVIDIA's compute API + deep-learning-optimized primitives library that frameworks (PyTorch, TensorFlow) build on. |
| ROCm |
AMD's equivalent compute stack to CUDA, for AMD GPU hardware. |
| nvidia-smi / nvtop |
CLI/TUI tools for live GPU state — utilization, VRAM usage, temperature, running processes. First tools to check in any GPU incident. |
| NVIDIA DCGM Exporter |
Exports deep hardware telemetry (VRAM allocation, Tensor Core usage, thermal data) into Prometheus for Grafana dashboards — the production-grade successor to manually polling nvidia-smi. |
| NVIDIA MIG (Multi-Instance GPU) |
Hardware-level partitioning of a single physical GPU (A100/H100) into isolated smaller instances — lets multiple smaller inference jobs share one card safely. |
| NVLink |
High-bandwidth GPU-to-GPU interconnect within a server; topology awareness matters when placing multi-GPU jobs so they land on directly-linked cards. |
| Thermal limits & PCIe bottleneck profiling |
Sustained load can throttle GPUs before they hit a hard failure — profiling PCIe bus and thermal headroom explains "mysteriously slow" nodes that aren't erroring out. |
2. Containerization & Orchestration
| Term |
Description |
| NVIDIA GPU Operator |
Kubernetes operator that automates driver/toolkit installation and maps physical GPUs into container runtimes — the standard way GPUs become schedulable K8s resources. |
| nvidia-container-toolkit |
The runtime component that actually exposes a host GPU inside a container. |
| Volcano / Kubeshare |
Batch-scheduling plugins for Kubernetes purpose-built for deep learning. Volcano's key feature is gang scheduling — a multi-node job only starts once all required GPU + network resources are simultaneously available, avoiding partial-start deadlocks. |
| Ray Core / Ray Cluster / KubeRay |
Distributed compute framework for scaling Python/ML workloads across a cluster; KubeRay runs Ray natively on Kubernetes. |
| ArgoCD / GitOps |
Declarative, Git-driven deployment for the cluster — config drift is caught by diffing against the repo rather than manual kubectl apply. |
3. Data Center Networking & Fabric
The layer that makes multi-node training/inference possible at all.
| Term |
Description |
| Leaf-Spine architecture |
Standard datacenter topology: Spine switches ↔ Leaf switches ↔ GPU servers, chosen for predictable low-latency, non-blocking paths. |
| BGP / iBGP / ECMP |
Routing protocols directing traffic between racks. ECMP (Equal Cost Multi-Path) splits large flows (e.g. one server streaming model layers to another) evenly across all available spine links simultaneously. |
| EVPN-VXLAN |
Overlay network technology (paired with iBGP) that virtualizes Layer 2 connectivity across racks/pods. |
| RoCE (RDMA over Converged Ethernet) / InfiniBand |
Kernel-bypass fabrics — data moves GPU-to-GPU without going through the CPU/OS network stack, essential for multi-node training/inference latency. |
| NCCL (NVIDIA Collective Communications Library) |
The library actually performing multi-GPU/multi-node collective operations (all-reduce, etc.) over RoCE/InfiniBand. nccl-tests is the standard tool to isolate whether a slowdown is a software bug or a bad physical link. |
| PFC (Priority Flow Control) / ECN (Explicit Congestion Notification) |
Congestion-control mechanisms preventing packet drops during massive multi-GB model transfers; switch telemetry tracks PFC pause frames to spot chokepoints. |
| NVMe-oF (NVMe over Fabrics) |
Extends NVMe storage access over the network fabric (via RoCE or TCP) — supports up to 65,535 I/O queues vs iSCSI's 256, enabling the parallel data lanes deep learning storage needs. |
| GPUDirect Storage (GDS) |
Lets GPUs read/write storage directly, bypassing the CPU — pairs with NVMe-oF for model-loading throughput. |
4. Storage & Data Architecture
| Term |
Description |
| High-speed NVMe storage |
Baseline requirement for model weight loading — spinning disk or standard network storage becomes the bottleneck otherwise. |
| Distributed filesystems |
WEKA (proprietary, ultra-high-performance, natively integrates with GDS), JuiceFS (open-source alternative), Lustre / IBM Spectrum Scale (GPFS) (HPC-heritage parallel filesystems) — all solving the same problem of feeding many GPUs from shared storage in parallel. |
| Vector Databases (Milvus, Qdrant, Chroma) |
Store embeddings for RAG retrieval. Index parameters like HNSW and IVFPQ trade off query speed vs recall accuracy vs memory footprint — the key tuning knob for RAG latency. |
| Elasticsearch / OpenSearch / Apache Lucene |
Traditional text search engines, often used alongside vector search for hybrid (keyword + semantic) retrieval. |
| Apache Spark |
Batch processing framework, used here for bulk/offline embedding generation at scale. |
5. Model Serving & Inference
| Term |
Description |
| vLLM / Triton Inference Server / Hugging Face TGI |
The three dominant inference engines. Core techniques they implement: continuous batching (don't wait for a full batch — add/remove requests mid-flight) and PagedAttention (manage the KV cache like OS virtual memory, avoiding fragmentation). |
| Quantization (AWQ, GPTQ, GGUF) |
Shrinks 16-bit floating-point weights to 8-bit or 4-bit, trading a small accuracy loss for large VRAM/throughput gains. |
| Token-level metrics |
TTFT (Time To First Token) and tokens/sec are the two numbers that actually describe user-perceived performance — more meaningful than raw GPU utilization alone. |
Reference: Production vLLM Config Parameters
| Parameter |
Purpose |
tensor_parallel_size |
Splits a single model's layers across N GPUs (intra-node parallelism). |
pipeline_parallel_size |
Splits model layers across pipeline stages (often across nodes). |
gpu_memory_utilization |
Fraction of GPU VRAM vLLM is allowed to claim for KV cache + weights. |
max_model_len |
Maximum context length (tokens) the engine will serve. |
block_size |
KV-cache page size for PagedAttention. |
max_num_seqs |
Maximum concurrent sequences (requests) batched together. |
enable_chunked_prefill |
Splits long prompt prefill into chunks so it can interleave with ongoing decode steps, reducing latency spikes. |
max_num_batched_tokens |
Caps total tokens processed per batching step across all sequences. |
kv_cache_dtype |
Data type for the KV cache (e.g. fp8) — trades memory for precision. |
quantization: "awq" |
Selects the quantization method for the loaded weights. |
disable_log_requests |
Turns off per-request logging — a production throughput/log-volume tradeoff. |
trust_remote_code |
Allows executing custom model code from the Hugging Face repo — a supply-chain security consideration (see Section 9). |
6. AI Pipelines & Lifecycle
| Term |
Description |
| RAG (Retrieval-Augmented Generation) |
The overall pattern: chunk source data → embed → store in a vector DB → retrieve relevant chunks at query time → feed to the LLM as context. |
| Apache Airflow |
Orchestrates the RAG ingestion pipeline as a DAG — pull from internal databases, chunk text, run embedding models, push to the vector database on a schedule. Also used for CDC (Change Data Capture) delta-sync during migrations (Section 10). |
| Fine-tuning (LoRA, QLoRA) |
Parameter-efficient fine-tuning methods — adapt a base model without retraining all its weights. |
| Private Hugging Face Registries |
Self-hosted model registry so weights don't depend on (or leak to) the public Hub — pairs with the supply-chain security concerns in Section 9. |
7. API Gateway, Access & Security Boundary
| Term |
Description |
| AI Gateway (Apache APISIX, Envoy, PortKey) |
Layer-7 proxy sitting in front of inference engines — handles routing, streaming, rate limiting, and auth in one place rather than each engine handling it independently. |
| SSE (Server-Sent Events) |
The streaming mechanism behind token-by-token chat responses — the client keeps one connection open and receives incremental chunks. |
| AuthN/AuthZ (Keycloak, Active Directory via OIDC) |
Identity and authentication layer for who's allowed to call the gateway at all. |
| RBAC |
Role-based authorization — what an authenticated caller is allowed to do/see (including inside vector DBs like Milvus, which has its own RBAC audit log). |
| Service Mesh (Istio, Linkerd) |
Manages inter-pod traffic (mTLS, retries, observability) inside the cluster, distinct from the north-south AI Gateway at the edge. |
| Rate limiting — TPM (Tokens Per Minute) / RPM (Requests Per Minute) |
LLM-specific throttling; token-based limits matter more than request-based ones since one request's cost varies wildly by output length. |
| JWT expiration tuning |
A common real bug: short token TTLs (meant for typical web APIs) expire mid-generation, since LLM responses can legitimately take 30–60+ seconds — causing spurious 401s mid-stream. |
8. Monitoring & Observability
| Term |
Description |
| Prometheus + Grafana |
The standard metrics pipeline — scraping GPU telemetry (via DCGM Exporter) and application metrics, visualized on dashboards. |
| Key GPU metrics |
VRAM allocation, Tensor Core usage, thermal state, GPU utilization over time — used both for real-time alerting and for forecasting future hardware purchasing needs. |
| Key LLM metrics |
TTFT, tokens/sec, input/output token volume. |
| OpenTelemetry for LLMs (Langfuse, Arize Phoenix) |
Distributed tracing purpose-built for LLM pipelines — isolates exactly which hop (retrieval, embedding, generation) is causing latency. |
| NCCL tests |
The go-to diagnostic for distinguishing a software bug from a genuinely faulty network link in multi-node setups. |
9. Security & AISecOps
| Term |
Description |
| Supply chain security |
Prefer safetensors over legacy .bin/.pkl (Pickle) files — Pickle can execute arbitrary code on load, a real attack vector for "malicious model weights." |
| Model registry lockdown |
Sanitize and control what weights enter your internal Hugging Face registry before they reach any GPU node. |
| Prompt injection mitigation |
NeMo Guardrails and Llama Guard act as an input/output firewall — filtering malicious instructions in and sensitive data (SSNs, API keys, internal docs) out, before it reaches the network edge. |
| Infrastructure sandboxing |
Isolating untrusted code execution (e.g. a model's custom inference code, or agent tool-calls) from the rest of the cluster. |
| Exfiltration prevention |
The combined effect of guardrails + RBAC + registry lockdown — stopping sensitive internal data from leaving via model output. |
10. High Availability & Multi-Datacenter
| Term |
Description |
| Active-Passive (Warm Standby) |
Rack A serves live traffic (models resident in VRAM); Rack B keeps models pre-staged on NVMe with a synced local cache, ready to take over — not running a full duplicate hot cluster. |
| Global AI Gateway |
The single entry point that can redirect traffic from Active to Standby during failover. |
| State replication design principle |
Replicate the application layer and RAG data pipeline (vector DB state, prompt history) — don't attempt to replicate live in-VRAM GPU state, which isn't practical. |
| Datacenter migration phases |
1) Heavy-weight pre-staging (move large model weights first), 2) CDC-based vector delta sync (Airflow/Kafka), 3) Blue-green token cutover at the Global AI Gateway. |
11. Automation (Ansible for AI Infra)
| Term |
Description |
| Ansible for driver/CUDA install |
Standardizes GPU node bring-up: install NVIDIA drivers, CUDA toolkit, and NVIDIA Fabric Manager consistently across the fleet. |
| Ansible for Ray/vLLM deployment |
Same configuration-management discipline applied to the inference software stack, not just the OS/driver layer. |
(See the earlier Ansible chapters for the full idempotency/role/Vault treatment — the same principles apply directly to GPU node provisioning.)
12. Troubleshooting Reference
| Symptom |
Likely Cause |
Where to Look |
| HTTP 504 / connection drops mid-sentence |
Gateway proxy timeout shorter than generation time |
proxy_read_timeout, AI Gateway (APISIX) logs |
| HTTP 429 (Too Many Requests) |
Rate limit (TPM/RPM) exceeded |
apisix_http_status metrics |
| HTTP 401 mid-stream (RAG) |
JWT expired during a long generation |
JWT TTL config vs actual generation time |
| Missing RAG response text |
Vector DB RBAC blocking the query |
Milvus RBAC audit log |
| CUDA Out of Memory (OOM) |
KV cache / batch size exceeds VRAM budget |
gpu_memory_utilization, vLLM engine VRAM metrics |
| Container dies on startup |
Driver/CUDA version mismatch |
nvidia-smi, host driver logs |
| Slow first-token delivery |
Vector DB index lookup or context chunking latency |
Langfuse / Arize Phoenix trace |
| Multi-node cluster stalls |
Missing/misconfigured RoCE or PFC, dropped packets |
nccl-tests, switch telemetry |
| Node genuinely slow but not erroring |
PCIe bottleneck or thermal throttling |
nvidia-smi, nvtop, thermal metrics |
| Storage-bound slowness |
Disk I/O can't keep up with model loading |
iostat -x, fio, gdscheck |
How the Layers Compose (Request Flow)
User Request
│
▼
Global AI Gateway (APISIX/Envoy) ── AuthN/AuthZ, rate limiting, SSE streaming
│
▼
Active Layer (Rack A) ──────────────── Warm Standby (Rack B, failover target)
│ │
▼ ▼
Inference Engine (vLLM/Triton/TGI) Pre-staged models on NVMe
│ PagedAttention + continuous batching
▼
RAG Retrieval ── Vector DB (Milvus/Qdrant) ── Embeddings from Airflow pipeline
│
▼
GPU Compute ── MIG partitioning, NVLink topology, K8s GPU Operator scheduling
│
▼
Storage Fabric ── NVMe-oF/GDS, RoCE/InfiniBand, NCCL for multi-node
│
▼
Observability (Prometheus/Grafana/DCGM, Langfuse/Phoenix) watches every hop
│
▼
Guardrails (NeMo/Llama Guard) sanitize input/output at every boundary