Skill to Learn for LLMOPS

Skills to learn for AI/LLM/AISEC OPS

• GPU and Hardware management
• NVIDIA Drivers
• CUDA toolkit
• CUDA, cuDNN
• ROCm
• Troubleshoot driver mismatches and kernel panics
• nvidia-smi
• nvtop
• isolate memory leaks
• trace VRAM usage
• profile PCIe bus bottlenecks
• Hardware thermal limits
• monitoring VRAM

• NVIDA GPU Operator to map Physical GPUs directly to container runtimes

• Hardware Partitioning

• Learn to slice physical hardware resources
• NVIDIA MIG (Multi-Instance GPU (like an H100 or A100), Instances for smaller, parallel model tasks.

AI Containerisation and Orchestration

• Kubernetes NVDIA GPU Operator
• Schedule and allocate GPU Resources efficiently to containers
• Inference Engines and Model Serving
• vLLM
• Triton Inference (TIG)
• Optimize Token Latency and memory

Vector Datanases & Storage

• RAG (Retrieval agumented generation)
• Administer Vector Database like (Milvus, QDrant or Chroma)
• Understand high-speed NVME storage for model weights

Monitoring & Security

• Track GPU Metrics
• Token Throughput
• memory bottlenecks using prometheus and grafana
• Implement API security, Rate limiting
• LLM Guardrails ( NeMo, Guardrails) to prevent data leaks

Data Center Networking Requirements

• High speed storage and Model Loading
• Model Loading Monitoring
• Implement dedicated 10GbE or 25 GbE network paths

• minimum between storage arrays (NVMe-based SAN/NAS) GPU computing nodes.

• Multi-node Distributed Training & Inference

• Implement RoCE(RDMA over Converged Ethernet)
• InfiniBand switched
• layer-7 proxy routing
• Understanding Streaming responses (Server-Sent Events)

APi Management & Security

• Apache APISIX
• Apache Airflow
• Airflow DAGa to pull data from internal databases, chunk text,
• Embedding models

• push data to vector database on a schedule.

• Apache spark (Batch embeddings)

• Apache Lucene
• Elasticsearch
• OpenSearch

Local Inference Server

• vLLM

• Triton Inference Server

• Linux Performance tuning

• CUDA, ROCm, Core Driver Stack

• K8s GPU Operators & MIG

• ROCE / InfiniBand

• Multi-Node GPU Scaling (NCCL)
• Layer 7 Streaming
• AI Gateway

Model Serving & Inference

• Engines ( vLLM, Triton, TGI)
• Quantization (AWQ, GPTQ)
• high-throughput Token Tuning

Data & Vactor Architecture

• Vector DBs (Milvus, Qdrant)
• Storage (NVMe, Ceph, MinIO)
• Data Streaming (Apache Spark)

AI Pipelines & Lifecycle

• Orchestration (Apache Airflow)
• Fine-tuning (LoRA, QLoRA)
• Private Hugging Face Registriges

Monitoring, Security & Guardrails

• AI Telemetry (Prometheus, DCGM)
• Threat Vectors (Prompt Injection)
• Local Guardrails & Auditing

Advanced AI Networking

• kernel-Bypassing Fabrics
• RoCE (RDMA over Converged Ethernet)

• NCCL (NVIDIA Collective Communications Library)

• NVMe-oF(NVMe over Fabrics)

• Layer 7 Streaming Proxies API Gateways ( Apache APISIX or Envoy )
• Stream token chunk, Server-Sent Events (SSE) with triggering connection drops.

Distributed Inference Engines:

• vLLM
• Triton inference server
• Hugging Face TGI
• Continuous batching and Paged Attention Techniques

Model Quantization Operations

• AWQ, GPTQ, GGUF to shrink 16-bit floating-point weight into 8-bit or 4-bit configurations

• Context Optimiztation

• Vactor Database Administration

• Configure index parameter (HNSW, IVFPQ) in (Mivus, Qdrant, Chroma)

• Hybrid Data Processing

Monitoring Security and Guardrails

• NVIDIA DCGM Exporter
• Export deep GPU hardware execution matrics into Prometheus via the NVIDIA DCGM Exporter
• Displaying Live VRAM allocation
• Tensor Core Usage
• Thermal Tracking

• Grafana Dashboards

• Time to first token (TTFT)

• Tokens per seconds

• Total input/output volumes to forecast future hardware purchasing neede

• Local Policy Guardails

• Deploy defensive sanitization middleware
• NeMo Guardrails directly into your API Boundaries
• Flag and filter out prompt injection attacks
• Probhibited internal data outputs before they reach the network edge

• NVLink Layouts and topology awareness

• Placing tasks on GPUs

• NVIDIA GPU Operator

• nvidia-container-toolkit
• Kubeshare / volcano
• advanced batch-scheduling plugins for kubernetes
• Volcano optimizes scheduling for deep learning workloads
• learning workloads, ensuring that multi-node tasks start only when all required network and gpu resources are simultaneously free (Gang Scheduling)

Cluster

• Kubeshare / Volcano
• Ray Core
• Ray Cluster

Ansible

• Ansible Installs CUDA
• Ansible for Fabric Manager

• Ansible for Host Drivers

• Ray, vLLM

• ArgoCD

• GitOps Engine

Netowrking

• BGP
• iBGP
• Leaf-Spine Network Architecture

• Spine Switch > (eBGP / RoCE ) > Leaf Switch > GPU Server N > iBGP Pod)

• ECMP (Equal Cost Multi Path) networking

• When Server A needs to stream model layers to Server C, BGP splits the massive traffic load evently across all available spine switch simultaneously.

• iBGP overlay and EVPN

• iBGP inside your pods and compute racks, paired with EVPN (Ethernet VPN) and VXLAN

Tracing Debugging & Observablity

• PFC (Priority Flow Control)
• ECN (Explicit Congestion Notification)

• Traffic Priority queue

• Switch Telemetries to track PFC pause Frame and to spot network chokepints

• RoCE Debugging. Learn to look at /sys/class/infiniband/

• Debug RDMA

• OpenTelemetry for LLMs

• Asunchronous Streams
• Phoenix (by Arize)/ Langfuse
• NVIDIA NCCL tests

AISecOps

• Supply chain security
• safetensors file instead of legacy .bin or .pki (Pickle) fukes
• lockdown local Hugging face model registry
• Exfiltration Presentation
• Prompt Injection Mitigation
• Input/Output Guardrails
• Liama Guard
• NeMo Guardrails (Act as local firewall)
• SSNs or API keys
• Infrastructure Sandboxing
• Untrusted Code Execution

Netowrking

• Fabric Routing ( BGP, iBGP , EVPN-VXLAN)
• Congestion Control (PFC/ECN ) Prevent packet drops during massive multi-gigabyte model data transfers

• AI Tracing (Langfuse / Phoenix. OpenTelemetry) for Isolate Token latency bottlenecks across the AI serving Pipeline

• Performance Testing ( NCCL Tests ), Diagnose whether a Slow Model is caused by software bugs or a faulty hardware link

• Model Security

• Safetensors

• Internal Registries (Sanitize and lock down incoming AI weights against supply chain hacks)

• Llama guard, NeMo Guardails : Filter Prompt injections and block internal corporate data leakes.

Token Aware and Memory Aware routing

• AI Gateway (Apache APISIX AI GATEWAY or PortKey AI Gateway
• front of your inference engines (vLLM / Triton)

HTTP Server-Sent Events (SSE, so the user sees text appearing token-by-token.

User Request > Global AI Gateway > Active Layer > Server Room Rack A , Models Active in VRAM > Live PagedAttention Queue

Warm Standby Layer > Server Room Rack B > Models Pre-staged on NVMe > Local Model Cache synced

• Active- Passive (Warm Standby)

• Global AI Gateway

Distributed Storage Engines:

Continuously sync vector database changes (Milvus / Qdrant) and prompt history indexes across physical data center clusters.

State Replication:

The Design: Replicate your application layer and RAG data pipeline rather than trying to replicate live GPU States

GPUDirect Storage (GDS). (IMP)

• NVMe-Of (NVME over Fabrics)

• NVMe over RoCE (RDMA) or NVMe over TCP

• iSCSI use 256 commands

• NVMe-oF supports upto 65535 I/O queues . This creates the massive parrllel data lanes reqyired for deep learning

• WEKA Data platform (Proprietary ultrahigh performance distributed filesystem that ingegrates natively with GDS, delivering millions of IOPS acrsoss local NVMe storage pools.

• JuiceFS: (opensource distributed filesystem)

• Luster or IBM Spectrum Scale (GPFS)

AI Datacenter migration:

• Phase 1: Heavy Weight Pre--staging
• Phase 2: Vector Delta Synchronization (CDC) Change Data Capture (CDC) pipeline via apache airflow or Apache Kafka
• Phase3: The Blue-green Token Cutover ( Global AI Gateway ( Apache APISIX)

Unified Management Frameworks:

• KubeRay
• ClearML/ LMFlow
• NVIDIA NIM / vLLM Traker

• AuthN/AuthO

• Keycloak

• Active Directory via OIDC (OpenID Connect)
• RBAC
• Service Mesh (Inter-Pod Traffic) Istio or Linkerd
• AI proxy Gateway
• Apache APISIX or Envoy Proxy

• Token Per Minute (TPM)

• RPM (Requests per minutes)

• API Gateway Throttling

• JWT token expiration times are set too low, because LLM generation can take 30 - 66- seconds

• k9s or kubectl logs

• milvus-cli

• HTTP 504 Gateway timeout or connections drop mid-sentance

• HTTP 429 (Too many requests)

• HTTP 401 (Unauthorized) errors during live streaming for RAG

• HTTP 401 (Unauthorized) errors during live streaming for RAG

• apisix_http_status

• proxy_read_timeout

• CUDA out of Memory (OOM)

• gpu_memory_utilization)

• vllm: engine_vram_skyrocketing
• gdscheck
• iostat -x
• fio to trace read/write IOPS
• NCCL timeouts
• nccl-tests

• Drops mid-response: Apache APISIX logs
• container death on startup: nvidia-smi / host logs (CUDA/Driver version mismatch or VRAM allocation error

• slow first-token delivery : Arize Phoenix / Langfuse --> Context chunking or vector db index lookup latency

• Multi-node cluster stalls --> (nccl-test) switch CLI --> Missing RoCE/PFC configuration causing packet drops

• Missing RAG response text: Milvus RBAC Audit Log ( JWT security claims blocking)

• production-vllm-config.yaml
• model
• tensor_parallel_size
• pipeline_parallel_size
• gpu_memory_utilization
• max_model_len
• block_size
• max_num_seqs
• enable_chunked_prefill
• max_num_batched_tokens
• kv_cache_dtype
• quantization : "awq"
• disable_log_requests

• trust_remote_code

---

LLM OPS Solution Architect