Distributed LLM

Key Concepts on Distributed LLM

To run a distributed LLM, you must split the model across multiple GPUs or nodes using strategies like tensor parallelism (splitting layers within a layer) and pipeline parallelism (splitting layers across nodes). This is essential when model weights exceed the memory capacity of a single GPU.

---

Questions:

What is Tensor Parallelism?

Pipeline Parallelism?

---

Choose a Framework

• vLLM: Best for high-throughput production serving. It uses Ray for multi-node orchestration and supports tensor parallelism (within a node) and pipeline parallelism (across nodes).

• Llama.cpp: Best for heterogeneous hardware (e.g., mixed Mac/Linux devices) or local setups. It uses an RPC backend to distribute GGUF model layers across machines.

• exo: Handles heterogeneous fleets gracefully, allowing devices like Mac Studios, iPhones, and Linux PCs to participate in a single inference cluster.

Configure Parallelism

• Single-Node Multi-GPU: Use tensor parallelism if the model fits in one node but requires multiple GPUs. Set tensor_parallel_size to the number of GPUs.

• Multi-Node Multi-GPU: Use pipeline parallelism combined with tensor parallelism if the model exceeds a single node's capacity. Set tensor_parallel_size for GPUs per node and pipeline_parallel_size for the number of nodes.

Infrastructure Requirements

• Network: High-speed interconnects like InfiniBand or NVLink are critical for tensor parallelism to avoid bottlenecks from inter-node communication traffic.

• Setup: For vLLM, initialize a Ray cluster and ensure all nodes have identical environments (e.g., via Docker). For Llama.cpp, start rpc-server on workers and connect via --rpc on the head node

---

Questions:

What is InfiniBand?

What is NVLink?

What is Ray cluster?

Explore the Llama.cpp

---

from vllm import LLM

llm = LLM(
    model="meta-llama/Llama-2-70b-chat-hf",
    tensor_parallel_size=4,       # 4 GPUs in one node
    pipeline_parallel_size=2,     # 2 nodes
    dtype="float16"
)   

---

Core Concepts

Distributed Inference splits model computations across multiple GPUs or nodes to handle models exceeding single-device memory.

• Tensor Parallelism (TP): Splits individual layers across GPUs within a node. Requires high-bandwidth interconnects (NVLink).

• Pipeline Parallelism (PP): Splits model layers across nodes. Tolerates slower networks better than TP.

• Data Parallelism: Replicates the model across nodes; each handles different batches. Often combined with TP/PP for MoE models.

---

Frameworks & Setup

vLLM (Production Serving)

Best for high-throughput, multi-node clusters with identical hardware.

• Runtime: Uses Ray for orchestration (default for multi-node).

• Configuration:

-- Single Node: tensor_parallel_size = N (N GPUs).

-- Multi-Node: tensor_parallel_size = GPUs_per_node, pipeline_parallel_size = Num_nodes.

• Network: Requires private, high-speed network (InfiniBand/RoCE). Set VLLM_HOST_IP and NCCL_SOCKET_IFNAME.

vllm serve <model_path> \
  --tensor-parallel-size 8 \
  --pipeline-parallel-size 2 \
  --distributed-executor-backend ray   

---