Tensor Parallelism vs Pipeline Parallelism: Multi-GPU Scaling
When one GPU isn't enough, you need multi-GPU parallelism. Understand TP vs PP, when to use each, and how NVLink/InfiniBand affect scaling efficiency.
Why Multi-GPU?
Modern LLMs exceed single-GPU VRAM. Llama 70B at FP16 = 140 GB — more than any GPU's capacity. Even at FP8 (70 GB), high-context inference pushes beyond 80 GB. Multi-GPU parallelism splits the model across GPUs so that no single GPU holds the entire model. The challenge: inter-GPU communication introduces latency. The speedup depends on the parallelism strategy and interconnect bandwidth (NVLink at 900 GB/s vs PCIe at 64 GB/s vs InfiniBand at 400 Gb/s).
Tensor Parallelism (TP)
Tensor parallelism splits individual layers across GPUs. Each GPU holds a slice of every weight matrix and computes a partial result, then all-reduce to combine. TP requires fast interconnect: NVLink 4.0 (900 GB/s) on H100 delivers near-linear speedup up to TP=8. PCIe (64 GB/s) limits TP to 2 GPUs as communication overhead becomes significant. TP is optimal for inference: it reduces latency proportionally to GPU count (TP=2 = 2x faster generation). The all-reduce happens every forward pass, so interconnect bandwidth directly determines scaling efficiency.
Pipeline Parallelism (PP)
Pipeline parallelism assigns different layers to different GPUs. GPU 1 runs layers 0-19, GPU 2 runs layers 20-39, etc. Each GPU processes a micro-batch and passes activations to the next GPU. PP requires less interconnect bandwidth than TP because activations are smaller than weight gradients. However, PP introduces pipeline bubbles: GPUs idle while waiting for the previous stage to finish. PP is optimal for training: it scales to hundreds of GPUs with moderate interconnect (InfiniBand 400 Gb/s) because gradient synchronization happens once per training step, not per token.
TP vs PP: Decision Matrix
Use TP for: inference latency reduction, single-node multi-GPU (NVLink-connected), models up to 4x single-GPU VRAM. Use PP for: training at scale, multi-node clusters (InfiniBand-connected), models requiring >8 GPUs. Use TP+PP combined for: 70B+ training across 8-64 GPUs, where TP handles intra-node communication and PP handles inter-node. For inference: TP=2 on 2x H100 is the most common production config for 70B FP8. For training: TP=8 on a single HGX node + PP=4 across nodes for 70B fine-tuning.
Interconnect Impact on Scaling
NVLink 4.0 (900 GB/s, H100): TP=2 delivers 1.85x speedup, TP=4 delivers 3.4x, TP=8 delivers 6.5x. NVLink 5.0 (1.8 TB/s, B200): TP=2 delivers 1.95x, TP=4 delivers 3.8x, TP=8 delivers 7.6x. PCIe 4.0 (64 GB/s, RTX 4090): TP=2 delivers 1.4x, TP=4 delivers 1.8x — poor scaling. InfiniBand NDR (400 Gb/s): suitable for PP across nodes but too slow for TP. The takeaway: NVLink is mandatory for efficient tensor parallelism. PCIe GPUs should use model parallelism (PP) or quantization to avoid multi-GPU communication.