H100 vs H200 for Long-Context Workloads: The 141GB KV-Cache Advantage

Executive TL;DR

For long-context LLM workloads (128K+ tokens), the NVIDIA H200's 141GB HBM3e provides a decisive advantage over the H100 SXM5's 80GB. At 128K context, Llama 3.3 70B in FP8 occupies ~72GB for model weights and ~32GB for KV-cache — totaling ~104GB. The H200 fits this comfortably on a single GPU; the H100 requires tensor parallelism across 2 GPUs, introducing NVLink all-reduce latency that increases time-to-first-token by 15-20%.

The KV-Cache Math That Decides Everything

At FP8 precision, Llama 3.3 70B model weights occupy approximately 72GB (84B params × 1 byte/param + overhead). At 128K context with batch=1, the GQA KV-cache (8 KV heads, 128 head dimension, 32 layers) consumes approximately 32GB. Total VRAM budget: ~104GB.

TTFT Comparison: Single-GPU vs Tensor-Parallel

Tensor parallelism across 2 H100s introduces NVLink all-reduce latency on every forward pass. The H200 eliminates this entirely — all attention computations happen on a single GPU with no cross-GPU synchronization.

Cost-Benefit Analysis

H100 spot: $1.89/hr (2 GPUs = $3.78/hr). H200 spot: $2.20/hr (1 GPU). The H200 at 1.15x the hourly cost of a single H100 (or 0.58x the cost of 2 H100s) delivers stronger long-context throughput.