KV-cache memory grows linearly with context length but quadratically with batch size. At 128K context, a single Llama 70B request consumes ~32GB — nearly half an H100's entire VRAM budget.
KV-cache memory grows linearly with context length but quadratically with batch size. At 128K context, a single Llama 70B request consumes ~32GB — nearly half an H100's entire VRAM budget.
The KV-cache is the silent budget killer in LLM inference. At 128K context, a single Llama 3.3 70B request consumes ~32GB of VRAM for KV-cache alone — consuming 40% of an H100 SXM5's 80GB total memory budget. This forces tensor parallelism across 2 GPUs, doubling infrastructure cost for what appears to be a single request.
KV-cache memory = 2 × num_layers × num_kv_heads × head_dim × seq_len × bytes_per_element
= 2 × 32 × 8 × 128 × seq_len × 1 byte
= 65,536 × seq_len bytes = 0.0625 MB per token
| Context | KV-Cache (FP8) | Total (Model + KV) | H100 (80GB) | H200 (141GB) |
|---|---|---|---|---|
| 4K | 256 MB | 72.3 GB | ✓ Single GPU | ✓ Single GPU |
| 32K | 2 GB | 74 GB | ✓ Single GPU | ✓ Single GPU |
| 128K | 32 GB | 104 GB | ✗ TP=2 | ✓ Single GPU |
DeepSeek-R1 and DeepSeek-V3 use Multi-head Latent Attention (MLA), which compresses the KV-cache by a factor of ~4-8x compared to standard GQA. This reduces the 128K KV-cache from 32GB to approximately 4-8GB, allowing single-GPU serving on H100-class hardware.
KV-cache formula verified against vram-canonical-engine.ts: KV_cache = 2 × num_layers × num_kv_heads × head_dim × seq_len × bytes_per_element. Llama 70B parameters: 32 layers, 8 KV heads (GQA), 128 head_dim, FP8 = 1 byte/param. Verified against gpu-specs.ts maxContextLlama70b and fp8Throughput fields. Spot pricing from gpu-pricing.json.