āš”Under $0.50/hršŸ§ VRAM Estimatorāš–Compare GPUsšŸŽFree LLM APIsšŸŽÆModel Index
Deep Dive12 min

What is KV Cache? Memory Scaling in Long-Context Inference

KV-cache is why LLMs need so much more VRAM than their weight size suggests. Learn how it scales with context length and how to optimize it.

What is KV Cache?

During autoregressive text generation, the LLM processes one token at a time. To avoid re-computing attention for every previous token, the model caches the Key and Value tensors from previous steps ā€” this is the KV-cache. Without KV-cache, generating N tokens would require O(NĀ²) compute. With KV-cache, each step is O(N) because previous K/V vectors are reused. The tradeoff: KV-cache grows linearly with context length, batch size, and number of layers, consuming significant VRAM that could otherwise hold model weights.

VRAM formula breakdown ā†’VRAM calculator ā†’

KV-Cache Memory Formula

KV-cache size = 2 Ɨ num_layers Ɨ num_heads Ɨ head_dim Ɨ context_length Ɨ batch_size Ɨ precision_bytes. For Llama 3.3 70B at FP16: 2 Ɨ 80 layers Ɨ 8 heads Ɨ 128 dim Ɨ 32k context Ɨ 1 batch Ɨ 2 bytes = ~8 GB. At 128k context: ~32 GB. At batch=32 with 32k context: ~256 GB. The multiplier is dramatic: going from 8k to 128k context increases KV-cache by 16x. This is why single-GPU 128k inference requires the H200 (141 GB) or B200 (192 GB).

H200 141GB analysis ā†’DeepSeek R1 context limits ā†’

KV-Cache Optimization Techniques

PagedAttention (vLLM): allocates KV-cache in non-contiguous pages, reducing internal fragmentation from ~60% to ~4%. This alone can 2x your effective batch size. FlashAttention-3: fuses attention computation to reduce peak KV-cache memory by 20-40% through kernel-level optimization. Grouped Query Attention (GQA): reduces KV head count from num_heads to num_kv_heads (e.g., 8 KV heads vs 64 attention heads in Llama 3), cutting KV-cache by 8x. Quantized KV-cache: storing K/V in INT8 or INT4 reduces KV-cache memory by 50-75% with minimal quality loss.

vLLM production deployment guide ā†’InfiniBand vs NVLink analysis ā†’

Context Length vs VRAM: Practical Limits

RTX 4090 (24 GB): 7B FP16 at 128k (~4 GB KV-cache), 70B INT4 at 8k (~0.5 GB KV-cache). L40S (48 GB): 32B FP8 at 32k (~2 GB KV-cache), 70B INT4 at 16k (~4 GB KV-cache). H100 (80 GB): 70B FP8 at 32k (~16 GB KV-cache), 70B FP8 at 128k (OOM ā€” needs H200). H200 (141 GB): 70B FP8 at 128k (~16 GB KV-cache, 53 GB headroom). B200 (192 GB): 70B FP16 at 128k (~32 GB KV-cache, 90 GB headroom). The H200 is the minimum for single-GPU 128k inference at 70B scale.

Compare H100 vs H200 ā€” context window analysis ā†’Cheapest cloud GPU finder ā†’

Related Models

GPU Price Trackers

Engineering Guides

Frequently Asked Questions

Why does KV-cache grow with context length?ā–¾
Each new token in the context adds a Key and Value vector for every attention layer and head. These vectors must be stored to avoid recomputation. For Llama 70B, each token adds ~256 KB of KV-cache ā€” at 128k tokens, that's ~32 GB just for the cache.
Can I reduce KV-cache without losing quality?ā–¾
Yes. PagedAttention reduces fragmentation overhead. GQA reduces KV head count. Quantized KV-cache (INT8/INT4) cuts memory 50-75% with <1% quality loss. FlashAttention reduces peak memory. Combine these for 3-5x effective KV-cache reduction.
How does batch size affect KV-cache?ā–¾
KV-cache scales linearly with batch size: batch=32 uses 32x more KV-cache than batch=1. This is why high-concurrency serving requires GPUs with large VRAM. The H200's 141 GB can sustain batch=32 at 32k context for 70B models.
AI Compute 101 ā€” Educational Reference | OpenGPU RadarMore Guides ā†’