LLM Cloud GPU Cost & VRAM Calculator
Estimate VRAM overhead, compare cluster configurations, and calculate burn rate across H100, H200, B200, and RTX 4090 cloud GPUs.
The Formula: How VRAM is Calculated
Total VRAM = Model Weights (bytes) + KV Cache + Activation Memory + CUDA Context (~5%)
// Model Weights (FP16) = Parameters × 2 bytes // Example: Llama 70B = 70 × 10^9 × 2 = 140 GB (FP16) // KV Cache = 2 × num_layers × num_heads × head_dim × seq_len × batch_size × 2 bytes // Llama 70B at 32k ctx, batch=1: ~16 GB // Activation Memory ≈ 10-20% of model weights (workload dependent) // Total: 140 + 16 + 21 + 8.85 ≈ 185.85 GB → requires 2× H100 80GB (TP=2)
For INT4 quantization (AWQ/GPTQ), weights shrink by 75%: Llama 70B INT4 = ~35 GB weights. This is why a single RTX 4090 (24 GB) can run Llama 8B INT4 but not Llama 70B INT4.
Memory Bandwidth vs Compute: Why Context Window Length Causes Memory Saturation
LLM inference is fundamentally memory-bandwidth bound, not compute-bound. Each token generation requires reading the entire model weights from VRAM once. At 70B parameters in FP16 (140 GB), a GPU with 3.35 TB/s bandwidth (H100) reads weights in 42ms per token = ~24 tokens/sec theoretical maximum.
As context window grows, KV-cache expands linearly. At 128k context, KV-cache consumes ~32 GB for Llama 70B — leaving less VRAM for batch processing. The GPU shifts from compute-bound (small batch, short context) to memory-bandwidth bound (large batch or long context).
RTX 4090
1.0 TB/s
Practical max: 8B INT4 only
H100 SXM5
3.35 TB/s
Practical max: 70B FP8, batch=8
B200
8.0 TB/s
Practical max: 70B FP4, batch=64+
Cost per Million Tokens: Self-Hosted Cloud Pods vs Direct API Inference
Self-hosting on cloud GPUs is cheaper than API pricing only above a sustained throughput threshold. Below that threshold, you pay for idle VRAM.
| Method | Llama 70B $/1M Tokens | Break-Even |
|---|---|---|
| OpenAI API | $3.00 | N/A (pay per token) |
| Anthropic API | $15.00 | N/A (pay per token) |
| Self-Hosted H100 ($1.89/hr) | $0.28 | ~6.7M tokens/day sustained |
| Self-Hosted RTX 4090 ($0.34/hr) | $0.42 | ~1.6M tokens/day (8B INT4) |
At 10M tokens/day sustained throughput, self-hosted H100 delivers 10.7x cost savings vs OpenAI API and 53.6x vs Anthropic API. Below 1M tokens/day, API pricing is preferred because you are not paying for idle GPU time.