LLM GPU & VRAM Memory Calculator
Estimate VRAM requirements, compare cluster configurations, and calculate burn rate across H100, H200, B200, and RTX 4090 cloud GPUs. All calculations are labeled CALCULATED ESTIMATE (Not physical hardware benchmark).
Understanding Model Weights Memory
Model weights consume the largest portion of VRAM during inference. The memory footprint depends on the number of parameters and the quantization precision used.
// FP16 (half precision): Parameters × 2 bytes // Example: Llama 3.1 8B = 8 × 10^9 × 2 = 16 GB (FP16) // Example: Llama 3.3 70B = 70 × 10^9 × 2 = 140 GB (FP16) // INT4 (4-bit quantization): Parameters × 0.5 bytes // Example: Llama 3.1 8B INT4 = ~4 GB weights // Example: Llama 3.3 70B INT4 = ~35 GB weights // FP8 (8-bit quantization): Parameters × 1 byte // Example: Llama 3.3 70B FP8 = ~70 GB weights (Hopper GPUs only)
CALCULATED ESTIMATE — These are architectural weight-size calculations based on published parameter counts, not physical hardware benchmark measurements. Actual VRAM usage may vary with inference engine optimizations, kernel fusion, and framework overhead.
KV-Cache Scaling Formula
The Key-Value (KV) cache stores intermediate attention results for each token generated, growing linearly with context length and batch size. This is the second-largest VRAM consumer after model weights.
// KV Cache = 2 × num_layers × num_heads × head_dim × seq_len × batch_size × bytes_per_param // Llama 3.1 8B (32 layers, 32 heads, 128 head_dim): // At 4k ctx, batch=1 FP16: ~2 GB // At 32k ctx, batch=1 FP16: ~16 GB // At 128k ctx, batch=1 FP16: ~64 GB // Llama 3.3 70B (28 layers, 64 heads, 128 head_dim): // At 4k ctx, batch=1 FP16: ~14 GB // At 32k ctx, batch=1 FP16: ~112 GB // At 128k ctx, batch=1 FP16: ~448 GB (requires multi-GPU)
CALCULATED ESTIMATE — Formula-based calculation using published architecture parameters. Actual KV-cache usage depends on the inference engine's memory management strategy (PagedAttention, prefix caching, etc.).
CUDA Runtime Overhead
Beyond model weights and KV-cache, the CUDA runtime, framework libraries, and GPU context consume a baseline amount of VRAM. This overhead is relatively constant across workloads but becomes significant when running smaller models on GPUs with limited VRAM.
// CUDA Context + vLLM runtime: 1.2 GB baseline (fixed) // Activation buffer (single-batch inference): 0.4 GB (fixed) // Memory fragmentation safety headroom: 10% of subtotal // Total overhead per model: CUDA (1.2 GB) + Activation (0.4 GB) + Fragmentation (10%) // This is why Llama 3.1 8B INT4 (~4.9 GB total) fits comfortably // on RTX 4090 (24 GB), but Llama 3.3 70B FP8 (~188 GB) requires multi-GPU.
CALCULATED ESTIMATE — Deterministic values from the Canonical VRAM Engine. CUDA overhead is a fixed 1.2 GB baseline. Activation buffer is a fixed 0.4 GB for single-batch inference. No arbitrary 10-20% activation multipliers. Fragmentation headroom is a deterministic 10% of the subtotal.
Precision Trade-offs (FP16 vs FP8 vs INT4)
| Precision | Bytes/Param | Llama 3.1 8B | Llama 3.3 70B | Hardware Support |
|---|---|---|---|---|
| FP16 | 2 bytes | 16 GB | 140 GB | All GPUs |
| FP8 | 1 byte | 8 GB | 70 GB | H100/H200/B200 (Hopper+) |
| INT4 | 0.5 bytes | ~4 GB | ~35 GB | All GPUs (AWQ/GPTQ) |
CALCULATED ESTIMATE — Precision-based weight-size calculations from published parameter counts. Actual quantization performance depends on the quantization method (AWQ, GPTQ, bitsandbytes), group size, and calibration dataset. FP8 requires native Tensor Core support (Hopper architecture or later).
Worked Examples: VRAM Sizing Comparison
Example 1: Llama 3.1 8B at INT4 on RTX 4090 (24 GB)
Model weights (INT4): ~4 GB
KV-cache (4k ctx, batch=1, FP16): ~2 GB
CUDA runtime overhead: ~4 GB
Total: ~10 GB → Fits on RTX 4090 (24 GB)
CALCULATED ESTIMATE — Based on published model architecture and quantization benchmarks. Actual usage varies with inference engine and context length.
Example 2: Llama 3.3 70B at FP8 on H100 SXM5 (80 GB)
Model weights (FP8): ~70 GB
KV-cache (4k ctx, batch=1, FP16): ~14 GB
CUDA runtime overhead: ~4 GB
Total: ~88 GB → Requires 2× H100 (TP=2)
CALCULATED ESTIMATE — Based on published model architecture. Tensor parallelism across 2 GPUs reduces per-GPU KV-cache by half, making total VRAM fit within 2× 80 GB = 160 GB.
The Formula: How VRAM is Calculated
Total VRAM = Model Weights + KV Cache + Dynamic Activation Buffer + CUDA Runtime Overhead
// Model Weights (FP16) = Parameters × 2 bytes // Example: Llama 70B = 70 × 10^9 × 2 = 140 GB (FP16) // KV Cache = 2 × num_layers × num_heads × head_dim × seq_len × batch_size × 2 bytes // Llama 70B at 32k ctx, batch=1: ~16 GB // CUDA overhead: 1.2 GB baseline // Activation buffer: 0.4 GB (single-batch autoregressive decode) // Memory fragmentation headroom: 10% of subtotal // Total = Weights + KV Cache + CUDA + Activation + Fragmentation // Llama 70B at 32k ctx, batch=1 FP16: 140 + 16 + 1.2 + 0.4 + 15.76 = ~173.36 GB → requires 2× H100 80GB (TP=2)
For INT4 quantization (AWQ/GPTQ), weights shrink by 75%: Llama 70B INT4 = ~35 GB weights. This is why a single RTX 4090 (24 GB) can run Llama 8B INT4 but not Llama 70B INT4.
The Formula: How VRAM is Calculated
Total VRAM = Model Weights (bytes) + KV Cache + Dynamic Activation Buffer + CUDA Runtime Overhead
// Model Weights (FP16) = Parameters × 2 bytes // Example: Llama 70B = 70 × 10^9 × 2 = 140 GB (FP16) // KV Cache = 2 × num_layers × num_heads × head_dim × seq_len × batch_size × 2 bytes // Llama 70B at 32k ctx, batch=1: ~16 GB // CUDA overhead: 1.2 GB baseline // Activation buffer: 0.4 GB (single-batch autoregressive decode) // Memory fragmentation headroom: 10% of subtotal // Total = Weights + KV Cache + CUDA + Activation + Fragmentation // Llama 70B at 32k ctx, batch=1 FP16: 140 + 16 + 1.2 + 0.4 + 15.76 = ~173.36 GB → requires 2× H100 80GB (TP=2)
For INT4 quantization (AWQ/GPTQ), weights shrink by 75%: Llama 70B INT4 = ~35 GB weights. This is why a single RTX 4090 (24 GB) can run Llama 8B INT4 but not Llama 70B INT4.
Memory Bandwidth & Context Saturation: Why Longer Contexts Need More VRAM
LLM inference is fundamentally memory-bandwidth bound, not compute-bound. Each token generation requires reading the entire model weights from VRAM once. At 70B parameters in FP16 (140 GB), a GPU with 3.35 TB/s bandwidth (H100) reads weights in 42ms per token = ~24 tokens/sec theoretical maximum.
As context window grows, KV-cache expands linearly. At 128k context, KV-cache consumes ~32 GB for Llama 70B — leaving less VRAM for batch processing. The GPU shifts from compute-bound (small batch, short context) to memory-bandwidth bound (large batch or long context).
RTX 4090
1.0 TB/s
Practical max: 8B INT4 only
H100 SXM5
3.35 TB/s
Practical max: 70B FP8, batch=8
B200
8.0 TB/s
Practical max: 70B FP4, batch=64+
Cost per Million Tokens: Self-Hosted Cloud Pods vs Direct API Inference
All figures below are labeled with their data provenance. API prices are provider-published; self-host costs are calculated estimates.
Self-hosting on cloud GPUs is cheaper than API pricing only above a sustained throughput threshold. Below that threshold, you pay for idle VRAM.
| Method | Llama 70B $/1M Tokens | Break-Even | Provenance |
|---|---|---|---|
| OpenAI API | $3.00 | N/A (pay per token) | PROVIDER_PUBLISHED |
| Anthropic API | $15.00 | N/A (pay per token) | PROVIDER_PUBLISHED |
| Self-Hosted H100 ($1.89/hr) | $0.28 | ~6.7M tokens/day sustained | CALCULATED_ESTIMATE |
| Self-Hosted RTX 4090 ($0.34/hr) | $0.42 | ~1.6M tokens/day (8B INT4) | CALCULATED_ESTIMATE |
At 10M tokens/day sustained throughput, self-hosted H100 delivers 10.7x cost savings vs OpenAI API and 53.6x vs Anthropic API. Below 1M tokens/day, API pricing is preferred because you are not paying for idle GPU time.
Why these numbers? ▾
OpenAI API $3.00 and Anthropic API $15.00 per 1M tokens are provider-published pricing from their official documentation.
Self-host H100 cost ($1.89/hr) is based on observed spot pricing from public cloud providers. Cost per token calculated as: (gpu_hourly_rate × num_gpus × overhead) / tokens_per_hour.
Break-even assumes 95% GPU utilization and steady-state inference. Actual costs vary with model size, context length, and batch size.
Source: https://opengpuradar.com/methodology · Data freshness: Verified via Public Cloud APIs & Market Scraping | Refreshed Daily (UTC)