Fundamentals8 min

What is VRAM and How Much Do LLMs Actually Need?

VRAM is the single most important spec for running LLMs locally. This guide explains exactly how much VRAM you need for every model size, from 7B to 700B+.

What is VRAM?

VRAM (Video Random Access Memory) is the dedicated memory on a GPU that stores model weights, KV-cache, and activation tensors during LLM inference. Unlike system RAM, VRAM is directly accessible by the GPU's compute cores via high-bandwidth memory buses (HBM3 at 3.35 TB/s on H100, or GDDR6X at 1 TB/s on RTX 4090). The critical constraint: an LLM cannot run if its total memory footprint exceeds available VRAM. This is why VRAM capacity — not TFLOPS — is the primary bottleneck for LLM deployment.

GPU Hardware Radar — all tracked GPUs →VRAM Calculator — estimate your needs →

The VRAM Formula: Weights + Context + Overhead

Total VRAM = Model Weights + KV-Cache + CUDA Overhead. Model weights scale linearly with parameter count and inversely with precision: a 70B model at FP16 = 140 GB, at FP8 = 70 GB, at INT4 = 35 GB. KV-cache scales with context length, batch size, and number of layers: at 32k context with batch=1, KV-cache adds ~8 GB for a 70B model. CUDA overhead (kernels, activations,碎片) adds 5-15% on top. Rule of thumb: multiply FP16 weight size by 1.1 for real-world VRAM usage.

DeepSeek R1 VRAM breakdown →Llama 3.3 70B VRAM breakdown →

VRAM Requirements by Model Size

7B models: 14 GB FP16, 7 GB FP8, 4 GB INT4 — fits RTX 4090 (24 GB) with headroom. 13B models: 26 GB FP16, 13 GB FP8, 7 GB INT4 — fits RTX 4090 at FP8/INT4. 32B models: 65 GB FP16, 33 GB FP8, 18 GB INT4 — needs L40S/A100 for FP8, fits 4090 at INT4. 70B models: 140 GB FP16, 71 GB FP8, 38 GB INT4 — needs 2x H100 or 1x H200. 700B+ models: 1.4 TB FP16, 700 GB FP8, 350 GB INT4 — needs 4-8x H200 cluster.

Cheapest cloud GPU finder →Model hosting directory →

Cloud GPU VRAM Tiers

Entry tier (24 GB): RTX 4090 — best for 7B-13B INT4 models, QLoRA fine-tuning, Stable Diffusion. Mid tier (48 GB): L40S, RTX 6000 Ada — ideal for 30B FP8 or 70B INT4 with context headroom. High tier (80 GB): H100 SXM5, A100 — handles 70B FP8 at 32k context, multi-GPU training building block. Ultra tier (141 GB): H200 — single-GPU 70B at 128k context, no tensor parallelism needed. Frontier tier (192+ GB): B200, B300 — trillion-parameter training, full-context 70B at FP16.

Compare H100 vs H200 →Compare H100 vs B200 →

Related Models

GPU Price Trackers

Engineering Guides

Frequently Asked Questions

How much VRAM do I need to run a 70B LLM?▾

At FP8 precision, a 70B model requires ~71 GB for weights plus ~16 GB for KV-cache at 32k context = ~87 GB total. A single H200 141GB handles this. At INT4, the requirement drops to ~46 GB, fitting an L40S 48GB.

Can I run LLMs without a GPU?▾

Yes, but extremely slowly. CPU inference using llama.cpp with mmap is possible for small models (7B-13B), but expect 1-5 tok/s vs 100+ tok/s on GPU. For any production workload, GPU acceleration is mandatory.

Does VRAM speed matter as much as capacity?▾

For LLM inference, capacity is the hard constraint — you cannot run a model that doesn't fit. Once capacity is met, bandwidth determines throughput: HBM3 (3.35 TB/s) delivers 3-5x more tokens/second than GDDR6X (1 TB/s) for memory-bound workloads.

AI Compute 101 — Educational Reference | OpenGPU Radar|More Guides →