Fundamentals10 min

LLM Quantization Explained: FP16 vs FP8 vs INT4/AWQ

Q: Does quantization reduce LLM quality?

FP8 has negligible quality loss (<2% perplexity degradation) and is indistinguishable for most users. INT4/AWQ introduces 3-5% degradation — noticeable on complex reasoning tasks but acceptable for code generation, summarization, and chat. Always benchmark on your specific workload.

Q: Can I mix FP8 and INT4 in the same model?

vLLM supports mixed-precision inference where attention layers use FP8 and linear layers use INT4. This optimizes both quality (FP8 for sensitive layers) and memory (INT4 for bulk weights). The quality loss is between pure FP8 and pure INT4.

Q: Which quantization is fastest?

FP8 on H100/H200 is fastest because it's hardware-native — the Tensor Cores process FP8 at full speed. INT4 on RTX 4090 is fast for small batch sizes but scales poorly because the dequantization overhead exceeds normal levels at batch≥16. FP16 is universally fast but requires 2x the VRAM.

Quantization cuts VRAM usage by 2-8x with minimal quality loss. Understand FP16, FP8, INT8, and INT4/AWQ precision formats and when to use each.

What is Quantization?

Quantization reduces the numerical precision of model weights and activations — from 16-bit floating point (FP16) to 8-bit (FP8) or 4-bit integer (INT4). Each step halves memory usage: a 70B model goes from 140 GB (FP16) to 70 GB (FP8) to 35 GB (INT4). The tradeoff is numerical accuracy: lower precision introduces quantization error that degrades output quality. Modern quantization techniques (AWQ, GPTQ, FP8 native) minimize this error through calibration data and hardware-native support.

VRAM sizing calculator →Model GPU requirements →

FP16 / BF16: The Precision Baseline

FP16 (16-bit floating point) uses 2 bytes per parameter. A 70B model = 140 GB. BF16 (Brain Float 16) has the same memory footprint but larger dynamic range — preferred for training. FP16/BF16 produces the highest quality outputs with zero quantization error. The downside: VRAM requirements are prohibitive for large models. A 70B FP16 model needs 140 GB — more than any single GPU except the B300 (288 GB). This is why production inference almost always uses FP8 or INT4.

FP8: The Production Sweet Spot

FP8 (8-bit floating point) uses 1 byte per parameter — exactly half of FP16. A 70B model = 70 GB. Native FP8 support on H100/H200/B200 Tensor Cores means zero software overhead: the model runs at full hardware speed with <2% perplexity degradation. FP8 is the recommended precision for production inference on Hopper/Blackwell GPUs. It fits 70B models on a single H200 (141 GB) with ample KV-cache headroom, or on 2x H100 (80 GB each) with tensor parallelism.

H100 FP8 analysis →L40S FP8 inference →

INT4 / AWQ: Maximum Compression

INT4 (4-bit integer) uses 0.5 bytes per parameter — 4x smaller than FP16. A 70B model = 35 GB. AWQ (Activation-aware Weight Quantization) calibrates the quantization process on representative data, preserving the weights that matter most for output quality. Quality tradeoff: 3-5% perplexity degradation on standard benchmarks — noticeable on edge cases but acceptable for most production workloads. INT4 enables 70B models on a single L40S 48GB or 2x RTX 4090, reducing cloud costs by 60-80% vs FP16.

RTX 4090 INT4 inference →Llama 3.3 70B INT4 sizing →

When to Use Each Format

Use FP16/BF16: training, fine-tuning, and when maximum quality is non-negotiable. Use FP8: production inference on H100/H200/B200 — the best quality-per-dollar ratio. Use INT4/AWQ: budget inference on consumer/enterprise GPUs (RTX 4090, L40S), high-throughput batching where 3-5% quality loss is acceptable. Use INT8: middle ground for A100 (no FP8 support) — 2x compression with <1% quality loss. Never mix precisions within a single model without framework support (vLLM handles mixed precision automatically).

Compare RTX 4090 vs L40S →Production vLLM deployment guide →

Related Models

GPU Price Trackers

Engineering Guides

Frequently Asked Questions

Does quantization reduce LLM quality?▾

FP8 has negligible quality loss (<2% perplexity degradation) and is indistinguishable for most users. INT4/AWQ introduces 3-5% degradation — noticeable on complex reasoning tasks but acceptable for code generation, summarization, and chat. Always benchmark on your specific workload.

Can I mix FP8 and INT4 in the same model?▾

vLLM supports mixed-precision inference where attention layers use FP8 and linear layers use INT4. This optimizes both quality (FP8 for sensitive layers) and memory (INT4 for bulk weights). The quality loss is between pure FP8 and pure INT4.

Which quantization is fastest?▾

FP8 on H100/H200 is fastest because it's hardware-native — the Tensor Cores process FP8 at full speed. INT4 on RTX 4090 is fast for small batch sizes but scales poorly because the dequantization overhead exceeds normal levels at batch≥16. FP16 is universally fast but requires 2x the VRAM.

AI Compute 101 — Educational Reference | OpenGPU Radar|More Guides →