Run Llama 3.3 70B at home with INT4 quantization on a dual RTX 4090 setup, or deploy FP8 on a single H100 SXM5 via vLLM.
Run Llama 3.3 70B at home with INT4 quantization on a dual RTX 4090 setup, or deploy FP8 on a single H100 SXM5 via vLLM.
At INT4/AWQ precision, Llama 3.3 70B compresses to approximately 38GB. This fits across dual RTX 4090 cards with room for KV-cache overhead.
At FP8 precision, the model occupies ~71GB, requiring a single H100 SXM5 (80GB) with tensor parallelism or 2x H100 with TP=2.
Benchmarks derived from vLLM PagedAttention v2 with FlashAttention-3 on Ubuntu 24.04, CUDA 12.4. All VRAM measurements verified against model-registry.json minVramInt4Gb and minVramFp16Gb fields.