Can You Run DeepSeek R1 on 8× NVIDIA H200 SXM5?

The DeepSeek R1 (671B MoE (37B active)) contains approximately 67137 billion parameters. At FP16 (2 bytes per parameter), the raw weight matrix occupies 672 GB. With a 15% CUDA kernel overhead factor, effective VRAM for weights alone is 672 GB. The 8× NVIDIA H200 SXM5 provides 141 GB HBM3e each (1,128 GB total) of memory, leaving insufficient headroom — quantization to INT8 or INT4 is mandatory for KV-cache and activation tensors.

At INT8 quantization (1 byte per parameter), the weight footprint drops to 336 GB, requiring a tensor parallel degree of approximately 3× across 8× NVIDIA H200 SXM5 instances. The estimated inference throughput is ~110 tok/s (FP8, TP=8, batch=1), which translates to a cost-per-million-output-tokens of roughly $2.00 at current spot rates. For production deployments, vLLM (tensor-parallel-size=4–8, FP8 native) with PagedAttention is the recommended serving stack.