NVIDIA H100 SXM5 Cloud Pricing & Rental Rates
The H100 SXM5 is the workhorse of AI training clusters. Its 80 GB HBM3 memory handles 70B parameter models with tensor parallelism across 2-4 GPUs, while 3.35 TB/s bandwidth keeps the compute units fed during all-reduce operations.
Target Workload: Multi-node 70B+ LLM pre-training & fine-tuning
Live Cloud Pricing for NVIDIA H100 SXM5
Spot and reserved rates refreshed from provider APIs. Click a provider to deploy.
| Provider | GPU & VRAM | Interconnect | Spot Rate | On-Demand | Monthly | Status | Action | |
|---|---|---|---|---|---|---|---|---|
Community | NVLink 4.0 (900 GB/s) | $1.89 / hr | $4.72 / hr | $1,157 / mo | Instant | |||
Bare Metal | NVLink 4.0 (900 GB/s) | $2.29 / hr | $5.73 / hr | $1,401 / mo | Instant | |||
Dedicated | NVLink 4.0 (900 GB/s) | $2.99 / hr | $7.48 / hr | $1,830 / mo | Instant | |||
Cloud | NVLink 4.0 (900 GB/s) | $3.49 / hr | $8.73 / hr | $2,136 / mo | Instant |
Models That Fit NVIDIA H100 SXM5 (80GB HBM3)
Deterministic VRAM calculation (FP8) via entity graph. Only editorial and enriched models shown.
| MODEL | PARAMS | CONTEXT | FP8 VRAM | FIT? | CALCULATOR |
|---|---|---|---|---|---|
| DeepSeek R1 Distill Qwen 32B | 32B | 128K | 42.9 GB | ✓ fits | Pre-filled → |
| Llama 3.1 8B Instruct | 8.03B | 128K | 10.8 GB | ✓ fits | Pre-filled → |
| Llama 3.2 3B Instruct | 3.2B | 128K | 4.3 GB | ✓ fits | Pre-filled → |
| Llama 3.2 1B Instruct | 1B | 128K | 1.3 GB | ✓ fits | Pre-filled → |
| Qwen 2.5 Coder 32B | 32.5B | 128K | 43.6 GB | ✓ fits | Pre-filled → |
| Qwen 2.5 Coder 14B | 14.7B | 128K | 19.7 GB | ✓ fits | Pre-filled → |
| Qwen 2.5 Coder 7B | 7.61B | 128K | 10.2 GB | ✓ fits | Pre-filled → |
| Qwen 2.5 14B Instruct | 14.7B | 128K | 19.7 GB | ✓ fits | Pre-filled → |
| Qwen 2.5 7B Instruct | 7.61B | 128K | 10.2 GB | ✓ fits | Pre-filled → |
| Mistral NeMo 12B | 12B | 128K | 16.1 GB | ✓ fits | Pre-filled → |
| Gemma 2 27B | 27B | 8K | 35.7 GB | ✓ fits | Pre-filled → |
| Gemma 2 9B | 9.24B | 8K | 12.2 GB | ✓ fits | Pre-filled → |
Inference & Serving Capacity
Practical model feasibility, max batch sizes, and KV-cache retention limits for NVIDIA H100 SXM5.
Llama 3.3 70B
FEASIBLEFP8 native, 2-GPU tensor parallelism recommended
DeepSeek 671B
FEASIBLE8-way NVLink mesh, 80 GB per GPU fits 8 experts
Qwen 2.5 32B
FEASIBLEFP8 on single GPU, high batch throughput
vLLM Throughput (FP8)
~120 tok/s (vLLM, Llama 70B FP8, batch=1)
Estimated tokens/second, single GPU, Llama-class model
Max Context Window (Llama 70B)
32k tokens (FP8) — KV-cache consumes ~16 GB at batch=1
Maximum context length before KV-cache eviction
Hardware Bottleneck Analysis
Whether NVIDIA H100 SXM5 is compute-bound (TFLOPS) or memory-bandwidth bound (GB/s) across workloads.
Bottleneck Classification
Compute-bound at small batch; memory-bandwidth bound at batch≥64
Recommended Quantization
FP8 / FP4 native
Best Cluster Topology
8-way HGX Baseboard with NVLink 4.0 Mesh
Deep Analysis
H100 SXM5 8-Way HGX AllReduce Analysis: In an 8-way HGX baseboard, each GPU connects via NVLink 4.0 at 900 GB/s bidirectional bandwidth. For Llama 70B FP8 tensor parallelism, TP=2 splits the 72 GB model across 2 GPUs (36 GB each) with 16 GB headroom for KV-cache at 32k context. TP=4 splits across 4 GPUs (18 GB each) but introduces 3 all-reduce sync points per forward pass — the 900 GB/s NVLink mesh absorbs this with <5% communication overhead. At TP=8, the model shards to 9 GB per GPU, but NCCL all-reduce latency increases to ~8% due to the ring-all-reduce topology across 8 nodes. The optimal config is TP=2 for inference (lowest latency) and TP=4 for training (better compute utilization). NVLink saturation occurs at batch≥128 when the 3.35 TB/s HBM3 bandwidth cannot sustain the 1,979 FP8 TFLOPS — this is the fundamental ceiling for single-node throughput.
Architecture & Die Breakdown
Architecture
Hopper GH100 — 4nm TSMC
TDP
700W
Memory Subsystem
80GB HBM3 at 3.35 TB/s bandwidth. High Bandwidth Memory provides the throughput needed to keep tensor cores fed during large batch inference.
Interconnect
NVLink 4.0 (900 GB/s). Enables multi-GPU tensor parallelism with high-bandwidth, low-latency GPU-to-GPU communication.
Precision Performance
| Precision | TFLOPS | Use Case |
|---|---|---|
| FP8 | 1,979 | Training & inference with mixed-precision |
| FP16 | 989 | Full-precision training, fine-tuning, evaluation |
Break-Even ROI Calculator
Monthly hours where reserved pricing beats spot for NVIDIA H100 SXM5. Above the break-even point, reserve commits save money.
Spheron
RunPod
Lambda Labs
Break-even at 320 hours/month: if you run NVIDIA H100 SXM5 more than 320 hours per month, reserved pricing on all three providers saves money. At 720 hours/month (24/7), reserved saves $510+/mo vs on-demand.
Related GPUs
Frequently Asked Questions
How much does it cost to rent an NVIDIA H100 SXM5 per hour?▾
What is the monthly reserved pricing for NVIDIA H100 SXM5?▾
Can an NVIDIA H100 SXM5 run 70B parameter LLMs?▾
Is spot pricing reliable for distributed NVIDIA H100 SXM5 training?▾
Compare alternatives & next steps
Hardware Analysis & Practical Guides
How to Run Llama 3.3 70B Locally: VRAM, Quantization & Deployment
8 min read
how-toProduction vLLM Deployment: PagedAttention, KV-Cache & Continuous Batching
newsNVIDIA Blackwell B200 Compute Impact: FP4 Tensor Cores, NVLink 5.0 & VRAM Density
researchReal-World Cost of Hosting a 70B LLM: Spot Pricing vs API Breakeven Analysis
guideQuantization Formats Explained: FP8 vs INT4 vs AWQ for LLM Serving
7 min read