H100 SXM5
80GB HBM3, 3.35 TB/s — proven workhorse for 70B TP=2 inference
H200 SXM5
141GB HBM3e, 4.8 TB/s — single-GPU 70B at 128k context
Head-to-Head Hardware Specs
Side-by-side hardware architecture comparison for H100 SXM5 vs H200 SXM5.
H100 SXM5
H200 SXM5
Which GPU Fits Your Workload?
Pre-Training
H100 — cost-efficiency at scale, 2-4 GPU TP optimal
Fine-Tuning
H100 — FP8 fine-tuning fits 70B on 2-GPU TP=2
Inference
H200 — single-GPU 70B at 128k context, no TP overhead
NVIDIA H100 vs H200 SXM5: Memory Bandwidth & 70B Model Serving Shootout
80GB HBM3 (3.35 TB/s) vs 141GB HBM3e (4.8 TB/s) — Evaluating whether the H200's +76% VRAM and +43% bandwidth eliminates 2-GPU tensor sharding for Llama 3.3 70B.
Technical Scorecard
Side-by-side infrastructure specs using live pricing data for H100-class hardware.
| Metric | H100 SXM5 | H200 SXM5 |
|---|---|---|
| Network Fabric | NVLink 4.0 (900 GB/s) | NVLink 4.0 (900 GB/s) |
| Storage Throughput | Local NVMe (7,000 MB/s) | Local NVMe (7,000 MB/s) |
| Egress Pricing | Provider dependent | Provider dependent |
| SLA Guarantee | 99.9% (bare-metal) | 99.9% (bare-metal) |
| 8-GPU 100h Cost | $1,512 (8× H100 @ $1.89/hr) | $2,000 (8× H200 @ $2.50/hr) |
Live Pricing Comparison
Spot and reserved rates refreshed from provider APIs. Filtered to H100-class hardware.
| Provider | GPU & VRAM | Interconnect | Spot Rate | On-Demand | Monthly | Status | Action | |
|---|---|---|---|---|---|---|---|---|
Community | NVLink 4.0 (900 GB/s) | $1.89 / hr | $4.72 / hr | $1,157 / mo | Instant | |||
Bare Metal | NVLink 4.0 (900 GB/s) | $2.29 / hr | $5.73 / hr | $1,401 / mo | Instant | |||
Dedicated | NVLink 4.0 (900 GB/s) | $2.99 / hr | $7.48 / hr | $1,830 / mo | Instant | |||
Cloud | NVLink 4.0 (900 GB/s) | $3.49 / hr | $8.73 / hr | $2,136 / mo | Instant |
When to Choose H100 SXM5
- Budget-sensitive 70B inference under 32k context
- Multi-node training where H100 cluster cost matters
- QLoRA fine-tuning with 2-GPU tensor parallelism
When to Choose H200 SXM5
- 128k+ context windows for Llama 70B
- High-concurrency serving (batch≥32) requiring large KV-cache
- Single-GPU deployment simplifying ops
Technical Deep-Dive
VRAM & Context Window Analysis
H100: 80GB HBM3 fits Llama 70B FP8 (72GB) with 8GB for KV-cache — limited to ~8k context at batch=1. H200: 141GB HBM3e fits the same model with 69GB remaining for KV-cache, enabling full 128k context on a single GPU. The H200 eliminates the need for KV-cache offloading or multi-GPU context splitting.
Memory Bandwidth Impact
H100: 3.35 TB/s sustains ~120 tok/s on Llama 70B FP8. H200: 4.8 TB/s (+43%) sustains ~135 tok/s — a 12.5% throughput improvement. The bandwidth advantage compounds at higher batch sizes: at batch=32, the H200 delivers 18% more tokens/second before hitting memory stalls.
Cost Per Token Analysis
H100 at $1.89/hr: $0.0157 per million tokens (70B FP8, 120 tok/s). H200 at $2.50/hr: $0.0154 per million tokens (70B FP8, 135 tok/s). At 128k context, the H200 is cheaper on cost-per-token because it avoids the 2-GPU TP=2 overhead. Below 32k context, the H100's lower hourly rate delivers lower total cost.
Final Verdict
The H200 eliminates 2-GPU tensor sharding for Llama 70B at 128k context. The H100 remains the lower-cost option for sub-32k workloads at $1.89/hr vs $2.50/hr.