Batch Inference GPU Economics: Offline Throughput Cost (2026)
Offline batch inference economics: why the cheapest large-VRAM GPU leads when latency is free and preemption risk is priced in.
The fast answer
Cheapest verified GPU: GeForce RTX 4090 at $0.34/hr on-demand (Vast.ai) among 6 candidate GPUs. Lowest on-demand hourly row for this GPU's providers.json key; refreshed daily.Observed On-Demand
VRAM needed: DeepSeek V3 at INT4 needs 381.0 GB full-stack (weights 336 GB + KV-cache + overhead) at 128,000 tokens.
Cost driver: Tokens per GPU-hour at full VRAM utilization: batch workloads should pick the largest model that fits the card before comparing rates.
Candidate GPUs for Batch Inference
Fit = full-stack VRAM total for DeepSeek V3 (INT4/FP16 at model context) ≤ GPU VRAM. Rates are observed rows from data/providers.json, refreshed daily.
| GPU | VRAM | Bandwidth | INT4 fit | FP16 fit | On-demand | Spot |
|---|---|---|---|---|---|---|
| B200 Blackwell | 192 GB | 8.0 TB/s | OOM | OOM | $3.99/hr | $3.99/hr |
| H100 SXM5 | 80 GB | 3.35 TB/s | OOM | OOM | $1.89/hr | $1.89/hr |
| A100 80GB SXM4 | 80 GB | 2.0 TB/s | OOM | OOM | $1.59/hr | $1.59/hr |
| H200 SXM5 | 141 GB | 4.8 TB/s | OOM | OOM | $2.79/hr | $2.79/hr |
| L40S | 48 GB | 864 GB/s | OOM | OOM | $0.69/hr | $0.69/hr |
| GeForce RTX 4090 | 24 GB | 1.0 TB/s | OOM | OOM | $0.34/hr | $0.34/hr |
Reference models for this workload
Methodology & provenance
Batch jobs trade latency for utilization: large batches push every GPU toward its bandwidth/compute ceiling, so cost-per-token ordering uses observed hourly rates against VRAM fit (registry weight model + KV-cache at batch context). Spot rows are shown where they exist because batch workloads can checkpoint and requeue — preemption tolerance is a workload property, not a pricing assumption.
Next steps