AI Cluster Architecture Guide: Interconnects, Memory & Topologies
A systems engineer's reference for avoiding multi-node training bottlenecks, hypervisor tax, and network saturation.
Memory Bandwidth vs. Raw TFLOPS: The Real LLM Bottleneck
Why Bandwidth Beats Compute for Inference
LLM inference is memory-bandwidth bound, not compute bound. A 70B parameter model in FP16 requires 140 GB of VRAM just for weights. At 4.8 TB/s (H200 HBM3e), loading the full model takes ~29 ms. At 3.35 TB/s (H100 HBM3), it takes ~42 ms. The difference is 31% latency reduction — directly translating to higher tokens-per-second at identical batch sizes. Raw TFLOPS are irrelevant if the memory bus cannot feed the compute units fast enough.
KV Cache Expansion and Batch Size Scaling
The KV cache grows linearly with sequence length and batch size. For a 70B model at batch=256 with 8K context, the KV cache alone consumes ~32 GB. H200's 141 GB HBM3e allows batch sizes 2.4× larger than H100's 80 GB before hitting memory limits. This is why H200 commands a premium for high-concurrency serving: it's not about TFLOPS, it's about fitting more concurrent requests in VRAM without swapping to host memory.
GDDR6X vs. HBM3: The PCIe Tax
RTX 4090's 1.0 TB/s GDDR6X bandwidth looks competitive on paper, but PCIe Gen4 x16 caps host-to-device transfer at 64 GB/s. In fine-tuning workflows where gradients must flow back to host memory, this creates a 15.6× bottleneck relative to the GPU's internal bandwidth. HBM3's 3.35 TB/s is accessed over NVLink, avoiding the PCIe lane entirely. For training, HBM is not optional — it's mandatory.
Interconnect Topologies: NVLink Mesh vs. InfiniBand NDR vs. RoCE v2
NVLink 4.0 / 5.0: The Intra-NODE Fabric
NVLink 4.0 (H100) provides 900 GB/s bidirectional between 8 GPUs in a single node — enough for tensor parallelism across all 8 GPUs with minimal all-reduce overhead. NVLink 5.0 (B200) doubles this to 1.8 TB/s, enabling pipeline parallelism split across NVLink without跨node penalties. The mesh topology means any GPU can reach any other GPU in 1 hop, keeping all-reduce latency under 2 μs for 4096-element vectors.
InfiniBand NDR: The Inter-NODE Fabric
400 Gb/s InfiniBand NDR provides 50 GB/s per port with RDMA semantics — zero-copy, kernel-bypass transfers. For a 256-GPU cluster across 32 nodes, the all-reduce communication volume is O(N) where N is the number of GPUs. At 400 Gb/s per link, a 70B model gradient sync takes ~1.2 seconds per step. RoCE v2 (RDMA over Converged Ethernet) offers similar throughput but requires explicit PFC/ECN configuration to avoid packet drops under congestion.
Rail-Optimized Switching
Rail-optimized topology connects GPU 0 on every node to the same top-of-rack switch, GPU 1 to a second switch, and so on. This eliminates intra-switch congestion and ensures all-reduce traffic is evenly distributed across the IB fabric. Without rail optimization, hot-spots on switch ports cause PFC credit exhaustion and 10-100× latency spikes that stall NCCL collectives.
Bare-Metal vs. Virtualized Cloud Pods
Hypervisor Tax: The Hidden 8-15% Cost
Virtualized GPU clouds (cloud providers using vGPU or MIG partitioning) introduce 8-15% overhead on memory bandwidth and compute throughput. The hypervisor intercepts MMIO calls and virtualizes the NVLink fabric, adding ~3 μs per memory access. For inference at batch=1, this translates to 12-18% lower tokens-per-second. For fine-tuning, the gradient sync overhead compounds across 1000+ steps.
GPU-Direct RDMA Pass-Through
Bare-metal providers (Spheron, Lambda Labs) expose GPU-Direct RDMA, allowing network interface cards to write directly into GPU VRAM without CPU involvement. This eliminates the host-memory bounce buffer, reducing all-reduce latency from ~15 μs (virtualized) to ~3 μs (bare-metal). For multi-node training, this 5× latency reduction compounds into 20-30% faster training completion.
Container Isolation Without Virtualization
Bare-metal pods use Linux namespaces and cgroups for isolation — zero hypervisor overhead. GPU access is direct via /dev/nvidia* device nodes. Container startup is <500 ms vs. 5-15 seconds for VM-based clouds. The tradeoff: no live migration, no fault tolerance at the hypervisor layer. Spot preemption handling must be implemented at the application level.
Spot Instance Architecture
Checkpointing Cadence: The 60-Second Rule
Spot instances on Vast.ai and RunPod receive SIGTERM 30 seconds before preemption. At 60-second checkpoint intervals, you lose at most 60 seconds of compute per preemption event. For a 70B model fine-tuning job at $1.86/hr (H100 spot), that's $0.031 per preemption. Over 24 hours with 4 preemptions, checkpoint overhead costs $0.12 — negligible vs. the 50-70% savings over on-demand.
Ephemeral NVMe Caching
Local NVMe drives (2-7 GB/s) on spot instances serve as write-back caches for checkpoint data. Write checkpoint to local NVMe in <2 seconds, then asynchronously replicate to S3. If preemption hits during replication, the local cache is lost — but the previous 60-second checkpoint is safe in S3. This two-tier strategy keeps checkpoint latency under 2 seconds while maintaining durability.
Graceful Shutdown Signal Handling
Register SIGTERM handlers in your training loop: save optimizer state, flush gradient buffers, and write a metadata file indicating checkpoint completeness. Resume logic reads the metadata file on startup — if the checkpoint is complete, skip the last 60-second window. If incomplete, roll back to the previous checkpoint. NCCL timeout must be set to 600 seconds (10 minutes) to survive spot preemption cycles without triggering false failure detection.
Ready to deploy your cluster?
Compare live spot pricing across all providers and find the optimal configuration for your workload.
View Live GPU Pricing →