Do You Need InfiniBand? NVLink vs InfiniBand NDR for Multi-Node LLM Training

Multi-node LLM training bottlenecks at the AllReduce step — where gradients are synchronized across GPUs. For a 70B parameter model with FP16 gradients, each AllReduce transfers ~140 GB across all GPUs. In a ring-all-reduce topology, the communication volume per GPU is 2(N-1)/N × 140 GB, where N is the GPU count. At 8 GPUs: 175 GB per GPU. At 64 GPUs: 275 GB per GPU. Without InfiniBand, this transfer crosses Ethernet at 100 Gbps (12.5 GB/s), taking 22 seconds per AllReduce step. With InfiniBand NDR at 400 Gbps (50 GB/s), it takes 5.5 seconds. Across 10,000 training steps, that's 4.7 hours saved.

Within a single node (8 GPUs), NVLink 4.0 at 900 GB/s per-GPU provides the interconnect. NVLink is not Ethernet — it's a direct GPU-to-GPU link with zero CPU involvement. For multi-node training, the question is how to connect NVLink domains across nodes. Two options: (1) InfiniBand NDR — dedicated RDMA fabric with GPU-Direct, kernel-bypass transfers. (2) RoCE v2 (RDMA over Converged Ethernet) — Ethernet-based RDMA with lossless fabric requirements. InfiniBand delivers 30-40% lower AllReduce latency because it avoids Ethernet's TCP/IP stack overhead and relies on hardware-based congestion control.

Modern AI clusters use rail-optimized switching: each GPU connects to a dedicated top-of-rack (ToR) switch, and ToR switches connect via a spine fabric. For 8-node clusters (64 GPUs): Each node has 8 GPUs, each connecting to one of 8 ToR switches via InfiniBand NDR 400 Gbps. The spine fabric aggregates 8 × 400 Gbps = 3.2 Tbps per rail. This topology ensures that AllReduce traffic is evenly distributed — no single switch becomes a bottleneck. Rail-optimized is mandatory for NCCL to achieve near-linear scaling beyond 32 GPUs.

InfiniBand has near-zero packet loss (<10^-15 BER) because it uses hardware-based flow control. Ethernet, even with RDMA over Converged Ethernet (RoCE v2), requires priority flow control (PFC) to achieve lossless behavior. A single packet loss event in RoCE v2 triggers a retransmission that stalls the entire AllReduce — impacting all 64 GPUs simultaneously. For training jobs running 24/7, even 0.001% packet loss translates to ~30 minutes of lost compute per day. InfiniBand eliminates this risk entirely.