NCCL bus bandwidth is typically 2x achieved bandwidth for inter-node collectives

NCCL all-reduce performance over InfiniBand achieves only ~50-60% of the theoretical bus bandwidth, and this gap is rarely stated explicitly.

Bus bandwidth (also called busbw or algorithmic bandwidth) is calculated as (N-1)/N * data_size / time for all-reduce, which assumes perfect overlap of communication and computation. Achieved bandwidth is the actual data moved divided by time. The gap comes from: (1) synchronization overhead between ranks, (2) the reduce-scatter + all-gather pattern of all-reduce means data is sent twice (once in each phase), (3) network protocol overhead, (4) the ring algorithm's serialization - each rank sends to only one neighbor at a time. For a ring all-reduce with N ranks, the ring algorithm achieves (N-1)/N of peak bandwidth in the ideal case, but real-world factors reduce this further.

Use tree algorithm for small message sizes (lower latency). Use ring algorithm for large message sizes (higher bandwidth). For inter-node, ensure NCCL is using the correct IB transport (not falling back to socket). Set NCCL_IB_HCA to specify the correct HCA. Use NCCL_NET_GDR_LEVEL=5 for GPUDirect RDMA.

The 2x gap between bus bandwidth and achieved bandwidth is one of the most commonly misunderstood performance metrics in distributed training. When a benchmark reports 'NCCL achieved 400 GB/s all-reduce bandwidth,' it may mean bus bandwidth (which counts data twice) or achieved bandwidth (which counts once). Always check which metric is being reported. The NCCL documentation and nccl-tests benchmark report bus bandwidth by default.