GPU interconnect and cluster costs - NVLink, InfiniBand, and Ethernet options

DERIVED ANALYSIS from existing repo records (acc-fill-dcd-h800-interconnect-halved, h100-pcie-air-cooled-vs-sxm-liquid, aws-ec2-p5-h100). Interconnect is a critical cost component for multi-GPU inference. Key data points: (1) NVLink (H100 SXM): 900 GB/s bidirectional, included in SXM module price. (2) NVLink (H200 SXM): 900 GB/s bidirectional, included in SXM module price. (3) NVLink (B200): 1,800 GB/s bidirectional, included in SXM module price. (4) PCIe (H100 PCIe): 64 GB/s bidirectional, included in PCIe card price. (5) InfiniBand (HDR): 200 Gb/s per port, $500-$1,000 per port for NIC + switch port. (6) InfiniBand (NDR): 400 Gb/s per port, $1,000-$2,000 per port. (7) Ethernet (100GbE): 100 Gb/s per port, $100-$300 per port. The H800's interconnect was halved to 400 GB/s (from 900 GB/s on H100) to comply with export controls, which directly impacts multi-GPU inference performance. For an 8-GPU node, the interconnect cost can be $4,000-$16,000 for InfiniBand, which is a significant addition to the GPU cost. The existing repo record (aws-ec2-p5-h100) notes that p5.4xlarge (single H100) does NOT support GPUDirect RDMA, making it unusable for distributed training.