InfiniBand XDR (Quantum-2)

PER LANE, PER DIRECTION - CORRECTED IN AUDIT 2026-10-03, THE UNIT ERROR, same class as the ualink-200g-1-0 correction. The IBTA states the physical layer is defined 'at data rates up through XDR at 200 Gb/s per lane' (link-ibta-release-2-1) - 200 GIGABITS. The previous value of 200 in this GB/s field read that as 200 GIGABYTES, overstating the per-lane rate by 8x. Correctly converted, 200 Gb/s = 25 GB/s per lane per direction, which is the recorded value - and is exactly the same per-lane rate as NVLink 2/3/4 and UALink 200G, since all four are built on comparable per-pair PAM4/NRZ SerDes rates. link_count 4 is the standard 4-lane NDR/XDR port width, giving the familiar 800 Gb/s port: 4 lanes x 200 Gb/s = 800 Gb/s = 100 GB/s PER DIRECTION PER PORT. NAMING WARNING, which is why the correction is easy to get wrong in the other direction too: InfiniBand's speed nomenclature quotes WHOLE-PORT figures while the underlying rate is per-lane. 'NDR' is the 400 Gb/s generation and 'XDR' is 800 Gb/s - both per port per direction - each corresponding to 100 Gb/s and 200 Gb/s per lane. So 'XDR = 200' is right per lane and wrong per port, and 'XDR = 800' is right per port and is 8x the per-lane rate. THE 8x ERROR MATTERS DECISION-WISE: at 200 'GB/s' per lane, an XDR port looked like 800 GB/s per direction, which would have exceeded NVLink 4's 450 GB/s per direction per GPU and inverted the repo's central claim that inter-node RDMA is roughly an order of magnitude below intra-node NVLink. At the corrected 100 GB/s per direction per port, XDR is about 4.5x below NVLink 4 per direction - the gap is real but smaller than 'an order of magnitude', which is the honest statement.

Switched fat-tree with a centrally managed subnet manager providing routing and multiple paths between nodes, so simple leaf/aggregation/spine switches suffice without per-switch management.

Managed switched fabric. A fat-tree with a typical 1:1 uplink-to-downlink ratio is non-blocking; a 2:1 ratio caps aggregate bisection bandwidth at half the sum of downlink bandwidth, which is exactly where all-to-all collectives (MoE expert parallel, TP across nodes) lose their bandwidth.

The scale-out baseline for AI clusters, and the counterweight to NVLink. Reach differentiates it: copper to 400 Gb/s in-rack, active optical cabling to 100 m, fiber to 10 km. Current specification state: Volume 1 and Volume 2 Release 2.1, both published 2026-07-31, with Volume 2 explicitly covering XDR at 200 Gb/s per lane and adding CMIS 5.4 support and updated XDR data rate support; Volume 1 grew from 2156 to 2294 pages and added Multipath Reliable Connection work. Inference consequence: scale-out fabrics are the tier where the oversubscription ratio, not the link rate, decides performance - a 2:1 fat-tree halves effective all-to-all bandwidth regardless of the 200 Gb/s lane rate. Pair with nvswitch-4/5 for the scale-up tier: they are different jobs and different failure modes.