Cloud TPU v5e ICI (inter-chip interconnect), 2D torus, 256-chip pod

PER CHIP, PER DIRECTION, PER PORT (400 GB/s per direction). Google's v5e spec table publishes 'Bidirectional inter-chip interconnect (ICI) bandwidth (per chip) 400 GBps' together with 'ICI ports per chip 4'. The published figure is therefore 400 GB/s bidirectional-aggregate per chip across 4 ports, i.e. 200 GB/s per direction per chip in aggregate over its ports, and 100 GB/s per direction per port if the four ports are symmetric. The typed value records the per-direction half of Google's published aggregate so that it is comparable with nvlink-4's per-link per-direction 25 GB/s and NOT with nvswitch-4's 450 GB/s per GPU per direction. Google writes 'GBps' here meaning gigabytes per second, not gigabits - the same table's DCN row writes 'Gbps'/'Tbps' for bits, so the two units are mixed within one spec sheet.

2D TORUS across a 256-chip pod, per Google's own 'Interconnect topology' row. This is the part a reader planning a multi-host job will get wrong: a 2D torus is NOT a uniform mesh and its bisection bandwidth does not follow from total bandwidth by any simple division. Google publishes the pod-level consequences as explicit numbers rather than leaving them to be derived: all-reduce bandwidth per pod 51.2 TB/s against bisection bandwidth per pod 1.6 TB/s, a factor of 32. Any sizing that treats the 256 chips as an unordered set of equivalent devices, or that assumes any-to-any fabric behaviour, will overestimate the cost of a global collective by roughly that 32x. The wrap-around edges that close each torus axis are the expensive part of the fabric, and collectives are shaped around them.

ICI is Google's TPU scale-up fabric and its scale-up domain is the POD, not the host: chips within one pod communicate chip-to-chip over ICI, while chips in different pods communicate by staging data through the host CPUs onto the data-centre network. Google's own words for the boundary are ICI versus DCN, not 'scale-up' and 'scale-out' - the Multislice Overview defines ICI as 'high speed, low latency internal links that connect TPUs within a TPU Pod' and DCN as 'a higher latency, lower-throughput network (when compared with ICI) that connects TPU slices in a Multislice configuration'. The scale-up boolean here is therefore this repo's mapping of that documented ICI/DCN boundary, not a quotation. SCALE-OUT FIGURES AS DOCUMENTED, for v5e: per-host NIC configuration 2 x 100 Gbps NIC, and data centre network bandwidth per pod 6.4 Tbps (BITS - Google writes Tbps, unlike the ICI GBps row). Both directions are unstated for the DCN figures, so they are recorded here as published and flagged unstated rather than halved. Host packing: 8 chips per host, so a 256-chip pod is 32 hosts, and v5e's published slice table runs 1x1 (1 chip, 1/8 host) up to 16x16 (256 chips, 32 hosts) - the smallest slices are SUB-HOST, which is why v5e serving is documented as single-host only up to 8 chips.

The cheapest TPU generation on which to learn the ICI/DCN split, because the pod is small enough (256 chips, 8 chips per host, 32 hosts) that the scale-up boundary lands inside a realistic serving job while the numbers are still fully published. PER-POD ICI AGGREGATE BANDWIDTH IS NOT PUBLISHED by Google for this generation - the spec sheet gives per-chip ICI bandwidth, per-pod all-reduce bandwidth and per-pod bisection bandwidth, and nothing that could be summed into a 'total pod ICI bandwidth'. That gap is deliberate here rather than filled by multiplication, because summing per-chip figures across a torus is exactly the operation that overstates a torus; the all-reduce and bisection pair is the published, load-bearing form of the pod-level answer. USE FOR FIT DECISIONS: 256 chips x 16 GB HBM is the memory ceiling of the scale-up domain, so a model whose weights plus KV cache exceed that is sharded across hosts over DCN, not merely across ICI. WHY THIS MATTERS INFERENCE-SIDE: the 32x all-reduce-to-bisection ratio means a naively-sharded batch-parallel decode over a full pod pays for cross-pod traffic that a host-confined deployment does not, and v5e's own docs steer multi-host inference to Sax and single-host inference to up to 8 chips (1x1, 2x2, 2x4 slices) - i.e. Google itself confines the latency-sensitive path to the scale-up domain. COMPARE: nvswitch-4's 8-GPU domain is non-blocking, so there is no equivalent penalty at small tensor-parallel degree on NVIDIA hardware; the torus penalty appears only as the scale-up domain grows from 8 devices to 256. CONTRAST WITH V5P: v5e is a 2D torus in a 256-chip pod, while v5p is a 3D torus in an 8960-chip pod, so per-chip ICI bandwidth and pod size both rise but the bisection-to-all-reduce relationship is not simply scaled. Sources: asic-tpu-v5e-specs for every per-chip and per-pod figure, cloud-tpu-multislice-overview for the ICI/DCN definitions, alt-tpu-system-arch for the pod/slice vocabulary. === THE HALVING IS NOT AN ASSUMPTION — CONFIRMED 2026-10-08 (w8-reverify) AGAINST GOOGLE'S OWN POD-LEVEL FIGURES. === Wave 7 recorded Google's ICI figures as bidirectional aggregates with the typed value being the per-direction half. That is a halving decision, and a halving decision is only defensible if something OUTSIDE the spec table forces it. Two independent things do. (1) Google's own row label, verbatim on both the v5e and v6e spec tables: 'Bidirectional inter-chip interconnect (ICI) bandwidth (per chip) 400 GBps' / '... 800 GBps'. The word 'Bidirectional' is Google's, and the table gives the port count beside it ('ICI ports per chip 4'), so the figure is an aggregate per chip, not per port and not per direction. (2) THE CLOSING ARITHMETIC, which is the real proof and is reproduced here so it does not have to be re-derived: Google publishes the all-reduce bandwidth per pod for the same generations. v5e: 'All-reduce bandwidth per Pod 51.2 TB/s' with a 256-chip pod; 256 x 200 GB/s per direction = 51.2 TB/s, EXACTLY. v6e: 'All-reduce bandwidth per Pod 102.4 TB/s' with a 256-chip pod; 256 x 400 GB/s per direction = 102.4 TB/s, EXACTLY. Both close on the HALVED figure and both would be 2x too high on the unhalved one. So an all-reduce over 256 chips at the per-direction rate reproduces Google's published pod collective bandwidth to the digit - the halving is what Google's own numbers require, not a convention this repo imposed on them. (3) A unit trap in the same table, recorded so it is not re-fallen-into: Google writes 'GBps' for the ICI row and 'Gbps'/'Tbps' for the DCN and NIC rows of the SAME table, so the units are mixed within one sheet and ICI figures are gigabytes. WHAT WAS ALSO VERIFIED AND IS CORRECT: v4 has no per-chip ICI bandwidth published (Google gives six links per chip and pod-level 1.1 PB/s all-reduce / 24 TB/s bisection but no per-chip GB/s), so its typed null is the honest value and not a gap to fill by dividing 1.1 PB/s by 4096 - that would be an artefact of the collective algorithm's overlap, not a link rate. v5p's 1200 GBps -> 600 per-direction is labelled 'Bidirectional inter-chip interconnect (ICI) bandwidth per chip' on the live v5p table and carries no port count, so no per-port figure is derivable and none is recorded.