Google Cloud TPU Inter-Chip Interconnect (ICI)

PER CHIP, PER DIRECTION, v5e GENERATION AS THE REFERENCE POINT. Google's per-chip spec tables publish ICI bandwidth as a BIDIRECTIONAL aggregate per chip, so this record's typed value is one generation's per-direction half and the per-generation figures are all carried explicitly here: v5e publishes 'Bidirectional inter-chip interconnect (ICI) bandwidth (per chip) 400 GBps' across 4 ICI ports, giving the recorded 200 GB/s per direction; v6e publishes 800 GBps bidirectional across the same 4 ports, giving 400 GB/s per direction; v5p publishes 1200 GBps bidirectional per chip. v4 is NOT recorded as a number: Google's v4 docs give a link count ('Six interconnect links per chip') and topology but no per-chip ICI bandwidth figure, so v4's value is deliberately absent rather than inferred from another generation. Do not compare these per-chip numbers directly against nvlink-4 or amd-xgmi without checking direction - NVIDIA's per-GPU NVLink figures are bidirectional aggregates too, which is why nvswitch-4 records 450 GB/s per direction from a published 900 GB/s.

Switched mesh, NOT a full mesh, and this is the defining difference from the GPU scale-up fabrics. Per generation: v4 uses a 3D mesh that can be reconfigured into a 3D torus, with 6 interconnect links per chip, a 4096-chip pod, and optical circuit switches between cubes - the v4 architecture paper is the source for the optical switching. v5p uses a 3D torus. v5e and v6e both use a 2D TORUS across a 256-chip pod. Google's pod-level docs give the consequences directly as published numbers, which is why those are worth reading instead of deriving a mesh diameter: v5e all-reduce 51.2 TB/s per pod against a BISECTION bandwidth of only 1.6 TB/s; v6e all-reduce 102.4 TB/s against bisection 3.2 TB/s. That ratio is the whole story of ICI versus NVSwitch - a torus gives strong nearest-neighbour and all-reduce performance at a bisection bandwidth roughly 30x lower than the all-reduce figure, whereas nvswitch-4's 8-GPU domain is non-blocking.

The fabric is packet-switched with topology awareness, and Google publishes the bisection and all-reduce bandwidths separately precisely because they diverge so much. Beyond the pod, scaling is by POD and then by data-centre network, not by extending ICI: v5e/v6e publish 2x100 Gbps and 4x200 Gbps per-host NIC configurations respectively, with 6.4 TB/s and 25.6 TB/s of data-centre network bandwidth per pod. Note that for TPU the 'scale-up' and 'scale-out' boundary sits at the pod, unlike the GPU parts in this repo where it sits at the 8-GPU NVSwitch domain - a 4096-chip v4 pod is ONE scale-up domain, and the practical consequence is that TPU topology is fixed at slice granularity by how the slice is carved, which is a different engineering constraint from choosing a tensor-parallel degree.

The scale-up fabric for every Google TPU in this repo, and the clearest example in the knowledge base of why a torus is not a mesh. PER-GENERATION PUBLISHED ICI FIGURES (all from Google's own per-chip spec tables): v4 - 6 links per chip, 3D mesh reconfigurable to 3D torus, 4096-chip pod, optical circuit switches between cubes, NO per-chip bandwidth published; v5p - 3D torus, 1200 GBps bidirectional per chip; v5e - 2D torus, 400 GBps bidirectional per chip over 4 ports, 256-chip pod, all-reduce 51.2 TB/s vs bisection 1.6 TB/s; v6e - 2D torus, 800 GBps bidirectional per chip over 4 ports, 256-chip pod, all-reduce 102.4 TB/s vs bisection 3.2 TB/s, plus 4x200 Gbps per-host NICs and 25.6 TB/s DC network per pod. INFERENCE CONSEQUENCE, and the reason this belongs in a memory-bound study: a torus deliberately trades bisection bandwidth for cheap nearest-neighbour links, and collectives are shaped around that. Ring and torus-aware algorithms are what the topology wants, and a naive flat all-reduce pays for the wraps. For a 2D torus the wrap-around edges are the expensive ones, so the standard mitigation is to arrange the tensor-parallel group to match the chip arrangement rather than to treat the pod as an unordered set of 256 equivalent devices. The 32x gap between v5e's all-reduce (51.2 TB/s) and bisection (1.6 TB/s) is the concrete number to reason from. COMPARE: amd-xgmi and amd-infinity-fabric-link get a fully connected 8-way domain with no wraps, so on an 8-GPU AMD node there is no torus penalty at all - AMD wins at small TP degree and loses at large pod scale, which is the same conclusion NVIDIA reaches from the other direction with a 72-GPU non-blocking NVSwitch domain. Note also that Google's own docs separate BF16/INT8 doubling behaviour across generations (v4 does NOT double INT8 over BF16, v5p's FP8 row is contested, v6e does double) - that is a compute-side inconsistency and must not be smuggled into a bandwidth comparison. KIND: filed as 'other' because ICI is a proprietary vendor fabric with no entry in the schema's interconnect enum, and it is not a member of any of the named families (not NVLink, not Ethernet, not InfiniBand) despite doing a scale-up job analogous to NVSwitch. SUPERSEDED IN PART BY THE PER-GENERATION ICI RECORDS (w7c-google-tpu-ici-v3, -v4, -v5e, -v5p, -v6e, -tpu7x). This umbrella record's typed bandwidth_gbps of 200 is the v5e per-direction half, but its version field says 'v4 through v6e generations', so a reader filtering by generation cannot tell which figure belongs to which part - which is exactly the class of error the per-generation records exist to remove. Read the umbrella as the family-level explanation (torus vs mesh, ICI vs DCN) and take any specific number from the per-generation record, which states its own direction convention, chips per pod, and where Google publishes nothing. Three corrections this umbrella carried that the per-generation records fix: it does not give v5p's pod size (8960 chips, published), it does not mention that TPU7x/Ironwood exists at all, and it treats v4's topology as a plain '3D mesh reconfigurable to 3D torus' without Google's published shape condition (2A=B=C or 2A=2B=C, each dimension a multiple of 4) that decides whether a given v4 slice is a torus or a mesh at all. === THE HALVING IS NOT AN ASSUMPTION — CONFIRMED 2026-10-08 (w8-reverify) AGAINST GOOGLE'S OWN POD-LEVEL FIGURES. === Wave 7 recorded Google's ICI figures as bidirectional aggregates with the typed value being the per-direction half. That is a halving decision, and a halving decision is only defensible if something OUTSIDE the spec table forces it. Two independent things do. (1) Google's own row label, verbatim on both the v5e and v6e spec tables: 'Bidirectional inter-chip interconnect (ICI) bandwidth (per chip) 400 GBps' / '... 800 GBps'. The word 'Bidirectional' is Google's, and the table gives the port count beside it ('ICI ports per chip 4'), so the figure is an aggregate per chip, not per port and not per direction. (2) THE CLOSING ARITHMETIC, which is the real proof and is reproduced here so it does not have to be re-derived: Google publishes the all-reduce bandwidth per pod for the same generations. v5e: 'All-reduce bandwidth per Pod 51.2 TB/s' with a 256-chip pod; 256 x 200 GB/s per direction = 51.2 TB/s, EXACTLY. v6e: 'All-reduce bandwidth per Pod 102.4 TB/s' with a 256-chip pod; 256 x 400 GB/s per direction = 102.4 TB/s, EXACTLY. Both close on the HALVED figure and both would be 2x too high on the unhalved one. So an all-reduce over 256 chips at the per-direction rate reproduces Google's published pod collective bandwidth to the digit - the halving is what Google's own numbers require, not a convention this repo imposed on them. (3) A unit trap in the same table, recorded so it is not re-fallen-into: Google writes 'GBps' for the ICI row and 'Gbps'/'Tbps' for the DCN and NIC rows of the SAME table, so the units are mixed within one sheet and ICI figures are gigabytes. WHAT WAS ALSO VERIFIED AND IS CORRECT: v4 has no per-chip ICI bandwidth published (Google gives six links per chip and pod-level 1.1 PB/s all-reduce / 24 TB/s bisection but no per-chip GB/s), so its typed null is the honest value and not a gap to fill by dividing 1.1 PB/s by 4096 - that would be an artefact of the collective algorithm's overlap, not a link rate. v5p's 1200 GBps -> 600 per-direction is labelled 'Bidirectional inter-chip interconnect (ICI) bandwidth per chip' on the live v5p table and carries no port count, so no per-port figure is derivable and none is recorded.