Cloud TPU v5p ICI (inter-chip interconnect), 3D torus, 8960-chip pod

PER CHIP, PER DIRECTION (1200 GB/s per direction). Google's v5p spec table publishes 'Bidirectional inter-chip interconnect (ICI) bandwidth per chip (GBps) 1200'. The published figure is a BIDIRECTIONAL AGGREGATE per chip, so the typed value is its per-direction half: 600 GB/s per direction per chip. Note that v5p does NOT publish an ICI port count, unlike v5e and v6e which both state 'ICI ports per chip 4' - so no per-port figure can be derived for v5p and none is recorded here. Compare with v5e's per-chip aggregate of 400 GBps bidirectional: v5p is 3x v5e per chip in each direction, which is why v5p's per-chip ICI is the largest of any generation in this set except TPU7x, which publishes the same 1200 GBps. Google writes 'GBps' (gigabytes) for the ICI row and 'Gbps' (gigabits) for the DCN row of the same table.

3D TORUS, published as 'Interconnect topology: 3D torus *'. v5p is also the generation where Google's own configuration docs say plainly that the default pod topology is NOT what a job gets: 'There are 8960 chips in a v5p Pod. The largest job that can be scheduled is a 96 cube (6144 chip) job', and the supported shapes table tops out at 16x16x24 = 6144 chips, 1536 hosts, 96 cubes. A full pod is 8960 chips / 17920 TensorCores / 2240 hosts / 140 cubes. So POD SIZE AND MAXIMUM SINGLE-SLICE SIZE DIFFER BY 8960 vs 6144 CHIPS, and that gap - not the per-chip ICI bandwidth - is the number that decides whether a job must be sharded across slices over the data-centre network rather than living inside one ICI domain. TWISTED TORUS: v5p supports 'twisted' topologies on slices where each dimension is equal to or twice the smallest, listed as 4x4x8, 8x8x16, 4x8x8, 8x16x16; Google publishes the theoretical bisection-bandwidth gain versus an untwisted torus as ~70% for 4x4x8 and ~40% for 4x8x8. The twist re-routes wrap-around edges to make an asymmetric torus symmetric, which improves load balancing, bisection bandwidth and packet route lengths. v5p also supports ICI resiliency by default for slices of one cube or larger (4x4x4), which routes around optical circuit switch and optical ICI faults between cubes - copper links within a cube are unaffected - at the cost of temporary ICI performance degradation.

ICI is the scale-up fabric; the scale-up domain is the POD (8960 chips). Google's documented boundary is ICI versus DCN, not 'scale-up'/'scale-out': ICI is 'high speed, low latency internal links that connect TPUs within a TPU Pod', while data crossing between slices travels 'by transferring data to CPUs (hosts) which in turn transmit the data over the data-center network (DCN)', described as 'a higher latency, lower-throughput network (when compared with ICI)'. The booleans are this repo's mapping of that boundary. SCALE-OUT FIGURES AS DOCUMENTED, v5p: 'Data center network (DCN) bandwidth per chip (Gbps) 50' - the only generation here whose DCN figure Google states PER CHIP rather than per host or per pod - and, from the host-properties table, NIC throughput 200 Gbps per host. Direction is not stated for either, so they are recorded as published and flagged unstated. Note the two DCN figures disagree in unit base (per chip vs per host, 4 chips per host) and are not reconciled here; v5p hosts carry 4 chips, so 50 Gbps per chip and 200 Gbps per host are consistent if the per-chip figure is the per-direction share and the NIC is bidirectional, but Google does not say so and neither figure is converted. v5p scales beyond one pod via Multislice to 18432 chips, which is explicitly a DCN construct.

The generation where chips-per-pod most often gets mis-stated, and the reason is worth recording: 8960 is the POD size while 6144 is the largest schedulable SINGLE SLICE, so 'the v5p pod' and 'the v5p slice you can get' are different numbers and only the first is a scale-up domain. Host packing is 4 chips per host (ct5p-hightpu-4t), 16 hosts to a cube of 64 chips, so the 8960-chip pod is 2240 hosts and the 6144-chip max slice is 1536 hosts. PACKING AND TOPOLOGY CHOICE INTERACT: Google states that the default topology for a chip count is the one most similar to a cube because that shape is likely best for data-parallel training, that other arrangements suit workloads with multiple kinds of parallelism, and gives the concrete rule - for an 8x16x16 slice, 8-way or 16-way model parallelism mapped to one physical topology dimension is more performant, while 4-way model parallelism is sub-optimal on that slice but optimal on a 4x16x32 slice of the same chip count. THAT IS THE SHARDING CONSTRAINT a reader planning multi-host will otherwise miss: on TPU the choice of tensor-parallel degree is constrained by which slice shape you can provision, not free as it is on a GPU node. PER-POD ICI AGGREGATE AND PER-POD BISECTION ARE NOT PUBLISHED for v5p - the page gives per-chip ICI bandwidth and per-chip DCN bandwidth but no pod-level all-reduce or bisection row, unlike v5e and v6e. The v3 and v4 pages do publish all-reduce and bisection per pod, so the gap is specific to v5p's page and is recorded as unpublished rather than estimated. Sources: asic-tpu-v5p-specs for every per-chip figure, the pod/slice/cube tables and the twisted-topology section, w7c-google-tpu-multislice-overview for the ICI/DCN definitions and for Multislice as a DCN construct. === THE HALVING IS NOT AN ASSUMPTION — CONFIRMED 2026-10-08 (w8-reverify) AGAINST GOOGLE'S OWN POD-LEVEL FIGURES. === Wave 7 recorded Google's ICI figures as bidirectional aggregates with the typed value being the per-direction half. That is a halving decision, and a halving decision is only defensible if something OUTSIDE the spec table forces it. Two independent things do. (1) Google's own row label, verbatim on both the v5e and v6e spec tables: 'Bidirectional inter-chip interconnect (ICI) bandwidth (per chip) 400 GBps' / '... 800 GBps'. The word 'Bidirectional' is Google's, and the table gives the port count beside it ('ICI ports per chip 4'), so the figure is an aggregate per chip, not per port and not per direction. (2) THE CLOSING ARITHMETIC, which is the real proof and is reproduced here so it does not have to be re-derived: Google publishes the all-reduce bandwidth per pod for the same generations. v5e: 'All-reduce bandwidth per Pod 51.2 TB/s' with a 256-chip pod; 256 x 200 GB/s per direction = 51.2 TB/s, EXACTLY. v6e: 'All-reduce bandwidth per Pod 102.4 TB/s' with a 256-chip pod; 256 x 400 GB/s per direction = 102.4 TB/s, EXACTLY. Both close on the HALVED figure and both would be 2x too high on the unhalved one. So an all-reduce over 256 chips at the per-direction rate reproduces Google's published pod collective bandwidth to the digit - the halving is what Google's own numbers require, not a convention this repo imposed on them. (3) A unit trap in the same table, recorded so it is not re-fallen-into: Google writes 'GBps' for the ICI row and 'Gbps'/'Tbps' for the DCN and NIC rows of the SAME table, so the units are mixed within one sheet and ICI figures are gigabytes. WHAT WAS ALSO VERIFIED AND IS CORRECT: v4 has no per-chip ICI bandwidth published (Google gives six links per chip and pod-level 1.1 PB/s all-reduce / 24 TB/s bisection but no per-chip GB/s), so its typed null is the honest value and not a gap to fill by dividing 1.1 PB/s by 4096 - that would be an artefact of the collective algorithm's overlap, not a link rate. v5p's 1200 GBps -> 600 per-direction is labelled 'Bidirectional inter-chip interconnect (ICI) bandwidth per chip' on the live v5p table and carries no port count, so no per-port figure is derivable and none is recorded.