Cloud TPU Multislice Overview

Google's own definition of the ICI/DCN boundary: 'TPU chips in each slice communicate through inter-chip-interconnect (ICI). TPU chips in different slices communicate by transferring data to CPUs (hosts) which in turn transmit the data over the data-center network (DCN).' Defines DCN as 'a higher latency, lower-throughput network (when compared with ICI) that connects TPU slices in a Multislice configuration', and ICI as 'high speed, low latency internal links that connect TPUs within a TPU Pod'. Also documents that inter-slice DCN code is generated by XLA, that all slices in a Multislice must be the same shape, and that heterogeneous slice shapes are unsupported. Source for the Scale-Up (ICI, within pod) vs Scale-Out (DCN, between slices and pods) split.