The physical picture
Chips in a TPU pod connect to their neighbors over ICI links (4.5e10 bytes per second each way per link on v5e), and slices along a mesh axis form physical rings. Every collective you have ever called resolves to movements over these links: an all-gather is N−1 hops around the ring, each chip pushing one shard to its neighbor; a reduce-scatter is the same ring with an add at each hop.
The TPU's native distributed operation is the async remote DMA: a chip pushes a buffer directly into a neighbor's VMEM and signals a semaphore, while its compute units keep working. That last clause is the entire economics of distributed kernels: transfers that hide behind compute are free, and transfers that do not are the dominant tax at scale. The craft is arranging the overlap.
The deep continuity with everything above: sharding decides where data lives, and a kernel is always a per-device program on its local shard. The streaming-attention algebra from stage 3 does not care whether a KV block arrived from local HBM or from the neighbor chip, which is why ring attention (stage 4) is a composition, not a new derivation. EX·04 shows it: the same schedule as EX·03, longer arrows.