Cost formulas you can do in your head

The nice thing about ring collectives is that their cost is arithmetic you can do at your desk, not a number you have to go measure first. Every formula below reduces to the same shape: bytes moved, divided by how fast the link under you can move them, scaled by a factor that comes from the ring's geometry rather than from anything the compiler decides on its own. Once you have that shape, you can sanity-check a profiler trace before you even open it, and notice quickly when a run is spending time on something other than the ring itself.

For a ring all-gather across N chips, the time is about $(N-1)/N$ times the total bytes being gathered, divided by the per-direction link bandwidth. The $(N-1)/N$ factor is just "how much of the data wasn't already local to you," which is why it shrinks toward 1 as N grows: with more chips, almost none of the final payload started where it ends up. On a unidirectional ring that's the whole story. Most TPU interconnects are bidirectional, meaning data can move both ways around the ring at once, which roughly halves the time versus sending everything in a single direction.

All-reduce costs about twice what a reduce-scatter alone costs, since it is built from a reduce-scatter followed by an all-gather of a payload the same size. If you already have the reduce-scatter number worked out for your chip count and byte count, double it and you have all-reduce, no separate formula to memorize. It's worth holding onto that relationship specifically, because all-reduce is the collective you will use most often, in gradient synchronization and in any place where every chip needs the same combined answer.

The numbers below are computed, not measured on hardware: 1 GiB all-gathered across 8 chips on v5e ICIThe inter-chip links (4.5e10 bytes per second each way per link on v5e); every collective resolves to hops over these.taught in /l/ici → links, at 4.5e10 bytes per second per direction. One direction: $(7/8) \times 2^{30}\text{ bytes} / 4.5\text{e}10\text{ B/s} \approx 21$ ms. Bidirectional: roughly half that, about 10 ms. Those are the numbers a ring all-gather owes you before a single flop of compute happens alongside it, and they scale the way you'd expect: double the bytes and you double the time, double the chips and the $(N-1)/N$ factor barely moves once N is already large.

Treat every number above as a floor, not a runtime you should expect to see in a profile. A real kernel overlaps the ring's hops with compute that doesn't depend on the data still in flight, so the collective's cost gets hidden behind work the chip was going to do anyway rather than sitting on top of it. Getting that overlap right, choosing what compute to schedule against which hop, is the entire subject of stage 4, and it's the difference between a collective that shows up as dead time in a trace and one that barely shows up at all.