The vocabulary, as ring movements

You already know the ring: chips wired so each one talks only to its two neighbors, moving shards with remote DMAA chip pushes a buffer straight into a neighbor’s memory and signals a semaphore, while its compute keeps working. The native distributed operation.taught in /l/ici → instead of routing through a switch. This chapter puts names on the specific movements that ring supports. Five words cover nearly everything a distributed kernel does on a pod: all-gather, reduce-scatter, all-reduce, collective_permute, and all-to-all. Each is the same physical ring, with a different rule for what a chip forwards and what it keeps. Once you can picture the ring turning, these API names stop being jargon and start reading as a literal count of hops and payload.

All-gather is the simple case: every chip starts with one shard and ends with all of them. On an N-chip ring, that takes N-1 hops. Each chip receives a shard from one neighbor, keeps a copy, and forwards it to the next neighbor in the same step. After N-1 of those steps, every chip holds a full copy of every shard that started anywhere on the ring. Nothing gets combined along the way; the ring just circulates data until it has covered every chip.

Reduce-scatter runs on the same ring, but it carries partial sums instead of raw copies. Each chip starts holding a full-size buffer contributed by every participant, and at every hop it adds the incoming piece to its own running total for that piece. After N-1 hops, each chip ends up owning one reduced shard, the sum across all chips for that one slice, and nothing else. The traffic pattern is identical to all-gather. The only difference is that addition happens at every hop instead of a plain copy.

All-reduce is what you get when you chain the two: a reduce-scatter, so every chip ends with one correctly summed shard, followed by an all-gather of those reduced shards, so every chip ends with the full reduced result. It looks like a single operation from the caller's side, but the ring runs through both phases in sequence, and the cost formulas in the next section reflect that.

Two more names round out the vocabulary. collective_permute moves data exactly one hop, chip to a chosen neighbor, no accumulation and no further forwarding. All-to-all is the busy one: every chip sends a distinct piece to every other chip, not one shared payload circulating but N different messages per chip. On a bidirectional ring, all-to-all costs roughly a quarter of an all-gather; the scaling book works through why the geometry favors it that way.

None of these five names describe different hardware paths. They describe five different rules for the same ring: what a chip keeps at each hop, what it adds, and how many hops it takes before every chip is done. Learn the ring once and the vocabulary stops being five things to memorize separately; it becomes one mechanism read five different ways, which is exactly how the rest of this chapter treats it.