The collective is a kernel

Nothing about a collective is special hardware sitting outside the kernel you already know. It's the same async remote DMAA chip pushes a buffer straight into a neighbor’s memory and signals a semaphore, while its compute keeps working. The native distributed operation.taught in /l/ici → plus semaphoreThe counter a DMA signals on completion and a kernel waits on; the synchronization primitive under every transfer.taught in /l/ici → pairing this chapter has been building toward, just run in a loop shaped like a ring. A chip issues a remote DMA that writes directly into a neighbor's buffer, then waits on a semaphore that neighbor signals once the write has landed. Repeat that N-1 times, with the right forwarding rule at each step, addition for reduce-scatter, plain copy for all-gather, and you have every collective this chapter named. There is no separate collective engine sitting beside the kernel.

Because the mechanism is only DMAAn asynchronous copy between memories that runs while compute continues; the grid pipeline is DMAs the runtime writes for you.taught in /l/pallas → and semaphoresThe counter a DMA signals on completion and a kernel waits on; the synchronization primitive under every transfer.taught in /l/ici →, a collective's schedule looks in a trace exactly like any other Pallas pipeline: a sequence of issued transfers with wait points between them, laid out to keep the chip doing useful work while data is still in flight. The site's ring exhibits make that literal, drawing flash's schedule with longer arrows than a naive one's, which is the visual difference between a hop that overlaps compute and one that stalls waiting on it. Once you can read that arrow length, you're reading the same picture a profiler would give you, just drawn by hand.

The labs that follow have you build all-gather from raw remote DMAsA chip pushes a buffer straight into a neighbor’s memory and signals a semaphore, while its compute keeps working. The native distributed operation.taught in /l/ici → directly, with no collective call standing between you and the hops. Writing the loop yourself, issuing each transfer, placing each wait, choosing what each chip forwards, is the fastest way to see that "collective" is a name for a pattern of movement rather than a primitive the hardware treats differently from any other kernel. Once you've built one ring by hand, reduce-scatter and all-reduce read as the same loop with a different rule at each hop, not as new mechanisms to learn from scratch.

That's why this chapter treats the collective as a kernel rather than as a library call to trust and move past. Once you've reasoned about arrows and wait points for one ring, the same reading applies to every collective the rest of this course touches, gradient all-reduce, expert-parallel all-to-all, anything sharded across a pod. The mechanism underneath never changes: remote DMAA chip pushes a buffer straight into a neighbor’s memory and signals a semaphore, while its compute keeps working. The native distributed operation.taught in /l/ici → to write, a semaphoreThe counter a DMA signals on completion and a kernel waits on; the synchronization primitive under every transfer.taught in /l/ici → to know it landed, and a ring shape that decides how many hops it takes.