the gap

Above the gap: XLAThe compiler: brilliant at fusing along dataflow edges, structurally unable to change your algorithm. That gap is why kernels exist.taught in /l/xla →, where the compiler decides everything and you cannot intervene. Below it: Pallas, where you decide everything: block shapes, memory residency, pipeline depth, all of it, tangled together with your math. There is no dial between the modes. Either you accept the compiler's ceiling, or you take over completely.

This is why the famous kernels exist as hand-written artifacts maintained by a small number of specialists. Flash attention is the online-softmax theorem plus expert memory choreography, welded together in one file. Every kernel like it is a crossing of this gap on foot: someone applied an algebraic identity the compiler could not, then hand-scheduled the result.

The track teaches you to make that crossing yourself. Stage 2 shows you the ceiling from above, in the compiler's own dumps. Stage 3 hands you the theorem and has you build the crossing: derive online softmax, prove its combine associative, then write the streaming kernel the compiler could not produce. After you have crossed once, the priesthood kernels read as engineering rather than magic.

A field note from this site's own profiler, and it sharpens the claim rather than weakening it: on current XLAThe compiler: brilliant at fusing along dataflow edges, structurally unable to change your algorithm. That gap is why kernels exist.taught in /l/xla →:TPU, the device timeline shows the compiler recognizing naive attention and dispatching its own hand-written online-softmax kernel (GYM·08 has the trace). Read the distinction exactly. Pattern recognition ships crossings humans already built; it does not derive new ones. For the shapes the matcher knows, the compiler now carries you over the gap. For everything else, and for every variant one mutation away, the gap is where it always was.