Six generations of the same idea

The architecture of the previous lesson has held for a decade: a matrix unit, a vector unit, a software-managed scratchpad, HBMThe chip’s main memory: large, far, and the resource memory-bound ops spend. 8.2e11 bytes per second on v5e, 1.6e12 on v6e.taught in /l/tpu → behind them, links to neighbors. What moves between generations is the numbers, and the numbers move unevenly. Compute has grown faster than memory bandwidth in almost every step, which drags one derived quantity upward: the ridgeThe FLOP-per-byte ratio where an op flips from memory-bound to compute-bound: about 240 on v5e, about 575 on v6e.taught in /l/tpu →, peak FLOPs divided by HBM bytes per second. That single ratio is the personality of a generation. It says which kernels the chip rewards and which it starves.

Read the table as columns of the same machine growing at different rates. Units are exact: FLOPs per second at bf16, bytes and bytes per second for memory, one-way bytes per second per link for ICI.

Take the step from v5e to v6e, the two chips this site's bench actually ran. Compute jumped 4.7x, from 1.97e14 to 9.20e14, because the MXUThe systolic matmul array: 128x128 on v5e, 256x256 on v6e. Matmuls only; everything else is the VPU’s job.taught in /l/tpu → doubled in both dimensions. HBMThe chip’s main memory: large, far, and the resource memory-bound ops spend. 8.2e11 bytes per second on v5e, 1.6e12 on v6e.taught in /l/tpu → bandwidth only doubled, 8.2e11 to 1.6e12. Divide and the ridgeThe FLOP-per-byte ratio where an op flips from memory-bound to compute-bound: about 240 on v5e, about 575 on v6e.taught in /l/tpu → moves from about 240 FLOPs per byte to about 575. Every op whose intensity sits between those two numbers changed teams: compute-bound on v5e, memory-bound on v6e. A kernel tuned on one chip can cross the ridge on the next without a single line changing, and the table told you before the profiler did.

Notice also what v5e is in this table: the only generation whose compute went down relative to its predecessor's line. It is the efficiency part, 16 GB of HBMThe chip’s main memory: large, far, and the resource memory-bound ops spend. 8.2e11 bytes per second on v5e, 1.6e12 on v6e.taught in /l/tpu → and a quarter of v5p's bandwidth, priced for serving. The lettered split matters when you rent: e parts trade capacity and bandwidth for cost, p parts keep the headline numbers. Stage 0 taught the rooflineThe floor model: latency is at least the larger of FLOPs over peak compute and bytes over bandwidth. Predict first, measure second.taught in /l/tpu → as a skill; this table is the terrain the skill runs on, and the roofline playground on that page now carries every generation listed here.

One wrinkle keeps confusing profiler output and device counts: a v5p chip carries two cores, each with its own MXUThe systolic matmul array: 128x128 on v5e, 256x256 on v6e. Matmuls only; everything else is the VPU’s job.taught in /l/tpu → and VMEMThe TPU’s software-managed vector scratchpad, about 128 MiB. Blocks must be staged here before compute touches them; what is resident is what your schedule staged.taught in /l/tpu →, and jax.devices() shows each as a device. v5e and v6e run one core per chip, so chip and device coincide. The table's host rows explain the other ratio you meet in practice: a v5e host manages a 4x2 block of chips, a v5p host a 2x2x1 block. None of this changes the math of a kernel; all of it changes what the numbers in your tooling refer to, and reading them wrong by a factor of two is a rite of passage this paragraph exists to skip.