SparseCore

A ranking model's first layer is a table read. The ISCA paper's example is a table with 80,000 rows, one per word in the English language, each row 100 numbers wide; one training example looks up a single row, or a small and dynamic number of rows that are then summed. Nothing in that is a matrix multiply. It is a gather on the way forward and a scatter on the way back, over tables the paper sizes anywhere from O(10 MiB) to O(100 GiB), with all the tables of one model reaching several TiB together.

The TPU chip lesson gave the reason a systolic array punishes small work: fill and drain dominate whenever the operand does not fill the grid. Lookups are not merely small, they are shaped wrong. The paper describes them as small gather or scatter memory accesses with low arithmetic intensity, so what decides their end-to-end speed is memory bandwidth, memory capacity and vector throughput rather than chip FLOPS per second. Hand that to an MXUThe systolic matmul array: 128x128 on v5e, 256x256 on v6e. Matmuls only; everything else is the VPU’s job.taught in /l/tpu → and most of the grid holds still while the memory system does all the work.

The tables are also too large for one chip, so they get cut up three ways the paper names: column sharding along the width, row sharding along the vocabulary, or whole tables placed on different chips. Traffic follows from the cut. Under model parallelism the pattern is a variable-length all-to-all whose ceiling is bisection bandwidth, and the fabric lesson in the ICIThe inter-chip links (4.5e10 bytes per second each way per link on v5e); every collective resolves to hops over these.taught in /l/ici → unit carries that arithmetic. The sparsity is unstructured on top of that, since a few feature values are far hotter than the rest, so compute, memory and network all skew together, and deduplicating the frequent values before the lookup is part of what the hardware has to make cheap.

Before there was a third core, the work had two possible homes, and the paper argues against both in a paragraph each. The TensorCore is the obvious candidate and the wrong one: wide VPUThe vector unit for elementwise work, organized as (8, 128) lanes; the origin of the tiling lattice every layer above obeys.taught in /l/tpu →, large matrix units, tuned end to end for dense operations, asked here to perform small gathers and scatters and to exchange variable-length data with other chips. It would run the work. The paper calls placing embeddings there suboptimal, and names those two access patterns as the reason.

The host CPUs are the other candidate. Push every lookup out to host memory and the CPU DRAM interface becomes the Amdahl bottleneck, amplified on v4 by four chips sharing one CPU host, with tail latency and data-center network bandwidth constraining whatever survives. This is not a thought experiment in the paper. They built the configuration and measured it, and that measurement is where the widely quoted 5x to 7x comes from.

What the codesign chose instead was the supercomputer's own memory. Every chip's HBMThe chip’s main memory: large, far, and the resource memory-bound ops spend. 8.2e11 bytes per second on v5e, 1.6e12 on v6e.taught in /l/tpu → joins one flat, globally addressable space, 128 TiB of it on a v4 pod, reached over a dedicated ICIThe inter-chip links (4.5e10 bytes per second each way per link on v5e); every collective resolves to hops over these.taught in /l/ici → network with gather and scatter support in hardware. The paper's term for the arrangement is a sea of cores. Because the SparseCores are separate cores rather than a mode of an existing one, dense compute, embedding work and ICI traffic proceed at the same time instead of taking turns.

Open one v4 SparseCore and nothing inside resembles a grid of multipliers. The paper calls it a dataflow architecture, meaning data flows out of memory into a set of directly connected specialized units, and a v4 chip carries four of these next to its two TensorCores.

Sixteen compute tiles do the general work. Each tile owns an HBMThe chip’s main memory: large, far, and the resource memory-bound ops spend. 8.2e11 bytes per second on v5e, 1.6e12 on v6e.taught in /l/tpu → channel and keeps many memory accesses outstanding at once, which is the design answering the access pattern directly: thousands of independent small reads, none of them waiting on the answer to the last. Inside a tile sit three units in a row. A Fetch Unit reads activations and parameters out of HBM. A programmable 8-wide SIMD vector unit, the scVPU, does the arithmetic on them, reusing the same ALUs as the TensorCore's VPU. A Flush Unit writes updated parameters back to HBM during the backward pass.

The tiles work out of Spmem, the sparse vector memory: 2.5 MiB per SparseCore, with each tile reading and writing its own slice of it. Multiply that out and the per-chip totals in the paper table fall out exactly, four SparseCores on v4 for 10 MiB of spMEM and two on v3 for 5 MiB. It is a small memory next to the tens of MiB the TensorCore side works from, and it is sized for vectors a hundred numbers wide rather than for the tiles of a matmul.

Five cross-channel units sit beside the tiles and perform the embedding operations no single tile can, because they operate across all sixteen banks of Spmem collectively. They take CISC-like instructions over variable-length inputs, so how long an instruction runs depends on the data handed to it. That is the opposite property from the array next door, where a matmul costs a cycle count fixed by its shape before it begins.

The abstract prices the whole thing in one line, and section 3.5 repeats it with a tilde on each figure: about 5% of the die area and about 5% of the power.

SparseCores, dataflow processors that accelerate models that rely on embeddings by 5x-7x yet use only 5% of die area and power.

The 5x to 7x is a subtraction rather than a headline. The paper takes TPU v4, changes nothing else, moves the embeddings into CPU host memory, and measures what the production model DLRM0 loses at 128 chips: a factor of 5 to 7, bottlenecked on CPU memory bandwidth. Read it as the price of not having the core. An area-and-power claim can only be honest if the comparison names what was removed.

The neighbouring numbers set the scale. On the same model against 576 Skylake sockets, TPU v3 is 9.8x faster; v4 beats v3 by 3.1x and the CPU configuration by 30.1x. Moving from the 2D torus of v3 to the 3D torus of v4 raises bisection bandwidth 2x to 4x at a given chip count and accelerates embeddings by 1.1x to 2.0x, which is the fabric doing embedding work rather than the core. Past about 1024 chips the paper reports SparseCore overheads starting to dominate, at which point bisection bandwidth matters less.

From the pod, the SparseCores look like a single machine. The Hot Chips deck describes non-coherent shared memory spanning the pod and millions of outstanding references reaching any node in it, hidden behind multithreading, with a full v4 pod presenting 8K TensorCores and 16K SparseCores across the optical switches.

SparseCore has been part of TPUs since v2, and the per-chip count has moved twice in the public record. The ISCA paper gives 2 on v3 and 4 on v4. The Cloud TPU architecture page gives 4 on v5p and on TPU7x, and 2 on v6e. The v5e page lists none at all, which is a gap in the documentation and not a documented zero: write unknown, not absent.

The generation labels deserve more care than the counts. The Hot Chips 2023 deck lists a third-generation embeddings coprocessor among the innovations of the fourth-generation system, meaning v4. The 2024 Trillium blog says v6e is equipped with third-generation SparseCore. Two chips, two generations apart, both called third. Nothing published reconciles the two, so carry the source with the label every time and never quote the number on its own.

One piece of hygiene before the principle, because two vendors put sparse in the name of very different things. NVIDIA's structured sparsity is a Tensor Core mode: prune the weights until at least two values in every group of four contiguous ones are zero, and the unit processes only the nonzeros for roughly twice the dense rate. What that does to a datasheet is read out in the machine stage, at /s/machine/inside-the-sm. SparseCore is not a mode of anything. It is a separate programmable processor with its own memory, its own instructions and its own HBMThe chip’s main memory: large, far, and the resource memory-bound ops spend. 8.2e11 bytes per second on v5e, 1.6e12 on v6e.taught in /l/tpu → channels, and the sparsity it serves is the unstructured kind that arrives from a lookup rather than from pruning.

The general move is worth taking away from the TPU entirely. When a workload class is large enough to matter and its access pattern fights the execution model of the main core, the answer that keeps winning is a sibling core beside it, not a wider or more general version of it. Widening the array would have cost area on every workload and still left gather latency where it was.

The condition on that move is the arithmetic in this lesson. DLRMs are about a quarter of Google's ML workload by the paper's own table, the core costs around 5% of area and power, and its absence costs 5x to 7x on that quarter. Change any of the three and the answer changes: a workload at one percent of the fleet does not earn silicon, and a workload that tiles into the array does not need it. The TPU chip lesson put SparseCore on the die in a single sentence and moved on, which was the right depth for a first pass; this is what that sentence was standing on.