The bundle, and the floor below it

A TPUv2 core fetches one 322-bit instruction at a time, and that instruction speaks to several units at once. Two of its slots carry scalar work. Four carry vector work, two of those reserved for vector load and store. Two more carry matrix work, one push and one pop. Then a miscellaneous slot, and six immediates. Read the layout as a budget rather than a format: those slots are what a compiler has to fill, per cycle, for the machine to be doing anything at all.

A machine whose one instruction word issues to several units at once, with the parallelism settled before the program runs, is a VLIW machine, for very long instruction word. The design paper says why the TPU is one. A VLIW architecture was the simplest way for the hardware to express instruction level parallelism, and it let the team use compiler techniques that already existed.

The scalar unit is the front of this machine. It pulls whole bundles from a local instruction memory, runs the scalar slots itself, and forwards the decoded rest onward to the vector and matrix units, where execution happens later and decoupled from the scalar side. The chip lesson at /l/tpu/tpu-chip introduced that unit from the data side, as the thing that decides which block comes next and fetches it. This is the same unit from the instruction side, and it is one job, not two.

One vector slot is also not one number. The vector unit is 128 lanes wide and each lane carries an eight-deep sublane dimension, so a vector slot operates on eight sets of 128-wide values in a cycle. The (8, 128) shape the course keeps meeting as a tiling rule is that geometry, read from the instruction that drives it.

The scalar unit is where computation originates. It fetches complete VLIW
bundles from a local instruction memory, executes the scalar operations slots
locally, and then forwards decoded instructions on to the vector and matrix
units where execution happens later, decoupled from scalar execution. The
VLIW bundle is 322 bits and is composed of two scalar slots, four vector
slots (two used for vector load/store), two matrix slots (a push and a pop),
one miscellaneous slot (a simple example would be a delay instruction), and
six immediates.

Those bundles have to reach the core somehow, and the answer is the one this chip gives for everything else. There is no instruction cache backed by HBM. There is a local instruction memory that DMAAn asynchronous copy between memories that runs while compute continues; the grid pipeline is DMAs the runtime writes for you.taught in /l/pallas → writes into, and the paper is candid about the decision: a cache would have been nice, and a DMA target for software-managed instruction overlays was easier to build. Code gets staged the way data gets staged, by a transfer that something scheduled.

That symmetry closes a gap in the picture the unit has been building. VMEMThe TPU’s software-managed vector scratchpad, about 128 MiB. Blocks must be staged here before compute touches them; what is resident is what your schedule staged.taught in /l/tpu → holds what software put there. SMEMScalar memory: lengths, flags, and indices live here, feeding control flow without ever entering the vector datapath.taught in /l/tpu → holds what software put there. So does the memory the program itself lives in. Nothing on this chip fetches ahead on your behalf, and the part that fetches your instructions is not an exception to its own design.

Put a rate on the stream and the tradeoff gets concrete. One 322-bit bundle per cycle at v3’s 940 MHz is 3.78e10 bytes per second of instruction fetch, about 4 percent of the 9.0e11 bytes per second v3’s HBMThe chip’s main memory: large, far, and the resource memory-bound ops spend. 8.2e11 bytes per second on v5e, 1.6e12 on v6e.taught in /l/tpu → delivers. Two assumptions sit under that arithmetic and neither is small. It assumes a bundle issues every cycle, which is the ceiling and not the average. And 322 bits is the published width for v2 and v3 alone, which the next two sections are about.

The shorthand for a VLIW machine is that it has no interlocks, and the TPU papers do not say that. What the design paper describes is hold conditions on instructions for execution interlock, plus synchronization flags for interlocking against software-managed DMA. Stalls exist on this chip. A wait is a real event with real cycles in it.

What is missing is the other half of a modern CPU. Nothing here picks a different instruction to run while one waits. A hold stalls the pipeline and does not fill it, because there is no pool of ready work to draw from and no unit whose job is to choose. The latency lesson at /l/tpu/two-ways-to-hide-latency drew that contrast against a GPU’s warp32 GPU threads scheduled as one unit; the GPU hides latency by switching among resident warps rather than by pipelining a scratchpad.taught in /l/tpu → scheduler; this is the same fact one level lower, in the instruction word itself.

One piece of hardware exists to buy the compiler slack, and it shows how tight the schedule is otherwise. Matrix results land in a Result FIFO, which the paper says lets them avoid strict execution schedule constraints for the long-latency matrix operations and shorten register lifetimes, simplifying the compiler. That is why push and pop are two separate slots. The two ends of a matrix operation get scheduled independently, and the FIFO absorbs the distance between them.

Static scheduling puts the performance in the compiler, and the TPU team has published both the risk and the return. Compiler problems likely sank the Itanium’s VLIW architecture, the TPUv4i paper says, and it names that before observing that many domain-specific accelerators rely on VLIW anyway. Then it gives the return. Over the twenty months from MLPerf Training 0.5 to 0.7, CUDA compilation improved the GPU by 1.8x and XLAThe compiler: brilliant at fusing along dataflow edges, structurally unable to change your algorithm. That gap is why kernels exist.taught in /l/xla → raised the TPU by 2.2x, against the 1 to 2 percent a year that C compilers move general-purpose code.

Twenty months of compiler work moved the same silicon 2.2x.

You have already scrolled past this compiler estimating a schedule. Every TPU fusionSeveral ops compiled into one kernel so intermediates stay in fast memory instead of round-tripping through HBM. XLA’s central optimization, with an exact limit.taught in /l/xla → in this repo’s capture carries a window_config inside its backend_config, and two of its fields are the compiler talking about cycles and bundles for that window. The cycle number is populated here and the bundle count is not, which is worth taking as the shape of the boundary rather than as a shortage. The plan has a place to say how many bundles it expects. What the dump hands you is the cycle estimate alone.

%fusion = bf16[1024,1024]{1,0:T(8,128)(2,1)S(1)} fusion(%copy-done, %k.1), kind=kOutput, ...
  backend_config={... "window_config":{"buffering_level":"2",
    "cost_model_type":"COST_MODEL_TYPE_CLASSIC",
    "estimated_bundle_count":"0","estimated_cycles":"3528", ...}}

A 322-bit word with a fixed slot layout sounds like an instruction set you could target, and Google decided it would not be one. TPUv2 and TPUv3 share the bundle length. TPUv4i broke with it on purpose, and the paper spends a section saying why rather than leaving it to be discovered.

The argument starts from what VLIW was for. Putting the parallelism in the instruction word is what lets a recompile use a wider machine, and binary compatibility freezes exactly that. The paper adds an admission worth reading twice: many engineers built Itanium compilers, some of them now on the XLAThe compiler: brilliant at fusing along dataflow edges, structurally unable to change your algorithm. That gap is why kernels exist.taught in /l/xla → team, where they learned the drawbacks of binary compatibility for a VLIW compiler and its hardware.

So the stable thing is a split inside the compiler. XLAThe compiler: brilliant at fusing along dataflow edges, structurally unable to change your algorithm. That gap is why kernels exist.taught in /l/xla → divides the compiling task into High-Level Operations that are machine independent and Low-Level Operations that are machine dependent, and the second name is LLO. Optimizations at the HLO level apply to every platform. A new TPU that confines its compiler changes to the LLO half, a wider VLIW for instance, keeps compiler compatibility while breaking every binary. The contract with the outside world is HLO, and the bundle sits on the far side of it.

The GPU stack draws its line one level lower, which is the comparison to hold. PTX is a published virtual instruction set with a documented stability story, and SASS below it is per-generation and undocumented, so a GPU author gets one readable layer past the compiler’s portable output. There is no PTX on the TPU side. The last level anyone outside can print is the one the MosaicThe MLIR dialect Pallas lowers to, and the last layer of the TPU stack you can read; only LLO below it is closed.taught in /l/mosaic → lesson at /l/mosaic/to-machine-code already draws the boundary at, and below it the encoding is per-generation because being per-generation is the design.

322 bits is published for v2 and v3, and for nothing since.

Start with what you can see, because it is more than the usual telling admits. XProf has a public LLO surface, documented in the open-source repo rather than inferred. The current instructions set two XLAThe compiler: brilliant at fusing along dataflow edges, structurally unable to change your algorithm. That gap is why kernels exist.taught in /l/xla → flags together, xla_xprof_enable_custom_call_tracing and xla_xprof_register_llo_debug_info, and what the doc shows those producing is LLO traces in the trace viewer, at the level of ops and of individual instructions. A Pallas kernel arrives as a custom call, so the surface is pointed at exactly the kernels this course writes.

An older recipe sits further down the same file under a heading that marks it old, and it is the only place the utilization view is documented. There, xla_enable_custom_call_region_trace pairs with the same debug-info flag, and the doc says an LLO utilization line then appears in the trace viewer for each TPU core or device executing the custom call. Nothing in the file attributes that line to the newer tracing flag. If you want the utilization view, the old flag is the one to set.

Back on the current recipe, the parameters that tune trace insertion are where bundles stop being a paper fact. Trace insertion is configured in units of bundles. trace_best_effort_frequency sets the target interval for opportunistic traces packed into existing bundles, and trace_guaranteed_frequency sets the maximum number of bundles allowed between two traces. The doc is exact about the difference: the best-effort pass will not create new bundles, and when the guarantee cannot be met by packing, the compiler creates a new bundle and places a trace there by itself. A profiler that can slip its own instruction into a bundle’s spare slot is describing a real slot budget.

The same document publishes the per-instruction cycle costs the compiler models, which is the nearest thing to an LLO timing manual in public. It names two units this course has not: XLU for transposes, and EUP, which the doc glosses as vector math functions like tanh, log and exp. Read the table as the model and not the measurement, in the doc’s own terms. The compiler calculates an intrinsic cycle cost per LLO instruction from the target generation and the execution unit resolving it, and XProf interpolates between its trace points with those numbers.

The 2026 XProf work goes a step further and says so in the same vocabulary. LLO bundle data is now exposed for Pallas authors, described there as the specific machine instructions issued to the TPU’s functional units during every clock cycle, with traces inserted by dynamic instrumentation so the times are exact rather than static compiler estimates. The worked example is idle cycles inside the MXUThe systolic matmul array: 128x128 on v5e, 256x256 on v6e. Matmuls only; everything else is the VPU’s job.taught in /l/tpu → pipeline, latency between a vmatmul and a vpop. Two instruction names in public, doing precisely what the push and pop matrix slots of a 2021 paper describe.

Now the other half, stated as plainly as the XLAThe compiler: brilliant at fusing along dataflow edges, structurally unable to change your algorithm. That gap is why kernels exist.taught in /l/xla → course states its own limits at /xla/pathways. The instruction set is not published. The encoding is not published past v3’s width. The scheduler that fills the slots ships inside libtpu and no flag prints it. Even the name is unsettled: the scaling book and the XProf docs expand LLO as low-level optimizer, while the TPUv4i paper and Google’s own 2026 announcement expand it as Low-Level Operations. Four public sources, two expansions, no correction anywhere. Treat everything here about the bundle’s internals as coming from papers about v2, v3 and v4i, and everything about the tooling as documentation for chips those papers never described.