The scale-up domain

Two GPUs in the same node are wired together by NVLink, and that wiring buys something a network does not. One GPU can issue a request against memory that physically sits on the other chip, and the request is routed there by GPU physical address, because inside the wiring all the GPUs share one address space. Nothing is packed into a message, addressed to a peer, and unpacked at the far end. The remote memory is memory.

That property is what scale-up names. You make the machine bigger by making one memory system bigger, as far as the fabric reaches, and code written against it reads like code written against a single very large GPU. Scale-out is the other move: more machines, each with its own memory, talking in messages. Nearly every confusing bandwidth figure in an NVIDIA rack becomes readable once you ask which of the two it is describing.

The GPU fabric lesson (/l/ici/gpu-fabric) counted the links and switches inside a node and set the per-GPU numbers against measured collectives. This lesson asks a different question about the same wires: how far the shared address space reaches, what the published figures are actually summing, and what changes in your program at the edge.

Eight GPUs sharing one address space is a manageable thing to build. Two hundred and fifty six of them across many chassis is a different problem, because any GPU could then reach any address on any other with no isolation between tenants, so one job's stray write lands in another job's memory. Hopper's answer was to keep the wires and change the semantics.

The whitepaper states the split plainly. Regular NVLink is the case where "all GPUs share a common address space and requests are routed directly using GPU physical addresses"; NVLink Network "introduces a new Network Address Space supported by new address translation hardware in H100 to isolate all GPUs' address spaces from one another and from the network address space". The wires below are still NVLink. What sits on them is no longer one memory system.

The consequence lands in your program, and NVIDIA names the comparison itself: because those endpoints do not share a common memory address space, connections "are not automatically established across the entire system", and instead, "similar to other networking interfaces such as InfiniBand, the user software should explicitly establish connections between endpoints as needed". So the place where InfiniBand takes over is not first a bandwidth cliff. It is the point where a peer stops being an address and starts being an endpoint you connect to, and NVLink Network has already crossed that line while still running on NVLink cable.

Inside the domain a peer is an address. Outside it, a peer is a connection.

Hopper put 256 GPUs behind that scheme. The Blackwell brief says the fifth generation "can scale up to 576 GPUs". Those are the ceilings on how far one vendor-supported NVLink fabric stretches, not on how many GPUs share a single address space, which is the smaller number set by the node or rack you actually bought.

Three generations of link are three different decisions, and the wire count tells you which. A100's NVLink 3 used four differential pairs in each direction to make one link carrying 25 GB/s each way, and put 12 links on the chip. H100's NVLink 4 got the same 25 GB/s each way out of two pairs, so the rate per wire doubled, and NVIDIA spent the saved wires on more links: 18 instead of 12. Blackwell's NVLink 5 keeps both the two pairs and the 18 links, and doubles the link itself to 50 GB/s in each direction.

Now the totals. NVIDIA writes that H100 "includes 18 fourth-generation NVLink links to provide 900 GB/sec total bandwidth", and 18 links at 25 GB/s each way comes to 450 in one direction, 900 with both summed. Blackwell's brief spells the convention out rather than leaving it to arithmetic: "1.8 TB/sec total bandwidth, 900 GB/sec in each direction". Every NVIDIA total for NVLink is both directions added together.

The scaling book counts the same eighteen links one direction at a time and prints 450 GB/s for an H100, which is where the GPU fabric lesson got its per-GPU figure. Neither source is wrong and neither is being sloppy. Before you compare two NVLink numbers from two authors, divide or multiply by two until they are describing the same directions.

NVIDIA's own ratios confirm they never switch conventions mid-document. Fourth-generation NVLink is "7x the bandwidth of PCIe Gen 5" and fifth-generation is "over 14X", which puts the PCIe Gen5 baseline at about 129 GB/s under both, and that is the x16 figure with both directions counted. Same convention, twice, two generations apart.

The GB200 NVL72 headline is that its NVLink Switch "enables 130TB/s GPU bandwidth in one 72 GPU NVLink domain". Do the multiplication before you do anything else with it. Seventy-two Blackwell GPUs at 1.8 TB/s each is 129.6 TB/s, so the figure is the sum of every GPU's own link capacity, both directions, and it says nothing at all about the switch tier those links plug into.

That makes it the wrong number for the question people usually ask of it. Bisection bandwidth is what crosses a cut, and cutting a 72-GPU domain leaves 36 GPUs on each side, so at most half the endpoints can be pushing across it in a given direction. Under the same both-directions accounting that produced 130, a 36-against-36 cut tops out at 64.8 TB/s, and only if every GPU on both sides sent exclusively across the cut. An all-GPU sum can never be a bisection figure, for any fabric, because half the endpoints are always on the wrong side.

The real ceiling could be lower still, and the public record will not tell you. NVIDIA publishes no NVL72 switch-tier capacity: the per-chip port count and the number of NVSwitch chips in the rack appear in secondary reporting but were not found in an NVIDIA document. Treat 64.8 TB/s as an upper bound you derived, not as a spec.

Compare a figure NVIDIA does construct carefully. For the 256-GPU Hopper NVLink Switch System, the whitepaper says each node "exposes a 2:1 tapered level of all the NVLink bandwidth of the GPUs in the node" and the connected nodes are "capable of delivering 57.6 TBs of all-to-all bandwidth". Work it through: 8 GPUs at 900 GB/s is 7.2 TB/s per node, the taper halves that to 3.6, and 32 nodes give 115.2 TB/s. The published 57.6 is exactly half of that, which is the one-direction reading. NVIDIA never states which convention the 57.6 uses. The arithmetic closing to the digit is the evidence.

So the same family of wires produces figures that look alike and mean four different things, and a fifth kind hides in the SuperPod material: the "9x increase in bisection bandwidth" quoted there is a ratio against the previous generation's InfiniBand system, not an absolute figure about anything.

Turn all of that into one decision you make while sharding. The collective that runs several times per layer, the tensor-parallel all-reduce, wants to stay inside the domain, and the domain size is therefore a hard cap on how wide you can make that axis: eight on an HGX node, seventy-two in an NVL72 rack. The collective that runs once per step, the data-parallel gradient reduction, is the one you can afford to send across the boundary.

The cost of crossing is not only the bandwidth drop. Inside, a peer read is an address the hardware routes; outside, it is an endpoint your software connected to first, with its own setup, its own failure modes, and a different piece of the stack responsible for it. Two collectives with identical shapes are different programs depending on which side of that line they land on.

TPUs draw no line of this kind, which is worth one paragraph before you carry the GPU habit across. ICIThe inter-chip links (4.5e10 bytes per second each way per link on v5e); every collective resolves to hops over these.taught in /l/ici → never offered a shared address space to begin with: every remote access is already an explicit DMAAn asynchronous copy between memories that runs while compute continues; the grid pipeline is DMAs the runtime writes for you.taught in /l/pallas → to a named neighbor, at one hop or at twenty, so there is no semantic edge to fall off. The boundary a TPU programmer manages is the slice edge where the data-center network takes over, and it is a bandwidth cliff rather than a change of model. The hop arithmetic that prices it lives in the ICI unit (/l/ici/tpu-fabric), and the numbers there are the ones to use.

That is the last of the hardware in this stage. You can read a rack spec sheet the way you now read a chip spec sheet, asking of every figure what it sums over and where it stops being true. Stage 1 picks it up at Pallas fundamentals (/s/pallas), where the questions stop being about what the machine is and start being about the schedule you write for it.