The GPU fabric

The unit above a single GPU is the node, which is 8 GPUs for everything through B200 and up to 72 for GB200. Inside it they are wired all-to-all through NVSwitches, so any GPU reaches any other in one hop at full bandwidth. The topology has moved around across generations; on an H100 node there are 4 NVSwitches and the GPUs attach to them in a 5 + 4 + 4 + 5 link pattern.

Each NVLink link on Hopper carries 25 GB/s full-duplex, meaning 25 each way with the two directions independent, so 50 total but never more than 25 in one direction. Each GPU has 18 of those links, which gives 18 * 25 = 450 GB/s of full-duplex bandwidth from every GPU into the network.

The switches are not the constraint. An NVSwitch carries up to 64 NVLink ports, so four of them come to 64 * 25e9 * 4 = 6.4 TB/s. The GPUs get there first: at 450 GB/s each, 8 of them cap the node at 3.6 TB/s. Split that node in half any way you like and the same 3.6 TB/s crosses the cut, which is the node's bisection bandwidth.

What changed generation to generation is worth reading as three separate decisions. Ampere had 12 links per GPU at 25 GB/s, for 300. Hopper left the link speed alone and added ports, 18 of them, for 450. Blackwell kept 18 ports and doubled the link to 50 GB/s, for 900. Meanwhile the switch count inside a node fell from 6 to 4 to 2, because each switch generation carries more.

The last row is really two machines. A B200 node is still 8 GPUs with 2 switches. A GB200 NVL72 puts 72 GPUs into one NVLink domain behind 18 switches, at the same 900 GB/s per GPU.

Past the NVLink domain the fabric changes technology. Every GPU in a reference H100 node has its own 400 Gbps CX7 NIC into an InfiniBand network, so the node egresses 8 * 400 / 8 = 400 GB/s into what NVIDIA calls the scale-out network, as against the NVLink domain below it. Thirty-two of those nodes, 256 GPUs, sit under a single set of 8 leaf InfiniBand switches and are called a Scalable Unit. The cabling is NDR, 50 GB/s full-duplex per port, through 64-port switches.

Four Scalable Units under 16 top-level spine switches make the reference SuperPod: 1024 GPUs, 512 node-level NVSwitches, 32 leaf IB switches, 16 spine IB switches, 560 switches in total. All leaf switches connect to all spine switches, and the cable and port counts are chosen so that bisecting the pod anywhere still gives every node its full 400 GB/s. That is what fat tree means here, and it is why AllReduce bandwidth in the scale-out network stays roughly constant as you add nodes.

So the shape of the whole machine is two numbers. 450 GB/s per GPU while you stay inside a node, 400 GB/s per node once you leave it. Every sharding decision on a GPU cluster is an argument about which side of that boundary your biggest collective lands on.

GB200 NVL72 racks put 72 GPUs in one NVLink domain, each with the full 900 GB/s to every other. Counting the egress from that node into the leaf level gives 4 * 18 * 400 / 8 = 3.6 TB/s, which is 9x an H100 node's 400 GB/s, exactly matching the 9x in GPU count.

Divide it out and the per-GPU story is flat: 400 / 8 is 50 GB/s of egress per GPU on an H100 node, and 3600 / 72 is also 50 GB/s on an NVL72. The rate at which a GPU can talk to the world outside its domain did not change. The domain got nine times bigger, so nine times as much of your traffic never has to leave it, and the rooflinesThe floor model: latency is at least the larger of FLOPs over peak compute and bytes over bandwidth. Predict first, measure second.taught in /l/tpu → you compute for a sharding move accordingly.

An 8xH100 node claims 450 GB/s per GPU. Measured AllReduce with SHARP disabled gets close to 370 GB/s, and only at array sizes around 10 GB per device. The estimate is not wrong. It just takes an enormous message to get there.

Smaller messages are much worse, and they are the ones you actually send. A LLaMA-3 70B MLP with 8-way model sharding is bf16[8192, 3584], about 58 MB, and an AllReduce over it achieves only around 150 GB/s against the nominal 450. TPUs reach peak bandwidth at considerably smaller messages, which is one of the sharper practical differences between the two fabrics.

Since Hopper, NVIDIA switches support in-network reduction (SHARP): the switch performs the reduction itself and multicasts the result, so each GPU does not have to egress its data twice. In theory that close to halves an AllReduce. In measurements it gives about 30%.

TPUs are wired as 2D or 3D tori, where each chip connects only to its neighbours. A message between two distant TPUs passes through every chip in between, and that forces uniform communication patterns across the mesh. In exchange, the number of links per chip is constant, so a pod grows without any per-chip bandwidth loss, up to 8960 chips on a v5p. A v5p link carries about 90 GB/s, or 540 GB/s summed across all axes of the torus.

The GPU tree buys the opposite property. Any GPU can send to any other in an arbitrary pattern, and you can extend the fabric to any size by adding switches or another layer of indirection, which the 4096-GPU configuration does with 128 spine and 64 core switches. You pay for that in latency and in switches, at every level, forever. The generational split is in the protocols themselves: NVLink behaves like a souped-up PCIe with low latency and little protocol overhead but no design for scale or fault tolerance, while InfiniBand is closer to Ethernet, built for large lossy networks.

A torus costs the same to wire at any size. A tree costs another layer of switches.