Inside the SM, multiplied out

Read the H100 whitepaper's own configuration list and there are three chips on the page, not one. The full GH100 carries 144 SMs across 8 GPCs, 6 HBMThe chip’s main memory: large, far, and the resource memory-bound ops spend. 8.2e11 bytes per second on v5e, 1.6e12 on v6e.taught in /l/tpu → stacks behind 12 512-bit memory controllers, and 60 MB of L2. The SXM5 part in the rack carries 132 SMs, 5 HBM3 stacks behind 10 controllers, 50 MB of L2. The PCIe part carries 114 SMs and HBM2e. One design, three harvests, and the SKU decides which counts your arithmetic is allowed to start from.

Twelve SMs and one memory stack are switched off somewhere between the drawing and the server. Why an 814 mm² die ships with pieces disabled is the yield story, and this lesson needs only the consequence: put 144 into a FLOPS calculation for a part that has 132 and every number after it is high by about 9%.

The tour of one SM, the SIMT model, and the divergence a TPU cannot have are the previous lesson (/s/machine/gpu-chip). What follows treats the same silicon as a parts list with a missing constant in it.

An SM is four processing blocks and a shared floor. Each block holds an L0 instruction cache, one warp32 GPU threads scheduled as one unit; the GPU hides latency by switching among resident warps rather than by pipelining a scratchpad.taught in /l/tpu → scheduler issuing 32 thread/clk, one dispatch unit at the same rate, a register file of 16,384 32-bit words, 32 FP32 lanes, 16 INT32, 16 FP64, 8 load/store units, one SFU, and one fourth-generation Tensor Core. The floor underneath them is 256 KB of combined L1 data cache and shared memory, plus a Tensor Memory Accelerator and four texture units.

Multiply the blocks out and the whitepaper’s comparison table confirms every total in text: 128 FP32 cores per SM, 64 FP64 excluding the Tensor Cores, 64 INT32, 4 Tensor Cores, and 65,536 32-bit registers, which is the 256 KB register file. Shared memory is carved out of the 256 KB block and is "configurable up to 228 KB"; the whitepaper describes the block itself only as "1.33x larger than A100", which is how you get A100’s 192 KB.

Keep the A100 column in view. It is the one that carries a clock, and that is what makes it useful for anything past nostalgia.

The shipping datasheet lists BF16 Tensor Core at 1,979 TFLOPS with an asterisk, and the asterisk resolves at the bottom of the page to "With sparsity". The whitepaper writes the same kind of entry as a pair, 1000/2000, under footnote 2: "Effective TOPS / TFLOPS using the Sparsity feature". Two notations, one convention, and the larger number is never the one you multiply against.

What is being assumed is 2:4 structured sparsity, a Tensor Core mode that the whitepaper says works by "exploit[ing] fine-grained structured sparsity in deep learning networks, doubling the performance of standard Tensor Core operations". Half of every group of four weights is zero, the hardware skips them, and the marketing number doubles. Dense is the starred value halved. FP32, FP64 and FP64 Tensor Core carry no asterisk and no multiplier.

Get this wrong and the error is loud rather than subtle. Feed 1,979 into the clock derivation below and it asks for 3.66 GHz, which no GPU has ever run at. On a narrower question the same mistake would have passed quietly, which is the argument for doing the multiplication at all.

One constant is missing and NVIDIA does not publish it: FLOPs per SM per clock through the Tensor Cores. Pin it on the chip whose clock is printed. The A100 row gives 108 SMs at 1410 MHz and FP16 Tensor 312/624, so dense is 312 TFLOPS, and 312e12 / (108 x 1.41e9) is 2048.9. Call it 2048 and the A100 reproduces itself to a tenth of a TFLOP.

Hopper is then one sentence away. The whitepaper says its Tensor Cores "deliver 2x the MMA (Matrix Multiply-Accumulate) computational rates of the A100 SM on equivalent data types", per SM and per clock, so the constant is 4096 per SM, 1024 per Tensor Core. Across 132 SMs that is 540,672 FLOPs per clock, and the datasheet’s dense BF16 rate of 989.5 TFLOPS divides out to 1.830 GHz.

The previous lesson runs the same equation with the other unknown fixed: it takes the scaling book’s 1.76 GHz as given and solves for the per-Tensor-Core rate, landing near 1024. Pinning the constant instead and solving for the clock is the version you can source end to end, because 2048 comes from a table that prints its own clock. Third-party GPU databases list 1830 MHz boost for H100 SXM5, which agrees. Treat that agreement as a check on the arithmetic, not as the missing datasheet line.

pin the constant where the clock is published
  A100    108 SMs x 2048 FP16 FLOP/SM/clk x 1.41 GHz = 311.9 TFLOPS
  printed                                              312   TFLOPS   [Table 3, p.39]

carry it across one documented sentence
  "2x the MMA ... rates of the A100 SM"             -> 4096 FLOP/SM/clk [p.22]
  132 SMs x 4096                                    =  540,672 FLOP/clk
  datasheet BF16 Tensor  1,979 TFLOPS "* With sparsity"
  dense                                             =  989.5 TFLOPS
  implied clock  989.5e12 / 540,672                 =  1.830 GHz

Now run the same move on the rates that do not go through a Tensor Core. FP32 is 67 TFLOPS over 132 x 128 x 2 = 33,792 FLOPs per clock, which asks for 1.983 GHz. FP64 is 34 TFLOPS over 132 x 64 x 2 = 16,896, which asks for 2.012 GHz. Neither is 1.830.

Some of that spread is rounding, and the reconciliation is worth doing rather than waving at. A printed 67 covers any clock from 1.968 to 1.998 GHz, and a printed 34 covers 1.983 to 2.042, so FP32 and FP64 are both consistent with a single clock in a narrow band just under 2 GHz. The 1.830 GHz the Tensor rows demand is nowhere near that band. Two families of numbers, two clocks, one page.

One row on the same page cannot be run through this arithmetic at all. FP64 Tensor Core prints 67 TFLOPS, which is the FP32 number over again, and the whitepaper repeats that coincidence in all three of its columns: 60 for both on SXM5, 48 for both on PCIe, 19.5 for both on A100. Set it against the plain FP64 row instead and the ratio is exactly two in the whitepaper’s H100 columns, 60 over 30 and 48 over 24, and only roughly two on the shipping datasheet, where doubling 34 gives 68 against a printed 67. Rounding accounts for that gap: a single underlying rate between 33.5 and 33.75 TFLOPS prints as 34 and doubles into a 67, the same way A100’s 9.7 is a rounded 9.746. But NVIDIA never prints an FP64 rate per clock through the Tensor Cores, so that row’s divisor can only be inferred from the doubling rather than counted off a table, and the table below records it as unavailable rather than filling it in.

The whitepaper’s preliminary column does the same thing with different values: its FP32 60 and its FP64 30 TFLOPS both imply 1.776 GHz to four figures, while its dense FP16 Tensor rate of 1000 implies 1.850. The PCIe column repeats the pattern, 48 and 24 both giving 1.645 GHz against 1.713 from its Tensor row. The gap survives the change of document and of SKU, and it even flips direction between the preliminary table and the shipping one.

Which leaves the clock itself. The whitepaper prints "Not Finalized" where the boost clock goes, for both H100 parts, and the shipping datasheet page lists no clock at all. So there is no vendor number to reconcile these two families against, and the honest end of the chain is to say that. A derivation that fails to close and reports why is worth more than one that quietly picks the clock that made it work.

A peak-FLOPS table is not necessarily quoted at one clock.

Bandwidth divides as cleanly as FLOPS multiply. Five HBM3 stacks carry the datasheet’s 3.35 TB/s, so each stack delivers 670 GB/s, and 80 GB over five stacks is 16 GB a stack. The 5120-bit memory interface across 10 512-bit controllers puts 1024 bits on each stack, which turns 670 GB/s into 5.23 Gb/s per pin.

That last figure is the one with an external yardstick. JEDEC’s HBM3 update tops out at 6.4 Gb/s per pin and 819 GB/s per device, so an H100 runs its memory at about 82% of the standard’s ceiling. This division closes because both of its inputs are printed, which is the difference between it and the clock.

Check which document you are holding here too. The whitepaper’s preliminary table says 3000 GB/sec for the same part that the shipping datasheet sells at 3.35 TB/s, and a rooflineThe floor model: latency is at least the larger of FLOPs over peak compute and bytes over bandwidth. Predict first, measure second.taught in /l/tpu → built on the older figure is off by 12% before you start.

The habit underneath all of this is short. Write the multiplication before you look up the answer, then reconcile: wrong SKU, sparsity convention, preliminary document, unpublished constant, in that order. When the gap survives every one of those, you have found something about the vendor’s numbers rather than about your own arithmetic.