Introduction

Full disclosure before we start: I’ve never run a program on a TPU. For most of the last decade that was the normal condition of anyone who wrote about accelerators from outside Google.

The chips lived in Google’s data centers, the papers described them a generation late, and the cloud instances were something you read about more often than something you rented.

2026 is the year that stopped being true. Anthropic is bringing up to a million TPUs online, and Broadcom has booked $21 billion of Ironwood racks for it, sold as complete systems rather than chips.

Meta signed a multi-year TPU rental in February and is negotiating to buy them for its own buildings next year. MediaTek’s TPU business is limited by how much wafer capacity it can get, not by demand. Last week Google even put four of them in orbit on a prototype satellite, running Gemma for fifteen minutes at a time. The TPU has real tenants now.

The most interesting part of the TPU, though, is not a chip. It is a 683 MB file. The runtime JAX uses to talk to TPUs ships on PyPI as a wheel called libtpu, and it contains the entire TPU compiler. That compiler will happily compile for a TPU that isn’t there.

You describe a topology, say four Ironwood chips in a 2x2x1 slice, and it hands back an executable, a cost estimate, a memory report and, if you ask with the right options, the VLIW bundles it would have sent to the chip.

The feature exists so that people can check whether a model fits on a slice before they pay for one. We used it the way we used ptxas in our pieces on NVIDIA’s compiler and on Triton: as a microscope pointed at a vendor that has never published an instruction set.

We compiled the same programs for v4, v5e, v5p, Trillium (v6e) and Ironwood (TPU7x) on a container with no accelerator at all, and read back everything the compiler was willing to say. Then we checked it against the best public description of the hardware, a five-generation retrospective by Google’s own architects (Jouppi, Lakshmanamurthy, Young and Patterson) posted in June and due in IEEE Micro.

Where the two overlap they mostly agree. The three places they don’t are Ironwood’s HBM bandwidth, the count of its matrix units, and the compute peaks of v5p and Ironwood.

A word on what kind of evidence this is. When the compiler tells us which instruction it emitted, where an array lives or which core runs a collective, that is a fact about the program it produced, and we report it as a measurement. When it tells us how many cycles something will take, that is the compiler’s model of the chip, and we label it that way every time.

The model matters on its own terms, because it is what the compiler optimizes against, but it is not a stopwatch. The public libtpu we used, version 0.0.48, does not know TPU 8t or TPU 8i. For those we have Google’s April deep dive and arithmetic.

Subscribe now

The very short version

• The compiler is on PyPI. Google’s TPU compiler ships inside libtpu, a 683 MB library, and compiles for TPUs that aren’t attached. We used it on a CPU-only container for v4, v5e, v5p, Trillium (v6e) and Ironwood (TPU7x).

• The critical batch. Trillium’s is 560, the highest of any shipping accelerator we have measured. Ironwood’s is 313 on Google’s datasheet and 270 by the compiler’s own constants, in the range of NVIDIA’s H100 (295) and B200 (281).

• The accumulator. In the final instruction stream for the same 1024³ bf16 matmul, Ironwood pops 1,024 results out of its matrix units and Trillium 4,096. Ironwood does no vector adds where Trillium does 3,072, and issues 80% fewer spill loads. Ironwood’s matrix instructions address a result buffer, mrb[0] to mrb[255]; Trillium’s results come out of an unaddressed queue.

• Precision. By the compiler’s cycle estimates, Ironwood runs an FP8 matmul in 0.48 times the cycles of bf16 and an int8 matmul in 1.13 times. On v5e, v5p and Trillium, int8 takes 0.47 to 0.62 times and FP8 e4m3 gains little. No chip the public compiler knows runs FP4 faster than its eight-bit path.

• Batch one. On v4 through Trillium, the compiler rewrites a batch-1 matmul into a vector multiply and reduction that its own cycle model prices at 2.75 to 3.50 times the matrix-unit version. One option turns the rewrite off. Ironwood doesn’t apply it.

• SparseCores. On Ironwood, every collective we compiled across four or more chips ran on the SparseCores, including the ragged all-to-all that MoE dispatch uses. The one exception was the dense all-to-all. No other generation offloaded anything. TPU 8i replaces those SparseCores with an engine built only for collectives.

• Storage. Arrays live in 4 KiB tiles, 128 elements wide. A single row of FP4 stores eight times its logical size. A KV cache with 64-wide heads costs twice its size in the row-major layout an attention kernel wants, and exactly its size in the layout the compiler picks when nobody asks.

• Prices. On Google Cloud’s on-demand rate card, the six accelerators we priced cost $1.40 to $1.65 per TB/s of HBM bandwidth per hour, TPU or GPU. Commitments follow a fixed formula: 70% of on-demand for one year, 45% for three. The estimated anchor rate for Anthropic is 13%.

• Money. Broadcom expects Anthropic to go from 1 GW of Ironwood in 2026 to 5 GW of TPU 8i in 2027 and 10 GW in 2028, and guides its AI revenue from $58 billion this fiscal year to $230 billion in fiscal 2028. Alphabet books multi-gigawatt TPU system sales in a $514 billion Cloud backlog and is raising up to $70 billion of equity for its build-out.

Subscribe now

Compiling for TPUs we don’t own

The whole setup is one line, pip install "jax[tpu]", which in October 2026 brings jax and jaxlib 0.11.2 and libtpu 0.0.48. JAX can describe a TPU topology that isn’t attached through jax.experimental.topologies, and any program lowered against that description is compiled by the real TPU compiler inside libtpu. Each compile took from under a second to about eight seconds on a single CPU core.

The topology names are stricter than the marketing names. v4:2x2x1, v5e:2x2, v5p:2x2x1, v6e:2x2 and tpu7x:2x2x1 are accepted. v7x:2x2x1 is rejected with “Invalid TPU external name: TPU v7x”, and tpu8t, tpu8i and every other spelling of TPU 8 we tried are rejected the same way. The largest slice we asked for, tpu7x:16x16x16, came back as 8,192 devices.

import numpy as np, jax, jax.numpy as jnp
from jax.experimental import topologies
from jax.sharding import Mesh, NamedSharding, PartitionSpec as P

topo = topologies.get_topology_desc(platform="tpu", topology_name="tpu7x:2x2x1")
mesh = Mesh(np.array(topo.devices[:1]), ("x",))
spec = lambda s: jax.ShapeDtypeStruct(s, jnp.bfloat16, sharding=NamedSharding(mesh, P()))

c = jax.jit(lambda x, w: x @ w).lower(spec((8, 8192)), spec((8192, 28672))).compile()
c.cost_analysis()["optimal_seconds"]   # the compiler's roofline estimate
c.memory_analysis()                    # argument, output, temp and code bytes
c.as_text()                            # optimized HLO, one backend_config per fusion

Five kinds of output came back, and the rest of this piece is built from them. The cost analysis gives FLOPs, bytes accessed and a time called optimal_seconds. The memory analysis gives argument, output and temporary bytes and the size of the generated code.

The optimized HLO attaches a JSON backend_config to every fusion, with the tiling windows the compiler chose, an estimated cycle count, the name of the emitter that generates the code and the on-chip memory it reserves. A dump directory, enabled with XLA_FLAGS, adds 18 entries per module, including one that lists every compiler option with its value.

And two options passed through LIBTPU_INIT_ARGS, --xla_jf_dump_to and --xla_jf_dump_llo_text=true, make the compiler write every stage of its low-level backend to disk, down to the final bundles, together with a report of how many instructions of each kind a bundle can hold.

This is what one fusion looks like in the optimized HLO for a 4096³ bf16 matmul on Ironwood, trimmed to the interesting fields:

ROOT %fusion = bf16[4096,4096]{1,0:T(8,128)(2,1)} fusion(%x.1, %y.1), kind=kOutput,
  backend_config={"window_config":{"kernel_window_bounds":["512","8"],
    "output_window_bounds":["64","8"],"input_window_bounds":["64","32"],
    "estimated_cycles":"324772","iteration_bounds":["4","8","1"],
    "cost_model_type":"COST_MODEL_TYPE_CLASSIC","ml_estimated_microseconds":0,
    "buffering_level":"2"},
  "scoped_memory_configs":[{"memory_space":"1","size":"33554432"}],
  "used_scoped_memory_configs":[{"memory_space":"1","size":"31297536"}],
  "convolution_algorithm_config":{"emitter":"EmitAllBatchInSublanes"}}

The compiler split the output into 32 windows of 512 by 1,024, kept the whole K dimension of each operand window on chip, double-buffered it, reserved 32 MiB of on-chip memory and used 29.8 MiB of it, and predicted 324,772 cycles. Every number in that line is a decision someone at Google tuned, and none of it appears in the documentation.

Two fields in that config deserve a second look. cost_model_type says COST_MODEL_TYPE_CLASSIC, and next to it sits ml_estimated_microseconds, set to zero, in every matmul fusion we inspected on Trillium and Ironwood.

Google published a learned performance model for TPU programs at MLSys in 2021, and showed that it beat the production analytical model on exactly the decision this config records, tile-size selection, as well as on fusion.

TpuGraphs, a dataset for training such models, followed in 2023. In this release the slot for a learned estimate is present in every matmul fusion, and the analytical model fills in every number we report.

Share

The 683 MB file

libtpu.so in the 0.0.48 wheel is 683,032,200 bytes, with a second 37,636,776-byte library, sdk.so, beside it.

For scale, ptxas from CUDA 13.3 is 48 MB, tileiras, the closed back end of NVIDIA’s Tile IR, is 95 MB, and Triton’s libtriton.so is 180 MB. libtpu is fourteen times ptxas.

The size stops being surprising once you look at what is inside. ptxas is one stage of a pipeline. libtpu is the pipeline plus the runtime. Among its 4,237,056 printable strings, the token sparse_core appears 51,866 times, memory_space_assignment 3,100 times and mosaic, the Pallas back end, 828 times.

Source paths survive as strings: the TPU back end lives under platforms/xla/service/jellyfish, its debugger under platforms/deepsea/jellyfish/xdb.

The chips are numbered inside the binary. The protocol-buffer descriptor that defines the chip enum is embedded in the file, and decoding it gives TPU_VERSION_INVALID = 0, JELLYFISH = 1, DRAGONFISH = 2, PUFFERFISH = 3, VIPERFISH = 4, GHOSTLITE = 5 and 6acc60406 = 6, one value per family since TPU v2.

Ask the compiler which chip it is targeting and the target description it dumps answers: PUFFERFISH for a v4 compile, VIPERFISH for both v5e and v5p, GHOSTLITE for Trillium, and TPU_VERSION_6acc60406 for Ironwood. Every older family kept its codename in the shipped binary. The newest one ships as a string that looks like a hash.

A second enum, for core types, holds a small piece of history. It has TPU_CORE_TYPE_TENSOR_CORE = 1, TPU_CORE_TYPE_SPARSE_CORE = 3, and value 2 under two names: TPU_CORE_TYPE_BARNA_CORE and TPU_CORE_TYPE_SPARSE_CORE_V0. Google’s retrospective says the SparseCore was already on the TPU v2 die but was only disclosed in the TPU v4 paper. The binary still carries the first SparseCore’s other name, 7,570 times.

The option surface is just as large. The dump of the compilation environment lists 1,184 top-level options. 861 begin with xla_tpu_, 106 with xla_jf_, and 33 with xla_sc_ for the SparseCore.

Some read like a research agenda: the scheduler options include a biased random-key genetic algorithm with a generation limit that defaults to 1,200. Most will never matter to anyone outside Google. A handful decide things this piece is about, and we name them where they come up.

Subscribe now

What the TPU compiler thinks each chip is

XLA keeps a roofline for every program it compiles. Each instruction gets a time equal to the larger of its FLOPs divided by a peak rate and its bytes divided by a bandwidth, and the sum is reported as optimal_seconds.

The rates are per-target constants, which means they can be read back. Compile a matmul large enough to be compute-bound and divide its FLOPs by its optimal_seconds. Compile an elementwise add large enough to be bandwidth-bound and do the same with its bytes.

We swept the matmul from 1,024 to 16,384 and the add from 64 MiB to 1 GiB, and both constants plateau exactly.

Two critical things in that table need explaining before anything else.

The first is the devices. In these topologies a v4 chip and an Ironwood chip each appear as two devices, with core_on_chip 0 and 1 at the same coordinates, while v5p appears as one device per chip because its two TensorCores are fused into one logical core called megacore.

Every training TPU since v2 has had two TensorCores that share only HBM, according to Google’s retrospective; megacore is a compiler illusion that started with v4. Ironwood drops it. Google’s TPU7x documentation says frameworks see each Ironwood chip as two devices, one per chiplet, a programming model closer to TPU v3 than to v4 or v5p.

Each Ironwood device has 94.74 GiB of HBM available, half of the chip’s 192 GiB minus a reserve, and 64 MiB of VMEM. Code written for v5p that assumed one device per chip sees twice as many on Ironwood, each with half the memory.

The second is the ratios between the compiler’s constants and Google’s published figures.

On Trillium the compiler’s constants are the datasheet: 918 TFLOP/s and 1,638 GB/s. On Ironwood its bandwidth is 3,686 GB/s per core, 7,372 per chip, within 0.11% of the 7,380 GB/s in Google’s documentation.

Google’s own sources don’t agree with each other here: the architects’ paper lists 7,300. Its compute is 996 TFLOP/s per core, 1,992 per chip, or 86.3% of the 2,307 Google lists. v5p carries the same kind of discount on both axes: 394 of 459 TFLOP/s and 2,350 of 2,765 GB/s, 85.8% and 85.0%. v5e and v4 match on compute and are discounted on bandwidth, to 85.9% and 81.9%.

One more cross-check falls out of the memory report: v5p’s 95.73 GiB of free HBM per device fits the 96 GiB in the retrospective better than the 95 GiB in the documentation table.

The binary doesn’t say whether the compute discounts are clock assumptions or margins, but it gives a hint. The low-level backend reports how many instructions of each kind fit in one bundle, and the MXU count changes between generations: four MXU slots per bundle on v4, v5e and v5p, two on Trillium and Ironwood.

Divide the compiler’s cycle estimates for large matmuls by the number of MXU slots and you get the work one slot does per cycle. On the four-slot chips it is 15,350 to 15,860 bf16 multiply-adds, close to the 16,384 of one 128 by 128 systolic array.

That matches Google’s counts: four MXUs per TensorCore on v5e, eight 128 by 128 MXUs per chip on v4 and v5p. On Trillium a slot reaches 113,900 multiply-adds per cycle and on Ironwood 108,800, which is impossible for one 256 by 256 array (65,536) and consistent with two arrays per slot, or one array retiring two rows per cycle.

Combine slot rates with the published peaks and you get clocks the datasheets don’t print: 1.05 GHz for v4, which matches the 1,050 MHz in Google’s TPU v4 paper, 1.50 GHz for v5e, 1.75 GHz for v5p and Trillium, and 2.20 GHz for Ironwood. The retrospective describes Ironwood’s matrix units as four 256 by 256 arrays for bf16.

Counted per chip, reaching 2,307 TFLOP/s with four such arrays would need a 4.4 GHz clock; counted per TensorCore, it needs 2.2 GHz, which is what the compiler’s slot rates imply.

We read the paper’s four arrays as per TensorCore. Run the same arithmetic on the compiler’s constants instead of the datasheet and v5p drops to 1.50 GHz and Ironwood to 1.90 GHz.

One consistent reading is that the compiler models those two chips at a lower clock than their peak figures assume. Another is that it simply derates them. Either way, the compiler’s own roofline for Ironwood uses 996 TFLOP/s per core, not 1,153.

Subscribe now

The critical decode batch, v4 to TPU 8i

In earlier pieces we kept coming back to one number: the decode batch at which a chip stops waiting for its weights. A decode step has to stream every weight from HBM at least once.

At small batch that stream is the whole cost, and adding sequences is nearly free until the arithmetic catches up. The crossover is B* = P·b / (2·B_hbm), where P is the FLOP rate at the precision you use, b the bytes per weight and B_hbm the bandwidth.

Because P usually doubles whenever b halves, B* barely moves with precision. On an H100 it is 295 whether you run bf16 or FP8, on a B200 about 281.

Between TPU generations it moves a lot.

Trillium’s critical batch is 560, by the datasheet and by the compiler, which agree exactly. That is the highest of any shipping accelerator in this piece, NVIDIA’s included.

Trillium raised compute 4.7 times over v5e, and the slot table shows how: four times the multiply-adds per cycle, from half as many slots each eight times larger, at 1.75 GHz instead of 1.50. Bandwidth only grew 1.9 times, so Trillium needs about twice as many concurrent sequences per chip as an H100 before its FLOPs start to pay.

That’s an odd profile for a chip Google’s architects describe, in a footnote, as focused on inference. It suits batched, throughput-bound serving, and it punishes latency-bound decode.

Ironwood undid it. It raised bandwidth 4.5 times over Trillium, to 7,380 GB/s, while raising compute 2.5 times, which puts its critical batch at 313 on paper and 270 by the compiler’s constants, close to the H100’s 295 and the B200’s 281.

Google introduced Ironwood as its first TPU built for inference, and its architects list it as the fifth of their training supercomputers. In roofline terms both descriptions fit: it is the generation that brought the critical batch back down.

The compiler’s estimates agree when you compile real shapes. We compiled one 8192 by 28672 projection, the shape of a large dense MLP layer, at every batch from 2 to 2,048 and plotted the estimated time against batch 2.

On Ironwood the estimate at batch 2 is 127.5 µs. It is 7.5% higher at batch 256 and 1.97 times higher at 512.

On Trillium it is 287 µs at batch 2, only 15.1% higher at batch 512 and 1.97 times higher at 1,024. v5p, the chip with the most bandwidth per FLOP, bends first: 17% above its floor at batch 192 and 56% above at 256. In absolute terms Ironwood is still the fastest at every batch, 251 µs at batch 512 against Trillium’s 330. What changes is where each chip stops being free.

The eighth generation splits the difference in an instructive way. TPU 8t keeps HBM bandwidth modest at 6,528 GB/s while doubling MXU throughput with native FP4, so its critical batch is about 482 at FP8 or FP4.

That’s a training chip. TPU 8i has more bandwidth, 8,601 GB/s, and by Google’s own numbers the same throughput at FP4 as at FP8: 10.1 PFLOP/s at FP4 per chip, and 11.6 EFLOP/s at FP8 across a 1,152-chip pod, which is 10.07 per chip. If those figures mean what they appear to mean, 8i’s critical batch is 585 with FP8 weights and 294 with FP4 weights.

That breaks the precision invariance our earlier pieces relied on, and it breaks it in a useful direction. On 8i, FP4 is not a way to get more FLOPs. It is a way to halve the bytes each decode step has to read, which halves the batch you need before the chip is busy.

It is a compute format on the training chip and a bandwidth format on the inference chip. One caveat: the per-chip FP8 rate comes from a pod total. If Google counts only the 1,024 active chips its Boardfly description mentions, FP8 would be 11.3 PFLOP/s per chip and the two formats would no longer match.

Subscribe now

A batch of one

Below the critical batch every extra row is nearly free, so a batch of one should cost what a batch of eight costs. On Ironwood it does. On every older TPU, by the compiler’s own estimate, it costs about three times as much.

The reason is visible in the optimized HLO. Compile a [1, 8192] by [8192, 28672] product for v4, v5e, v5p or Trillium and there is no matrix multiply left in it. The compiler has rewritten the dot into an elementwise multiply followed by a reduction, which runs on the vector unit.

Give it two rows and you get a convolution, generated by the MXU emitter EmitAllBatchInSublanes. The rewrite comes from a pass whose source file name survives in the binary, tpu_dot_strength_reducer.cc, and it is controlled by the option xla_tpu_enable_dot_strength_reduction, which defaults to true.

The cycle estimates attached to each program price the decision. On v4 the rewritten product takes 3,154,024 cycles against 1,077,235 on the MXU, 2.93 times as many.

On v5e it is 3,280,393 against 1,147,598 (2.86 times), on v5p 1,292,915 against 470,757 (2.75 times), and on Trillium 1,986,785 against 568,013 (3.50 times). Turn the option off with LIBTPU_INIT_ARGS=--xla_tpu_enable_dot_strength_reduction=false and batch one costs what batch two costs.

On Ironwood the pass doesn’t fire for this shape at all, and batch one stays on the MXU at 352,341 cycles either way.

The final bundles make the difference concrete. With the rewrite, Trillium’s program for a [1, 8192] by [8192, 1024] product contains no matrix instruction. It’s 855 vector multiplies, 506 vector adds, 966 unpack instructions that widen bf16 to f32, and 244 cross-lane broadcasts, permutes and rotates.

Without the rewrite, the same product is 128 matmul issues fed by 2,048 weight pushes, the same instruction stream the compiler emits for batch eight.

We don’t know why the rewrite is on. A plausible reason is that it was tuned for small matrices, where pushing weights into the MXU costs more than the arithmetic they feed. For an 8,192-wide projection the compiler’s cost model disagrees with its own pattern pass, and the pattern pass runs first.

In production this rarely bites, because serving engines usually pad the decode batch to a few fixed sizes so that each size compiles once, and the smallest size is usually larger than one. It does bite in hand-written single-stream decode loops, in draft models for speculative decoding that run at batch one, and in mixture-of-experts code that issues one dot per expert when an expert receives a single token.

If you run any of those on v5e or Trillium, the option is worth a test on real hardware. We can only tell you what the compiler predicts.

Subscribe now

The tile and the KV cache

Every TPU array lives in tiles, and the layout string in the optimized HLO says which. A bf16 matrix is bf16[4096,4096]{1,0:T(8,128)(2,1)}: the last dimension varies fastest, the array is cut into tiles of 8 by 128 32-bit words, and (2,1) means two bf16 rows are packed into each word. f32 is T(8,128).

Eight-bit types are T(32,128)(4,1) and four-bit types T(64,128)(8,1). Every one of these tiles is 4 KiB. On the training chips from TPU v2 through v5p that is exactly one vector register, 8 sublanes by 128 lanes of 32 bits.

Ironwood’s registers are 16 by 256 according to the retrospective, four times larger, but its memory layouts use the same 8 by 128 tiles as every other generation we compiled for. Narrow types don’t get smaller tiles. They get more rows per word.

The compiler is careful with small arrays. For an array with one row it picks a T(1,128) tile instead of padding to eight rows, and for two or four rows T(2,128) or T(4,128).

What it can’t do is store less than one 32-bit word per lane. So on all five generations a single row of bf16 occupies twice its logical size, a single row of FP8 four times and a single row of FP4 eight times. FP4 activations need eight rows before they stop paying for air; FP8 needs four.

The minor dimension is less forgiving. Storage tiles are 128 elements wide on every generation, and a minor dimension narrower than that is padded unless the compiler can move a larger dimension into its place. When nobody constrains the layout, it does.

A [4096, 64] array is stored transposed, layout {0,1}, with the 4,096 dimension innermost, and costs exactly its logical size. A KV cache shaped [4096 pages, 16 tokens, 8 heads, 64 dims] gets layout {0,3,2,1}, with the page index innermost, and also costs exactly its size, as do the same cache with 16,384 pages, 16 heads or 32-token pages.

That default is a storage decision, not an access decision. It scatters each token’s 64-wide head across 64 sublane rows and interleaves 128 different pages in every vector, which is the opposite of what an attention kernel wants. Force the same cache into row-major order, layout {3,2,1,0}, and it costs 2.0 times its logical size, in bf16 and FP8 alike.

A small block such as [64, 16, 8, 64], which has no dimension of 128 or more to move inward, pays 2.0 times even in the default layout. Fold the heads into the minor dimension, [64, 16, 512], and row-major order pays nothing. With 128-wide heads the problem doesn’t exist.

Several open models use 64-wide heads, and on a TPU the layout of their KV cache decides whether half of it is padding. Kernels fix their own layouts, so this is a decision the kernel author makes once for everyone who uses the kernel.

This is the TPU version of a constraint we met on Blackwell, where the smallest UMMA tile has 64 rows. On NVIDIA the floor is in the math unit. On the TPU it is in the storage format, which means it costs HBM capacity and bandwidth, not just idle multipliers.

Share The Software Frontier

Precision: bf16, int8, FP8 and FP4 per generation

Datasheets list one number per format, when they list it at all. The compiler has to decide how to run every format on every chip, and its cycle estimates show what it decided.

We compiled one 8192 by 8192 by 8192 matmul in seven input types on all five generations and divided each estimate by the bf16 one.

Ironwood runs FP8 in 0.48 times the cycles of bf16, which is the 2x Google advertises: 4,614 TFLOP/s against 2,307. It runs int8 in 1.13 times the cycles of bf16, slower than bf16, and int4 and FP4 in 1.15 times. Nothing in the optimized HLO converts the inputs. The emitter handles each narrow type itself, and on Ironwood only FP8 has a fast path in the cost model.

The older chips are the mirror image. int8 runs in 0.47 times the bf16 cycles on v5e, 0.53 on v5p and 0.62 on Trillium, consistent with the int8 rate Google publishes for v5e, 393 TOPS, twice its bf16 peak.

FP8 e4m3 gets little or nothing: 0.94, 1.03 and 0.86, in line with Google’s own table, which lists v5p and Trillium at the same FP8 rate as bf16. One result we can’t explain: FP8 e5m2 runs exactly as fast as int8 on v5e, v5p and Trillium, in the same number of cycles to the unit.

Either the cost model files e5m2 under the eight-bit integer path, or the hardware has a fast path for it that no datasheet mentions. Without a chip we can’t tell which. v4 accelerates nothing: every narrow type costs 1.08 to 1.17 times bf16.

Two practical consequences follow. If you serve int8-quantized weights on v5e or Trillium, those checkpoints lose their speed on Ironwood and should be re-quantized to FP8 before the migration, not after. And no chip the public compiler knows runs FP4 faster than its eight-bit path.

TPU 8t is the first TPU with native FP4, and on 8i, as above, FP4 appears to run at the FP8 rate. Our FP4 piece compared NVIDIA’s NVFP4 with the OCP MXFP4 format that AMD uses. Google hasn’t said, in the material we read, which FP4 the eighth generation uses or how it scales its blocks.

Subscribe now

Inside one matmul on Trillium and Ironwood

The deepest thing libtpu will show you is its low-level IR, which it calls LLO, all the way to the final bundles. For a single 1024 by 1024 by 1024 bf16 matmul, the two dump options write 72 stages for Ironwood and 73 for Trillium.

The names read like a compiler textbook written for a VLIW machine: pre-auto-mxu-assigner, bf16-coalescing, x8-coalescing, MXU-assigner, vmac-transform, critical-path-scheduler, vliw-packed-bundles, vliw-bundle-scheduler, register-pressure, post-ra, delay-converter and, last, final_bundles. Trillium has one stage Ironwood doesn’t, post-allocation-offset-shuffling.

A bundle is the set of instructions issued together. TPU v2’s scalar unit fetched bundles of 322 bits, according to the retrospective, and Ironwood’s are more than 50% wider.

The slot table above says what one can hold. On both Trillium and Ironwood it is two MXU instructions, four vector ALU operations, two cross-lane operations, two result pops, one transcendental, three vector loads, two vector stores and two scalar operations, plus dedicated slots for spill traffic.

At the level of the instruction format the two TensorCores are identical. The programs they get are not.

Trillium’s program is 4,342 bundles and 17,639 instructions. Ironwood’s is 3,094 bundles and 8,677 instructions. Strip out the spills and both load the same 2,048 vectors and store the same 512 results, and both issue the same 1,024 matrix multiplies after the same 256 weight pushes.

Everything else is the cost of getting partial sums out of the matrix unit. Trillium pops 4,096 results and Ironwood 1,024. Trillium executes 3,072 vector adds and Ironwood none. Trillium issues 2,384 spill loads and 1,396 spill stores, Ironwood 476 of each.

Here are three consecutive bundles from the middle of Trillium’s program, with registers renamed and VMEM addresses shortened:

vmatmul.mubr.bf16.gmra.mxu0 v51 ;; v21 = vpop.f32.mrf.mxu0 ;; vmatprep.mubr.bf16.mxu1 v61 ;; v30 = vld [buf+0xc30]
vmatprep.mubr.bf16.mxu0 v44 ;; v24 = vpack.c.bf16 v41, v23 ;; v3 = vadd.f32 v19, v5 ;; v57 = vadd.f32 v21, v22
    ;; v1 = vpop.f32.mrf.mxu1 ;; vmatpush1.bf16.msra.mxu1 v45 ;; v41 = vld [spill] ;; v23 = vld [spill] ;; v21 = vld [spill]
v31 = vpop.f32.mrf.mxu0 ;; v45 = vld [spill] ;; v22 = vld [spill]

and four from Ironwood’s:

v21 = vpop.f32.mrb[225].mxu0 ;; v43 = vpack.c.bf16 v37, v19 ;; v0 = vpop.f32.mrb[226].mxu1
v27 = vpack.c.bf16 v21, v62 ;; v12 = vpop.f32.mrb[226].mxu0 ;; v40 = vpop.f32.mrb[227].mxu1 ;; v21 = vld [spill]
v33 = vpop.f32.mrb[227].mxu0 ;; vst [buf+0xe08] v43 ;; v41 = vpack.c.bf16 v40, v0
vst [buf+0xe00] v27 ;; v63 = vpack.c.bf16 v33, v12 ;; vmatmul.mubr.bf16.gmra.mrb[76].mxu1 v2 ;; v27 = vld [spill]

On Trillium, results leave the matrix unit through something called mrf, with no address, and are added together on the vector unit while partial sums wait in registers and spill to VMEM. On Ironwood the matrix instruction names a destination, mrb[76], and each pop names a source, mrb[225], and nothing is added afterwards. Across the whole program the indices run from mrb[0] to mrb[255].

The counts fit one reading exactly. The 1024 by 1024 f32 output is 1,024 vector registers’ worth of 8 by 128 tiles. Trillium pops each of them four times, because one 256-deep pass through the matrix unit covers a quarter of K = 1024, and sums the four partials with three adds: 4,096 pops and 3,072 adds. Ironwood pops each one once.

A decode-shaped product, [8, 8192] by [8192, 1024], has 8 output tiles and a K that takes 32 passes. Trillium pops 256 times and adds 248 times; Ironwood pops 8 times and adds nothing.

We read mrf as matrix result FIFO and mrb as matrix result buffer: on Ironwood, partial sums accumulate inside the matrix unit, in addressed entries, for as many passes as K needs. That interpretation is ours, not Google’s. It is the same move NVIDIA made between Hopper and Blackwell, when the tensor core’s accumulator left the register file for tensor memory.

In our TMEM piece the reason was that Blackwell’s largest accumulator no longer fit in 255 registers. Here the measurable effect is that the vector unit stops doing the matrix unit’s bookkeeping. The result-pop slots are busy 16.5% of the time on Ironwood against 47.2% on Trillium, the vector ALUs 12.4% against 26.5%, and the spill-store slots 7.7% against 16.1%, while the MXU slots stay above 90% on both.

Those idle slots now have a second use. The retrospective describes a hardware replay unit in Ironwood’s vector unit that samples vector bundles at random and re-executes them in idle VLIW slots, replaying odd-lane operations on even lanes to catch silent data corruption without slowing the program. A matmul that leaves seven in eight vector ALU slots empty gives it plenty of room.

A caution about where the accumulator matters. In the decode-shaped program both chips spend about 4,100 bundles, dominated by roughly 8,200 vector loads and 2,048 weight pushes. When a program is a stream of weights, an accumulator saves little.

It matters for prefill, for training and for any matmul deep enough in K that partial sums pile up, which is exactly the work that turns into spills on Trillium.

Leave a comment

SparseCores, collectives and MoE dispatch

SparseCores are small dataflow processors built for embedding lookups. Google’s retrospective counts two per chip on TPU v2 and v3 and four on v4, v5p and Ironwood; Google’s documentation gives Trillium two. Each has 16 compute tiles; together they take about 5% of the die area and of the power, and Ironwood’s are 2.4 times faster than v5p’s.

The retrospective also says they began serving as offload engines for collectives such as AllReduce, AllGather, ReduceScatter and Broadcast once Transformers took over the workload. In the public compiler, with default options, that only happens on Ironwood.

Compile an all-reduce across four Ironwood chips, eight devices, and the optimized HLO contains no all-reduce running on the TensorCore.

It contains an asynchronous call on an execution thread named sparsecore, with a configuration that says "device_type":"DEVICE_TYPE_SPARSECORE" and "offload":"OFFLOAD_COLLECTIVE", and a ring description in which one phase is marked "across_cores_on_chip":true, the hop between the two TensorCores of a chip. The TensorCore starts the call and carries on. The SparseCores move the data.

We compiled all-reduce, all-gather, reduce-scatter and the dense all-to-all at 64 KiB, 1 MiB and 16 MiB per device on Ironwood slices of 4, 8 and 64 chips and on v5e, v5p and Trillium slices, plus the ragged all-to-all that JAX uses for mixture-of-experts dispatch.

On Ironwood, all-reduce, all-gather and reduce-scatter were offloaded to the SparseCores in every case that compiled, and so was the ragged all-to-all, on 8, 16 and 128 devices. The dense all-to-all never was, on any generation. On v5e, v5p and Trillium nothing was offloaded, although v5p has four SparseCores per chip.

Their collectives run on the TensorCore, with a short-message all-reduce emitter and a ring emitter above it; on the 2- to 4-device slices we bisected, the switch happens just under 1 MiB per device, and around 2 MiB on v5p. A two-device collective on Ironwood, whether between the two cores of one chip or between neighboring chips, also stays on the TensorCore.

The library behind these choices is large. libtpu’s strings name 146 distinct identifiers ending in Emitter or Strategy, among them a D2DUniDirRingStrategy for the die-to-die hop inside a chip, an InferenceShortRingSumEmitter, a RaggedAllToAllEmitter and a RotatedPincerQuantizedEmitter, whose name suggests collectives on quantized data. Our compiles used a handful of them, and each choice is recorded in the HLO.

The switches are in the compilation environment. xla_tpu_enable_sparse_core_collective_offload_all_reduce, _all_gather and _reduce_scatter are set to AUTO, and xla_tpu_sparse_core_all_reduce_offload_min_size_in_bytes is 65,536, so very small all-reduces stay where they are.

Read that next to Google’s description of TPU 8i. Each 8i chip has two TensorCores on core dies and one Collectives Acceleration Engine on the chiplet die, “replacing four SparseCores” of Ironwood, with five times lower on-chip collective latency, plus a new topology, Boardfly, that cuts the worst path across a 1,024-chip pod from 16 hops to 7 to speed up all-to-all traffic.

On Ironwood the SparseCores are already the TPU’s collectives engine, MoE dispatch included. TPU 8i replaces them with hardware built only for that job and attacks the remaining cost, hop count, through the network. TPU 8t keeps its SparseCores, and Google describes them as offloading data-dependent all-gathers and other collectives too.

For serving, the reason to care is tensor parallelism. A Megatron-style transformer layer performs two all-reduces per layer in decode, each of batch times hidden times two bytes: 1 MiB at batch 64 and a hidden size of 8,192, right where the older chips switch emitters.

On Ironwood that traffic, and the MoE dispatch next to it, leaves the TensorCore, which can keep multiplying while it moves.

Subscribe now

VMEM and TPU 8i’s 384 MB

VMEM is the TPU’s software-managed on-chip memory, where a Pallas kernel stages its tiles. The compiler enforces its size and says so when you ask for too much.

We compiled a Pallas kernel with an ever larger scratch buffer until it failed, which on Ironwood reads “Allocation (size=68157440) would exceed memory (size=67108864)”. The limits per device: v4 16 MiB, v5e 128 MiB, v5p 64 MiB, Trillium 128 MiB, Ironwood 64 MiB per core.

Per chip that is 32 MiB on v4 and 128 MiB on v5e, v5p, Trillium and Ironwood, which matches the retrospective’s table for the training chips; on v5p a megacore program is still limited to one core’s 64 MiB.

XLA’s own fusions stay well below that: on Ironwood the large matmul fusions carried a 32 MiB scoped budget, and the largest fusion allocations we saw were about 15 MB on v4, v5e and v5p and up to 33 MB on Trillium and Ironwood. A kernel that wants the rest has to ask for it.

The same trick gives HBM. Compile something larger than memory and the error reports the space available per device: 15.25 GiB on v4, 15.75 on v5e, 95.73 on v5p, 31.24 on Trillium and 94.74 per Ironwood core.

On-chip memory per chip has been 128 MiB on every TPU since v5e and v5p. The retrospective’s explanation is that SRAM density grew more slowly than logic density, so VMEM only quadrupled from TPU v2 to Ironwood while the die grew much more.

TPU 8i is the first chip in this lineage to break the plateau, with 384 MB of on-chip SRAM, three times Ironwood, and Google says it can host a larger KV cache entirely on silicon. It is worth putting a number on larger.

A 70B-class model with grouped-query attention, 80 layers and 8 KV heads of 128, stores 163,840 bytes of KV per token in FP8. 384 MB holds about 2,300 of those tokens, and only if nothing else lived in VMEM. Ironwood’s 128 MiB holds about 820. A model with DeepSeek’s compressed latent attention, 576 values per token per layer over 61 layers, needs 35,136 bytes per token, so 8i holds about 10,900 tokens of it.

Per chip that is one modest conversation. Per pod it is another matter: 1,152 chips carry 442 GB of SRAM, about 2.7 million tokens of the 70B-class cache. On-chip KV is a pod-scale idea.

It works when the cache is sharded across hundreds of chips that can reach each other quickly, which is what the collectives engine and Boardfly are for. The SRAM, the CAE and the topology only make sense together.

Subscribe now

The bill: TPU vs GPU, per byte and per FLOP

As of September, Google’s pricing page lists Ironwood at $12.00 per chip-hour on demand in Iowa, $8.40 on a one-year commitment and $5.40 on three years.

Trillium lists at $2.70 on demand and $1.22 on three years, v5p at $4.20 and $1.89, v5e at $1.20 and $0.54. SemiAnalysis estimates that Anthropic pays about $1.60 per Ironwood chip-hour on Google Cloud. For GPUs on the same cloud, a September snapshot of public prices put an H100 at $5.38 and a B200 at $11.28 per GPU-hour on demand.

The transacted neocloud market, as measured by Ornn’s index on 28 September, was $2.56 for an H100 and $8.11 for a B200.

Every hourly rate hides two prices. One is what you pay per unit of HBM bandwidth, which is what a decode step below the critical batch consumes. The other is what you pay per unit of compute, which is what you consume above it.

On demand, the first price is almost the same for all six accelerators we priced: $1.40 per TB/s-hour for v5e, $1.41 for the B200, $1.52 for v5p, $1.61 for the H100, $1.63 for Ironwood and $1.65 for Trillium.

The second ranges more than threefold, from $1.47 per dense 8-bit PFLOP/s-hour on Trillium to $4.58 on v5p. Whether by design or by convergence, Google’s on-demand rate card prices bandwidth and lets FLOPs fall where they may.

That has a direct consequence for tokens. Take a 70B-class dense model with 8-bit weights and short contexts, and assume a perfect roofline: no KV traffic, no communication, no idle time, and per-chip throughput that doesn’t depend on how many chips the model is split across. These are floors, not forecasts.

At batch 64 every chip in the table is bandwidth-bound, and the on-demand floor per million output tokens lands between $0.42 and $0.50 on all of them. The chip doesn’t matter, because the rate card has already priced the bandwidth.

At each chip’s critical batch the floor falls to between $0.057 on Trillium and $0.18 on v5p, and now the chip matters a great deal. Trillium is the cheapest FLOP Google sells, as long as you can keep 560 sequences per chip in flight.

Ironwood’s floor at its critical batch is $0.10 on demand, $0.045 on a three-year commitment, and $0.013 at the estimated Anthropic rate, about a quarter of the transacted H100 market’s $0.050 and a fifth of the B200’s $0.070.

Long contexts push every chip back toward the bandwidth price, whatever the batch. With an FP8 KV cache, the same model reads 163,840 bytes per token for every position of context, so at 32K tokens each generated token reads 5.4 GB of cache no matter how many sequences share the weights.

That caps utilization at 4.2% of peak on Ironwood, 2.3% on Trillium, 4.4% on an H100 and 4.6% on a B200, and it puts the on-demand floor at between $2.08 and $2.46 per million output tokens on every chip in the table, about five times the short-context floor at batch 64.

For agentic workloads with long contexts, the rate card’s bandwidth price is the price of a token, which is why so much attention research in the last year has been about reading fewer bytes of cache.

The comparison at the critical batch is the economic story of the TPU in 2026. At list price Ironwood is priced like a B200 on Google Cloud, $12.00 against $11.28, and the same rate card puts every chip near the same bandwidth price.

At the price a tenant with a gigawatt-scale contract reportedly pays, Ironwood is the cheapest accelerator in this piece on both axes, by 1.9 to 2.9 times over the next cheapest option. The scale of those contracts is public now. Broadcom booked $10 billion and then another $11 billion of Ironwood racks for Anthropic, describing them as complete system sales, and in September put Anthropic’s deployment at 1 GW of Ironwood in 2026, 5 GW of TPU 8i in 2027 and 10 GW in 2028.

Meta signed a multi-year TPU rental in February and is negotiating to buy chips for its own data centers from 2027.

Tenants also change what the compiler is. A chip that only Google ran could keep its codenames, its 1,184 options and its rewrites to itself. A chip rented by Anthropic and Meta has customers who will read the same dumps we read, find the same options and tune them. For now, the most detailed description of the TPU’s internals that anyone outside Google can read is a 683 MB binary.

Subscribe now

Economic and financial implications

Everything above is a property of a chip or of a compiler. Money turns those properties into decisions, and in 2026 the decisions are large enough to show up in the earnings of three companies. Start with what the findings are worth to a buyer, then follow the money up the stack.

What the compiler’s findings are worth to a buyer

Four results in this piece change the cost of a token directly, using the same floor model as the table above.

Precision on Ironwood. The compiler prices an int8 matmul at 1.13 times the bf16 cycles and an FP8 matmul at 0.48 times. Above the critical batch, where compute sets the price, an int8 checkpoint costs about 2.3 times as much per token as the same model in FP8. Below it, both read one byte per weight and cost the same. The penalty lands on the high-batch traffic that pays for a deployment.

The KV layout. At 32K tokens of context the cache dominates every byte a decode step reads. For a model with 64-wide heads and the same cache size per token as our reference model, a row-major cache stores and moves twice its logical size, which doubles the long-context floor, from $2.08 to $2.46 per million output tokens on demand to $4.17 to $4.92.

Batch one. For single-stream decode on v4 through Trillium, the multiply-and-reduce rewrite costs 2.75 to 3.50 times the cycles of the matrix-unit path in the compiler’s model. A latency-bound product that runs one sequence per replica pays that multiple on every token, unless someone changes one option.

Which generation. At three-year list prices Trillium’s dense 8-bit compute costs $0.66 per PFLOP/s-hour, the cheapest in this piece, and its bandwidth $0.74 per TB/s-hour, the same as Ironwood’s $0.73. Above 560 concurrent sequences per chip Trillium’s floor is $0.026 per million tokens against Ironwood’s $0.045.

Below that, the two cost the same per byte. Offline work that can keep hundreds of sequences in flight, such as evaluations, synthetic data and reinforcement-learning rollouts, is Trillium’s economic home. Interactive serving is Ironwood’s.

The rate card is a formula

Google’s TPU prices follow a fixed schedule. For every generation with a published price, the one-year commitment is 70% of the on-demand rate, the three-year commitment 45% and Flex-start 50%: $0.84, $0.54 and $0.60 against $1.20 for v5e; $2.94, $1.89 and $2.10 against $4.20 for v5p; $1.89, $1.22 and $1.35 against $2.70 for Trillium; $8.40, $5.40 and $6.00 against $12.00 for Ironwood.

The estimated rate for Anthropic, $1.60 per Ironwood chip-hour, is 13.3% of on-demand. It is not on the card, and it is a different kind of price: a tenant price for gigawatts rather than a list price for hours.

What a gigawatt costs and earns

Broadcom booked $21 billion of Ironwood racks for Anthropic and describes Anthropic’s 2026 deployment as 1 GW of Ironwood: roughly $21 billion of rack hardware per gigawatt, before buildings, power and the network outside the racks.

On the revenue side, one Ironwood chip rented for 90% of the hours in a year earns $12,600 at the estimated anchor rate, $42,600 at the three-year list rate, $66,200 at the one-year rate and $94,600 on demand.

Google doesn’t publish Ironwood’s power per chip, so the two numbers can’t be joined directly. They can be joined conditionally. For the anchor rate to repay $21 billion of racks in four years at 90% utilization, a gigawatt has to hold about 416,000 chips, which means each chip, with its share of host, network and cooling, can draw no more than about 2.4 kW.

At the three-year list rate the same payback allows 8.1 kW per chip, and on demand 18 kW. At any published price the racks pay for themselves in well under four years. At the anchor price the answer depends on watts per chip, a number Google hasn’t published. Energy and buildings come on top of all of these.

Who captures the margin

A GPU buyer pays NVIDIA’s gross margin: 75.0% in the quarter to July, guided to 74.0% for the next and, according to the company’s call, heading for a trough of 71% to 72% in its fiscal fourth quarter as memory costs rise.

A TPU buyer pays Google, which pays Broadcom, whose company-wide gross margin was also 75% in its quarter to August but fell 210 basis points in a single quarter because AI chips, 73% of them custom accelerators, made up more of its revenue.

The gap between those two margin stacks is the room Google has to price anchor tenants far below its own list. It is also how the B200 rental index could rise 79% in a quarter while Ironwood capacity went to Anthropic, by SemiAnalysis’s estimate, at about a fifth of the B200’s market rate.

Alphabet

Alphabet’s filings now describe the TPU as something it sells. Its first-quarter 10-Q says Google Cloud has agreements to supply multiple gigawatts of TPU hardware to customers who run their own infrastructure, with the revenue counted in the Cloud backlog.

That backlog reached $514 billion at the end of June, up more than $50 billion in a quarter. Google Cloud revenue grew 82% to $24.8 billion in the second quarter, with a 35.6% operating margin, and management said growth accelerated even after excluding TPU system sales.

The bill is on the other side of the ledger. Alphabet raised its 2026 capital-expenditure guidance to $195 billion to $205 billion, spent $44.9 billion in the second quarter alone and reported negative free cash flow of $5.9 billion for the quarter.

On 1 June it announced up to $70 billion of new equity: $15 billion of mandatory convertible preferred stock, $15 billion of common stock and a $40 billion at-the-market program, to fund AI infrastructure. It has also reportedly agreed a joint venture with a large investment firm to lease TPUs to other customers.

A company that funds part of its build-out with equity has to care about the utilization and the price of every chip, which is the trade-off an 87% discount to list makes visible.

Broadcom

Broadcom’s AI semiconductor revenue was $16.7 billion in its quarter to August, up 221% and 56% of total revenue, with custom accelerators 73% of it.

It shipped Ironwood in high volume to Anthropic and to Google, began production shipments of TPU 8i for Google, and guided AI revenue to $58 billion for fiscal 2026, $115 billion for fiscal 2027 and $230 billion for fiscal 2028. On the same call it gave Anthropic’s roadmap: 1 GW of Ironwood in 2026, 5 GW of TPU 8i in 2027 and 10 GW in 2028.

Put the two together and something has to give. If TPU 8i racks cost what Ironwood racks cost per gigawatt, Anthropic’s 2027 deployment alone would be about $105 billion of hardware, close to Broadcom’s entire $115 billion fiscal-2027 AI guidance, which also has to cover Google’s own TPUs, other custom-chip customers and networking.

Either TPU 8i is much cheaper per gigawatt than Ironwood, or a large share of the value flows through Google’s TPU system sales rather than through Broadcom’s revenue, or both.

Reports conflict on which partner designs which eighth-generation chip, and Google hasn’t confirmed any of them; Broadcom’s own call places TPU 8i in its shipments.

MediaTek and the inputs

MediaTek is Google’s second TPU partner. It reportedly holds orders for Google’s v7e and v8e, asked TSMC for a sevenfold increase in CoWoS packaging capacity for its Google projects by 2027, and expects more than $1 billion of ASIC revenue in 2026 and several billion in 2027.

Digitimes reports that its TPU orders are now limited by capacity. The binding inputs for every TPU, as for every GPU, are advanced packaging and HBM. NVIDIA attributes its margin step-down to memory costs, and each new TPU carries more HBM than the last: 192 GiB on Ironwood, 216 GB on TPU 8t and 288 GB on TPU 8i.

A rate card that is flat in dollars per unit of bandwidth is consistent with HBM being the input that sets the cost.

Nvidia

NVIDIA’s own numbers show no damage yet: revenue of $96.2 billion in the quarter to July, up 106%, data-center revenue of $89.0 billion, guidance of $108 billion for the next quarter and about 70% growth for fiscal 2028.

Demand still exceeds supply. What TPUs change is the price at the margin. The largest buyers outside Google now have a second supplier: Anthropic is scaling toward 10 GW of TPUs by 2028, Meta rents TPUs alongside its NVIDIA and AMD contracts, and Google sells TPU systems for customers’ own buildings. At list price Google matches NVIDIA’s cost per byte. In contracts it doesn’t have to.

Compute markets

GPU rental indices, from Ornn’s transacted index to Silicon Data’s H100 index, price GPU-hours. TPU hours have no index, and comparing a TPU-hour with a GPU-hour is a category error unless you normalize.

The normalization this piece suggests is bandwidth for inference below the critical batch and dense compute above it. On that basis Ironwood’s list price is a B200’s, its three-year price is below the B200 market and its anchor price has no GPU equivalent.

A market that wants to hedge inference capacity across vendors would need to quote it per TB/s-hour, not per chip-hour.

Subscribe now

Notes for anyone serving on TPUs

• Size the batch to the generation. Trillium needs about 560 concurrent sequences per chip before compute is the limit, Ironwood 270 to 313, v5p about 168. Below those numbers you are paying the bandwidth price and extra sequences are almost free. Quantizing weights from bf16 to FP8 does not change the number on Ironwood; FP4 weights on TPU 8i would halve it.

• Count devices, not chips, on Ironwood. Each chip is two devices with 94.74 GiB of usable HBM and 64 MiB of VMEM each. A sharding plan written for v5p’s megacore sees twice the devices with half the memory, and a tensor-parallel degree of two can live inside one chip.

• Move int8 checkpoints to FP8 before moving to Ironwood. In the compiler’s model int8 takes 1.13 times the bf16 cycles there, and FP8 0.48 times.

• Fix the KV layout yourself. With 64-wide heads, a row-major cache costs twice its size. Fold heads into the minor dimension, or accept the compiler’s default only if no kernel needs row-major order.

• Watch for batch one on v4 through Trillium. Hand-written decode loops, draft models and per-expert dots can hit the multiply-and-reduce rewrite. Pad to two rows, or test xla_tpu_enable_dot_strength_reduction=false on real hardware.

• Ask Pallas for VMEM explicitly. XLA’s fusions budget 16 to 32 MiB; the hardware has 64 MiB per Ironwood core and 128 MiB on v5e and Trillium.

• Expect collectives to change hands at 64 KiB on Ironwood. Above it, all-reduce, all-gather, reduce-scatter and ragged all-to-all across four or more chips run on the SparseCores while the TensorCore computes. Only the dense all-to-all stays on the TensorCore.

• Buy bandwidth below the critical batch and FLOPs above it. Compare chips in dollars per TB/s-hour for interactive decode and long contexts, and in dollars per PFLOP/s-hour for batch work above the critical batch. The published commitment discounts are fixed at 30% and 55% off on-demand; anything better is a contract, not a list price.

Subscribe now

What the compiler can’t tell us

Every time in this piece is a model output. The cycle estimates and optimal_seconds are what the compiler believes, and the compiler is optimizing against them, but real programs also pay for DMA contention, ICI congestion, host stalls and everything else a static model leaves out.

The relative results, such as batch one against batch two or int8 against bf16 on the same chip, are much safer than any absolute time.

We can’t tell whether the compiler’s lower compute constants for v5p and Ironwood reflect a clock, a margin or something else, and the implied clocks depend on our reading of the slot rates and of the retrospective’s MXU count.

We can’t tell whether FP8 e5m2 really runs at the int8 rate on v5e, v5p and Trillium or is only filed that way by the cost model. The names matrix result FIFO and matrix result buffer are our reading of mrf and mrb, supported by the counts but not confirmed by Google.

Layouts in this piece are the ones the compiler chooses for a program’s parameters when nothing else constrains them, plus one forced row-major case. Inside a real model, kernels and neighboring operations constrain layouts, so the padding you pay depends on the code around the array.

Coverage is narrow in places: matmuls go up to 16,384 on a side, collective tests up to 64 chips and three message sizes, and everything uses the compiler’s default options except where we say otherwise. All of it comes from one release, libtpu 0.0.48, as of October 2026, and a later release can change any compiler decision described here.

The TPU 8 numbers come from Google’s April deep dive and from reporting on it, and the 8i FP8 rate per chip is derived from a pod total. The Ironwood list prices reach us through two reports quoting Google’s pricing page, the GPU prices on Google Cloud through a third-party snapshot, and the Anthropic rate is an analyst estimate.

GPU rental prices move weekly: SemiAnalysis’s index has the B200 up 79% in the three months to late September. Company figures come from filings and earnings calls; the gigawatt arithmetic combines numbers from different companies and periods, and the power per chip that would join cost and revenue is unpublished.

Subscribe now

Five predictions

1. By 30 June 2027, a public libtpu release on PyPI will accept a TPU 8t or TPU 8i topology name for compile-only use.

2. When it does, an all-reduce compiled for four or more TPU 8i chips will name an engine other than the TensorCore and the SparseCore, and the ragged all-to-all will run on the same engine.

3. Through 31 December 2027, no public libtpu release will estimate an int8 matmul on Ironwood as cheaper than the same matmul in bf16.

4. By 31 December 2027, vLLM’s TPU documentation will recommend FP8 rather than int8 weights for serving on Ironwood.

5. When TPU 8i gets a public on-demand price, it will be within 15% of $1.63 per TB/s-hour of HBM bandwidth, which means between $11.92 and $16.12 per chip-hour.

Share The Software Frontier

Doing this yourself

Everything in this piece came from one install and a handful of options. The topology call and the compile are in the method section above. The rest is environment variables:

XLA_FLAGS="--xla_dump_to=/tmp/hlo --xla_dump_hlo_as_text"                    # HLO stages, compiler options
LIBTPU_INIT_ARGS="--xla_jf_dump_to=/tmp/llo --xla_jf_dump_llo_text=true"      # every LLO stage, final bundles
LIBTPU_INIT_ARGS="--xla_tpu_enable_dot_strength_reduction=false"             # keep batch-1 dots on the MXU

libtpu prints warnings about TPU worker hostnames and accelerator types when no TPU is attached. For compile-only use they are harmless. The VMEM probe is a Pallas kernel with a pltpu.VMEM scratch shape that grows until the compile fails, and the HBM probe is any program whose temporaries exceed memory.

The chip enum comes from the protocol-buffer descriptor inside libtpu.so: find the string TPU_VERSION_JELLYFISH, and the byte after each name is a field tag followed by the value.

If you have access to real TPUs, the most useful thing you could do with this piece is time the batch-one option and the int8 against FP8 result on Ironwood and tell us what you see. We will publish corrections with credit.

Subscribe now

Confidence dossier

Tier A is a primary document: Google’s documentation, Google’s blog, Google’s architects’ retrospective, a company’s own statement. Tier B is reputable reporting, a published index or a figure from vendor material we did not re-read. Tier C is our arithmetic on A or B.

Tier D is our interpretation. Tier M is our own measurement with libtpu 0.0.48 and JAX 0.11.2 in compile-only mode, with no TPU attached. Every claim in the piece has a row.

Subscribe now

Sources

1. N. P. Jouppi, S. Lakshmanamurthy, C. Young and D. Patterson, “Google’s Training Supercomputers from TPU v2 to Ironwood,” to appear in IEEE Micro, July/August 2026. arxiv.org/abs/2606.15870

2. Google Cloud, “TPU7x (Ironwood),” Cloud TPU documentation. docs.cloud.google.com/tpu/docs/tpu7x

3. Google Cloud, “TPU v5e,” Cloud TPU documentation, last updated 11 August 2026. docs.cloud.google.com/tpu/docs/v5e

4. D. Gupta and S. Mugazambi, “Inside the eighth-generation TPU: An architecture deep dive,” Google Cloud Blog, 22 April 2026. cloud.google.com/blog

5. HPCwire, “Google Bolsters AI Hypercomputer with New TPU Chips, Virgo Interconnect, Speedier Lustre,” 22 April 2026. hpcwire.com

6. DatacenterDynamics, “Google unveils eighth-generation TPUs, two dedicated training and inference chips.” datacenterdynamics.com

7. Moor Insights and Strategy, “Research note: Google TPU 8: Architecture, Context, and Enterprise Relevance.” moorinsightsstrategy.com

8. N. Jouppi et al., “TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings,” ISCA 2023. arxiv.org/abs/2304.01433

9. S. J. Kaufman et al., “A Learned Performance Model for Tensor Processing Units,” MLSys 2021. arxiv.org/abs/2008.01040

10. P. M. Phothilimthana et al., “TpuGraphs: A Performance Prediction Dataset on Large Tensor Computational Graphs,” 2023. arxiv.org/abs/2308.13490

11. Google Cloud, “Cloud TPU pricing.” cloud.google.com/tpu/pricing

12. Google Cloud, “Dynamic Workload Scheduler pricing.” cloud.google.com/products/dws/pricing

13. shattered.io, “Google Puts $12/Hr Price on Ironwood TPU vs Nvidia,” September 2026. shattered.io

14. Spheron, “Google TPU v7 Ironwood vs NVIDIA B200: Inference Cost (2026),” citing a SemiAnalysis estimate of Anthropic’s rate. spheron.network

15. Ornn, Compute Price Index, settlement of 28 September 2026. data.ornn.com/markets

16. Silicon Data, H100 Rental Price Index (SDH100RT). silicondata.com

17. fastgpu, “What it costs to rent an H100, B200 or RTX 4090 in September 2026,” DEV Community. dev.to

18. shattered.io, “Nvidia B200 Cloud Price Hits $8.01/Hr,” on the SemiAnalysis GPU pricing index, September 2026. shattered.io

19. SemiAnalysis, “TPUv7: Google Takes a Swing at the King.” newsletter.semianalysis.com

20. TrendForce, “Anthropic Emerges as Broadcom’s Mega-Client; Margin Challenges Ahead,” 12 December 2025. trendforce.com

21. DatacenterDynamics, “Broadcom bulges with Anthropic AI orders.” datacenterdynamics.com

22. SDxCentral, coverage of Broadcom’s 2026 earnings call. sdxcentral.com

23. Dataconomy, “Meta signs multibillion-dollar deal to rent Google TPUs for AI training,” 27 February 2026, reporting The Information. dataconomy.com

24. Digitimes, “MediaTek TPU orders rise as chip capacity war intensifies,” 1 October 2026. digitimes.com

25. NPR, “Google launches Project Suncatcher, a step towards AI data centers in space,” 1 October 2026. npr.org

26. J. Austin et al., “How to Scale Your Model,” Google DeepMind. jax-ml.github.io/scaling-book

27. JAX documentation, “Ahead-of-time lowering and compilation.” docs.jax.dev

28. L. Bradanini and L. Tettamanti, “How Blackwell’s Tensor Memory Actually Works,” The Software Frontier, 3 August 2026. thesoftwarefrontier.com

29. Alphabet, “Alphabet Announces Second Quarter 2026 Results,” Form 8-K exhibit 99.1. sec.gov

30. Alphabet, Q2 2026 earnings call. abc.xyz

31. Alphabet, Form 10-Q for the quarter ended 31 March 2026. sec.gov

32. Alphabet, Form 8-K of 1 June 2026 on equity offerings. sec.gov

33. Yahoo Finance, “Alphabet Q2 2026 earnings: revenue up 24%, Cloud surges 82%.” finance.yahoo.com

34. Broadcom, “Broadcom Inc. Announces Third Quarter Fiscal Year 2026 Financial Results,” 2 September 2026. investors.broadcom.com

35. The Motley Fool, Broadcom Q3 2026 earnings call transcript. fool.com

36. NVIDIA, “NVIDIA Announces Financial Results for Second Quarter Fiscal 2027,” 26 August 2026. sec.gov

37. NVIDIA Q2 fiscal 2027 earnings call summary, on margin and fiscal 2028 outlook. transcripts.platformaeronaut.com

38. TrendForce, “MediaTek Reportedly Secures Google v7e, v8e TPU Orders, Requests 7-Fold CoWoS Increase from TSMC,” 15 December 2025. trendforce.com

39. TrendForce, “MediaTek Forecasts $1B in ASIC Sales for 2026.” trendforce.com

40. tech-insider, “Google TPU 8t and 8i: 121 Exaflops, $21B Nvidia Challenge,” with one account of the design-partner split. tech-insider.org

41. Midas, summary of a Commercial Times report with a conflicting account of the split. getmidas.com

42. Storyboard18, “Meta signs multi-billion dollar AI chip rental deal with Google,” reporting The Information on a TPU leasing joint venture. storyboard18.com

43. pump.co, “Google Cloud TPU: Pricing, Performance and Use Cases,” on commitment prices. pump.co

44. dataknobs, “Google TPU Pricing 2026,” on Ironwood Flex-start and commitment prices. dataknobs.com

Intro

The first thing about FP4 that I could not explain to myself was a byte. NVFP4 and MXFP4 use the same 4-bit element, E2M1: one sign bit, two exponent bits, one mantissa bit. A Blackwell tensor core multiplies the same sixteen codes in both formats.

What differs is how a block of values stores its scale: one byte for every 32 values in MXFP4, a different kind of byte for every 16 values in NVFP4.

Every comparison I read puts NVFP4 ahead, sometimes far ahead, and yet OpenAI’s gpt-oss, DeepSeek V4 and Moonshot’s Kimi K3 all shipped in MXFP4, and Moonshot’s previous flagship used neither format.

I wanted the difference in numbers rather than adjectives, and I wanted to know what the silicon does with each format. Almost everything below comes from work that needs no GPU.

We rebuilt the rounding rules of every block format we could find in NumPy and checked them against two independent references: a published table of Gaussian errors, and AMD’s measurements on the activations of two real models.

Then we asked NVIDIA’s own assembler, ptxas 13.4.92 from the PyPI wheels, which FP4 instructions eight targets from Hopper to Rubin accept and what machine code each one becomes. Where measurement was impossible we read the vendor code that ships with the hardware.

At a glance

• One byte of scale separates NVFP4 from MXFP4. It is worth 1.6 dB of signal-to-noise on Gaussian data, 4.2 dB on activations and about 9% of perplexity under plain rounding.

• Under the open standard’s scale rule, 41.5% of MXFP4 blocks clip their own maximum, and the rule that rescued training is the worse one for inference error.

• One constant turns format noise into perplexity, explaining 98.7% of the variance across 21 formats; its value changes from model to model.

• FP4 arithmetic needs both operands in FP4. The big open releases ship FP4 weights with 8-bit or 16-bit activations, so they collect the bandwidth half only.

• INT8 is gone from the tensor-core instructions of B300 and Rubin. Rubin adds a wider scale format, UE5M3, and 3-bit lookup-table weights.

Subscribe now

Sixteen codes

E2M1 has a sign bit, two exponent bits with a bias of 1 and one mantissa bit. Its non-negative values are 0, 0.5, 1, 1.5, 2, 3, 4 and 6. Negative zero is a separate code with the same value, so sixteen codes give fifteen distinct numbers.

The grid is a floating-point format in miniature: neighbours are 0.5 apart up to 2, then 1 apart, then 2 apart. Nothing lies between 4 and 6, so a value that lands on 5 after scaling ends up 20% away from wherever it rounds.

An INT4 grid with the same maximum puts its levels 6/7 apart everywhere. Which grid is better depends entirely on where values land after scaling (Figure 1).

Every block format scales a block so that its largest value lands near the top of the grid, which pushes the other values of a bell-shaped block toward zero, where the FP4 grid is dense: in our simulation 50% of the values of a Gaussian block land below 2, where half of the non-negative levels live.

That is the case for floating point at four bits, and also its weak spot: the few values near the top of each block fall into the coarsest part of the grid.

The yardstick throughout this piece is the signal-to-quantization-noise ratio (SQNR): ten times the base-10 logarithm of signal power over error power. Each extra bit of a well-designed quantizer is worth about 6.02 dB.

The best possible 16-level quantizer for a Gaussian, the Lloyd-Max quantizer, needs no scales because it is told the variance in advance; it reaches 20.19 dB in our simulation, against 20.22 dB in the textbooks.

Real formats must discover the scale from the data, block by block, and they pay for it in bits. MXFP4 spends 8 bits of scale on every 32 values, 4.25 bits per element in total. NVFP4 spends 8 bits on every 16 values plus 32 bits per tensor, 4.5 bits per element.

NVFP4 therefore shrinks an FP8 checkpoint by 1.78 times rather than 2, a detail that matters again when we get to the critical batch.

Subscribe now

One byte of scale

MXFP4 comes from the Open Compute Project’s Microscaling specification (OCP MX v1.0, in september 2023). Its scale is E8M0: eight exponent bits and no mantissa, so every scale is a power of two.

The specification sets the shared exponent of a block to floor(log₂ amax) - 2, where amax is the largest magnitude in the block and 2 is the exponent of 6, the largest E2M1 value.

Divide the block by that scale and its largest value lands at 4 x 2^f, where f is the fractional part of log₂ amax. Because f is somewhere in [0, 1), the block maximum lands somewhere in [4, 8), and everything above 6 saturates to 6.

The maximum is clipped whenever 2^f exceeds 1.5, that is whenever f exceeds log₂ 1.5 = 0.585. If block magnitudes are spread evenly on a log scale, which is roughly what happens across the thousands of blocks and dozens of layers of a model, that is 1 - log₂ 1.5 = 41.5% of all blocks.

Our simulation over 128 tensor scales gives 41.51%. A clipped value can lose up to a quarter of its magnitude: 7.9 becomes 6.

NVFP4, introduced with Blackwell, stores the block scale as E4M3, the FP8 format with three mantissa bits, and computes it as amax / 6 rounded to the nearest E4M3 value (NVIDIA technical blog, June 2025). The block maximum then lands within about 6% of 6 instead of anywhere between 4 and 8, and the small overshoot rounds back to 6.

The price is range. Unsigned E4M3 runs from 2^-9 to 448, about 18 binades, which is not enough to hold the scales of an entire model. NVFP4 therefore adds a second level: one FP32 number per tensor, equal to the tensor’s amax / (6 x 448), divides the whole tensor before the block scales are computed.

There is a second way to build an E8M0 scale. Asit Mishra, Dusan Stosic, Simon Layton and Paulius Micikevicius showed that the OCP rule can make MXFP8 pretraining drift away from its BF16 baseline, and fixed it by rounding the scale up, so that amax divided by the scale never exceeds the largest element (Recipes for Pre-training LLMs with MXFP8, arXiv:2506.08027).

Rounding up removes clipping, but the block maximum now lands anywhere in (3, 6], which leaves the top of the grid unused for most blocks (Figure 2). Both rules exist in silicon.

On every Blackwell target we tried, cvt.rz.satfinite.ue8m0x2.f32 and its .rp twin each assemble to a single F2FP.SATFINITE.E8 instruction with a .RZ or .RP modifier, so the choice costs nothing at run time.

For inference the two rules are not equivalent, and the rule that rescued training is the worse one for error. Averaged over tensor scales, the OCP rule gives 18.75 dB on Gaussian data against 18.65 dB for rounding up.

On the activation proxy introduced below the gap is 0.50 dB, again in favour of the OCP rule (16.69 against 16.19 dB). Clipping one value per block costs less squared error than coarsening every other value. Training is hurt by bias, and a clipped maximum is biased; a single forward pass is hurt by squared error.

Clipping one value per block costs less squared error than coarsening every other value in it.

Subscribe now

The sawtooth

Power-of-two scales have a property that is easy to miss: the error depends on where the tensor sits relative to powers of two. Multiply a tensor by a constant c and every MX scale shifts with it, but the fractional part of log₂ c moves every block maximum within its binade, and with it the clipping fraction.

We multiplied the same 262,144 Gaussian values by 2^t for 128 values of t between 0 and 2 (Figure 3). NVFP4 does not move at all: 20.44 dB at every t, because the FP32 tensor scale absorbs the constant exactly.

MXFP4 under the OCP rule moves between 18.58 and 18.96 dB with a period of exactly one binade, and the share of clipped blocks swings between 26.8% and 55.9%. Rounding up moves between 18.52 and 18.79 dB.

The swing is small on Gaussian data, about 0.4 dB from trough to peak, but it means that an MXFP4 model’s accuracy depends on constants nobody chose for this purpose, such as the gain of the normalization layer in front of a projection. It also points at a cheap remedy.

A per-tensor multiplier that moves each tensor to its best phase recovers the distance from the average to the peak, 0.20 dB here.

That remedy is exactly NVFP4’s structure, an FP32 number per tensor above the block scales, and a November 2025 preprint applies the same two-level idea to MXFP8 training, keeping power-of-two block scales under an FP32 tensor scale (arXiv:2511.05811).

Subscribe now

Where the error lives

Before trusting any of these numbers we checked the simulator against someone else’s. Jack Cook and colleagues at MIT and NVIDIA publish the mean squared error of five block formats on standard normal data (Adaptive Block-Scaled Data Types, arXiv:2603.28765, Table 1).

Our implementation reproduces all five within 1%: 13.23 against their 13.2 (in units of 10^-3) for MXFP4, 9.04 against 9.0 for NVFP4, 7.56 against 7.5 for NVFP4 with Four Over Six, 7.44 against 7.4 for NVINT4 and 6.16 against 6.2 for their IF4 format (Figure 4, left).

The agreement says that the rounding details (ties to even, saturation, scale rounding, the FP32 tensor scale) match theirs, not only the headline numbers.

With the simulator anchored, the next question is where inside each block the error comes from (Figure 5).

Under the OCP rule, the single largest value of a 32-value MXFP4 block carries 23.1% of the block’s squared error on Gaussian data and 39.9% on activations, against the 3.1% it would carry if error were spread evenly.

Its average relative error is 11.9%. In NVFP4 the block maximum carries 2.1% and 5.5% of the error, at or below its even share of 6.25%, with an average relative error of 2.25%.

NVFP4 still has a soft spot near the top of its grid, because nothing exists between 4 and 6. Four Over Six, from Cook, Junxian Guo, Song Han and colleagues (arXiv:2512.02010), scales each block either to 6 or to 4, whichever gives less error.

Scaling to 4 gives up the levels 6 and -6 but makes 3 a level at 75% of the block maximum. In our simulation the choice is worth 0.78 dB on Gaussian data and 0.30 dB on activations.

IF4, from the same group, goes further: each block chooses between the FP4 grid and an INT4 grid rescaled to the same maximum, and records the choice in the sign bit of its E4M3 scale, a bit NVFP4 never uses because scales are positive.

It reaches 22.11 dB on Gaussian data, 1.7 dB above NVFP4 at the same 4.5 bits.

Subscribe now

Tails

Gaussian data is the friendly case, and activations are not Gaussian. The input to the down projection of a SwiGLU MLP is silu(a) x b, the product of two roughly Gaussian projections, and products of Gaussian variables have heavy tails.

Our proxy is exactly that product, with independent standard normal a and b. Its kurtosis is about 22, against 3 for a Gaussian.

A proxy that simple needs a reality check, and AMD published the one we needed. In June 2026 its ROCm team measured SQNR on the post-SiLU down_proj inputs of every layer of Llama-3.1-8B and Qwen3.6-27B, using the OCP reference quantizers (Atre, Bao, Tiwari and Sirasao, AMD ROCm blog, 26 June 2026).

Their layer means were 16.8 and 16.4 dB for MXFP4, 29.3 and 28.6 dB for MXFP6 E2M3, and 32.8 and 32.1 dB for FP8 with one scale per token. The proxy gives 16.69, 28.97 and 31.69 dB (Figure 4, right).

The first two fall between AMD’s two models and FP8 is within 1.1 dB. For a distribution defined in one line that is closer than we expected, and it is the activation model we use from here on.

On heavy tails the formats separate (Figure 6). NVFP4 improves, from 20.43 to 20.90 dB, because a floating-point grid suits blocks with one large value and many small ones. MXFP4 falls, from 18.79 to 16.69 dB. The NVFP4 advantage grows from 1.64 dB on Gaussian data to 4.21 dB on activations. Integer grids suffer most.

NVINT4, which beats NVFP4 on Gaussian data at 21.29 dB, drops to 18.55 dB, and INT4 with a scale per 32 values, the grouping of Moonshot’s Kimi K2 Thinking checkpoint (we model the scale as BF16), drops from 20.26 to 16.64 dB.

Weights sit in between. We extracted the 84.9 million linear-layer weights of a real trained transformer, the RoBERTa-base encoder packaged in spaCy’s en_core_web_trf 3.8.0, read them without PyTorch and quantized every matrix along its input dimension.

Their kurtosis is 6.0 and the largest weight lies 27 standard deviations from zero. NVFP4 reaches 20.51 dB, MXFP4 18.58 dB, NVINT4 20.87 dB, Four Over Six 21.23 dB and IF4 21.98 dB.

The NVFP4 advantage over MXFP4 is 1.94 dB pooled, and between 1.67 and 2.55 dB across the 48 matrices. RoBERTa is a 2019 encoder, not a modern decoder, so we read it as a sanity check on the synthetic numbers rather than as a benchmark.

Share The Software Frontier

Rotations

The usual cure for heavy tails is a rotation. Multiply blocks of values by a Hadamard matrix and each output becomes a signed average of all inputs, so a single outlier is spread across the block and the distribution moves toward Gaussian.

QuaRot and SpinQuant made this standard practice for INT4. For FP4 it is less obviously useful. Vage Egiazarian, Dan Alistarh and colleagues show that rotations help MXFP4 but hurt NVFP4 under round-to-nearest, and prove that NVFP4’s small 16-value groups already neutralize the outlier mitigation that rotations provide (Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization, ICLR 2026, arXiv:2509.23202).

Our simulation reproduces the sign of every effect and gives the sizes (Figure 7). A 16-point Hadamard rotation leaves Gaussian data unchanged within 0.03 dB. On activations it costs NVFP4 0.88 dB, gives MXFP4 2.13 dB and gives NVINT4 4.07 dB.

With eight outlier channels at twenty times the scale of the others the changes are -1.03, +2.27 and +3.21 dB. Larger rotations hurt NVFP4 less: a 128-point rotation costs it 0.51 dB on activations.

On RoBERTa’s weights, whose kurtosis is 6 rather than 22, a 16-point rotation changes NVFP4 by -0.09 dB, MXFP4 by +0.16 dB and NVINT4 by +0.42 dB: the same signs, much smaller sizes.

The best 4-bit recipe in our sweep is a rotation followed by NVINT4, integers under an E4M3 scale per 16 values: 22.63 dB on activations, 1.74 dB above plain NVFP4 and 1.36 dB above IF4.

Cook and colleagues found the same ordering in real W4A4 runs on Qwen3.5, where NVINT4 after a Hadamard transform beat NVFP4. The catch comes in the section on the assembler: no tensor-core instruction on a shipping NVIDIA datacenter part multiplies INT4 values under E4M3 block scales.

The block-scaled FP4 instructions accept only E2M1 elements, and the only integer tensor-core instruction left on datacenter Blackwell takes INT8 and is being withdrawn.

Subscribe now

From decibels to perplexity

Signal-to-noise ratios matter only if they predict something people care about. Cook and colleagues report the average WikiText-2 perplexity of four Qwen3.5 models (9B, 27B, 35B-A3B and 122B-A10B) for 23 block formats, with the weights and activations of every linear layer quantized by round-to-nearest (arXiv:2603.28765, Table 6).

We simulated 22 of them, all but a block-8 variant of Four Over Six. We computed the noise-to-signal power ratio (NSR) of each format on our activation proxy and fitted the simplest law that could work: the logarithm of the perplexity ratio is proportional to the noise power.

With one free parameter, ln(ppl / ppl_BF16) = 8.25 x NSR explains 98.7% of the variance across the 21 formats that stay above 12 dB, from 3.5 to 6.5 bits per element, integer and floating point, MX and NV (Figure 8). The rank correlation is 0.986.

Dropping any one format moves the fitted constant between 7.95 and 8.33, and the median error of the fit is 0.6% of perplexity. The same fit on Gaussian noise explains 95.4%, so the heavy-tailed proxy predicts better, as the comparison with AMD suggested.

W4A4 adds weight noise to activation noise, so we also charged each format twice, once on Gaussian data for its weights and once on the proxy for its activations. That doesn’t do better (98.3% with one constant, 98.8% with a constant per term), and the two-term fit puts most of the weight on the activation term.

The two terms rise and fall together across formats, so that split is a hint rather than a measurement. One format breaks the law: MXFP3, at a perplexity of 70. Below roughly 12 dB the models stop degrading gracefully and start to fail.

Read backwards, the fit gives targets. Staying within 1% of BF16 perplexity needs about 29.2 dB on the proxy, within 2% about 26.2 dB, within 5% about 22.3 dB. NVFP4 sits at 20.9 dB, predicted +6.9% against a measured +6.4%; MXFP4 sits at 16.7 dB, predicted +19.3% against a measured +16.0%.

The measured gap between them is 9.1% of perplexity. That is the size of one byte of scale per block when nothing else is done to the model. Quantization-aware training, rotations and better rounding all shrink it;

MR-GPTQ, the method Egiazarian and colleagues propose, brings MXFP4 to within 1 to 2% of NVFP4 accuracy.

The law travels, but its constant does not. AMD’s appendix gives WikiText-2 perplexities for the two models it profiled, quantized W4A4 by round-to-nearest.

For Llama-3.1-8B the law predicts +19.3% for MXFP4 against a measured +20.0% (8.47 against 7.06), and +0.6% for per-token FP8 against a measured +0.7%.

For Qwen3.6-27B it predicts the same +19.3% for MXFP4 against a measured +6.1% (7.53 against 7.10): the larger and newer model tolerates about three times the noise. The constant is a property of the model, not of the format, and 8.25 is an average over a family that runs from 9 to 122 billion parameters.

Leave a comment

What the assembler accepts

The numerics are half the story; the other half is which instructions exist. ptxas, nvdisasm and cuobjdump ship as ordinary x86 programs inside NVIDIA’s PyPI wheels, so the question can be asked on any laptop: wrap one instruction in a minimal PTX kernel, assemble it for a target and read the SASS that comes out.

We did this for the FP4-related instructions of PTX ISA 9.4 on eight targets with CUDA 13.4.92 (Figure 9). The targets are sm_90a (H100), sm_100a (B200), sm_100f (the portable Blackwell family), sm_103a (B300), sm_107a (Rubin, compute capability 10.7 according to NVIDIA’s TensorRT 11.3 release notes), sm_110a (Jetson Thor), sm_120a (RTX 50 and RTX PRO) and sm_121a (GB10).

Hopper has no FP4 at all. ptxas rejects conversion to FP4 on sm_90a as a feature the target does not support, rejects the conversion back, rejects conversion to E8M0 scales and has no tcgen05 instructions.

FP4 weights on an H100 must be decoded into BF16 or FP8 with integer tricks or table lookups before they reach a tensor core, and at that point FP4 offers nothing that INT4 does not.

That is the practical background to the Kimi K2 Thinking team choosing INT4 weights with 16-bit activations in November 2025: weight-only INT4 kernels such as Marlin were mature on Hopper (LMSYS blog, January 2026).

On datacenter Blackwell, FP4 reaches its own datapath only through the block-scaled FP4 kinds. tcgen05.mma with .kind::mxf4 or .kind::mxf4nvf4 assembles to UTCOMMA.

The same E2M1 operands sent through .kind::f8f6f4, the kind that also accepts FP8 and FP6, assemble to UTCQMMA, the opcode FP8 uses. The names fit H for half, Q for quarter and O for one eighth of 32 bits, and they fit NVIDIA’s documented rates: FP4 through the mixed kind runs at FP8 speed.

Any model that multiplies FP4 weights by FP8 activations is in this position by construction, because one of its operands is eight bits wide.

NVFP4 with 16-value blocks assembles to UTCOMMA.4X on sm_100a, sm_100f and sm_110a and to UTCOMMA.BLOCK16 on sm_103a and sm_107a. The older spelling of the qualifier, .scale_vec::4X, is rejected for sm_100f, the family target meant to run on both B200 and B300, so portable builds must use .block16.

INT8 is on its way out. tcgen05.mma.kind::i8 assembles to UTCIMMA on sm_100a and on Thor’s sm_110a, and is rejected on sm_100f, sm_103a and sm_107a.

In August 2026 Teng-Ruei Chen traced the same withdrawal for B300 through the spec sheet, the PTX ISA, CUTLASS and the two main open serving engines (Spec Sheets Are Not Kernels, arXiv:2608.11693). The assembler adds two facts: Rubin continues it, and NVIDIA left INT8 out of the portable Blackwell family entirely.

The spec sheets agree. GB300 lists about 0.165 POPS of dense INT8 against 5 PFLOPS of dense FP8, a ratio of 30; a Rubin GPU lists 0.25 POPS against 17.5 PFLOPS, a ratio of 70. On B200 the two were equal.

Consumer Blackwell is a different machine. sm_120a and sm_121a have no tcgen05, but they have warp-level FP4: mma.sync with shape m16n8k64 assembles to OMMA.SF.16864, with UE4M3 scales for NVFP4 through .kind::mxf4nvf4 or E8 scales for MXFP4 through .kind::mxf4.

One more result surprised us: on every datacenter target, Hopper included, the warp-level FP8 instruction mma.sync.m16n8k32 with E4M3 operands assembles to an F2FP unpack to FP16 followed by HMMA.16816, a half-precision multiply.

Only the consumer parts run it natively, as QMMA.16832. On a B200, FP8 tensor throughput is reachable only through tcgen05.

Rubin accepts four instructions that no other target does: conversion to the new UE5M3 scale format (F2FP.SATFINITE.UE5M3), an FP4 conversion that applies a power-of-two scale on the way (F2FP.SATFINITE.E2M1 with a .SCALE_BY_C modifier), FP4 conversion with round-toward-zero, and a matrix multiply that decompresses its weights from a lookup table (UTCQMMA.LUTB). We come back to all four.

Share

K

Spec sheets quote FP4 and FP8 throughput as separate numbers, and the instruction descriptors that CUTLASS builds for each architecture show where the ratio comes from (CUTLASS main branch, October 2026). A dense block-scaled FP4 multiply reduces K = 64 elements per instruction on B200, 96 on B300 and 128 on Rubin.

A dense FP8 multiply reduces 32, 32 and 64. The ratios, 2, 3 and 2, are exactly the ratios of dense FP4 to dense FP8 on the spec sheets: 10 and 5 PFLOPS on GB200, 15 and 5 on GB300, 35 and 17.5 on Rubin (Figure 10).

B300’s extra FP4 throughput is one bit. In the 32-bit Blackwell instruction descriptor for block-scaled FP4, bit 31 selects the K size. A 0 means dense K = 64 or sparse K = 128; a 1 means dense K = 96, with the sparse version marked invalid.

That single comment in mma_sm100_desc.hpp explains why GB300’s dense FP4 rose by half over GB200, from 10 to 15 PFLOPS, while its sparse FP4 stayed at 20: the fast dense mode has no sparse form, so sparse code still runs the K = 128 path both chips share.

Rubin widens the field to two bits, split between bits 3 and 31: dense K of 64, 96 or 128, with sparse forms at 128 and 192 and none for the K = 128 mode. Rubin’s densest sparse mode is therefore 1.5 times its densest dense mode.

NVIDIA’s Rubin architecture overview describes the 50 PFLOPS headline as sparse NVFP4 (as reported by Guru3D, July 2026), and 1.5 times the dense 35 is 52.5, so the descriptor and the spec sheet agree to within 5%. Rubin’s FP4 multiply also requires both operands to be K-major; there is no transposed form.

Subscribe now

Making FP4 on the fly

Weights are quantized once, offline. Activations are quantized at every step by the kernel that produces them, and that costs instructions. We wrote the quantization of one block in PTX the way a fused epilogue would: absolute maximum, scale, scale encoding and decoding, multiply, convert, pack.

We assembled it with ptxas -O3 for four targets and counted SASS instructions, excluding loads, stores, address arithmetic and control flow (Figure 11).

NVFP4 costs 2.38 instructions per element on B200, B300 and Rubin alike. For 16 values that is seven three-input FMNMX3 and one FMNMX for the maximum, 18 FMUL, 10 F2FP conversions, one MUFU reciprocal for the decoded scale and one half-precision add. MXFP4 costs 2.16 per element, because its scale is a power of two and its reciprocal takes three integer instructions.

The two scale-rounding rules compile to the same count. Consumer Blackwell pays 2.81 and 2.59, because it has no three-input FMNMX3 and needs 15 comparisons for 16 values.

Rubin’s fused conversion removes the multiplies for MX formats entirely: 1.19 instructions per element. NVFP4 gets no fused path, because a scale that is not a power of two cannot be applied by adjusting an exponent.

Whether 2.4 instructions per element matter depends on how much tensor-core work each quantized value feeds. A quantized activation is reused for every one of the N output columns of the GEMM that consumes it.

If the quantization cannot overlap with other work, its share of the GEMM’s time is about I x F / (256 x S x f x N), where I is instructions per element, F the dense FP4 rate, S the number of SMs and f the clock, since each SM issues 128 lane-instructions per cycle and each multiply-add counts as two FLOPs.

At an assumed 1.9 GHz that is about 330/N for NVFP4 on B200, 460/N on GB300 and 760/N on Rubin: 4.6%, 6.4% and 10.6% at N = 7168, the hidden size of DeepSeek-V3-class models. Rubin’s fused conversion halves its own figure for MX formats.

Tensor cores have grown faster than the general-purpose ALUs around them, and quantization is one of the places where the difference shows. When AMD profiled MXFP4 and MXFP6 serving on MI355X, the standalone activation-quantization step was measurable overhead, and the team wrote a dedicated kernel to remove it (AMD ROCm blog, June 2026).

Subscribe now

What Rubin adds

The first addition is a scale format. UE5M3 has no sign bit, five exponent bits with a bias of 15 and three mantissa bits; CUTLASS defines its range as 0 to 114,688, with subnormals and a NaN code (include/cutlass/float8.h). It keeps E4M3’s three mantissa bits, so inside a block it behaves like NVFP4’s scale, but it covers about 34 binades against 18.

In Blackwell’s block-scaled instruction descriptor the scale format is one bit, E4M3 or E8M0; in Rubin’s it is two bits, and the value 2 means UE5M3 (mma_sm107_desc.hpp).

Conversions to UE5M3 assemble only for sm_107a, and we assembled both round-to-nearest and round-up forms.

The extra range matters because of NVFP4’s second level. We quantized Gaussian tensors in blocks of 16 FP4 values with no tensor scale at all, sweeping the standard deviation from 10^-8 to 10⁴ (Figure 12).

With UE4M3 block scales the result stays within 0.5 dB of the best only for standard deviations between 0.032 and 562. Weights of large models sit near one over the square root of the hidden size, about 0.011 at a width of 8,192, and a tensor at 0.01 loses 2.4 dB with UE4M3 scales and no tensor scale.

RoBERTa-base, with a width of 768 and a standard deviation of 0.053, happens to sit inside the window. That is why NVFP4 carries its FP32 tensor scale. With UE5M3 the window runs from 10^-4 to at least 10⁴, the end of our sweep.

NVIDIA has not said that UE5M3 exists to retire the second scale, but on these numbers it could, and kernels would save a global reduction and a multiply per tensor. The IF4 authors proposed spending the unused sign bit of the E4M3 scale on a choice between FP4 and INT4 grids; UE5M3 spends that bit on range instead.

The second addition is a weight format with no fixed grid. PTX ISA 9.4 adds a .decompress::lut::b qualifier to tcgen05.mma.

In this mode, according to SemiAnalysis’s description, each weight is a 3-bit index into a table of eight E4M3 values, one table per 8 x 64 tile of the weight matrix, so storage is 3 + 64/512 = 3.125 bits per weight; the table lives in tensor memory and the lookup happens inside the multiply.

In the assembler the instruction exists only for sm_107a and becomes UTCQMMA.LUTB, so the multiply itself runs on the FP8 path. A block-scaled variant adds E8M0 scales for every 32 weights.

We fitted eight-entry codebooks by k-means on each group of 512 values, rounded the entries to E4M3 and measured. On Gaussian data the 3.125-bit format reaches 14.76 dB, above NVFP3 at 3.5 bits (14.31 dB) and slightly above an ideal 3-bit quantizer that knows the variance (14.62 dB).

On RoBERTa’s weights, with codebooks shared over 8 x 64 tiles as in the hardware, the plain format reaches 13.87 dB, 0.34 dB below NVFP3 at 3.5 bits (14.21 dB): real weights change scale from row to row, and one table per tile follows that less well than a scale per 16 values.

The block-scaled variant, at 3.375 bits, recovers the loss and reaches 14.23 dB, just above NVFP3 with an eighth of a bit less. Against NVFP4 the plain format gives up 5.7 dB on Gaussian data and 6.6 dB on RoBERTa for 1.375 fewer bits. At 6.02 dB per bit a fair exchange would cost 8.3 dB, so per stored bit the lookup table is still the more efficient weight format.

It would be the wrong tool for activations, where one table cannot follow the local scale (11.73 dB on our proxy, below NVFP3’s 13.86), and the hardware offers it only for the B operand anyway.

The third addition is an operand type. CUTLASS lists an unsigned 8-bit E5M3 type among the A and B operand types of Rubin’s FP8 multiply (the CuTe DSL tcgen05 tables). An unsigned operand only makes sense for values that are never negative.

In a transformer the obvious candidate is the softmax output that multiplies V in attention, which E4M3 represents poorly below 2^-9. That reading is ours; the code says only that the type exists.

SemiAnalysis also reports that Rubin’s tensor memory grows from 512 to 576 columns, with the extra columns meant for block scale factors, which on Blackwell compete with accumulators for space.

Subscribe now

Who buys which half

FP4 sells two different things. Fewer bytes per weight make decoding faster and models smaller, and that half works whatever the precision of the activations.

Faster arithmetic needs both operands in FP4 and the block-scaled FP4 instructions, and that half needs W4A4. The open models of the past fourteen months bought mostly the first half (Table 1).

FP4 sells two things: fewer bytes and faster arithmetic. Only W4A4 collects both.

OpenAI’s gpt-oss quantizes its MoE weights, more than 90% of the parameters, to MXFP4 at 4.25 bits per parameter (model card, arXiv:2508.10925).

Moonshot moved from INT4 in K2 Thinking to MXFP4 weights with MXFP8 activations in K3, applied with quantization-aware training from the supervised fine-tuning stage onward and a group size of 32 (Kimi K3 model card, July 2026).

On Blackwell that combination runs through .kind::mxf8f6f4 on the FP8 path. On an MI325X, which has no FP4 matrix instructions, vLLM converts K3’s MXFP4 experts to group-32 INT4 at load time and serves them with BF16-by-INT4 kernels (Shakudo, September 2026).

DeepSeek V4 is the most careful case. Its report applies MXFP4 quantization-aware training to the MoE experts and to the query-key path of the attention indexer, then dequantizes the expert weights to FP8 for computation, and notes that this is lossless as long as the ratio between the largest and smallest FP4 sub-block scale inside each 128 x 128 FP8 block stays under a threshold (arXiv:2606.19348). The report does not state the threshold, so we enumerated it.

Every E2M1 value times 2^d is exactly representable in E4M3 if and only if d lies between -8 and 6, so the sub-block exponents inside a tile may span 14 binades, a ratio of 16,384, or 11 binades if FP8 subnormals are to be avoided.

Gaussian weights with log-normal row scales span 4.6 binades on average and 6 at most in our simulation, so the condition is loose and V4’s empirical check unsurprising.

NVIDIA’s own NVFP4 checkpoints are the exception. They quantize activations too, by post-training quantization, and NVIDIA reports DeepSeek-R1-0528 within one point of FP8 on six of seven benchmarks, with AIME 2024 two points higher.

AMD’s measurements show what the second half is worth when it is bought without help. On one MI355X, W4A4 MXFP4 delivered 5.1% more total tokens per second than FP8 on Llama-3.1-8B and 10.0% more on Qwen3.6-27B, from twice the peak arithmetic.

If only GEMM time halved, those speedups imply that GEMMs were 9.6% and 18.2% of the FP8 step, which says more about the rest of the step than about FP4.

Accuracy moved the other way: GSM8K on Llama-3.1-8B fell from 80.44 with FP8 to 62.55 with MXFP4. MXFP6 activations with MXFP4 weights recovered 76.42 at 2.8% less throughput than MXFP4, because MI355X runs FP6 at the FP4 rate.

NVIDIA runs FP6 at the FP8 rate, so that particular trade does not exist on Blackwell.

Subscribe now

The critical batch

A decode step reads every active weight once and performs two FLOPs per weight for each sequence in the batch. Below a critical batch B* = P x b / (2 x BW), where P is the dense FLOP rate, b the bytes per weight and BW the memory bandwidth, the step is limited by bandwidth; above it, by arithmetic.

Earlier pieces in this publication found B* nearly independent of precision, because P doubles when b halves. Block scales and Blackwell Ultra both break that (Figure 13).

Scales first. NVFP4 weights cost 0.5625 bytes, not 0.5, so on GB200 B* rises from 312.5 with FP8 to 351.6 with W4A4 NVFP4, 12.5% higher purely from scale bytes; MXFP4’s smaller overhead gives 332.0.

Then GB300: dense FP4 is three times dense FP8 there, so W4A4 NVFP4 pushes B* to 527.3, 1.69 times the FP8 value. The FP4-weight, FP8-arithmetic schemes go the other way.

Halving the bytes without speeding up the arithmetic roughly halves B*, to 166 on GB200 and GB300 and 211 on Rubin, and Rubin’s lookup-table weights take it to 155. Those formats make small batches much cheaper and become compute-bound at about half the batch. W4A4 keeps large batches cheap, and on GB300 it is worth three times FP8 arithmetic.

At batch 1, weight traffic alone caps decoding of a 70B dense model at 57 tokens per second in BF16 on 8 TB/s of HBM, 114 in FP8, 203 in NVFP4 and 215 in MXFP4; on Rubin’s 22 TB/s, at 559 in NVFP4 and 805 with lookup-table weights. Attention reads and synchronization come on top of these ceilings.

Subscribe now

What to use where

The measurements reduce to a handful of situations (Table 2). The table is our reading of the evidence in this piece, not a benchmark of anyone’s serving stack, and it applies to post-training quantization: models trained with quantization in the loop have already chosen their grid.

Subscribe now

Where this could be wrong

• Synthetic data. Every SQNR here except RoBERTa’s comes from synthetic distributions. The activation proxy matches AMD’s three numbers, but those are layer means over one tensor type in two models; attention inputs, residual streams and gradients have other shapes.

• One family, one recipe. The perplexity law is fitted on Qwen3.5 with round-to-nearest W4A4, where weight and activation noise mix and our single proxy cannot separate them. Models trained with quantization in the loop, which includes gpt-oss, DeepSeek V4 and Kimi K3, learn around their grid, and the law should not be applied to them. The constant also moves between models: AMD’s numbers put it near 8.5 for Llama-3.1-8B and near 2.8 for Qwen3.6-27B. The rank correlation of 0.986 is the more robust number.

• Rates from names. We infer that FP4 through .kind::f8f6f4 runs at FP8 speed from the opcode it becomes and from NVIDIA’s documentation, not from a timed kernel. The K values come from CUTLASS source, not from hardware, and a descriptor comment is a strong hint rather than a measurement.

• Our codebooks. The lookup-table numbers depend on plain k-means with E4M3 entries. NVIDIA’s own fitting, or quantization-aware training, would do better, so our LUT figures are a floor.

• Assumed clocks. The quantization-overhead estimate assumes 1.9 GHz, NVIDIA’s listed SM counts (148 for B200, 160 for B300, 224 for Rubin) and no overlap with other work. Real epilogues hide part of the cost.

• Intent. UE5M3 without a tensor scale is our reading of what the format allows, not NVIDIA’s stated purpose, and the softmax use of the unsigned E5M3 operand is a guess from the type’s existence.

• Moving spec sheets. Rubin’s numbers come from announcements and partner documents months before broad availability, and NVIDIA’s own figures for B300 range from 13 to 15 PFLOPS of dense FP4 depending on the system.

Share

Predictions

1. By 30 June 2027, at least one open-weight model above 100 billion parameters will ship its primary checkpoint in a format only Rubin runs natively: FP4 with UE5M3 scales, or 3-bit lookup-table weights.

2. Through the first PTX ISA release that adds Feynman (sm_140), tcgen05.mma.kind::i8 will not assemble for sm_103a or sm_107a.

3. By 31 March 2027, vLLM or SGLang will merge a Rubin MoE kernel that reads expert weights through .decompress::lut::b.

4. Moonshot’s next flagship after K3 will again pair 4-bit weights with 8-bit activations rather than move to W4A4.

5. By 31 December 2027, a revision of the OCP Microscaling specification will add a non-power-of-two scale type or change the default rounding of E8M0 scales.

Subscribe now

Confidence dossier

Every number in the text carries a tier. A: primary source (vendor documentation, archived paper, model card). B: vendor source code or reputable secondary reporting. C: our derivation from A or B sources, with stated assumptions. D: our inference. M: our own measurement.

Share

Sources

1. Open Compute Project, OCP Microscaling Formats (MX) Specification v1.0, September 2023.

2. B. Darvish Rouhani et al., Microscaling Data Formats for Deep Learning, arXiv:2310.10537.

3. E. Alvarez et al., NVIDIA, Introducing NVFP4 for Efficient and Accurate Low-Precision Inference, 24 June 2025.

4. NVIDIA, Pretraining Large Language Models with NVFP4, arXiv:2509.25149, v2, 4 March 2026.

5. A. Mishra, D. Stosic, S. Layton, P. Micikevicius, Recipes for Pre-training LLMs with MXFP8, arXiv:2506.08027, v2, 18 August 2025.

6. V. Egiazarian et al., Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization, ICLR 2026, arXiv:2509.23202.

7. J. Cook et al., Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling, arXiv:2512.02010, v5, 9 May 2026.

8. J. Cook et al., Adaptive Block-Scaled Data Types, arXiv:2603.28765, 30 March 2026.

9. S. Atre, B. Bao, S. Tiwari, A. Sirasao, AMD, MXFP6 and MXFP4 Mixed Precision for Accelerating Dense LLMs on AMD Instinct MI355X, 26 June 2026.

10. T.-R. Chen, Spec Sheets Are Not Kernels: An ISA- and Source-Level Audit of INT8 Availability on NVIDIA Blackwell Ultra, arXiv:2608.11693, August 2026.

11. NVIDIA, Inside NVIDIA Blackwell Ultra: The Chip Powering the AI Factory Era, NVIDIA technical blog.

12. NVIDIA, HGX platform specifications (HGX B300 and HGX B200).

13. Verda, GB300 NVL72 architecture (per-GPU dense FP4, FP8 and INT8 for GB300, GB200, B300, B200).

14. The Register, Nvidia unpacks its Vera Rubin CPUs and GPUs at CES, 5 January 2026.

15. NVIDIA, 2026 1H product guide (Vera Rubin NVL72 dense specifications), distributed by Gigabyte.

16. Guru3D, NVIDIA details Rubin GPU architecture with HBM4, NVLink 6 and 336 billion transistors, July 2026.

17. SemiAnalysis, Vera Rubin NVL72 vs GB200 NVL72? Inference TCO and Architecture Analysis, 23 July 2026.

18. NVIDIA, PTX ISA 9.4, CUDA 13.4.

19. NVIDIA, TensorRT 11.3.0 release notes (Vera Rubin, compute capability 10.7).

20. NVIDIA, CUTLASS, main branch, October 2026: include/cute/arch/mma_sm100_desc.hpp, mma_sm107_desc.hpp, mma_sm107_umma.hpp, include/cutlass/float8.h, python/CuTeDSL/cutlass/cute/nvgpu/tcgen05/mma.py.

21. Apache TVM, PTX instruction table and pull request 20271, September 2026.

22. OpenAI, gpt-oss-120b and gpt-oss-20b Model Card, arXiv:2508.10925.

23. DeepSeek-AI, DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence, arXiv:2606.19348.

24. vLLM, DeepSeek V4 model documentation (MXFP4 experts with UE8M0 scales).

25. Moonshot AI, Kimi K3 model card, as reported by SemiAnalysis InferenceX and the VESSL W4AFP8 card.

26. LMSYS, INT4 quantization-aware training, 26 January 2026.

27. Shakudo, How We Deployed Kimi K3 on AMD GPUs, updated 4 September 2026.

28. heise online, AMD Instinct MI350X and MI355X (FP6 and FP4 at twice the FP8 rate).

29. S. Lloyd, Least Squares Quantization in PCM, IEEE Transactions on Information Theory, 1982; J. Max, Quantizing for Minimum Distortion, IRE Transactions on Information Theory, 1960.

30. Two-level MXFP8 training with an FP32 tensor scale, arXiv:2511.05811, November 2025.

31. Y. Liu et al., RoBERTa, arXiv:1907.11692; Explosion, en_core_web_trf 3.8.0.

32. S. Ashkboos et al., QuaRot, arXiv:2404.00456; Z. Liu et al., SpinQuant, arXiv:2405.16406.

Subscribe now

Corrections

Instruction count. An early count of the NVFP4 quantization cost, 2.31 instructions per element, left out a half-precision add and the byte permutes. The counting rule used here, every instruction except memory access, address arithmetic and control flow, gives 2.38.

Codebook tiles. Our first pass over real weights shared lookup-table codebooks across 512 consecutive values in memory rather than across 8 x 64 tiles of the matrix, as the hardware does. The figures here use tiles.

Weight scale window. A draft said typical weights fall outside the no-tensor-scale window of UE4M3 while quoting RoBERTa’s standard deviation of 0.053, which is inside it. The text now uses the scale of large-model weights, about 0.01, where the loss is 2.4 dB.

Format count. A draft said the perplexity table holds 22 formats and that the fit starts at 3.25 bits. The table holds 23, we simulated 22, and the fit covers 3.5 to 6.5 bits.

Error message. A draft quoted an assembler error message from memory. It was removed; the text now describes the rejection without quoting it.

AMD quantization kernel. The previous version removed a statement that AMD wrote a dedicated activation-quantization kernel, because we could not re-verify it at the time. AMD’s post says exactly that, and the statement is back with its source.

INT8 figures. The previous version cited a secondary page for B300’s INT8 throughput that, on re-reading, lists INT8 at the FP8 rate. The figures now come from Chen’s audit and from per-GPU tables that match NVIDIA’s datasheet.

Kimi K2 Thinking. The previous version gave the scale type of Kimi K2 Thinking’s INT4 weights as BF16. Our sources confirm the grouping of 32 and the 16-bit activations, not the scale type, and the text now says so.

Subscribe now

Appendix: reproducing the measurements

The assembler and disassembler come from NVIDIA’s PyPI wheels and run on any x86 Linux machine without a GPU.

pip download --no-deps nvidia-cuda-nvcc nvidia-cuda-nvdisasm      # 13.4.92
python3 -m zipfile -e nvidia_cuda_nvcc-13.4.92-*.whl cuda/
python3 -m zipfile -e nvidia_cuda_nvdisasm-13.4.92-*.whl cuda/
chmod +x cuda/nvidia/cu13/bin/*
cuda/nvidia/cu13/bin/ptxas -arch=sm_107a -o k.cubin k.ptx
cuda/nvidia/cu13/bin/nvdisasm -c k.cubin | grep -E "UTC|F2FP|OMMA|QMMA"

A probe is one instruction inside a minimal kernel. This one asks for an NVFP4 block-scaled multiply; changing the target line and the qualifiers gives every cell of Figure 9.

.version 9.4
.target sm_107a
.address_size 64
.visible .entry k(.param .u64 out, .param .u64 in, .param .u32 idesc)
{
  .reg .b32 %r<8>; .reg .b64 %rd<8>; .reg .pred %p<2>;
  ld.param.u64 %rd1, [out]; ld.param.u64 %rd2, [in]; ld.param.u32 %r1, [idesc];
  ld.global.u32 %r2, [%rd2]; ld.global.u32 %r3, [%rd2+4];
  ld.global.u64 %rd3, [%rd2+8]; ld.global.u64 %rd4, [%rd2+16];
  setp.ne.u32 %p1, %r1, 0;
  tcgen05.mma.cta_group::1.kind::mxf4nvf4.block_scale.block16
      [%r2], %rd3, %rd4, %r1, [%r3], [%r3], %p1;
  ret;
}

The reference semantics of NVFP4 fit in a few lines of NumPy. Every other format in this piece changes the element grid, the block size or the way the scale is rounded.

import numpy as np
E2M1 = np.array([0, .5, 1, 1.5, 2, 3, 4, 6])
E4M3 = np.array(sorted({m / 8 * 2.0**-6 for m in range(8)} |
                       {(1 + m / 8) * 2.0**(e - 7) for e in range(1, 16) for m in range(8)}))[:-1]

def rne(a, grid):                      # nearest grid point, ties to even code, saturating
    i = np.clip(np.searchsorted(grid, a), 1, len(grid) - 1)
    lo, hi = grid[i - 1], grid[i]
    up = (hi - a < a - lo) | ((hi - a == a - lo) & (i % 2 == 0))
    return np.where(a >= grid[-1], grid[-1], np.where(up, hi, lo))

def nvfp4(x, block=16):
    g = np.abs(x).max() / (6 * 448)                                   # FP32 tensor scale
    xb = x.reshape(-1, block)
    s = rne(np.abs(xb).max(1, keepdims=True) / (6 * g), E4M3) * g     # E4M3 block scale
    s1 = np.where(s > 0, s, 1.0)
    return (np.sign(xb) * rne(np.abs(xb) / s1, E2M1) * s).reshape(x.shape)

Introduction

Every coding agent works the same way underneath. At each step it sends the model everything it has so far, the system prompt, the tool definitions, every file it opened and every command it ran, followed by one new tool result, and the model reads all of it to choose one small next action.

In the session we price later in this piece, the context starts at 12,000 tokens and ends at 85,500 after fifty steps, and the model is sent 2.44 million input tokens in order to write 20,000.

Almost no one of that is billed at full price. The provider has seen those tokens before, has kept what it computed the first time, and charges the repeat at a fraction of the normal rate. On Anthropic’s current price list the fraction is 10 percent for most models, 5 percent for Opus 5.5 and 2.5 percent for Fable 5.1 and Mythos 5.1. On DeepSeek it is 2 percent.

The discount comes with a clock: five minutes, thirty minutes, an hour or a day, depending on the provider and on what you pay.

I wanted to know where those tokens go between two calls, who pays to keep them, and why the numbers are what they are. Why a tenth and not a half. Why five minutes and not five hours. Why DeepSeek can charge a fiftieth while Google charges rent by the hour.

We ended up with one ratio that answers most of these questions, a replay of public production traces that answers most of the rest, and a view of where inference hardware is heading that neither of us expected when we started. It also produced a handful of rules that, in the agent session we model, cut the bill in half.

Subscribe now

The short version

1. What is cached. Not your text: the key and value tensors every attention layer computes for every token. Depending on the architecture that is 890 bytes to 512 KiB per token, so a 100,000-token agent context occupies anywhere from 89 MB to 52 GB.

2. Why a read costs a tenth. A hit costs the provider a reload and some rent, not a recompute, and on an H100 even a single SSD returns cache faster than the GPU can regenerate it for every model here released since 2024. How cheap a read can be depends on how much compute each stored byte saves, which varies 700-fold across the models we measured; DeepSeek’s V4.1-Flash saves the most per byte and charges 2 percent.

3. Why five minutes. GPU memory is the most expensive place to keep anything: an H100 fills its own HBM with fresh cache in about a minute on a 70B-class model, host memory pays for roughly an hour and SSD for days. Five minutes is also where most reuse happens; in Kimi’s production traces it captures 79 percent of the possible reuse for chat and 93 percent for agents.

4. Why writes cost extra. Keeping bytes is rent, but only for entries nobody reads again, because every read refreshes the lifetime for free. Converted to dollars per gigabyte-hour, most providers price that rent like GPU memory, far above what DRAM costs.

5. What to do about it. Caching everything pays only if more than 22 percent of the tokens that pass through the cache are hits (53 percent with the one-hour lifetime). Gaps of 5 to 36 minutes are cheapest to cross with a zero-output ping every four and a half minutes, longer ones with the one-hour lifetime. In our fifty-step agent session with nine long tool calls, that is the difference between $1.88 and $0.97.

6. Where it is going. A Rubin GPU fills its own memory with fresh cache 2.4 times faster than a B200. That is why NVIDIA is adding petabytes of flash to every pod and why DeepSeek shrank its cache 437-fold in three years.

Subscribe now

The price list in September 2026

Four providers sell the same physical thing in three different ways.

Anthropic charges a premium to write the cache and a small fee to read it. A write costs 1.25 times the base input price if the entry should live five minutes and 2 times if it should live an hour. A read costs 0.1 times the input price on most models, 0.05 on Opus 5.5 and 0.025 on Fable 5.1 and Mythos 5.1^.

Every read refreshes the lifetime at no charge, and the clock starts when the request that wrote or read the entry begins, not when its response finishes, so a response that streams for four minutes leaves about one minute for the follow-up. Anthropic also states that the cached key and value tensors are held in memory only and never stored at rest.

OpenAI has moved to the same structure. From GPT-5.6 onward a write costs 1.25 times the uncached input rate, a read 0.1 times, and an entry stays eligible for at least thirty minutes after its latest write or reuse^.

Earlier models charged nothing to write and kept entries in what OpenAI describes as volatile GPU memory for roughly five to ten minutes of inactivity; an extended policy kept them for up to 24 hours by offloading the key and value tensors to storage local to the GPU once memory filled up. The two companies differ in one detail that matters for agents at scale: Anthropic does not count cache hits against rate limits, and OpenAI does.

Google separates reading from keeping. Reading an explicit Gemini cache costs a tenth of the input price, and keeping the cache alive costs rent: $4.50 per million tokens per hour on Gemini 3.1 Pro and $0.50 on Gemini 3.8 Flash, a rate the price page schedules to double on January 1, 2027.

Gemini also caches without being asked: according to Google’s caching guide, implicit caching is on by default for Gemini 2.5 and newer models, discounting repeated prefixes of at least 4,096 tokens on the current 3.x models, with no storage fee and no guarantee of a hit.

DeepSeek charges for neither writing nor keeping. A hit on its current flash model costs $0.003 per million tokens off-peak against $0.15 for a miss, both doubling during two weekday windows^. DeepSeek has offered context caching since August 2024 and serves its hits from a cache on disk.

Read one way, the market has converged: a read at a tenth of the input price is the default at Anthropic, OpenAI and Google. Read another way, it has already broken below that point, with the newest models from Anthropic and DeepSeek at 5, 2.5 and 2 percent. The charges for writing and keeping have not converged at all. Both patterns come from what the provider is physically holding.

Subscribe now

What the provider keeps

A cached prompt is not text. When a transformer processes a token, every attention layer computes a key vector and a value vector for it, and every later token in the sequence reads them. Keeping those vectors lets the next request with the same beginning skip computing them again. OpenAI’s documentation says it directly: the cache holds key and value tensors, not the tokens themselves.

How many bytes that is depends on the attention design, and the range is wide. With grouped-query attention, the design used by Llama 3, Qwen3, GLM and MiniMax, every token costs two vectors per layer per key-value head:

bytes per token = 2 × layers × kv_heads × head_dim × bytes per element
Llama-3.1-70B:    2 × 80 × 8 × 128 × 2 (BF16) = 327,680 bytes = 320 KiB

A 100,000-token agent context for that model is 32.8 GB, two fifths of an H100’s memory for a single conversation. Multi-head latent attention, which DeepSeek introduced in V2 and Moonshot reused in Kimi K2, stores one compressed latent per layer instead of per-head keys and values: 512 values plus 64 for position.

In BF16 that is 1,152 bytes per layer and 70,272 bytes per token across DeepSeek-V3’s 61 layers, so the same 100,000 tokens take 7.0 GB. DeepSeek-V3.2 adds a small FP8 key per token for its sparse-attention indexer but stores the latent itself in FP8, which by our layout accounting comes to 48,068 bytes.

The newest designs compress across tokens as well. DeepSeek’s V4 family keeps compressed and heavily compressed attention caches, about 3,600 bytes per token for V4-Flash by our reconstruction in August, and DeepSeek’s own report puts V4-Flash at 7 percent of V3.2’s cache at a million tokens.

V4.1-Flash, released on September 10, goes further: its decoder layers take their global cache from a projection of the final encoder state instead of computing their own, and the cache is stored in FP4 with one scale per 16 channels. The model card gives 890 bytes per token. A 100,000-token context is 89 MB.

Two things in that chart are easy to miss. The first is that cache size does not follow model size. Llama-2-7B, with plain multi-head attention, keeps 512 KiB per token, more than Llama-3.1-70B; MiniMax-M2, with 10 billion active parameters, keeps 248 KiB because all 62 of its layers use full attention with eight key-value heads; GLM-4.6 keeps 368 KiB across 92 layers.

The second is how far one lab has pushed. DeepSeek’s first model, the 67B release of late 2023, kept 389,120 bytes per token: 95 layers, eight key-value heads of dimension 128.

Divide by 890 and the result is 437.2, which is the 437-fold reduction DeepSeek reports in the V4.1 model card. We computed the V1 figure from its architecture and got the same ratio, a useful check on both numbers.

Sliding windows and linear attention trade differently. gpt-oss-120b keeps full history in only 18 of its 36 layers and a 128-token window in the rest; Qwen3-Next keeps full attention in 12 of 48 layers and a fixed-size recurrent state in the others.

Both shrink the per-token cache, and both add a per-sequence state that can only be reused at the exact position where it was saved, which makes prefix matching harder than it is for a paged cache.

Subscribe now

Compute saved per byte

The size alone does not say whether keeping the cache is worth anything. What matters is the work the bytes save. Recomputing a token’s cache means running prefill for it again, which costs at least two floating-point operations per active parameter. Call that F and divide it by the bytes kept:

ρ = F / k        prefill FLOPs avoided per byte of cache kept
F = 2 × active parameters in prefill      (a floor: attention adds more at long context)

For Llama-3.1-70B, F is 141 GFLOP per token and k is 327,680 bytes, so ρ is 431 thousand: every byte kept saves 431 thousand operations. DeepSeek-V3 is at 1.05 million, V4-Flash at 7.2 million, and V4.1-Flash, whose prefill runs through only 8 billion active parameters, at 18 million.

At the bottom are Llama-2-7B at 26 thousand and MiniMax-M2 at 79 thousand. Across the fifteen models in this piece ρ spans a factor of about 700.

Mixture-of-experts made caching relatively less valuable before DeepSeek made it more valuable. Sparsity cuts compute per token without touching attention, so a model like Qwen3-235B-A22B or MiniMax-M2 carries the cache of a large dense model and the prefill cost of a small one.

DeepSeek compressed attention in the same generations in which it sparsified the experts, and every release since V2 sits further toward the upper left of the chart.

Our F is a floor. With dense attention, recomputing a long prefix also pays for attention itself, which grows with position. Averaged over a 64,000-token prefix, attention adds about 84 GFLOP per token to Llama-3.1-70B’s 141, and about 160 GFLOP to DeepSeek-V3’s 74 when V3 runs prefill without absorbing its projections.

At agent context lengths the real ρ of dense-attention models is 1.6 to 3.2 times the floors we use, so every conclusion below gets stronger when those terms are added back.

Subscribe now

How fast a GPU really writes cache

Put ρ next to a GPU and you get the quantity that ties the rest of the piece together. A GPU running prefill at peak throughput P and utilization u produces fresh cache at the rate

G = P × u / ρ        bytes of new cache per second

We use dense FP8 peak for P, 1,979 TFLOPS on an H100 and 4.5 PFLOPS on a B200, and u = 0.30. That is at the optimistic end of what has been published, so it is worth calibrating. SGLang’s open reproduction of DeepSeek’s serving system processed 52,300 input tokens per second per eight-GPU H100 node on 2,000-token prompts, which is about 0.26 of the FP8 peak once attention is counted.

On GB200 the same team reached 18,471 input tokens per second per GPU with FP8 experts, 0.29 of dense FP8 peak, and 26,156 with NVFP4 experts. DeepSeek’s own production prefill nodes, averaged over a day of real traffic, processed 73,700 input tokens per second per node including cache hits; removing the 56.3 percent that were hits leaves about 4,000 computed tokens per second per H800, roughly 0.16 of peak.

We keep 0.30 as the central value and carry 0.15 to 0.5 in every range below. On an H100 at our assumption, Llama-3.1-70B writes 1.38 GB of cache per second, DeepSeek-V3 0.56 GB, V4-Flash 83 MB and V4.1-Flash 33 MB.

G has one property worth stating separately. Move a model’s arithmetic from BF16 to FP8 and P doubles; store its cache in FP8 as well and k halves; G does not move.

Quantizing both sides together leaves the rate at which a GPU fills memory where it was, the same kind of precision invariance we actually found for the critical batch size in August, seen from the memory side. Quantizing only the arithmetic doubles it.

G answers three different questions. It is the break-even bandwidth for reloading, because any link faster than G delivers cached bytes faster than the GPU could regenerate them.

It sets the capacity bill for a lifetime, because holding everything a GPU produces for τ seconds takes G × τ bytes. And it sets the clock for the GPU’s own memory, because HBM capacity divided by G is how long the GPU takes to fill its HBM with fresh cache.

Reload or recompute

The first question has a lopsided answer. At 1.38 GB/s, a single Gen5 NVMe drive, which reads at about 14 GB/s, already returns Llama-3.1-70B’s cache faster than an H100 can recompute it, and every link above it in the hierarchy is faster still.

For the MLA and compressed models the margin over a single drive is 25 to 420 times. Only models at the bottom of the ρ scale need more than a drive: MiniMax-M2 produces 7.5 GB/s of cache on an H100 and about 67 GB/s on a Rubin GPU, a full PCIe Gen5 x16 link.

The latency tells the same story. Reloading a 100,000-token Llama-3.1-70B context, 32.8 GB, over a Gen5 x16 link at an effective 55 GB/s takes circa 0.6 seconds; recomputing it on eight H100s at our utilization takes about 3 seconds before attention is counted.

For V4.1-Flash the context is 89 MB and the reload takes a couple of milliseconds. Recomputation wins only where compute per byte is low and prefixes are short. A September characterization of SSD-backed caching for vLLM put the break-even at about 6,200 tokens for Qwen3-4B on its H100 setup.

Hiding the reload

A reload does not have to be waited on. The engine can load layer l + 1 of the cached prefix while it computes layer l of the new tokens, which is how the layer-wise pipelines in Mooncake and the systems that followed it work.

The reload disappears behind the computation when the link moves the cached bytes in no more time than the GPU spends on the new ones:

L_cached × k / W  ≤  L_new × F / (P × u)      which gives      W ≥ (L_cached / L_new) × G

This is where agents are the hard case. A late agent step carries a long cached prefix and a short new suffix: 84,000 tokens cached and 1,500 new in the last step of our session, a ratio of 56.

Hiding that reload for Llama-3.1-70B on an H100 needs 77 GB/s, more than a Gen5 x16 link delivers, although attention over the long prefix adds compute to the new tokens and cuts the requirement by more than half.

DeepSeek-V3 needs 32 GB/s and V4.1-Flash under 2 GB/s. The ratio of cached to new tokens is exactly what an agent pushes up, which is why coherent CPU links and small caches matter more for agents than for chat.

Share The Software Frontier

The clock

Reloading is relatively cheap. Keeping is certainly not free, and it is paid for by the second. An entry is worth keeping as long as the expected saving from a future hit exceeds the rent. Recomputing one byte of cache costs the GPU’s price per second divided by G, and holding one byte costs the tier’s price per byte-second.

The holding time at which the two are equal is the break-even:

τ* = (GPU cost per second) / (G × s)       s = holding cost of the tier per byte-second
τ*_HBM = H / G                             when HBM is priced at its share of the GPU

The HBM line needs one assumption. When serving is limited by memory, the normal state of a decode-heavy fleet, every gigabyte of HBM holding an idle cache entry is a gigabyte not holding an active request, so its price is its share of the GPU’s rent.

With that price the break-even reduces to something physical: the time the GPU takes to fill its own HBM with fresh cache. On an H100 serving Llama-3.1-70B that is 58 seconds. H/G is an upper bound, because part of HBM holds the weights: about 11 percent for Llama-3.1-70B spread over eight H100s, about 60 percent for a 671-billion-parameter FP8 model on eight H200s. The break-even shrinks in the same proportion.

For the other tiers we priced memory at 2026 levels, which are not those of a year ago. TrendForce expected conventional DRAM contract prices to rise 55 to 60 percent in the first quarter alone; by late August a 64 GB DDR5 server module cost 2.5 to 3 times its January price and a 3.84 TB enterprise NVMe drive 2.3 to 2.8 times, and TrendForce still expected both to rise in the third quarter.

We tend to assume $12 per GB of DRAM over four years, with 50 percent added for power, space and the CPU socket it hangs from, and $0.25 per GB of SSD over five years, doubled for the servers and network around it.

For the GPU we use an H100 at $2.50 an hour, inside the range specialist clouds charge in 2026.

Per byte-hour, HBM, DRAM and SSD come out at roughly 2,700 to 45 to 1. Each step down the ladder buys 45 to 60 times more holding time for the same money.

For Llama-3.1-70B the ladder reads about a minute in HBM, about an hour in DRAM and about two days on SSD. For DeepSeek-V3 it is two and a half minutes, two and a half hours and four and a half days. For V4.1-Flash it is 40 minutes, 41 hours and 77 days. MiniMax-M2 gets 11 seconds, 11 minutes and 8 hours.

Not a single one of these numbers is totally accurate. Across the ranges we consider plausible (utilization from 0.15, what DeepSeek’s production fleet achieves, to 0.5, an H100 from $1.50 to $4.00 an hour, the low and high ends of 2026 memory quotes, and 60 to 100 percent of HBM available to cache), the HBM break-evens move by a factor of 0.36 to 2.0, the DRAM ones by 0.2 to 5.8 and the SSD ones by 0.17 to 7.1. The whiskers in the chart show those ranges; the order of the tiers and of the models never changes.

Now lay the product menu over it. Five minutes is past the HBM break-even of every grouped-query model in our set even at the top of its range, and inside the DRAM break-even of all of them at our central assumptions, which says a five-minute cache for those models belongs in host memory rather than on the GPU.

One hour sits at the DRAM break-even of a 70B-class grouped-query model: 59 minutes at the center, between 12 minutes and almost 6 hours across the ranges. At the center, twenty-four hours is past the DRAM break-even of every model here except V4.1-Flash, and pays on SSD only for models above roughly 230 thousand FLOPs per byte, which leaves out most of the sparse grouped-query models.

The providers’ own descriptions match the ladder: OpenAI keeps short-lived entries in GPU memory and moves to local storage for the 24-hour policy, DeepSeek serves hits from disk, and Google charges rent for anything kept on request.

These are break-evens for an entry that is certain to be read again. With a probability p of reuse, each one shrinks by p. The probabilities are what the traces are for.

Share

What the traces say

Moonshot published request traces from Kimi’s serving system with the Mooncake paper, which won the best paper award at FAST 2025. Each request carries an arrival time, its input and output lengths, and one hash per 512-token block, remapped so that equal hashes mean reusable cache.

The conversation trace holds 12,031 requests and the tool-and-agent trace 23,608, both sampled from one hour of production traffic. Average inputs are 12,035 and 8,596 tokens and average outputs 343 and 182, ratios of 35 and 47 to one.

We replayed both traces block by block against a cache with unlimited space whose entries expire a fixed time after their last use. With no expiry at all, 37.4 percent of the conversation trace’s input tokens and 57.1 percent of the agent trace’s could have come from cache. Those ceilings sit where others have found them.

Alibaba’s study of its production traces found ideal hit ratios of 62 and 54 percent over a day, and in its API trace single-turn requests produced 97 percent of the hits. DeepSeek reported that 342 billion of the 608 billion input tokens it served in one day of February 2025, 56.3 percent, hit its on-disk cache.

A five-minute lifetime captures 79 percent of the ideal reuse in the conversation trace and 93 percent in the agent trace. Ten minutes captures 94 and 99 percent. Thirty minutes captures essentially all of it. Most of the value is in the first five minutes, and by thirty there is little left to buy.

The gaps between reuses show where the two workloads differ. In the agent trace about half of all reused tokens were reused within one second of their previous use (52 percent over the whole trace, 50 percent in the edge-corrected sample below): requests arriving together and sharing a long prefix, the signature of an agent fanning out parallel calls.

In the conversation trace that share is about 10 percent, and the median gap is about two minutes, roughly the time a person takes to read an answer and type the next question.

Fan-out is also why Anthropic’s documentation notes that an entry becomes available only after the first response starts, and suggests waiting for it before sending parallel requests. Ten identical prefixes sent in the same second are ten cache writes, not one write and nine reads.

The trace covers one hour, which biases every gap statistic toward short gaps, so for the chart above we counted only blocks whose previous use came at least thirty minutes before the end. Among those, 75 percent of the conversation reuse and 91 percent of the agent reuse that happens within half an hour happens within the first five minutes.

Capacity is the other half of the question. An LRU cache sized to hold ten minutes of the trace’s unique traffic captures 89 percent of the ideal reuse for conversations and 98 percent for agents, and twenty minutes captures 98 percent of the conversation ideal.

The unique traffic arriving at a prefill GPU is exactly what that GPU computes, which is G. So the capacity a ten-minute window needs per GPU, with prefill running flat out, is G × 600 seconds: 827 GB for Llama-3.1-70B on an H100, 338 GB for DeepSeek-V3 and 20 GB for V4.1-Flash.

An H100 has 80 GB of HBM, and an eight-GPU DGX H100 has 2 TB of system memory, about 256 GB per GPU. A ten-minute window for a 70B grouped-query model fits in neither, and has to live on SSD or in a pool shared across machines. For DeepSeek-V3 it spills just past DRAM. For V4.1-Flash it fits in HBM next to the weights. That comparison goes a long way toward explaining why DeepSeek can nearly give reads away and charge nothing for keeping.

It also explains why nobody lets you write everything for free. If every miss in the conversation trace were written at 1.25 times the input price and every hit read at 0.1, a five-minute cache would cut the input bill by 9 percent; on the agent trace the cut would be 36 percent.

The margin is in what gets written, which is why both Anthropic and OpenAI let callers place breakpoints, and why OpenAI’s new explicit mode charges no write for anything after the last breakpoint.

Leave a comment

Where the bytes live

NVIDIA describes the hierarchy its Dynamo framework manages in four levels: G1 is GPU memory for the cache of requests in flight, G2 is host DRAM for staging, G3 is SSD inside a node and G4 is shared storage for what has to survive.

At CES in January it added a level between the last two. The CMX context memory storage platform, first shown as ICMS, puts Ethernet-attached flash managed by BlueField-4 processors into each pod, with petabytes of shared capacity, and NVIDIA claims five times the tokens per second and five times the power efficiency of general-purpose storage for this job. Its own description of the data is the best argument for a separate tier: transient, derived, and recomputable if lost.

The rest of the rack is moving the same way. A GB200 superchip pairs two Blackwell GPUs with a Grace CPU carrying up to 480 GB of LPDDR5X. A Vera Rubin NVL72 rack carries 54 TB of LPDDR5X, 1.5 TB per Vera CPU and 750 GB per GPU, reachable over a coherent NVLink-C2C link of 1.8 TB/s.

The two labs that price hits lowest built their own versions of this years ago. Mooncake pools the DRAM and SSD of Kimi’s GPU servers into one distributed cache; the paper reports more than 100 billion tokens a day on thousands of nodes, and 115 and 107 percent more requests on A800 and H800 clusters than the previous system.

DeepSeek’s 3FS file system, built on SSDs and RDMA, reached 6.6 TiB/s of aggregate reads on a 180-node cluster, and its README shows cache reads for inference peaking at 40 GiB/s.

Flash has one limit that bandwidth arithmetic hides, which is wear. A prefill GPU that persisted everything it computed would write G bytes per second to flash, all day. For Llama-3.1-70B on an H100 that is 119 TB a day; a 30 TB drive rated for one full write per day absorbs 30. Endurance alone would call for four drives per GPU before capacity or bandwidth enter the picture.

For DeepSeek-V3 the figure is 49 TB a day, for V4.1-Flash under 3 TB. At these write rates, admission control, writing only the blocks likely to be reused, decides whether the tier outlives its warranty.

The V4.1 model card makes the same point from the model side: its bounded replay of the sliding-window layers exists so that their cache never has to be persisted to SSD, and it cuts the persistent footprint to about an eighth of V4-Flash’s.

Share The Software Frontier

Rubin and the context tier

The part we did not expect is what the next GPU does to all of this. The HBM break-even is H/G, and G is proportional to peak compute, so the clock for GPU memory is set by a single hardware constant: HBM capacity per unit of compute.

In megabytes of HBM per dense FP8 TFLOPS it is 40.4 on an H100, 71.2 on an H200, 40.0 on a B200 and 37.2 on a GB200. On Rubin, with 288 GB against 17.5 dense FP8 PFLOPS on NVIDIA’s specification page, it is 16.5. NVIDIA revised that page on September 21, lowering HBM bandwidth to 19.2 TB/s and NVLink to 3 TB/s per GPU; neither enters this ratio, and capacity and compute are unchanged.

That one constant moves the whole ladder. For Llama-3.1-70B the time to fill HBM with fresh cache falls from 58 seconds on an H100 to 24 on Rubin, for DeepSeek-V3 from 142 to 58, for V4.1-Flash from 40 minutes to 16.

The capacity a ten-minute window needs per GPU grows with compute instead: 7.3 TB for Llama-3.1-70B, 3.0 TB for DeepSeek-V3 and 175 GB for V4.1-Flash, against 288 GB of HBM and 750 GB of LPDDR5X per Rubin GPU.

A Rubin GPU produces cache faster than anything in its own rack can hold for more than a minute or two, unless the model’s cache is small. Its share of the rack’s host memory holds about a minute of Llama-3.1-70B’s output and about two and a half minutes of DeepSeek-V3’s; for V4.1-Flash it holds 43 minutes.

The flash tier in the pod is the hardware answer to that gap, and a 437-fold smaller cache is the model answer. They are substitutes, and different companies are pursuing them.

The same arithmetic casts a different light on a product that disappeared. In September 2025 NVIDIA announced Rubin CPX, a GPU with 128 GB of GDDR7 built for the compute-heavy prefill of long contexts and due at the end of 2026. It was missing from the roadmap at GTC in March.

Commentators have pointed to tight 3nm capacity and the price of GDDR7 during the memory shortage. Our reading, which is an interpretation and nothing more, is that caching also shrank the market the chip was designed for.

When half or more of the input tokens in agent traffic are cache hits, the prefill that remains is the new part of each request, and a pool of flash that keeps the rest of the context close does more for time to first token than a pool of extra prefill compute.

Subscribe now

How the lookup works

No one of this helps unless a request can find its cache, and finding it is stricter than it looks. Attention is causal: the cache for position i depends on every token before i. Change one token near the start of the prompt and every block after it has a different cache, even where the text afterward is identical.

So every system identifies cache by prefix. vLLM hashes each block of tokens together with the hash of the block before it, adding extra keys for images, adapters and a per-tenant salt, and a lookup walks that chain until the first miss.

The API products expose the same chain at a coarser grain. Anthropic writes an entry only at a breakpoint, as a cumulative hash of everything before it, and a read walks back at most 20 blocks looking for an entry an earlier request wrote; it will not discover stable content behind a changing block unless something wrote an entry there.

Minimum cacheable lengths run from 512 tokens on its newest models to 4,096 on Haiku 4.5. OpenAI’s GPT-5.6 has a 1,024-token minimum, places an implicit breakpoint at the end of the latest eligible message, allows up to four writes per request, and checks the first two and the latest fifty explicit breakpoints on a lookup. Gemini’s implicit cache starts at 4,096 tokens on its current 3.x models.

The minimums reflect bookkeeping. Every entry carries a hash, an index record and a routing decision, and below a few hundred tokens that overhead is comparable to recomputing. OpenAI’s guide works through the trade from the customer’s side and shows that padding a short shared prefix up to the 1,024-token minimum pays off once the prefix is longer than about 102 tokens and reused often enough.

Then the request has to arrive where the cache is. OpenAI’s cache lives on individual machines, and requests are routed by a hash of the initial tokens after its hidden system content plus an optional key; above about 15 requests per minute for one prefix and key, some requests overflow to machines that do not hold the entry.

Open-source stacks solve the same problem with cache-aware routers that trade some load balance for hits. For agents the difficult moment is the pause: while a tool runs, the engine is tempted to evict the conversation’s cache to make room for someone else.

Continuum, from Berkeley, pins the cache in GPU memory for a time-to-live predicted from the tool’s expected duration, and reports large reductions in job completion time on SWE-Bench and BFCL agent workloads.

Shared caches also leak. A hit is faster than a miss, and in audits run in September and October 2024, Stanford researchers detected prompt caching at 8 of 17 API providers and sharing across users at 7 of them, OpenAI among them.

The large providers now isolate: OpenAI by organization and processing region, Anthropic by workspace on its own API. OpenAI still recommends a separate cache key per end user inside one organization, which stops one customer from probing for another’s cached prefixes.

Subscribe now

What the prices imply

With the physics in place, the price lists can be read as statements about cost. Start with the question a developer can answer from them directly:

how many times must a cached prefix be read before caching beats resending it?

With a write multiplier w and a read multiplier r, writing once and reading n times costs w + n·r input-equivalents, and not caching costs 1 + n.

break-even reads:  n > (w - 1) / (1 - r)
5-minute write  (w = 1.25, r = 0.1):   n > 0.28    one read wins:  1.35 against 2.00
1-hour write    (w = 2.00, r = 0.1):   n > 1.11    one read loses: 2.10 against 2.00

A five-minute write pays for itself with one read and an hour-long write needs two. The second read is the price of the longer clock, and the hourly cost of holding can be backed out of each price list. Anthropic’s five-minute premium is a quarter of the input price for five minutes, three times the input price per hour; its one-hour premium is one times the input price per hour.

OpenAI’s is a quarter for at least thirty minutes, at most half the input price per hour. Google’s rent is 2.25 times the input price per hour on Gemini 3.1 Pro and two thirds of it on Gemini 3.8 Flash. DeepSeek’s is zero.

Those hourly figures describe an entry that is written and never read again. An entry that keeps being read costs nothing more to keep, because at both Anthropic and OpenAI every read refreshes the lifetime without a new write charge.

For an active agent the write premium is a deposit, not rent. The provider charges for the chance that you walk away and gives the holding away to anyone who comes back, which is the pricing you would design if idle bytes in fast memory were the cost you worried about.

Converting those charges to dollars per gigabyte-hour requires the cache size per token of the model behind each price, and none of these providers publishes it for its closed models. So we plotted every charge against every plausible size.

Across the cache sizes of the modern open models in this piece, from V4.1-Flash’s 890 bytes to GLM-4.6’s 368 KiB, every one of these charges is at least two and a half times our DRAM cost, and most are more than ten times it. For caches under 64 KB per token, all of them except Gemini Flash’s are at or above what HBM costs to rent.

Sonnet 5’s hourly rate, $2 per million tokens, equals H100 HBM rent at 64 KB per token and exceeds DRAM cost for any cache under 3.9 MB per token, a size no production model approaches. None of the providers prices its clock at DRAM cost, whichever tier the bytes actually sit in.

DeepSeek is the one provider for which both sides of the calculation are public. At 890 bytes per token a million cached tokens occupy 0.89 GB, and holding them on SSD for a full day costs about $0.0002 at our rates. Recomputing them costs at least 16 PFLOP, 27 seconds of an H800 at our utilization, about $0.015 at $2 an hour.

The off-peak read price, $0.003, is a fifth of that recompute floor and about twelve times a day of SSD rent; the miss price, $0.15, is ten times the recompute floor. Nineteen months earlier DeepSeek charged $0.14 for an R1 hit against $0.55 for a miss: 25 percent, on a model that kept 79 times more cache per token.

We cannot see inside Anthropic’s models, so we cannot say why a read on Fable 5.1 costs a quarter of what it costs on Sonnet 5 as a share of the input price. The arithmetic says a read gets cheaper to serve as the cache per token shrinks relative to compute, and a factor of four is the size of step DeepSeek’s generations have taken.

It is also what a company would do to win agent workloads, where reads are most of the bill. The two explanations are not exclusive.

Subscribe now

An agent session, priced

Here is what the clock does to a real bill. Take a coding agent that starts with a 12,000-token system prompt and tool catalog and takes fifty steps. Each step appends 1,100 tokens of tool output and 400 tokens of model output.

The context ends at 85,500 tokens; the session sends 2.44 million input tokens and receives 20,000 output tokens, a ratio of 122 to one. Manus, which runs this kind of loop at scale, reports a ratio around 100 to one and names cache hit rate as the metric it watches above all others.

Without caching the session costs $5.08, almost all of it input. With a five-minute cache and every tool call finishing inside the window it costs $0.88: the session reads 96.5 percent of its input tokens from cache, its effective input price falls to 14 percent of list, and 69 percent of what remains is reads.

Now let nine of the forty-nine gaps last eight minutes: the test suite, the build, the person who went for coffee. Each time the cache is gone, and the entire context is written again at 1.25 times the input price. The session costs $1.88, more than double. Writing everything with the one-hour lifetime brings it to $1.01; keeping the five-minute entries alive with one zero-output ping per gap brings it to $0.97.

The same session on Opus 5.5, with reads at 5 percent, costs $1.30 against $10.15 uncached. On DeepSeek’s V4.1-Flash off-peak it costs 3.2 cents against 38 cents. Because reads dominate an agent’s bill, the read multiplier matters more than any other number on the price list: halving it from 0.1 to 0.05 at Sonnet 5’s base price would cut the input side of this session by 34 percent.

Price lists also differ in structure, so the same session lands at very different fractions of each provider’s own input price. With a warm cache it runs at 0.14 of list on Sonnet 5 and GPT-5.6, 0.13 on Gemini’s implicit cache if every repeat hits, 0.09 on Opus 5.5, 0.07 on Fable 5.1 and 0.05 on DeepSeek’s V4.1-Flash.

The eight-minute gaps separate them further. GPT-5.6’s thirty-minute minimum absorbs them, and DeepSeek publishes no lifetime but serves hits from disk, so we actually assume its cache survives them too. Anthropic’s five-minute models need pings or the one-hour write to stay near their warm figures; left to expire, Sonnet 5 climbs to 0.34, Opus 5.5 to 0.30 and Fable 5.1 to 0.29. A lower read price does little for a session whose cache keeps expiring.

Subscribe now

A field guide

The arithmetic above turns into rules that can be applied without redoing it. Each comes with the number behind it, so it can be rechecked when a price list changes.

When caching pays at all

If every input token either writes to the cache or reads from it, the input bill as a multiple of the uncached bill is (1 - h)·w + h·r, where h is the share of those tokens that are hits. Caching pays when that is below one:

break-even hit rate:  h* = (w - 1) / (w - r)
5-minute write  (w = 1.25, r = 0.10):   h* = 21.7%
1-hour write    (w = 2.00, r = 0.10):   h* = 52.6%
Opus 5.5, 5-minute (r = 0.05):          h* = 20.8%
DeepSeek (no write premium):            any hit rate pays

Kimi’s chat trace at a five-minute lifetime has a hit rate of 29 percent, just above the line, which is why caching everything there saves only 9 percent. The agent trace sits at 53 percent and our modeled session at 96.5 percent, where the input bill falls to 14 percent of list. For chat-like traffic, mark what is shared across requests, such as the system prompt and the tool definitions, and let the rest go uncached; for agents, cache everything.

Crossing a long tool call

The expensive event in an agent session is an expiry in the middle of a long context, because it turns the cheapest tokens in the session into the most expensive ones. There are three ways across a gap longer than five minutes. Let the entry expire and rewrite the context at 1.25 times the input price.

Buy the one-hour lifetime and pay 2 times instead of 1.25 on the write. Or keep the five-minute entry alive with a request that repeats the prefix every four and a half minutes. Anthropic documents this as pre-warming: with max_tokens set to 0 the request reads the cache, refreshes its lifetime and generates nothing. Each ping costs one read, a tenth of the context’s input price.

Charging the whole one-hour premium to a single gap, pings are cheaper than the one-hour write for gaps up to about 36 minutes and cheaper than a rewrite up to about 54 minutes. When a session has several long gaps the one-hour premium is spread across them and wins sooner.

Pings have to reproduce the prefix and the settings exactly, thinking and effort configuration included, and Anthropic rejects the zero-output form when it is combined with streaming, extended thinking, structured outputs or a forced tool choice.

OpenAI’s GPT-5.6 needs none of this for gaps under half an hour, because its minimum lifetime is already thirty minutes. In our session, one ping per eight-minute gap brings the bill from $1.88 to $0.97.

Keeping the prefix identical

One changed token invalidates every block after it, so the rules are mechanical. Put what never changes first: tool definitions, then instructions, then reference material, then history. Append tool results and messages instead of editing or reordering earlier turns.

Serialize JSON with a fixed key order. Keep timestamps and per-request data at the end or in later messages. Do not toggle settings that are rendered into the prompt: Anthropic’s list includes tool definitions, web search and citation toggles, speed mode, and thinking or effort settings;

OpenAI’s includes tools, parallel tool calls, output format, reasoning effort and verbosity. Manus describes arriving at the first three rules after several rewrites of its agent framework.

Compaction is the deliberate exception. Replacing a context of C tokens with a summary of c tokens forfeits the cache and costs a summary generated at output prices. At Sonnet 5 prices, where output costs five times input, it pays back after about ((w - r + 5)·c + r·C) / (r·(C - c)) further steps. Compacting 80,000 tokens into a 5,000-token summary pays back in about five steps; into a 20,000-token summary, in about 22.

Measuring what you pay

Both APIs report the split. Anthropic returns cache_creation_input_tokens, cache_read_input_tokens and input_tokens, where input_tokens counts only tokens after the last breakpoint; OpenAI returns cached_tokens and cache_write_tokens inside input_tokens_details, and its input_tokens includes both.

The number to watch is the effective input multiplier:

effective input multiplier = f_uncached + w × f_written + r × f_read

where the f are shares of all input tokens. Our session runs at 0.14 with a warm cache and 0.34 with nine expiries. A rising multiplier with a steady hit rate usually means more writes: a breakpoint on content that changes, or a conversation that grows more than 20 blocks past its last write, beyond the reach of Anthropic’s lookback.

Sizing a cache for your own fleet

For a self-hosted deployment the same formulas become a sizing recipe. Per prefill GPU, a cache that holds τ seconds of fresh output needs G × τ bytes, and in Kimi’s traces ten minutes of unique traffic captured 89 to 98 percent of the achievable reuse.

On an H100 that is 827 GB for Llama-3.1-70B, 338 GB for DeepSeek-V3 and 20 GB for V4.1-Flash, which picks the tier for you: GPU or host memory for compressed caches, SSD or a shared pool for the rest. A flash tier has to be sized for writes as well as capacity, G × 86,400 bytes per GPU per day if everything is persisted, which is why admission control belongs in the design from the start.

And if reloads must hide behind prefill, the link needs (cached tokens / new tokens) × G: 77 GB/s for the last step of our session on Llama-3.1-70B, which a PCIe Gen5 x16 link cannot deliver and a coherent CPU link can.

Subscribe now

Where this could be wrong

Utilization and prices move G and the clock linearly. We used 30 percent of dense FP8 peak and an H100 at $2.50 an hour. Double the utilization and every break-even time halves; halve the GPU price and the DRAM and SSD break-evens halve. The ordering of tiers and models does not change, and neither do the log-scale charts, but individual minutes and hours can be off by a factor of two. Published measurements put real prefill between 0.16 of peak in DeepSeek’s production fleet and 0.29 in SGLang’s GB200 benchmark, so our central value is on the optimistic side: at production utilization every HBM break-even roughly doubles and every capacity figure halves.

The HBM price is an opportunity cost. Pricing idle HBM at its full share of GPU rent is right when serving is memory-bound. A compute-bound prefill fleet values HBM less at the margin, and its HBM break-even is longer than H/G.

Closed models are closed. Everything we say about Anthropic, OpenAI and Google infrastructure is inference from prices and documentation. If their caches per token are far smaller than the open models’ caches, which DeepSeek has shown is possible, the implied storage charges sit even further above HBM rent; if far larger, closer to it.

The traces are one sampled hour of one service. Kimi’s traces come from chat and tool traffic, not from long-running coding agents with multi-minute tool calls. Agent sessions in 2026 probably have longer gaps and higher reuse, which would favor longer lifetimes than our replay suggests. Our correction for the one-hour window removes the truncation bias only for gaps up to thirty minutes. Alibaba has published two-hour traces from its Bailian service in a compatible block-hash format, including a coding trace and a reasoning trace; we have not replayed them here, and they are the obvious next check.

Prices are strategy as much as cost. A provider can price reads below cost to win agent workloads, and rent above cost because explicit caches are bought by customers with few alternatives. Reading prices as cost signals assumes competition does some of the work.

F is a floor, and we left latency out. Attention makes recomputation more expensive at long context, and a hit also buys time to first token, which has a value we did not price. Both make caching more attractive than our numbers show.

The rules are only as durable as the price lists. Pings work because reads refresh lifetimes for free and zero-output requests are documented, and the crossovers in the field guide move with every multiplier. Either could change with the next pricing update, which is why every rule above is written as a formula that can be rerun.

Subscribe now

Predictions

Each of these can be checked against a public price list, model card or announcement.

1. By December 31, 2027, at least two of Anthropic, OpenAI and Google will list a cache-read price at or below 5 percent of the fresh input price for their flagship model. Today only Anthropic does.

2. By December 31, 2027, an open-weight model from a lab other than DeepSeek, competitive with frontier models on agentic coding benchmarks, will ship with a cache under 4 KB per token at its reference precision.

3. By December 31, 2027, DeepSeek will list a cache-hit price at or below 1 percent of its cache-miss price.

4. By December 31, 2027, at least two of the five largest GPU clouds will have announced a production flash tier dedicated to KV cache, built on CMX or an equivalent design.

Subscribe now

Confidence dossier

Every number in this piece is listed below with a tier. A means a primary source we read directly: documentation, a model card, a paper or a company post. B means secondary reporting, a preliminary specification or market data. C means our own derivation or model from stated assumptions. D means our interpretation of closed systems, strategy or vendor claims. M means our own measurement on public data, either the Mooncake traces or published model configurations.

Method

Cache sizes come from each model’s published configuration: layer counts, key-value heads and head dimensions for grouped-query models, latent and positional dimensions for MLA models, and the model card for V4.1-Flash.

The V3.2 layout counts 512 FP8 latent values, 16 bytes of scales and 64 BF16 positional values per layer, plus a 128-value FP8 indexer key with a 4-byte scale. The V4-Flash figure comes from our August reconstruction of its compressed attention. F counts two operations per active parameter and nothing else.

G, the break-even times and the capacities follow from the formulas in the text, with P at dense FP8 peak, u at 0.30, an H100 at $2.50 an hour, a B200 at $4.50 and the tier costs in the table. Rubin figures use NVIDIA’s specification page as revised on September 21, 2026.

The trace replay sorts requests by arrival time and treats each hash as one 512-token block. A request hits its longest prefix of blocks present in the cache and stops at the first miss, and every block it touches has its last-use time set to the request’s arrival. Tokens are counted with the final partial block capped at the request’s input length. Lifetimes are sliding: an entry expires a fixed time after its last use. The capacity replay evicts the least recently used block once the cache is full. For the gap statistics we counted only reuses whose previous use came at least 1,800 seconds before the end of the trace.

The session model has a 12,000-token starting prefix, fifty steps, 1,100 tokens of tool output and 400 of model output per step, and Claude Sonnet 5 list prices. A step after a long gap pays a full write for its whole context.

The whiskers in the clock chart vary utilization from 0.15 to 0.5, the H100 price from $1.50 to $4.00 an hour, DRAM from $8 to $16 per GB with 25 to 100 percent overhead, SSD from $0.15 to $0.35 per GB with 50 to 200 percent overhead, and the share of HBM available to cache from 60 to 100 percent.

The gap model sends a ping every 4.5 minutes after the first five, charges each ping one read of the context, and charges the whole one-hour premium to the gap it protects; in the session with pings, each of the nine long gaps lasts eight minutes and takes one ping. No number in this piece required a GPU.

Subscribe now

Sources

1. Anthropic. Prompt caching. Claude Platform Docs, accessed September 2026. https://platform.claude.com/docs/en/build-with-claude/prompt-caching

2. OpenAI. Prompt caching. OpenAI API documentation, accessed September 2026. https://developers.openai.com/api/docs/guides/prompt-caching

3. Google. Gemini Developer API pricing, accessed September 2026. https://ai.google.dev/gemini-api/docs/pricing

4. Google. Context caching. Gemini API documentation. https://ai.google.dev/gemini-api/docs/caching

5. DeepSeek. Models and Pricing. DeepSeek API Docs, accessed September 2026. https://api-docs.deepseek.com/quick_start/pricing

6. DeepSeek. DeepSeek-V4.1-Flash: Smarter, Faster, More Efficient. Release note, September 10, 2026; the news index dates Context Caching to August 2, 2024. https://api-docs.deepseek.com/news/news260910

7. DeepSeek. Day 6: One More Thing, DeepSeek-V3/R1 Inference System Overview. Open Infra Index, February 2025. https://github.com/deepseek-ai/open-infra-index/blob/main/202502OpenSourceWeek/day_6_one_more_thing_deepseekV3R1_inference_system_overview.md

8. Grattafiori et al. The Llama 3 Herd of Models. arXiv:2407.21783, 2024. https://arxiv.org/abs/2407.21783

9. DeepSeek-AI. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv:2405.04434, 2024. https://arxiv.org/abs/2405.04434

10. Moonshot AI. Kimi K2 model summary, README. https://github.com/MoonshotAI/Kimi-K2

11. DeepSeek-AI. DeepSeek-V3 inference configuration, config_671B.json. https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/configs/config_671B.json

12. DeepSeek-AI. DeepSeek-V3.2-Exp inference configuration, config_671B_v3.2.json. https://github.com/deepseek-ai/DeepSeek-V3.2-Exp/blob/main/inference/config_671B_v3.2.json

13. Bradanini and Tettamanti. DeepSeek V4-Flash: The Cost of Deciding What to Read. August 2026.

14. Simon Willison. DeepSeek V4, almost on the frontier, a fraction of the price. April 24, 2026 (quotes the V4 report on cache size at 1M tokens). https://simonwillison.net/2026/apr/24/deepseek-v4

15. DeepSeek-AI. DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression. Model card, September 2026. https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash

16. Touvron et al. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288, 2023. https://arxiv.org/abs/2307.09288

17. MiniMax. MiniMax-M2 configuration, config.json. https://huggingface.co/MiniMaxAI/MiniMax-M2/resolve/main/config.json

18. GLM-4.6 configuration, as published in RedHatAI’s NVFP4 release of zai-org/GLM-4.6, config.json. https://huggingface.co/RedHatAI/GLM-4.6-NVFP4/blob/main/config.json

19. DeepSeek-AI. DeepSeek LLM: Scaling Open-Source Language Models with Longtermism. arXiv:2401.02954, 2024. https://arxiv.org/abs/2401.02954

20. OpenAI. gpt-oss reference implementation, gpt_oss/torch/model.py. https://github.com/openai/gpt-oss/blob/main/gpt_oss/torch/model.py

21. Hugging Face Transformers v4.57.1. Qwen3-Next configuration defaults. https://github.com/huggingface/transformers/blob/v4.57.1/src/transformers/models/qwen3_next/configuration_qwen3_next.py

22. Qwen Team. Qwen3 Technical Report. arXiv:2505.09388, 2025. https://arxiv.org/abs/2505.09388

23. The SGLang Team. Deploying DeepSeek with PD Disaggregation and Large-Scale Expert Parallelism on 96 H100 GPUs. LMSYS blog, May 5, 2025. https://lmsys.org/blog/2025-05-05-large-scale-ep/

24. The SGLang Team. Deploying DeepSeek on GB200 NVL72 with PD and Large Scale EP (Part II): 3.8x Prefill, 4.8x Decode Throughput. LMSYS blog, September 25, 2025. https://www.lmsys.org/blog/2025-09-25-gb200-part-2

25. Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs. arXiv:2609.11744, September 2026. https://arxiv.org/abs/2609.11744

26. Qin et al. Mooncake: Trading More Storage for Less Computation, a KVCache-centric Architecture for Serving LLM Chatbot. USENIX FAST 2025, Best Paper Award. https://www.usenix.org/conference/fast25/presentation/qin

27. TrendForce. Memory Makers Prioritize Server Applications, Driving Across-the-Board Price Increases in 1Q26. January 5, 2026. https://www.trendforce.com/presscenter/news/20260105-12860.html

28. ITLDC. Server RAM and SSD market update, 2026 (supplier prices tracked from January 1 to August 21). https://itldc.com/es/blog/server-ram-ssd-market-update-2026

29. igor’sLAB. DRAM and NAND remain more expensive: TrendForce sees slowing but still rising memory prices in Q3 2026. https://www.igorslab.de/en/dram-and-nand-remain-more-expensive-trendforce-sees-slowing-but-still-rising-memory-prices-q3-2026/

30. IntuitionLabs. Data Center GPU Pricing 2026: The Full AI Pricing Index. July 20, 2026. https://intuitionlabs.ai/articles/data-center-gpu-pricing-2026

31. Moonshot AI and Tsinghua University. Mooncake FAST’25 trace release: conversation, tool and agent, and synthetic traces. https://github.com/kvcache-ai/Mooncake/tree/main/FAST25-release

32. Wang et al. KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider. USENIX ATC 2025. https://arxiv.org/abs/2506.02634

33. NVIDIA Technical Blog. Introducing NVIDIA BlueField-4-Powered CMX Context Memory Storage Platform for the Next Frontier of AI. March 16, 2026. https://developer.nvidia.com/blog/introducing-nvidia-bluefield-4-powered-inference-context-memory-storage-platform-for-the-next-frontier-of-ai/

34. NVIDIA. Vera Rubin NVL72, product page, preliminary specifications. https://www.nvidia.com/en-us/data-center/vera-rubin-nvl72

35. Tom’s Hardware. Nvidia launches Vera Rubin NVL72 AI supercomputer at CES. January 2026. https://www.tomshardware.com/pc-components/gpus/nvidia-launches-vera-rubin-nvl72-ai-supercomputer-at-ces-promises-up-to-5x-greater-inference-performance-and-10x-lower-cost-per-token-than-blackwell-coming-2h-2026

36. DeepSeek-AI. Fire-Flyer File System (3FS), README, performance section. https://github.com/deepseek-ai/3FS

37. NVIDIA. NVIDIA Unveils Rubin CPX: A New Class of GPU Designed for Massive-Context Inference. Press release, September 9, 2025, via AIwire. https://dev.aiwire.net/2025/09/09/nvidia-unveils-rubin-cpx-a-new-class-of-gpu-designed-for-massive-context-inference/

38. Tom’s Hardware. Nvidia removes Rubin CPX accelerators from its roadmap. March 2026. https://tomshardware.com/pc-components/gpus/nvidia-removes-rubin-cpx-accelerators-from-its-roadmap-groq-3-lpus-take-center-stage-as-cpx-is-removed

39. ASCII weekly. Analysis of the 2026 NVIDIA roadmap and the missing Rubin CPX, in Japanese. https://weekly.ascii.jp/elem/000/004/387/4387523/3/

40. vLLM. Automatic Prefix Caching, design document. https://docs.vllm.ai/en/latest/design/prefix_caching/

41. Li et al. Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live. arXiv:2511.02230. https://arxiv.org/abs/2511.02230

42. Gu, Li, Kuditipudi, Liang and Hashimoto. Auditing Prompt Caching in Language Model APIs. ICML 2025. https://proceedings.mlr.press/v267/gu25b.html

43. Yichao Ji. Context Engineering for AI Agents: Lessons from Building Manus. July 18, 2025. https://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus

44. Alibaba. Qwen-Bailian anonymized usage traces: to-C, to-B, thinking and coder workloads, README. https://github.com/alibaba-edu/qwen-bailian-usagetraces-anon

Introduction

Four days after Modular shipped Mojo 1.1, I did what I just always do with a brand new compiler. I installed it in a container with one CPU core and no GPU, wrote a kernel that doubles a vector, and asked for PTX for an H100. I asked for sm_90. The first two directives of what came back were these:

.version 8.5
.target sm_90a

That one letter is the subject of this piece. sm_90 is Hopper. sm_90a is Hopper with its architecture-specific instructions unlocked, among them the warpgroup matrix multiply and the warpgroup register reallocation that the fastest Hopper kernels are built on.

The price of the suffix is written into NVIDIA’s own assembler. PTX that declares .target sm_90a can only be compiled for sm_90a. It cannot be compiled for plain sm_90, and it cannot be carried forward to the next generation. I didn’t ask for that trade.

The compiler made it for me, without a warning, and it makes the same trade for every Hopper and Blackwell name it accepts but one.

I spent the following days pulling on that thread, and it runs through the whole stack. I installed Mojo 1.1.0 and MAX’s max-core 26.6.0 from PyPI, cloned the compiler Modular open-sourced in August, took NVIDIA’s ptxas 13.4 from PyPI to check what Mojo emits, and compiled well over a hundred small probe programs.

Nothing here ran on a GPU. Everything is about what the compiler decides before a GPU is involved. I’m not a compiler engineer, I’m someone who likes taking tools apart, and Mojo is now a tool you can take apart all the way down. Where I infer instead of measure, I say so, and the dossier at the end grades every claim.

The short version is basically this: Mojo’s portability is real, and it lives entirely in the source. The compiler treats a target name as a product rather than an instruction set and compiles for the exact chip it is told about. When the compiler travels with the program, as it does inside MAX’s runtime library, that costs nothing.

When it does not, as in an executable made with mojo build, the kernels inside are locked to one chip: an executable built for an H100 carries PTX that NVIDIA’s own assembler refuses to build for any Blackwell part. Either way, every NVIDIA kernel is finished by NVIDIA’s closed assembler, which MAX ships inside its wheel.

The design is, at least in my own view, pretty coherent. In several places the implementation is not: a device table that quietly compiles a Jetson Orin as an A100, a Blackwell guard that reads the machine you compile on instead of the machine you compile for, and a Jetson Thor entry that cannot reach the tensor memory its hardware has.

Subscribe now

What changed since the Mojo you remember

The Mojo of 2023 was (probably) pitched as a superset of Python with a split personality. def behaved like Python: dynamic, permissive, allowed to raise. fn was the strict twin: typed, non-raising unless declared, arguments immutable by default.

That split was the idea people remembered. It is gone. Modular said in August 2025 that Mojo may or may not grow into a full Python superset and that this was acceptable, which is how Simon Willison summarized the change when the compiler went open.

The repository’s own release notes date the rest. fn appears in the November 2022 notes as a new, stricter declaration. It is deprecated in 0.26.2, warned on by the compiler from 1.0.0b1 in May 2026, and finally removed in 1.1.0, released on 17 September 2026, together with alias, the __comptime_assert keyword, and the @parameter if and @parameter for forms.

The 1.0 announcement promised that changes during 1.x would be primarily additive, with breaking changes managed carefully in the way mature languages like C++ manage them. The first minor release after 1.0 deleted three keywords and two syntax forms.

To be fair, all of them were deprecated during the 1.0 cycle, so code that compiled without deprecation warnings under 1.0 should still compile under 1.1. The more interesting thing is what the removals leave behind. We wrote a batch of small programs to test what 1.1 actually enforces, and ran each one through the compiler:

def now means what fn used to mean. It does not raise unless it says raises, every variable is introduced with var, and arguments default to the immutable imm convention.

The memory model underneath is affine by default: a value moved out with ^ is uninitialized, and using it is a compile error. It is linear on request: a type declared with Deinitable where False never conforms to Deinitable, so it must be explicitly consumed, and the compiler rejects any path that drops it.

Rust has affine types and no native linear ones; Mojo has both in one type system. Exclusivity is enforced at call sites, and since 1.0 the lifetime checker, experimentally, understands the inside of containers, so a reference into a List held across an append is rejected instead of dangling after reallocation. Strings refuse to have a length until you say which length you mean.

What is still missing is also clear. Pattern matching exists on the main branch as an experimental __match statement, and the nightly changelog notes that exhaustiveness is not yet checked for enums or Bool. An async model and unions are on the roadmap.

So Mojo 1.1 is not Python with types anymore. It reads like a Rust whose surface borrowed Python’s indentation, attached to a compile-time metaprogramming system more capable than either. We have argued before that strict semantics and one way to do things suit a world where models write a growing share of the code.

Mojo 1.1 reads like a language that took that argument seriously, whether or not anyone at Modular would phrase it that way.

Subscribe now

What pip actually installs

pip install mojo pulls five packages: mojo itself (a 48.9 MB wheel holding the language server, the debugger and the REPL entry point), mojo-compiler (an 80.7 MB wheel) and its companion mojo-compiler-mojo-libs, mblack 26.6.0 for formatting, and mojo-lldb-libs.

The directory it installs is 675 MB. The compiler driver is a single binary of 142,215,032 bytes. It carries its own linker, an lld of 115,567,904 bytes, which is 81% of the size of the compiler that calls it. The standard library ships as one precompiled container, std.mojoc, 3,226,898 bytes, with the magic bytes MPKG and an entropy of 7.999 bits per byte.

Its entropy says it is compressed, or encrypted, and either way it is opaque. The source of that library is on GitHub under Apache 2.0; the file the compiler actually loads is not something you can read.

A “hello world” builds in 2.33 seconds cold and 0.60 seconds from a warm cache in our first session, and in 2.97 to 3.03 and 0.74 to 0.76 seconds across three later runs on the same, busier shared core. The ratio held: a warm build costs a quarter of a cold one.

The result is a 17,944-byte executable. The executable is not standalone. It links libKGENCompilerRTShared.so through an absolute RUNPATH that points into the Python environment Mojo was installed into, /usr/local/lib/python3.12/dist-packages/modular/lib in our case. Move the binary to a machine without that directory and the loader will not find its runtime unless you point it there. That is a normal trade for a young toolchain, and worth knowing before you ship anything built this way.

The bigger surprise is what the wheel does not contain. Mojo is sold as the language for GPUs, and after pip install mojo there is no GPU package. from gpu.id import block_idx fails with unable to locate module 'gpu'. The 1.0 changelog explains it: the accelerator APIs moved into a new max Mojo package and the layout package was bundled with MAX instead of Mojo.

Install max-core 26.6.0, an 85.2 MB wheel, and a max package appears with max.gpu inside it, one of 31 precompiled packages totalling 24.4 MB: nn at 6.8 MB, linalg at 4.8 MB, layout, kv_cache, quantization, comm, shmem, state_space, and five vendor binding packages we come back to later.

It also brings libmax.so at 162.6 MB, the GPU runtime libMGPRT.so at 44.7 MB, and a 37.7 MB file called libNVPTX.so that turns out to be NVIDIA’s assembler. With both installed, the tree is 982 MB.

The licenses deserve one careful paragraph, because the headline from August was that Mojo is now open source, and that is true of the source. The PyPI metadata for mojo 1.1.0 and for max 26.6.0 both declare LicenseRef-MAX-Platform-Software-License.

The Python shim that provides the wheel’s entry points opens with a header calling itself Modular proprietary. The repository’s LICENSE is Apache 2.0 with LLVM exceptions.

Reporting from ModCon by NAND Research and RuntimeWire says Modular also rewrote the MAX license there to remove device restrictions and is moving MAX toward a source-available model. We are not lawyers and read none of this as a problem.

What is measurable is narrower: the tree you clone and the wheel you install carry different license identifiers, and if you need the Apache terms end to end, Modular’s own instructions build the compiler from source and run a file with ./bazelw run --config=build-mojo KGEN:mojo -- run hello.mojo.

The same post notes that a prebuilt compiler is still required today if you customize MAX kernels or models.

Subscribe now

The compiler, now that it can be read

We cloned modular/modular at commit 26cfe94, dated 21 September 2026, and fetched the mojo/v1.1.0 tag (commit c6fa49f, 17 September) to compare against what actually shipped. The working tree is 11,054 files.

The compiler proper, the C++ and TableGen under Mojo/lib, Mojo/include and Mojo/tools, is 314,179 lines. The standard library, written in Mojo, is 185,284 lines. The compiler’s tests are another 149,354.

Then there is MAX: 1,048,124 lines of Mojo across kernels, models and tests, and 880,903 lines of Python. The Mojo code inside MAX alone is 3.3 times the size of the compiler. The language is the smaller asset in its own repository.

Under Mojo/include/Mojo there are five MLIR dialects, and their own descriptions say what each one is for. lit holds the language-level abstractions and lowers to kgen. pop defines parametric operations on top of MLIR’s LLVM dialect. kgen holds the top-level and support operations of what its description still calls the kernel generation framework. hlcf is a structured, high-level control-flow dialect, and co models async functions as coroutines with arbitrary suspension points.

Counting the TableGen definitions gives 196 operations, 114 types and 174 attributes across the five, and 45 passes declared next to them, plus 2 more in the shared Support library. The shape of kgen is the telling one: 39 operations, 64 types and 98 attributes, in 9,238 lines of TableGen, more than lit and pop combined.

Compile-time values in Mojo are carried as MLIR attributes; the device records we meet below are literally #kgen.target<...> attributes. That is why the dialect that carries a program’s compile-time structure is mostly types and attributes.

The name KGEN is everywhere once you look for it. Modular’s documented build target for the compiler is KGEN:mojo. The runtime every Mojo executable links is libKGENCompilerRTShared.so. The tools directory contains kgen, kgen-doc and kgen-reduce, and part of the test suite lives under test/kgen.

In our July piece on MLIR we described KGEN, from the public material available then, as a proprietary kernel generator that sat as a layer above Mojo. With the source open, that picture was wrong. KGEN is the compiler’s historical name and one of its dialects, sitting inside Mojo, not above it.

The directory list also has an Interpreter and an Elaborator. By their names and the passes they feed, the first runs comptime code inside the compiler and the second instantiates parametric code, which is where a large share of every build’s time goes.

Subscribe now

Where the compile time goes

Mojo can time its own pipeline. --mlir-timing reports every MLIR pass, --llvm-timing every LLVM pass, and --timing-json writes both as one object. Mojo caches compiled modules in MODULAR_CACHE_DIR and a warm cache skips most of the pipeline, so every measurement here points it at an empty directory first.

For the “hello world” example, 46 distinct passes run, plus five analyses. The four largest are importing the Mojo modules at 16.6% of pipeline time, VerifyParameters at 16.1%, LowerLIT at 13.1% and InlineParametric at 12.8%, followed by CheckLifetimes, the borrow checker, at 6.8%.

For a program that compiles the vector kernel for sm_90a as an offload target, the order shifts: imports 19.3%, ElaborateGenerators 15.4%, VerifyParameters 12.7%, InlineParametric 9.4%, LowerLIT 8.6%, CheckLifetimes 4.7%.

The three passes that exist to instantiate and check parametric code take 31.7% of the hello-world pipeline and 37.5% of the GPU one. The tree view shows the elaborator working in rounds: InlineParametric followed by VerifyParameters, repeated, with dead-symbol elimination and canonicalization in between.

Then there is LLVM. In the timing run for the sm_90a program, which the LLVM timers force onto one thread, the MLIR pipeline took 5.92 seconds and every LLVM pass on both the x86 host and the NVPTX offload added up to 15.6 milliseconds, or 0.26% of the build. Instruction selection for the whole GPU kernel took less than half a millisecond.

In our piece on Triton, Triton’s own MLIR pipeline was 81% of compile time and ptxas the other 19%. In Mojo the assembler is not in the build at all, because PTX is turned into machine code when the program loads, and LLVM is a rounding error. Nearly everything the compiler spends its time on is deciding what your program means.

That made us expect compile time to grow with the number of specializations, since generating kernels instead of enumerating them is the whole bet behind the language.

So we built programs that instantiate a small parametric function work[N] once, 8, 32, 128 and 256 times through a comptime for, each from an empty cache. We ran each three times. The medians were 3.08, 3.09, 3.08, 3.18 and 3.21 seconds, and a least-squares line through them gives about 0.56 milliseconds per specialization on a floor of about 3.1 seconds: two hundred and fifty-six specializations add roughly 0.14 seconds.

At this size an extra specialization costs almost nothing. The floor is paid before any user code. An empty def main(): pass builds in 2.92 seconds cold, median of three, and its pipeline looks like hello world’s: importing modules 18.1%, VerifyParameters 16.7%. With no user code at all, that is the standard library being imported and its parametric code verified. Adding print(1) costs about 0.13 seconds more, print("hello") about 0.06.

The honest caveat is that our bodies were quite small. A kernel that unrolls deeply at compile time will cost more, and the compiler has a --loop-unrolling-warn-threshold flag, defaulting to 1024, for when it does.

Subscribe now

The target sweep

mojo build --print-supported-accelerators lists every GPU the compiler will target. In 1.1.0 that is fourteen AMD architectures from gfx90a (MI250X) through CDNA3 and CDNA4 to RDNA2, RDNA3, RDNA3.5 and RDNA4 consumer parts, plus the mi300a APU name and two aliases.

Ten Apple entries, M1 through M5 each with and without Metal 4; and twenty NVIDIA names from sm_52 (Maxwell) to sm_121a (DGX Spark). --print-supported-targets lists the host CPU backends: the AArch64 family, 32 and 64-bit RISC-V in both endiannesses, and 32 and 64-bit x86. There is no Hexagon, no Qualcomm accelerator, no TPU and no Trainium in either list, which will matter at the end.

Modular’s GPU package exposes a function, _compile_code, that compiles a kernel for a named target at compile time and returns the assembly as a string. You don’t need to own any particular device.

We compiled the same small kernel for all twenty NVIDIA names and fed every result to NVIDIA’s ptxas 13.4.92, taken from the nvidia-cuda-nvcc wheel on PyPI.

from max.gpu.host.compile import _compile_code, get_gpu_target
from max.gpu import thread_idx, block_idx, block_dim

def vadd(c: Pointer[Float32, MutAnyOrigin], a: Pointer[Float32, ImmutAnyOrigin], n: Int):
    var i = Int(block_idx.x * block_dim.x + thread_idx.x)
    if i < n:
        c[unsafe_offset=i] = a[unsafe_offset=i] * 2.0

def main() raises:
    print(_compile_code[vadd, target = get_gpu_target["sm_90"]()]())

Five names come back with a suffix nobody asked for: sm_90 becomes sm_90a, sm_100 becomes sm_100a, sm_103 becomes sm_103a, sm_120 becomes sm_120a and sm_121 becomes sm_121a.

One name goes the other way: sm_87, Jetson Orin, comes back as .target sm_80. And sm_110a, Jetson Thor, loses the suffix it asked for and comes back as sm_110.

The PTX ISA version follows the family: 5.0 for Maxwell and Pascal, 6.3 for Turing, 8.1 for Ampere and Ada, 8.5 for Hopper, 8.7 for sm_120, 8.8 for sm_100, sm_103 and sm_121, and 9.0 for Thor. The path that real programs use behaves the same way.

Built with mojo build --target-accelerator sm_90 --emit asm, a program that compiles the kernel through DeviceContext.compile_function writes a PTX file next to its assembly that says .target sm_90a, and the same build for sm_100 and sm_87 says sm_100a and sm_80.

NVIDIA’s assembler makes the consequences concrete, and its own help text states the rules. PTX for sm_XY compiles to any target at or above XY, suffixed or not. PTX for the family target sm_XYf compiles to members of the same family at or above it.

PTX for sm_XYa compiles to sm_XYa and nothing else. Every one of Mojo’s upgraded outputs therefore fails when handed to ptxas with the architecture that was requested, for example:

$ ptxas -arch sm_90 vadd_for_sm90.ptx
ptxas fatal   : PTX with .target 'sm_90a' cannot be compiled for architecture 'sm_90'
$ ptxas -arch sm_100 vadd_for_sm90.ptx
ptxas fatal   : Program with .target 'sm_90a' cannot be compiled to future architecture

Change nothing but the directive back to .target sm_90 and the same body assembles for sm_100 and for sm_120. The same holds one generation later. Mojo’s output for a B200, sm_100a, is refused for the B300’s sm_103. Rewritten as the family target sm_100f, it assembles for sm_103 and is refused for sm_120, which belongs to a different family.

Of the twenty names Mojo accepts, twelve produce PTX that ptxas 13.4 will assemble for the architecture that was asked for. The three Maxwell and Pascal names fail earlier: ptxas 13.4 does not recognize sm_52, sm_60 or sm_61 as architectures at all.

Mojo lists them, and emits PTX 5.0 for them, but the current CUDA assembler cannot finish the job. Modular knows. A string inside the GPU runtime tells users of older hardware to point MODULAR_NVPTX_COMPILER_PATH at an external ptxas.

Subscribe now

A target is a product, not an instruction set

The suffix is not produced by a transformation anywhere in the compiler. It comes from a table. In the shipped 1.1.0 the table lives in info.mojo, where the H100 record already says sm_90a.

On the main branch it has moved to Mojo/stdlib/std/_gpu/host/_builtin_targets.mojo, and the first thing that file does is map command-line names to products:

        A10       ._with_cli_values[["sm_86"]],
        A100      ._with_cli_values[["sm_80"]],
        OrinNano  ._with_cli_values[["sm_87"]],
        L4        ._with_cli_values[["sm_89"]],
        RTX4090   ._with_cli_values[[]       ], # Could be ["sm_89"], but `sm_89` resolves to L4.
        B100      ._with_cli_values[[]                   ], # Could be ["sm_100", "sm_100a"], but both resolve to B200.
        B200      ._with_cli_values[["sm_100", "sm_100a"]],
        H100      ._with_cli_values[["sm_90" , "sm_90a" ]],

Each product carries a compilation target written as a kgen attribute, and the H100’s is where the letter comes from:

comptime _h100_target = CompilationTarget[
    _mlir_value=__mlir_attr[
        `#kgen.target<triple = "nvptx64-nvidia-cuda", `,
        `stdlib_plugin = "cuda", `,
        `arch = "sm_90a", `,
        `features = "+ptx85,+sm_90a", `,
        `tune_cpu = "sm_90a", `,
        ...

So sm_90 is not an instruction set to Mojo. It is a lookup key that resolves to “an H100”, and an H100 is compiled as sm_90a with PTX 8.5, 132 streaming multiprocessors in its record.

We parsed the whole table: 44 device records and 43 target records, because the B100 and the B200 share one. sm_89 resolves to an L4, so an RTX 4090 cannot be selected by name. sm_86 resolves to an A10.

On main, the lookup itself is a compile-time Mojo program that walks a type list of target collections, and the list has two members: BuiltinTargets and one called ADDITIONAL_TARGETS. The device registry is pluggable at compile time, which we come back to at the end.

Whether the suffix is a bug depends on where the compiler is when the kernel gets built. MAX carries it at runtime. libmax.so contains the pipeline we timed above: the pass names LowerLIT, ElaborateGenerators, InlineParametric, VerifyParameters, CheckLifetimes and LowerKGENToLLVM are all in it, next to 255 strings that mention NVPTX and 466 that mention AMDGPU. When MAX compiles a kernel for the GPU in the machine, sm_90a unlocks every instruction the H100 has and costs nothing.

An executable made with mojo build is a different object. We built one for sm_90. It is 44,512 bytes, links only libKGENCompilerRTShared.so, libAsyncRTMojoBindings.so and the C library, and contains no pass names at all. What it does contain is one 1,354-byte PTX module that says .target sm_90a. NVIDIA’s assembler builds that module for sm_90a and refuses it for sm_100, sm_103 and sm_120 as a future architecture.

Change the one directive to sm_90 and it builds for all three. So the kernels of an H100 executable cannot be carried to Blackwell by NVIDIA’s compiler, where a plain target would have let the driver compile them forward, and nothing in the build said so.

Triton makes the same choice as Mojo: in our Triton piece, its sm_arch_from_capability appends the suffix to every capability from 9.0 upward, with a live TODO next to it. NVIDIA’s own Tile IR makes the opposite one.

Its tileiras backend accepts eleven targets and none with the suffix, and buys forward compatibility by excluding the instructions that would need it. Three serious projects, two good answers. Mojo and Triton recompile for the exact chip. cuTile narrows what a kernel can say.

The family targets sit between them, and the NVIDIA assembler bundled in max-core knows sm_100f, sm_103f, sm_110f, sm_120f and sm_121f. Mojo’s table asks for none of them.

So the cost of the choice depends on where portability has to live. In MAX it lives in the source plus a runtime that carries a full compiler inside the 162.6 MB libmax.so, with NVIDIA’s assembler beside it in the wheel.

In a mojo build executable it ends at build time, and the executable is exactly as portable as a CUDA binary compiled only for sm_90a.

Subscribe now

Where the table is wrong

A device-centric design is only as good as the device table, and a hand-maintained table of 44 products accumulates small errors. We found four. They matter less individually than as a pattern.

The first explains Jetson Orin. The OrinNano record on main says arch = "sm_87", yet 1.1.0 emitted sm_80. The emitted LLVM IR already carried "target-cpu"="sm_80" with +ptx81,+sm_80, so the substitution happens in Mojo’s own code, before LLVM sees the kernel. The answer is in the 1.1.0 tag, where the table had a different structure.

Mojo/stdlib/std/_gpu/host/info.mojo maps "sm_87" to OrinNano correctly, then a method named target() turns a device into its compilation target through a chain of name comparisons that ends like this:

        if self.name == "A100":
            return _get_a100_target()
        ...
        if self.name == "Jetson Thor":
            return _get_jetson_thor_target()
        ...
        if self.name == "":
            return _get_empty_target()
        return _get_a100_target()

The chain has 43 name comparisons, and every product has a branch except one. There is no "Orin Nano" case, so the Orin falls through to the default, which is the A100.

Code built this way still runs on an Orin, because Ampere binaries are compatible within the family, so this is not a crash. It is a silent substitution, and the failure mode it represents is the one to worry about: an unrecognized device becoming an A100 instead of becoming an error.

Main has since restructured the table so that each device is declared together with its target record, as TargetAccelerator[GPUInfo, target], which removes this failure mode by construction. The shipped 1.1.0 still has it.

The second is an RTX 3090 record, unreachable from the command line because sm_86 resolves to the A10, whose features read +ptx63,+sm_86. PTX 6.3 predates sm_86. Compiling our kernel against that record, the way the standard library itself would, stops LLVM outright:

LLVM ERROR: PTX version 6.3 does not support target 'sm_86'. Minimum required
PTX version is 7.1. Either remove the PTX version to use the default, or increase
it to at least 7.1.

Like the GTX 1080 Ti, GTX 1060, GTX 970 and Tesla P100 records, its name is NVIDIA’s full marketing string, "NVIDIA GeForce RTX 3090", which suggests these entries exist to match physical cards by name at runtime. We could not test whether a real 3090 selects it. The entry is identical at the 1.1.0 tag and on main.

The third is cosmetic: the RTX 4090 and 4090 mobile records compile for sm_89 but set tune_cpu to sm_90a, Hopper’s tuning, in both versions. In practice the NVPTX backend does little architecture-specific scheduling, and ptxas does the real work, so we expect no measurable effect.

The fourth is Jetson Thor, whose record asks for sm_110 without the suffix. That one does have consequences, and they show up with the tensor cores.

Subscribe now

Where the tensor cores are

MAX exposes a warp-level matrix multiply-accumulate as one generic function, mma(d, a, b, c) in max.gpu.compute.mma, taking four SIMD values. Its body is a compile-time dispatch on the target: _mma_nvidia if the target is an NVIDIA GPU, _mma_amd if it is AMD, and _mma_apple only if the target is an Apple M5 with the metal4_0 feature.

We compiled a one-call kernel against every family and read what came out.

def k(o: Pointer[Float32, MutAnyOrigin]):
    var a = SIMD[DType.bfloat16, 8](1)
    var b = SIMD[DType.bfloat16, 4](1)
    var c = SIMD[DType.float32, 4](0)
    var d = SIMD[DType.float32, 4](0)
    mma(d, a, b, c)
    o[unsafe_offset=0] = d[0] + d[3]

With those operand widths, sm_80, sm_90a, sm_100a and sm_120a all produce the same PTX instruction, mma.sync.aligned.m16n8k16.row.col.f32.bf16.bf16.f32. On Turing, sm_75, which has no bfloat16 tensor cores, compilation does not fail with a Mojo diagnostic. LLVM aborts: Cannot select: intrinsic %llvm.nvvm.mma.m16n8k16.row.col.bf16.

Change the fragments to four lanes each and the AMD targets compile: gfx90a emits v_mfma_f32_16x16x16bf16_1k, while gfx942 and gfx950 emit v_mfma_f32_16x16x16_bf16. RDNA3 rejects four-lane fragments inside its own implementation file and accepts sixteen-lane bfloat16 inputs with eight accumulators, emitting v_wmma_f32_16x16x16_bf16.

The M5 under Metal 4 wants eight lanes of fp16 in and eight of fp32 out, and emits the AIR intrinsic air.simdgroup_matrix_16x16x16_multiply_accumulate. Hand it four lanes and a constraint explains that Apple MMA requires eight-element fragments.

On an M4, a clean compile-time error says the target does not support the operation at all.

So one function name really does reach five distinct matrix instructions in four families of matrix hardware. It does not do so from one piece of code.

The operand widths are the vendor’s register fragment layout, and they differ on every family: eight, four and four lanes on NVIDIA for this shape, four, four and four on CDNA, sixteen, sixteen and eight on RDNA3, eight, eight and eight on Apple.

The abstraction that actually hides this is the layout package. Its TensorCore type is parameterized on output type, input type, an MMA shape and a transpose flag, and computes the fragment shapes for the target at compile time.

This is the precise sense in which Mojo is portable: the portability is in parameters evaluated at compile time, not in the operation. It is a good design. It also means that the thing a kernel author writes against is layout, which ships with MAX, not with Mojo.

The Hopper warpgroup multiply shows the other edge. wgmma_async compiled for sm_90a emits wgmma.mma_async.sync.aligned.m64n64k16.f32.bf16.bf16, and NVIDIA’s assembler accepts it. Compiled for sm_100a, sm_120a or sm_80, it produces no Mojo diagnostic. LLVM aborts on %llvm.nvvm.wgmma.commit_group.sync.aligned. The function’s compile-time asserts check shapes and scale factors, and we found none that checks the target.

It’s a third, independent confirmation of what we measured in our piece on Blackwell’s tensor memory, that wgmma exists on no Blackwell target at all. It also shows where Mojo’s knowledge of the hardware ends and LLVM’s begins.

Subscribe now

The tcgen05 guard

Blackwell’s datacenter tensor cores are programmed through the tcgen05 family and a fifth memory space, tensor memory, which has to be allocated before use. MAX guards these instructions.

The file’s ten public tcgen05 functions make eleven calls to one constraint, identical at the 1.1.0 tag and on main:

# max/mojo/max/gpu/compute/arch/tcgen05.mojo
def check_blackwell_constraint():
    comptime assert _has_blackwell_tcgen05(), (
        "The tcgen05 instructions are only applicable on nVidia Blackwell"
        " (sm_100a, sm_101a) hardware."
    )

# Mojo/stdlib/std/sys/info.mojo
comptime _SM_101X_ARCHS: List[StaticString] = ["sm_101", "sm_101a"]

def _has_blackwell_tcgen05() -> Bool:
    return _has_nvidia_gpu_any[
        _SM_100X_ARCHS + _SM_101X_ARCHS + _SM_103X_ARCHS
    ]()

def _has_nvidia_gpu_any[archs: List[StaticString]]() -> Bool:
    comptime if not has_nvidia_gpu_accelerator():
        return False
    comptime for arch in archs:
        comptime if arch.removeprefix("sm_") in _accelerator_arch():
            return True
    return False

Two details in that code decide everything. The first is the choice of _has_nvidia_gpu_any. It asks whether the build has an NVIDIA accelerator configured, from the --target-accelerator flag or the detected GPU, and compares against that accelerator’s name. It does not ask what the kernel is being compiled for.

The same file has a sibling, _is_nvidia_gpu_any, that checks the compilation target. The second detail is the list: sm_101 was the name of Jetson Thor’s architecture before CUDA 13 renamed it sm_110, and Mojo’s own device table uses the new name. NVIDIA’s current assembler no longer accepts sm_101a at all.

We compiled a kernel that allocates tensor memory with every combination that matters, then gave each result to ptxas 13.4:

Kernel targetBuild acceleratorMojo 1.1ptxas 13.4sm_100anone (no flag, no GPU)refused by the guardnot reachedsm_100asm_100aemits tcgen05.alloc.cta_group::1.sync.aligned.shared::cta.b32acceptedsm_103asm_103aemits the sameacceptedsm_100asm_90arefused by the guardnot reachedsm_90asm_100aemits tcgen05 into sm_90a PTXrejected: tcgen05.alloc needs PTX ISA 8.6sm_110a (Thor)sm_110arefused by the guardaccepts tcgen05 on sm_110a and sm_110f, refuses it on sm_110sm_120a, sm_121asame as the targetrefused by the guardrefuses tcgen05 on both

On a GPU-less build machine, a common kind of CI runner, public tcgen05 code does not compile for a B200 unless someone passes --target-accelerator sm_100a. On a machine configured for an H100, a correct B200 kernel is refused. On a machine configured for a B200, the guard lets tcgen05 into Hopper PTX that NVIDIA’s assembler then rejects.

It’s possible that reading the build host is intentional, a way of instantiating Blackwell paths only when building for a Blackwell system, and that MAX’s own launch path always configures the accelerator to match.

The measured behaviour stands either way: in a language whose whole premise is compiling for hardware you are not running on, a hardware guard that consults the build host is the wrong primitive.

Jetson Thor gets both problems at once. Its architecture is missing from the guard’s list, which still says sm_101, and its device record compiles for sm_110 without the suffix, where ptxas refuses tcgen05 even when it is emitted. NVIDIA’s assembler accepts the identical instruction on sm_110a and sm_110f. As shipped, Mojo 1.1 cannot put Thor’s tensor memory to use, and fixing either blocker alone would not be enough.

The source explains why the Hopper and Blackwell failures looked so different: wgmma aborted inside LLVM, while tcgen05 sailed through LLVM into Hopper PTX.

They reach the instruction set by different roads. The warpgroup commit reaches LLVM as an NVVM intrinsic, the one named in LLVM’s error, and LLVM’s instruction selector checks intrinsics against the target. tcgen05.mojo contains 17 inline-assembly strings and no LLVM intrinsic calls: tcgen05.alloc is written out as text, and LLVM passes inline assembly through without reading it.

For tcgen05, the Mojo guard is the only check between the source and NVIDIA’s assembler. It is not an isolated choice. Across the max.gpu package we count 128 inline-assembly call sites against 76 LLVM-intrinsic call sites, alongside 296 compile-time asserts; across the kernel sources, 55 against 85, alongside 4,413 asserts.

Step back and there are three places where a portable stack can keep its knowledge of the hardware. The first is compile-time constraints written in the language, and when Mojo uses them the diagnostics are good: the M4’s clean refusal, the M5’s fragment-size message, the intent of the tcgen05 guard.

The second is LLVM’s instruction selector, which knows what each target can encode but can only say so by aborting the process. The third is the vendor’s assembler, which knows everything and speaks last. Inline assembly is a road around the middle layer: it goes from a Mojo string to the vendor’s assembler with nothing in between except whatever constraint the library author remembered to write.

A portable language is only as good as the first layer. In Mojo 1.1, a good part of the hardware knowledge still lives in the second and third.

Subscribe now

The last mile belongs to the vendor

On AMD, Mojo owns the path to machine code. The AMDGPU backend runs in process, the assembly it prints is the final GCN or RDNA instruction stream, and the runtime loads the result through hipModuleLoadData from libamdhip64.so.

We found no reference to AMD’s comgr library in the runtime. On Apple, Mojo stops earlier and deliberately: its output for an M-series target is LLVM IR with the air64-apple-macosx triple and Metal feature flags, handed to Apple’s toolchain to finish.

On NVIDIA, Mojo stops at PTX, and the rest is NVIDIA’s. The GPU runtime, libMGPRT.so, and libmax.so both reference cuModuleLoadDataEx from libcuda.so.1, the driver’s own just-in-time path, and the entry points of NVIDIA’s PTX compiler library, nvPTXCompilerCreate and nvPTXCompilerCompile. The library they look for, named in the runtime’s own strings, is the 37.7 MB libNVPTX.so in max-core, and despite the name it has nothing to do with LLVM’s NVPTX backend.

It exports 45,501 dynamic symbols, and 45,474 of them are prefixed libnvptxcompiler_static_. The other 27 are NVIDIA’s public API: fifteen nvPTXCompiler functions, eleven nvLinker functions for linking device code, and one JIT entry point. Its version string reads Cuda compilation tools, release 13.3, V13.3.33, and it depends on nothing but the C and C++ runtimes.

We’re talking about ptxas as a library, NVIDIA’s static PTX compiler repackaged as a shared object and redistributed inside the MAX wheel. The runtime’s error strings name the two ways around it: MODULAR_NVPTX_LIBRARY_PATH for a different copy of the library, and MODULAR_NVPTX_COMPILER_PATH for an external ptxas binary, recommended for older drivers and older hardware.

This is the part of the stack we called the load-bearing moat in our piece on NVIDIA’s compiler: since Kepler and Volta, the dependency interlocks for fixed-latency instructions live in control bits that ptxas writes, so the assembler is correctness-critical and not merely an optimizer.

Mojo replaces CUDA’s source language and a growing share of its library layer, and on NVIDIA it is finished by the same closed assembler as a CUDA C++ kernel. That’s not a criticism of Modular, to be fair. It is the most direct measurement we have of how far a language can reach on NVIDIA hardware, and the answer is up to PTX, one step short of the instructions the chip executes.

We measured nothing about speed, so the best evidence on what that reach buys comes from outside. Godoy and colleagues at Oak Ridge National Laboratory, in a paper at the SC’25 workshops, ported four scientific workloads to Mojo, a seven-point stencil, BabelStream, miniBUDE and Hartree-Fock, and compared them with CUDA on an H100 and HIP on an MI300A.

They found Mojo competitive with the vendor baselines on the memory-bound kernels, and gaps on AMD for atomic operations and for fast-math compute-bound kernels. The study predates 1.0 by almost a year, and it fits the picture here: the language reaches each vendor’s backend, and what remains is how well each backend is driven.

Share The Software Frontier

The vendor libraries never left

MAX is usually described, including by us in July, as a stack that needs no vendor libraries: its own kernels, generated by its own compiler. max-core contains five precompiled binding packages, _cublas, _cudnn, _cufft, _rocblas and _miopen, 1.25 MB together.

Seven source files in MAX’s linear-algebra kernels reference vendor BLAS, including the Hopper and Blackwell matmul dispatchers. The Hopper dispatcher’s own comment says that on a miss, an unsupported configuration or no tuning for the shape, it returns a miss so the caller can fall back to vendor BLAS.

A helper named _vendor_blas_fallback_disabled turns that fallback off, and its documentation says it returns true only when the fallback has been disabled globally or a benchmark asks specifically for the Mojo kernel. The fallback is on unless someone turns it off.

The tuning is where the work is. The matmul tree contains 188 Hopper and 148 Blackwell tuning-config entries, most of them in lists keyed by exact problem shapes. One list is headed as the GEMM shapes of Llama 405B in FP8, with entries such as M=64, N=16384, K=2048.

That is the same structure we found in AMD’s hipBLASLt, where 16,177 kernels cover 2.9 million shape mappings: the moat is not the code generator but the campaign of measurements that decides which generated kernel to use for which shape.

The phrase “no vendor library” describes the tuned path. The default dependency graph still contains cuBLAS. We repeated Modular’s framing uncritically in our July piece on MLIR, and this is the correction.

Share The Software Frontier

How portable the kernels are

The language reaches four families of matrix hardware. The question left is how much of MAX’s kernel library does. We classified every Mojo file under max/kernels/src, 544,039 lines, by its path.

Files that live in a directory or carry a name for one vendor or architecture, such as sm90, sm100, amd_structured or apple, hold 156,621 NVIDIA-specific lines (28.8%), 93,998 AMD-specific lines (17.3%) and 11,903 Apple-specific lines (2.2%).

The remaining 281,517 lines (51.7%) sit on generic paths, and those still branch on the target at compile time. At least 48.3% of the kernel library is written for one vendor or one architecture.

That is not a failure of the language. It is what fast kernels look like, and CUTLASS keeps separate kernel families per architecture generation for the same reason.

What Mojo changes is that the NVIDIA, AMD and Apple slices are written in one language, against one layout algebra and one set of compile-time abstractions, and share the half of the library that is generic. It also gives the ModCon claim a yardstick.

If adding an accelerator really takes ten times less effort than it used to, the next vendor’s slice should look more like Apple’s 11,903 lines than AMD’s 93,998. That is our inference, and it becomes checkable the day that slice appears.

Subscribe now

What Qualcomm bought, as far as the binary can tell

The price is public. Modular raised a $250 million round in September 2025 at a $1.6 billion valuation, bringing total capital to $380 million, according to RuntimeWire.

Qualcomm announced the acquisition on 24 June 2026 in an all-stock deal of about $3.9 billion and closed it on 29 July, with Chris Lattner named executive vice president of advanced AI software and platforms.

That is 10.3 times the capital raised and 2.4 times the last private valuation, the best outcome in the census of GPU software exits we assembled in August. Then came Mojo 1.0 on 11 August, the open compiler on 18 August, and Mojo 1.1 on 17 September, which also opened the compiler to outside contributions.

At ModCon, according to the NAND Research’s account, Modular announced platform support for AWS Trainium, Google TPUs and Qualcomm’s Cloud AI100 and Dragonfly accelerators, said that integrating a new accelerator now takes more than ten times less engineering effort than before, and announced an alliance program for hardware vendors, model providers and clouds.

None of those accelerators appears in what we measured. Mojo 1.1.0 lists NVIDIA, AMD and Apple GPUs and nothing else. No source file in the open repository mentions Qualcomm; the word occurs only inside four test-data files, two GGUF model files and two tokenizer vocabularies.

Hexagon appears in a comment in a C++ support file, in a test fixture, and in a documentation page whose sample target listing includes it next to amdgcn. The shipped binary lists neither: its host backends are AArch64, RISC-V and x86.

The mechanism for adding them is visible, though. On main, the target lookup walks two collections of devices, the builtin one and ADDITIONAL_TARGETS. A vendor can supply its own collection of device records at compile time without editing the table everyone else uses.

Our reading, which is inference and not measurement, is that the announced backends live outside the open tree, at least for now, or reach customers in a form we cannot see, and that the plumbing for plugging them in is already public.

The claim of ten times less effort is testable in exactly one way: the day a Qualcomm target appears in --print-supported-accelerators or in the open tree, the size of the change that added it will say how much effort it took.

What this says about the deal is the more interesting question. Nearly everything we counted is open: the compiler, the standard library and MAX’s kernels, all 523 kernel source files carrying the same Apache 2.0 with LLVM exceptions header as the compiler.

What stays closed is the runtime that compiles and runs models, the 162.6 MB libmax.so and the GPU runtime beside it, shipped under the MAX license, along with whatever target collections live outside the tree.

Opening the compiler commoditizes the frontend, much as NVIDIA let its own become a commodity years ago, building NVVM on LLVM and contributing the NVPTX backend upstream.

Qualcomm is not buying a way to make kernels portable; roughly half of them are written per architecture anyway. It is buying the team, the closed runtime and the plumbing that turns its own accelerators into one more target collection, next to a kernel library that was tuned on other people’s hardware first.

The repository’s root already carries governance for that future: an AI tool policy that asks contributors to label assisted work with an Assisted-by trailer, to keep pull requests small because AI lowers the cost of generating code and not of reviewing it, and to keep a human in the loop, alongside AGENTS.md and CLAUDE.md files written for coding agents.

Share

If you build on Mojo today

Build one executable per GPU architecture. A mojo build for sm_90 embeds sm_90a PTX that NVIDIA’s compiler will not build for Blackwell, and the Hopper outputs and nearly all the Blackwell ones are locked the same way. The family targets such as sm_100f would carry a B200 build to a B300, and Mojo does not ask for them.

The path that follows the hardware is MAX’s, which compiles on the machine the model loads on. On a CI runner without a GPU, pass --target-accelerator with the architecture your kernels target, or public tcgen05 code will not compile, and never build Hopper kernels on a runner configured for Blackwell.

On Jetson Orin, 1.1.0 compiles as sm_80; on Jetson Thor it cannot use tensor memory; on Maxwell and Pascal, point MODULAR_NVPTX_COMPILER_PATH at a ptxas old enough to know them.

If you need a guarantee that no vendor library is in the path, build with -D MODULAR_DISABLE_VENDOR_FALLBACK=true, the define the matmul dispatch reads to turn off the cuBLAS and rocBLAS fallback.

Executables built from a pip install carry an absolute RUNPATH into that Python environment. And don’t contort code to avoid Mojo’s 64-bit Int in index math: on our vector kernel, Int32 index math saved 2 of 20 PTX instructions and 4 of 11 64-bit operations, ptxas allocated 10 registers either way, and the cubin shrank by 128 bytes.

Three of the problems above, the Orin fallthrough, the RTX 3090 PTX version and the two Thor blockers, are small patches. Since 17 September the compiler accepts outside contributions, which may be the most useful thing the open-sourcing did.

Subscribe now

Where I might be wrong

The tcgen05 guard may read the build host on purpose. If MAX’s own launch path always sets the build accelerator to the device a kernel targets, the mismatch we produced can only happen when someone compiles by hand the way we did.

The consequence we measured, tcgen05 inside Hopper PTX, stands, but its practical reach may be small.

We compiled through _compile_code and, to check target selection, through DeviceContext.compile_function, and both picked the same targets. We never launched a kernel. Launching applies rules we never exercised, such as the requirement since 1.0 that kernel arguments be fixed-width types rather than Int. PTX that assembles is not code that is correct or fast.

Nor did we load a mojo build executable on a Blackwell GPU: the refusal of its embedded PTX is the verdict of NVIDIA’s assembler, which the runtime would also have to get past, not something we watched happen.

The kernel census classifies files by path. It is a lower bound on architecture-specific code, and a path that names a vendor can still hold helpers other vendors use.

Our timings come from a one-core container and small programs. Absolute numbers will differ on real machines. We expect the ratios to hold, the LLVM share especially, but the specialization curve is a result about small bodies only.

A PyPI metadata field is not a license text. We did not read the MAX license in full, and we are not lawyers.

The RTX 3090 record may be unreachable in practice. And backends absent from the open tree may exist and ship through channels we cannot see, such as MAX containers or partner builds. The claim is “not in the open tree”, not “does not exist”.

Share

Five dated predictions

First: by 31 March 2027, a stable Mojo release removes at least one of the two Jetson Thor blockers, either by adding sm_110 names to the tcgen05 guard or by compiling Thor as sm_110a or sm_110f.

Second: through 31 December 2027, the default target records for the H100, B200 and B300 stay architecture-specific, with the a suffix. MAX’s runtime compiler has no reason to give those instructions up, and we expect mojo build users to be told to build per architecture rather than see the default change.

Third: by 30 June 2027, a Qualcomm accelerator appears in --print-supported-accelerators of a public Mojo release or in the open repository.

Fourth: through 31 December 2027, every shipped configuration of Mojo and MAX that runs on NVIDIA hardware still finishes code with NVIDIA’s closed PTX compiler, whether bundled, invoked through the driver’s JIT, or as an external ptxas.

Fifth: fn does not return in any 1.x release.

Subscribe now

Confidence dossier

Every claim that carries weight in the piece, graded. M means we measured it and verify.py re-checks it; A means we read it in a primary source, usually the code at a named revision; B means it was reported or claimed and we did not reproduce it; C is our inference. 42 rows are M, 1 reuse our own earlier measurements, 15 are A, 4 are B and 6 are C.

Share

Reproduce it

Everything above ran in a one-core x86-64 container with 3 GB of memory and no GPU, on 21 and 22 September 2026. The probe scripts and the harness are published with this piece as mojo-compile-probe, MIT licensed. verify.py holds 187 checks and exits non-zero at the first number that no longer matches.

pip install mojo==1.1.0 max-core==26.6.0      # Mojo 1.1.0 (8189361e), MAX 26.6.0
pip download nvidia-cuda-nvcc==13.4.92 --no-deps  # ptxas V13.4.92, unzip and use bin/ptxas
git clone --depth 1 https://github.com/modular/modular.git   # measured at 26cfe94
git -C modular fetch --depth 1 origin tag mojo/v1.1.0        # c6fa49f, what shipped

python3 m2_target_sweep.py      # 30 targets, emitted .target and PTX version
python3 m3_ptxas_roundtrip.py   # every NVIDIA output through ptxas, forward-compat tests
python3 m4_tensor_cores.py      # mma, wgmma, tcgen05 per target and build flag
python3 m5_language.py          # the language probes
python3 m6_compile_time.py      # specialization scaling and timing census
mojo build aot.mojo --target-accelerator sm_90 -o aot_sm_90   # then extract the PTX and run ptxas
mojo build m7_launch_path.mojo --target-accelerator sm_90 --emit asm   # launch-path check
python3 verify.py --live        # re-checks every number quoted; exits non-zero on failure

Subscribe now

Corrections before publication

Writing the checks before the prose caught these, in the order they happened. We list them because the method only works if its failures are visible.

1. While probing imports, a shell redirection error made it look as if wildcard imports of missing modules were silently accepted. They are not; the compiler reports them correctly. Caught by re-running the probe with separate output files.

2. The family targets sm_100f through sm_121f were first attributed to Mojo’s LLVM NVPTX backend. The strings came from libNVPTX.so, which is NVIDIA’s nvptxcompiler, not LLVM.

3. The harness caught a conflation in the symbol count: libNVPTX.so exports 45,501 dynamic symbols, of which 45,474 carry the libnvptxcompiler_static_ prefix.

4. A chart subtitle said MAX’s code was five times the compiler. The measured ratio for MAX’s Mojo code is 3.3.

5. The Jetson Orin downgrade was first diagnosed against main, whose table is structured differently. The diagnosis in the text is against the v1.1.0 tag that actually shipped.

6. A draft said the RTX 3090 record was the only one named with a full marketing string. The GTX 1080 Ti, GTX 1060, GTX 970 and Tesla P100 records are too.

7. The first specialization-scaling run used one measurement per point and called the curve flat. Three runs per point show a small but real slope, about 0.56 ms per specialization; the text and Figure 4 use the repeated data.

8. Version 2 said the suffix costs nothing because Mojo compiles at load time and never ships PTX. That holds for MAX, whose runtime library carries the compiler. It is false for mojo build executables, which embed build-time PTX and no compiler; the targets section now separates the two.

9. Tuning-config counts included the struct definitions: the call sites are 188 and 148, not 189 and 149.

10. The pass count mixed 2 passes from the shared Support library into the Mojo compiler’s 45.

11. Hexagon also appears in a documentation page’s sample target list, not only in a comment and a test fixture.

12. Version 2 explained the compile floor as the standard library’s cost without measuring it; the empty-program census now does.

13. Final read-through: the opening said sm_90a unlocks the tensor memory accelerator. ptxas 13.4 accepts TMA copies on plain sm_90; the suffix is what wgmma and setmaxnreg need, and the text now names those.

14. Final read-through: version 3 said 1.1 deleted two keywords. The release notes list three keywords (fn, alias, __comptime_assert) and two syntax forms.

15. Final read-through: version 3 contrasted an open language with a kernel library it implied was not. All 523 kernel source files carry the same Apache 2.0 header; the closed part is the runtime, and the Qualcomm section now says so.

16. Final read-through: a sentence left over from version 2 still called the specialization curve flat.

17. Two framings from our July piece on MLIR are corrected in the text: KGEN is a dialect inside Mojo rather than a layer above it, and MAX’s matmul path keeps a vendor BLAS fallback.

Subscribe now

Sources

Measurements, analysis and conclusions are ours. Source excerpts from modular/modular are Apache 2.0 with LLVM exceptions; compiler and assembler messages are output we generated. Charts and harness: mojo-compile-probe.

Share

A very short version

NVIDIA’s data center revenue more than doubled in a year, to $89.0 billion a quarter. The company now sells whole racks, supplies reference designs for the buildings around them, and increasingly backs the financing as well.

The Vera Rubin rack started shipping in August, and the version of its product page updated on September 11 lists lower memory and NVLink bandwidth than the August version did, without explanation, which moves the batch size at which these machines stop being limited by memory.

Power, cooling and memory are the constraints that matter now, and the balance sheet shows it: $279 billion of supply commitments, mostly memory, up to $105 billion of guarantees behind one Ohio campus, and operating cash flow that fell to 40% of net income, partly because some customers were given longer to pay.

Subscribe now

The real introduction

We’ve alredy explored the fact that NVIDIA’s second quarter of fiscal 2027 closed on July 26, 2026, and the company reported it on August 26. On it we saw that revenue was $96.2 billion, up 106% from a year earlier, and $89.0 billion of it came from the data center.

But as many of you know, those are just the mere “headline numbers”. The part of the filing that says more about where the data center business is going towards sits further down the CFO commentary, in a section about commitments and guarantees.

That small section lists $279 billion of supply and capacity commitments, up from $119 billion just three months earlier, an increase NVIDIA itself attributes mainly to memory.

It lists $36 billion under agreements with AI clouds, a structure the CFO described on the call as a take-or-pay commitment on part of a facility’s capacity in exchange for a share of its revenue, and $20 billion of data center leases that NVIDIA signed and expects to reassign to third parties.

What sounds obvious is that neither of those two lines appears in the first quarter’s commentary. And it lists guarantees with a maximum gross exposure of $108.5 billion, of which $105 billion backs the land, power and shell for 4.25 gigawatts at a campus in Ohio that OpenAI will lease for 20 years.

Read next to the product news since the end of May, those tables describe a company whose unit of sale has moved from the chip to the rack and now to the gigawatt. At that scale the scarce inputs are memory, power, buildings and credit, and NVIDIA is now putting its own balance sheet behind all four.

Subscribe now

The quarter

In the filing’s own terms, second quarter revenue grew 18% sequentially, and Data Center grew 18% sequentially and 117% year over year. That put Data Center at 92.5% of the total, against 92.2% in the first quarter.

Everything else now sits in a platform called Edge Computing: $7.2 billion in the quarter, up 13% sequentially and 27% from a year ago. NVIDIA adopted this two-platform framework in the first quarter.

Data Center is split into Hyperscale and ACIE, and Edge Computing covers PCs, game consoles, workstations, AI-RAN base stations, robotics and automotive. GAAP gross margin was 75.0%, GAAP operating income $63.7 billion and GAAP net income $59.7 billion, which includes $7.8 billion of pre-tax net gains on equity securities.

The CFO pointed out on the call that growth accelerated for the fourth quarter in a row. Year over year, revenue grew 56% in the second quarter of fiscal 2026, then 62%, 73%, 85% and now 106%. The third quarter guide of $108.0 billion, plus or minus 2%, means 89% growth at the midpoint and 93% at the top of the range, so extending the streak would take a result above $117.3 billion, 8.7% over the midpoint.

NVIDIA beat its own midpoint by 4.6% in the first quarter and by 5.7% in the second. The comparison base is also harder: revenue in the third quarter of fiscal 2026 was 22% above the quarter before it.

For fiscal 2028, which roughly corresponds to calendar 2027, NVIDIA gave a preliminary outlook of about 70% revenue growth and described it as supply-constrained. Huang said the company had never guided a full year ahead before.

The CFO said customer forecasts point to growth doubling, and that supply should remain a bottleneck at least through the end of fiscal 2028.

The third quarter outlook assumes no Data Center compute revenue from China. Shipments of Hopper products to China were below 1% of Data Center revenue in the quarter; in the first quarter of fiscal 2026, before the license requirement, NVIDIA had sold $4.6 billion of H20.

Subscribe now

Who is buying

Hyperscale, which NVIDIA defines as the public clouds and the largest consumer internet companies, brought in $48.7 billion, up 13% sequentially and 102% from a year ago. ACIE, short for AI clouds, industrial and enterprise, brought in $40.3 billion, up 25% sequentially and 138% from a year ago.

On the call the CFO tied ACIE growth to capacity that neoclouds added for enterprises, AI startups and sovereign customers, and to hyperscalers buying capacity from AI clouds to supplement their own buildouts.

NVIDIA expects its neocloud partners to finish 2026 with 8 gigawatts installed, against roughly 3 gigawatts at the end of 2025, and said sovereign revenue, mostly booked through regional neoclouds, grew 35% sequentially and more than tripled year over year.

The second quarter commentary also contains a line that matters for anyone tracking this split: a company was moved from ACIE to Hyperscale because its business model changed, and prior periods were recast.

The size of the move can be read off the two commentaries. The first quarter was originally reported as $37.87 billion of Hyperscale and $37.38 billion of ACIE, so ACIE was 49.7% of Data Center.

In the recast the same quarter shows $43.05 billion and $32.20 billion, and ACIE drops to 42.8%. The difference between the two versions, $5.18 billion, is the first quarter revenue of that one customer, which NVIDIA does not name, and 6.9% of the whole Data Center line.

On the new basis, ACIE was 45.3% of Data Center in the second quarter. When NVIDIA describes the non-hyperscale business as roughly half of Data Center, as both executives did on the call, it helps to remember that a single classification decision moved that share by about seven points.

Two other statements from the call frame the customer mix. The CFO said the AI labs whose buildouts NVIDIA expects to support with its balance sheet should account for roughly a quarter of NVIDIA’s business next year.

On the hyperscale side, NVIDIA put cloud industry backlog above $2 trillion and capital expenditure by the five largest hyperscalers at nearly $800 billion in 2026 and $1.3 trillion in 2027. Those are NVIDIA’s figures, not a sum rebuilt here from each company’s guidance.

The same call announced that AWS will deploy an additional 2 million GPUs between this quarter and the second quarter of fiscal 2029.

Subscribe now

Timeline

The data center news of the last few months, in order, with the sources used for each entry:

• Jan 5. CES keynote: Vera Rubin in full production; Huang says 45°C water needs no chillers.

• May 18. Quarterly dividend raised from $0.01 to $0.25 a share; $80.0 billion added to the buyback authorization.

• May 31. Vera Rubin ramps into full production; DSX announced; Spectrum-X Ethernet Photonics in production.

• Early July. SemiAnalysis reports a Kyber delay to 2028; NVIDIA says its roadmap is intact.

• Aug 10. Financing platforms with six asset managers and banks, memorandums of understanding for more than $500 billion.

• Aug 17. Guarantees for SB Energy’s PORTS-Pike campus; 8-K filed.

• Aug 26. Second quarter results: $96.2 billion of revenue; Vera Rubin production shipments under way; Groq 3 LPX in full production.

• Sep 9. Australia: up to 2 GW with eight partners by 2027.

• Sep 10. Huang at a Goldman Sachs conference repeats $3 trillion to $4 trillion by 2030.

• Sep 11. Vera Rubin NVL72 product page updated; memory and NVLink bandwidth lower than in the August version.

• Oct 21. GTC Berlin keynote.

• Nov 17. Third quarter results.

Share The Software Frontier

The rack

Production shipments of Vera Rubin started precisely in August. NVIDIA says it already holds purchase orders from every major hyperscaler, AI cloud and system OEM, and it expects Vera Rubin to be about 20% of Data Center revenue in the third quarter.

If Data Center keeps the 92.5% share of revenue it had in the second quarter, it would be about $100 billion of the $108.0 billion guide, and 20% of that is roughly $20 billion of Vera Rubin in a single quarter.

NVIDIA’s specification table for the Vera Rubin NVL72 rack gives the shape of the machine; the figures here are from the version of the page updated on September 11.

The rack holds 72 Rubin GPUs and 36 Vera CPUs. Each GPU carries 288 GB of HBM4 with 19.2 TB/s of memory bandwidth, and the rack is listed at 20.7 TB and 1,400 TB/s. NVLink 6 gives every GPU 3 TB/s of scale-up bandwidth, 216 TB/s for the rack, and NVLink-C2C links each Vera to its GPUs at 1.8 TB/s.

Each Vera has 88 custom Olympus cores and 176 threads, 3,168 cores and 6,336 threads in the rack, with up to 1.5 TB of LPDDR5X per CPU and up to 54 TB per rack. Scale-out networking is listed at 0.45 TB/s per GPU and 32.4 TB/s per rack, both bidirectional, and the inlet temperature at 45°C.

NVIDIA’s own technical blog, which I tried to read carefully before writing all of this post, adds that HBM4 doubles the interface width of HBM3e and that Rubin nearly triples memory bandwidth over Blackwell.

The same table rates the rack at 3,600 PFLOPS for NVFP4 inference and 288 PFLOPS at BF16, a factor of 12.5 on identical silicon. Its footnotes also mark the NVFP4 inference row as a sparse specification and the training, BF16 and TF32 rows as dense ones, so the headline number is not only in the most compressed format in the table but also on a different basis from the rows below it.

A comparison between generations only means something when both sides use the same precision and the same basis.

A second ratio from the table: each GPU’s NVLink bandwidth is 16% of its memory bandwidth, 3 TB/s against 19.2 TB/s. Work that has to cross GPUs inside the rack, such as tensor parallel collectives or the all-to-all traffic of mixture-of-experts models, moves over about a sixth of the bandwidth each GPU has for reading its own weights and KV cache.

NVIDIA’s performance claims for Vera Rubin use at least three baselines. The product page says the NVL72 delivers up to 10x more tokens per megawatt and one tenth the cost per million tokens compared with GB200 NVL72, both measured on Kimi-K2-Thinking with 32K input and 8K output tokens, and that it trains a 10 trillion parameter mixture-of-experts model on 100 trillion tokens in a month with a quarter of the GPUs, in NVIDIA’s projection.

The August call claimed 30x higher throughput per megawatt and 35x lower token cost against Grace Blackwell Ultra, the GB300 generation that followed GB200. The May release claimed 10x agent throughput at scale against Grace Blackwell.

If GB300 is at least as efficient per megawatt as GB200, a 30x gain over it and a gain of at most 10x over GB200 cannot both hold on the same workload and configuration; the product page itself attaches a 35x per-megawatt figure to Vera Rubin paired with LPX. All of these remain vendor figures until independent benchmarks run on shipping racks.

Around the NVL72, NVIDIA now sells what it calls a POD of five rack types working as one system: the NVL72 itself, Vera CPU racks, Groq 3 LPX, Vera BlueField-4 STX storage and Spectrum-6 SPX Ethernet.

Groq 3 LPX, NVIDIA’s first rack-scale LPU system, is in full production, with volume shipments to early adopters expected later this quarter and Nebius first in line.

Jensen Huang described it as an add-on for services that will pay for very high interactivity, with lower throughput and a higher cost per token, and said most data centers will simply run Vera Rubin NVL72.

The Vera CPU is also sold on its own. Grace revenue passed $5 billion over the trailing twelve months, Vera is already shipping to lead partners with AWS starting this quarter, and NVIDIA expects CPU revenue to more than double in fiscal 2028.

Subscribe now

The revised table

The table above differs from the one quoted before September, and several write-ups published in the last month still carry the older numbers.

Spheron, which retrieved NVIDIA’s table on August 16, recorded 22 TB/s of HBM4 bandwidth per GPU, 1,580 TB/s per rack and 260 TB/s of NVLink 6, and NVIDIA’s own January technical blog gives NVLink 6 as 3.6 TB/s per GPU.

The page NVIDIA updated on September 11 lists 19.2 TB/s, 1,400 TB/s and 216 TB/s instead. The rack’s scale-out figure moved the other way, from the 28.8 TB/s reported in July to 32.4 TB/s. The page unfortunately does not say why.

The changes are not a single unit conversion, because the ratios differ, and the compute ratings and memory capacities match the January figures.

The new numbers can be checked against the rest of the page. The 100 MW factory table only reproduces with them: 40,000 GPUs at 19.2 TB/s is 768 PB/s, which the page rounds to 800, while 22 TB/s would give 880.

The rack total no longer equals 72 times the per-GPU figure exactly: that product is 1,382.4 TB/s against the 1,400 listed, a 1.3% rounding. Anything calculated from the old bandwidth, including the critical batch below, has to be redone.

Subscribe now

Ratings by precision

The ratings in the table span three orders of magnitude on the same silicon. Per GPU, NVIDIA lists 50 PFLOPS for sparse NVFP4 inference, 35 for dense NVFP4 training, 17.5 for FP8 or FP6 training, 4 at FP16 or BF16 and 2 at TF32, then 130 TFLOPS at FP32 and 33 TFLOPS at FP64.

The top of that list is 1,515 times the bottom. The steps between rows are simple ratios: dense NVFP4 is exactly twice FP8, FP8 is 4.375 times BF16, BF16 is twice TF32, and TF32 is 15.4 times the plain FP32 rating. The sparse inference row sits 1.43 times above dense NVFP4 training.

Each row answers a different question. The training rows describe dense matrix math, the sparse inference row folds in whatever sparsity NVIDIA assumed, and the FP64 row is the one that matters for classical simulation.

Ratios between a sparse row and a dense one, or between generations rated on different bases, look precise and mean little.

Subscribe now

The critical batch

A decode step reads every active weight once and does roughly two floating point operations per active parameter for every sequence in the batch. With N active parameters stored at b bytes each, a batch of B needs N × b bytes of memory traffic and 2 × N × B floating point operations per step.

The step is limited by memory bandwidth while N × b divided by the bandwidth is larger than 2 × N × B divided by the compute rating P, and by compute beyond that. N cancels, which leaves a property of the hardware alone:

B* = P × b / (2 × B_mem)

B* is the batch per model replica above which adding sequences stops being free. It counts weights only, at zero context. Longer contexts add KV cache reads for every sequence, which on their own push the real crossover higher, and at long enough contexts no batch makes the step compute-bound at all; attention compute, which also grows with context, pulls the other way.

B* also only makes sense with dense ratings, because the sparse row already assumes less work per parameter than the formula does. The model behind this post enforces that: each rating carries a basis field, and the function refuses a sparse one.

With the September 11 bandwidth of 19.2 TB/s, the dense rows give B* of 456 for NVFP4 and FP8 training and 208 for BF16 and TF32. The pairs match exactly because each rating scales with the inverse of its bytes per parameter: 35 × 0.5 and 17.5 × 1 give the same product, and so do 4 × 2 and 2 × 4.

With the old 22 TB/s, the same rows gave 398 and 182, so the revision moves the crossover 14.6% higher: a Rubin GPU stays memory-bound at larger batches than the earlier table implied. Plugging in the sparse inference row anyway would give 651, a number that describes nothing.

The memory ladder

Bandwidth falls sharply at each step away from the GPU’s own memory. Per GPU, NVIDIA lists 19.2 TB/s for HBM4, 3 TB/s for NVLink 6 and 0.45 TB/s for scale-out networking; each Vera connects at 1.8 TB/s over NVLink-C2C; and BlueField-4 runs at up to 800 Gb/s, which is 0.1 TB/s NVIDIA’s Rubin platform page gives Vera’s own LPDDR5X band.

width as up to 1.2 TB/s, the figure VideoCardz also reported from CES. On the figures as listed, local HBM4 is 6.4 times NVLink, 10.7 times NVLink-C2C, 16 times Vera’s memory and 42.7 times the scale-out figure.

Those ratios turn into time when data has to move. Assuming each interconnect figure counts both directions, as the product page states for scale-out and NVIDIA’s NVLink page states for NVLink, one direction carries half.

On that basis, and taking BlueField-4’s 800 Gb/s as a single direction, 100 GB takes about 67 milliseconds over NVLink 6, 111 milliseconds over NVLink-C2C, 444 milliseconds over the scale-out network and 1 second through BlueField-4, while a GPU reads 100 GB from its own HBM4 in about 5.2 milliseconds.

The rack holds 20.7 TB of HBM4 and up to 54 TB of LPDDR5X behind NVLink-C2C, 74.7 TB in all, and NVIDIA’s 100 MW factory table counts 12 PB of HBM4 and up to 30 PB of LPDDR5X across 40,000 GPUs, which it calls 42 PB of fast memory.

Whatever does not fit in that tier, KV cache for long agent sessions included, sits behind links that are several times slower.

Share

SRAM next to HBM

The Groq 3 LPX rack makes the opposite trade. NVIDIA’s page lists 256 LPUs per rack, each with 500 MB of SRAM, 150 TB/s of SRAM bandwidth and 2.5 TB/s of scale-up bandwidth, which gives 128 GB of SRAM, about 40 PB/s of SRAM bandwidth and 640 TB/s of chip-to-chip bandwidth per rack, plus 12 TB of DDR5 for larger models.

Against a Vera Rubin NVL72, an LPX rack has about 162 times less accelerator memory, SRAM against HBM4, and about 28 times more accelerator memory bandwidth.

A compact way to compare the two is the sweep rate, bandwidth divided by capacity: how many times per second a chip could read its entire memory. A Rubin GPU sweeps its 288 GB about 67 times a second. A Groq 3 LPU sweeps its 500 MB 300,000 times a second, 4,500 times as often.

If a chip’s memory were full of weights that every token has to read, the sweep rate would be its ceiling on tokens per second at batch one.

The limit is capacity: at 500 MB per chip, a trillion-parameter model cannot live in SRAM alone, which is why NVIDIA positions LPX as an accelerator for Vera Rubin and says the GPUs and LPUs jointly compute every layer for every output token.

Subscribe now

Networking

Networking is the part of the rack that grew fastest. Data Center networking revenue went from $3.0 billion in the fourth quarter of fiscal 2025 to $14.8 billion in the first quarter of fiscal 2027, 4.9 times in five quarters, and from 8.5% to 19.7% of Data Center revenue.

For the whole of fiscal 2026 it was $31.4 billion, up 142%. The fourth quarter commentary credited the NVLink compute fabric of GB200 and GB300 systems along with the growth of Ethernet and InfiniBand.

The second quarter commentary no longer splits Data Center into compute and networking. On the call NVIDIA said networking revenue grew 18% sequentially and that Spectrum-X Ethernet grew 2.6 times year over year.

If that 18% applies to the same line as the first quarter’s $14.8 billion, second quarter networking was about $17.5 billion. That is an estimate made here, not a reported number.

Spectrum-X Ethernet Photonics, which NVIDIA describes as the first co-packaged optics switches with 200 Gb/s SerDes, is in production, and NVIDIA claims 5x better power efficiency, 5x longer uptime and 1.3x faster deployment than networks built on traditional transceivers.

In the second quarter NVIDIA said Spectrum-6 switch systems supporting both pluggable and co-packaged optics are arriving in gigascale AI factories. NVIDIA’s case for co-packaged optics is made at the facility level: power the optics no longer burn is power the operator can give to compute.

Per GPU, the September 11 table puts scale-up at 3 TB/s and scale-out at 0.45 TB/s, as NVIDIA lists them, a ratio of 6.7. The same page names the rack’s scale-out fabrics, Quantum-X800 InfiniBand and Spectrum-X Ethernet, alongside the ConnectX-9 SuperNICs and BlueField-4 DPUs inside it.

Subscribe now

Kyber

The rack after this one is under more pressure. Jen Huang said at GTC in March 2025 that each Kyber rack for Rubin Ultra would draw 600 kW.

In early July, SemiAnalysis reported that Kyber had slipped to 2028 because its printed circuit midplane is hard to manufacture, and that a stopgap design joining two current racks back to back had been dropped after large customers objected.

Tom’s Hardware, reporting those claims, cites trade analyses describing a 78-layer board and notes that Kyber holds 144 GPU packages against 72 in the current Oberon rack.

NVIDIA’s reply to the publication was four words: “Our roadmap is intact.” The reported delay does not touch Vera Rubin, which uses the current rack.

The gigawatt

On the call NVIDIA offered a way to measure its share of a buildout: revenue opportunity per gigawatt. It put Hopper at about $18 billion, Grace Blackwell at about $25 billion and Vera Rubin at about $40 billion, with the Vera Rubin figure covering the Vera CPU, the Rubin GPU, NVLink, InfiniBand or Ethernet, and the Groq LPU.

That is 2.2 times the Hopper figure in two generations. Huang also said the total investment in a gigawatt of data center has gone from about $30 billion five years ago to about $60 billion today. If both statements describe the same gigawatt, NVIDIA’s content is about two thirds of it.

The Ohio disclosure gives a second way to check the $40 billion figure. NVIDIA says each generation of its infrastructure deployed at PORTS-Pike could represent about 1.5 million GPUs, or $150 billion to $200 billion of NVIDIA revenue.

On the call that statement follows the description of the initial 4.25 gigawatts, and reading it that way gives $35.3 billion to $47.1 billion per gigawatt; $40 billion times 4.25 is $170 billion, inside the range, which supports that reading.

The same numbers give about 353,000 GPUs per gigawatt, or 2.83 kW of IT load per GPU once everything else inside the IT envelope, from CPUs to switches to storage, is divided among the GPUs, and $100,000 to $133,000 of NVIDIA revenue per GPU.

The product page gives a third data point. NVIDIA’s 100 MW factory table, built on DSX with MaxLPS, counts 40,000 Rubin GPUs, which is 2.5 kW per GPU or 400,000 GPUs per gigawatt.

If the MaxLPS gain of up to 40% applied in full, the same site without it would hold fewer GPUs at 2.5 kW × 1.4, or 3.5 kW each. The Ohio figures, 2.83 kW per GPU and 88.2% of the MaxLPS density, fall between the two.

The bases are not identical, since the Ohio numbers are per IT gigawatt and the factory table does not say whether its 100 MW is IT load, so the comparison shows consistency rather than proof.

Power

Most of what changes at the facility level comes from power. NVIDIA’s own description of the problem is that racks today distribute 54 V DC over copper busbars, that GB200 and GB300 NVL72 racks use up to eight power shelves, that a Kyber rack at megawatt scale would need up to 64U of power shelves on 54 V, and that a single 1 MW rack on 54 V would need up to 200 kg of copper busbar.

Its answer is 800 V DC distribution. NVIDIA says row-level 800 V DC busways can move 85% more power than 415 V AC through the same conductor size and reduce copper requirements by 45%, and that the architecture improves end-to-end efficiency by up to 5%. It has also said full-scale production of 800 V DC data centers will coincide with Kyber.

The arithmetic behind the change is Ohm’s law. A 600 kW rack at 54 V draws about 11,100 amps; at 800 V it draws 750 amps, 14.8 times less. Resistive loss scales with the square of the current, so the same conductor would dissipate about 219 times less heat, and that margin is what lets a designer use far less copper instead.

The calculation leaves out conversion stages and distances, which differ between the two designs, but it shows why the voltage had to change. It also ties the facility to the rack: if Kyber slips, so does the point at which NVIDIA expects 800 V DC to reach full scale.

Cooling moved in the same direction. The September 11 table lists a 45°C inlet for the rack, and at CES in January Huang said the power of Vera Rubin is twice that of Grace Blackwell while the airflow is about the same and the water still enters at 45°C, and that at that temperature data centers need no water chillers. The product page does not list a rack power figure.

The physics of warm water is simple: heat carried equals mass flow times specific heat times the temperature rise, so removing 100 kW with a 10 K rise takes about 145 liters of water per minute, and doubling the rise to 20 K halves the flow to about 72 liters. Whenever the outdoor air is cooler than the water coming back from the racks, the heat can be rejected by dry coolers with fans and no compressor, and a warmer loop makes that true for more hours of the year than a colder one.

Assembly is the other constraint NVIDIA keeps pointing at. Its January technical blog says the modular compute and switch trays enable up to 18x faster assembly, its Rubin platform page says the comparison is against Blackwell, and Thunder Compute describes assembly time going from over an hour and a half to about five minutes, a ratio that matches. The product page adds that the rack uses cable-free modular trays and is supported by more than 80 MGX ecosystem partners.

The DSX platform is NVIDIA’s attempt to sell the facility layer as a reference design. It bundles validated designs that cover compute, networking, storage, power and cooling, a simulation layer called DSX Sim, grid integration called DSX Flex, open source operations software called DSX OS, and DSX MaxLPS, which NVIDIA says combines 45°C liquid cooling with in-rack technologies so an operator can run up to 40% more GPUs at their most efficient operating point inside a fixed power budget.

The trade is easy to state: 1.4 times the GPUs comes out ahead as long as each GPU gives up less than 28.6% of its throughput. NVIDIA says the impact on workload performance is minimal, and the announcement does not include the curve. DSX Flex is running a multi-megawatt pilot with Emerald AI and Silicon Valley Power that adjusts AI factory load to grid signals.

The sites are getting larger and more specific. PORTS-Pike is being developed on the grounds of the decommissioned Portsmouth Gaseous Diffusion Plant in Pike County, Ohio, with capacity coming online in phases from 2028. SB Energy and SoftBank plan at least 10 GW of new generation to support 8 GW of IT capacity, all of it for OpenAI, plus at least $4.2 billion of regional grid investment, and NVIDIA is investing $1.5 billion in SB Energy.

The ratio of planned generation to IT capacity there is 1.25. In Australia, NVIDIA announced on September 9 that eight cloud and data center partners there are expanding land, power and shell to host its DSX factories, with up to 2 GW by 2027.

At a Goldman Sachs conference the next day, Huang repeated his estimate of $3 trillion to $4 trillion of AI infrastructure spending by 2030 and acknowledged that land and power could slow deployment.

Memory

Supply-related commitments were $50.3 billion at the end of the third quarter of fiscal 2026, $95.2 billion a quarter later, $119.0 billion at the end of the first quarter of fiscal 2027, and $279 billion now.

That is 5.5 times in three quarters, with $160 billion added in the last quarter alone and attributed mainly to memory. By fiscal year, $92 billion falls due in the rest of fiscal 2027, $87 billion in fiscal 2028 and $88 billion in fiscal 2029, so $267 billion, or 96%, is due by the end of fiscal 2029.

The 10-Q says the commitments cover data center infrastructure systems, primarily memory and manufacturing facilities, and that some of the underlying agreements can be cancelled, rescheduled or adjusted before firm orders are placed, possibly at additional cost.

For scale: at the third quarter guide, one quarter of cost of revenue is about $28.1 billion, which is $108.0 billion times one minus the 74.0% margin. The $92 billion due in the rest of fiscal 2027 is about 3.3 quarters of that, and two quarters at that rate come to about $56.2 billion. Inventory rose from $19.8 billion to $31.6 billion over the same three quarters, and NVIDIA ties the latest increase to the Vera Rubin introduction.

Whatever is not consumed by shipments in those two quarters has to show up as inventory, as prepayments, or as capacity paid for ahead of fiscal 2028, when NVIDIA plans to grow about 70%. The filing does not break the $92 billion down.

The margin guide points the same way. The CFO described “extreme pricing conditions in memory,” said the increases had exceeded NVIDIA’s expectations and would go higher next year, and reset the outlook: 74.0% gross margin in the third quarter, plus or minus 50 basis points, a trough of 71% to 72% in the fourth quarter, and 72% to 73% in fiscal 2028 as price increases NVIDIA has already executed take effect in its first quarter.

On a GAAP basis the second quarter was 75.0%, the same as the fourth quarter of fiscal 2026. The only large dip in the eight quarters charted was the 60.5% of the first quarter of fiscal 2026, when NVIDIA took a $4.5 billion charge on H20.

Put per $100 of revenue, cost of revenue is $25.00 at the second quarter’s 75.0% margin, $26.00 at the third quarter guide, $28.50 at the midpoint of the fourth quarter trough and $27.50 at the midpoint of the fiscal 2028 range: 4%, 14% and 10% more cost for each dollar of sales than in the second quarter.

Rubin is built around HBM4, 288 GB per GPU in NVIDIA’s specification. On the call NVIDIA said it works with all three major memory suppliers to add the capacity its roadmap requires, and in the second quarter it announced a multiyear technology partnership with SK hynix.

Subscribe now

Credit

The full list, as of July 26, 2026, is in the CFO commentary. Commitments total $366 billion: $279 billion for supply and capacity; $29 billion of cloud service agreements that support NVIDIA’s own research, open models and autonomous vehicle software; $25 billion of data center leases that have not started, with terms of up to 20 years beginning between the third quarter of fiscal 2027 and fiscal 2033; $25 billion of equity investments in AI model makers, infrastructure financiers and other private companies; and $8 billion of capital expenditure.

Additional commitments total $56 billion: $36 billion of AI cloud agreements, and $20 billion of leases with terms of about 15 years, starting in fiscal 2028 or 2029, that NVIDIA expects to reassign.

Guarantees add $108.5 billion. The sum, $530.5 billion, is 5.8 times the $91.3 billion of total liabilities on the balance sheet and 1.75 times the $303.0 billion of revenue over the last four quarters. The sum mixes firm purchase obligations with a contingent maximum exposure, so it is a ceiling rather than a bill.

The AI cloud agreements are the new commercial structure. NVIDIA earns revenue on the upfront sale of the infrastructure and, if certain criteria are met, takes part of the revenue the AI cloud earns from its own customers.

On the call the CFO explained the mechanics. NVIDIA commits to take or pay for part of a facility’s capacity, which gives lenders a minimum revenue to underwrite, and in exchange shares in the neocloud’s revenue above that floor. NVIDIA says independent capital still underwrites each deal, that it is not making loans, and that it gets paid twice, once for the hardware and again from rental income.

The Ohio guarantees are residual value guaranties, and the 8-K describes how they work. There are several agreements tied to the leases, covering about 4.25 GW of IT load in total. Each generally becomes effective when its lease commences, total payment obligations are capped at $105 billion, and payment is conditional, among other things, on the lessor meeting ready-for-service conditions, expected from 2028.

The trigger events are an OpenAI insolvency that causes a lease default, or OpenAI failing to pay under a lease. In either case NVIDIA pays roughly the shortfall between a guaranteed minimum value of the lease and whatever is recovered by reletting or selling, and it can choose to assume the lease, have the lessor try to relet, start a sale, let the lease terminate, or defer for up to a year while covering specified costs.

The CFO commentary adds that the exposure declines as OpenAI pays rent, and that NVIDIA has the option to support about 3.8 GW more. Data Center Frontier reports that OpenAI has agreed to reimburse NVIDIA for amounts it actually pays.

In effect, NVIDIA is guaranteeing the longest-lived layer of the site (land, power and shell under 20-year leases) in order to sell the shortest-lived one (compute that NVIDIA expects to upgrade several times over those 20 years).

The cap works out to $24.7 billion per gigawatt, and to between 52.5% and 70% of the revenue NVIDIA attaches to a single generation of hardware at the site. What NVIDIA would actually lose in a default is not the rent itself but the gap between the guaranteed value and what a powered shell of that size would fetch from a new tenant or a buyer at that moment.

The rest of the lab exposure came out on the call. NVIDIA has invested nearly $50 billion in frontier AI labs. For another lab, which it did not name, it will provide selective credit enhancement for nearly 2 GW of compute. OpenAI’s existing and planned commitments add up to about 12 GW of NVIDIA compute.

The financing platforms NVIDIA announced on August 10 with six large asset managers and banks aim to mobilize more than $500 billion of third-party capital, but they rest on memorandums of understanding and remain subject to final agreements.

The CFO anticipated the obvious objection, that this is circular financing, and answered that NVIDIA compute is fungible and can be redeployed to other customers, which in NVIDIA’s view limits the risk.

Subscribe now

Cash

GAAP net income was $59.7 billion in the quarter, operating cash flow $24.1 billion and free cash flow $21.3 billion, against $58.3 billion, $50.3 billion and $48.6 billion in the first quarter. Operating cash flow was 40% of net income, down from 86%. The CFO commentary attributes the sequential decline to higher working capital and cash taxes.

The cash flow statement shows the working capital part: receivables absorbed $22.3 billion, inventories $5.8 billion, and prepaid expenses and other assets $5.5 billion, and net income included $7.8 billion of non-cash, pre-tax gains on equity securities.

Days sales outstanding rose from 45 to 60, which NVIDIA attributes to extended payment terms on large multi-quarter agreements with certain investment-grade customers.

The balance sheet has changed shape as well. Marketable equity securities of $42.8 billion and non-marketable securities of $51.2 billion total $93.9 billion, up from $35.1 billion at January 25 and now larger than the $56.6 billion of cash and marketable debt securities. In the first half of the fiscal year NVIDIA spent $42.4 billion on purchases of equity securities and booked $23.7 billion of net gains on them.

Total debt is $33.4 billion after $25.0 billion of senior unsecured notes issued in the quarter, against $8.5 billion at January 25, and financing activities include a $2.9 billion outflow labeled Groq, Inc.

Cash paid for buybacks and dividends was $25.8 billion ($19.7 billion and $6.0 billion), which NVIDIA describes as approximately $26.0 billion returned. The quarterly dividend went from $0.01 to $0.25 a share after a board decision on May 18, and $99.0 billion of buyback authorization remains.

November 17

NVIDIA reports its third quarter on November 17. The numbers above give a short list of things to check in that release: revenue against the $108.0 billion guide, and whether it clears the $117.3 billion that would keep growth accelerating;

Vera Rubin’s share of Data Center against the 20% NVIDIA expects; gross margin at 74.0% on the way to the fourth quarter trough; days sales outstanding against this quarter’s 60; and the supply line against inventory, which shows whether the commitments are turning into goods or into capacity reserved for later. Before that, Huang gives a keynote at GTC Berlin on October 21.

Reproduce the numbers

Every value marked M in the dossier comes from arithmetic on the sources, and the core of it fits in a short Python script with no dependencies. It prints the values used above.

# Inputs come from the NVIDIA filings and product pages listed under Sources.
rev = {"Q2FY26": 46743, "Q3FY26": 57006, "Q2FY27": 96221}   # $ millions

# Q3 FY27 revenue needed to keep year-over-year growth accelerating ($B)
print(round(rev["Q3FY26"] * rev["Q2FY27"] / rev["Q2FY26"] / 1000, 1))

# Q1 FY27 revenue moved from ACIE to Hyperscale by the recast ($B)
print((43050 - 37869) / 1000)

# Critical batch, weights only, zero context: B* = P * b / (2 * B_mem)
def b_star(pflops, bytes_per_param, bandwidth_tbs):
    return pflops * 1e15 * bytes_per_param / (2 * bandwidth_tbs * 1e12)

print(round(b_star(17.5, 1.0, 19.2)), round(b_star(4, 2.0, 19.2)))   # Sep 11 table
print(round(b_star(17.5, 1.0, 22.0)), round(b_star(4, 2.0, 22.0)))   # earlier table

# Sweep rate: bandwidth over capacity, per second
rubin = 19.2e3 / 288
lpu = 150e3 / 0.5
print(round(rubin, 1), round(lpu), round(lpu / rubin))

# Current for a 600 kW rack at 54 V and 800 V, and the resistive loss ratio
i54, i800 = 600e3 / 54, 600e3 / 800
print(round(i54), round(i800), round((i54 / i800) ** 2))

# Ohio: NVIDIA revenue per gigawatt ($B) and IT kW per GPU
print(round(150 / 4.25, 1), round(200 / 4.25, 1), round(4.25e6 / 1.5e6, 2))

# Water to remove 100 kW with a 10 K rise at 45 C (liters per minute)
print(round(100 / (4.18 * 10) / 0.990 * 60))

Glossary

Notes on the numbers

NVIDIA’s fiscal 2027 began on January 26, 2026, and its second quarter ended on July 26, 2026. Fiscal 2028 roughly corresponds to calendar 2027. Margins are GAAP throughout, because since the first quarter of fiscal 2027 NVIDIA’s non-GAAP measures include stock-based compensation and the historical non-GAAP figures were restated.

The fourth quarter trough and the fiscal 2028 range were given on the call without specifying GAAP or non-GAAP; the two measures were within 0.1 points of each other in each of the last two quarters.

The compute and networking split of Data Center was last given for the first quarter of fiscal 2027, rounded to $60.4 billion and $14.8 billion. The second quarter networking figure above is an estimate made here. Hyperscale and ACIE figures for the second quarter and the recast first quarter come from the second quarter commentary, and the as-reported first quarter figures come from the first quarter commentary.

Supply-related commitments are charted for the four quarters in which the commentaries used here report a total under that label. The first quarter of fiscal 2026 used a different definition, $29.8 billion of purchase commitments and obligations for inventory and manufacturing capacity, and is not charted.

Vera Rubin specifications are taken from NVIDIA’s product page as updated on September 11, 2026. The earlier values come from Spheron’s record of the page on August 16 and NVIDIA’s January technical blog, and the earlier scale-out figure from SiliconReport.

Transfer times assume each interconnect figure counts both directions and treat BlueField-4’s 800 Gb/s as one direction. Vera’s memory bandwidth comes from NVIDIA’s Rubin platform page, not from the NVL72 table.

Per-gigawatt revenue, the performance multipliers, the neocloud gigawatt figures and the hyperscaler capital expenditure figures are statements by NVIDIA, not audited numbers. The Kyber delay is a third-party report that NVIDIA has not confirmed. Values marked M in the dossier below are arithmetic on filed numbers, with the formula given.

Subscribe now

Sources

1. NVIDIA, CFO Commentary on Second Quarter Fiscal 2027 Results (8-K exhibit), Aug 26, 2026. https://www.sec.gov/Archives/edgar/data/1045810/000104581026000073/q2fy27cfocommentary.htm

2. NVIDIA, press release: Financial Results for Second Quarter Fiscal 2027, Aug 26, 2026. https://nvidianews.nvidia.com/news/nvidia-announces-financial-results-for-second-quarter-fiscal-2027

3. NVIDIA Q2 FY2027 earnings call, corrected transcript (FactSet, hosted by NVIDIA IR), Aug 26, 2026. https://s201.q4cdn.com/141608511/files/content_files/TRANSCRIPT_-NVIDIA-Corp-NVDA-US-Q2-2027-Earnings-Call-26-August-2026-5_00-PM-ET.pdf

4. NVIDIA, CFO Commentary on First Quarter Fiscal 2027 Results, May 2026. https://s201.q4cdn.com/141608511/files/doc_financials/2027/Q127/Q1FY27-CFO-Commentary.pdf

5. NVIDIA, CFO Commentary on Fourth Quarter and Fiscal 2026 Results (8-K exhibit), Feb 25, 2026. https://www.sec.gov/Archives/edgar/data/1045810/000104581026000019/q4fy26cfocommentary.htm

6. NVIDIA, CFO Commentary on Third Quarter Fiscal 2026 Results (8-K exhibit), Nov 19, 2025. https://www.sec.gov/Archives/edgar/data/1045810/000104581025000228/q3fy26cfocommentary.htm

7. NVIDIA, CFO Commentary on First Quarter Fiscal 2026 Results, May 2025. https://s201.q4cdn.com/141608511/files/doc_financials/2026/Q126/Q1FY26-CFO-Commentary.pdf

8. NVIDIA, press release: Financial Results for Second Quarter Fiscal 2026 (8-K exhibit), Aug 27, 2025. https://www.sec.gov/Archives/edgar/data/1045810/000104581025000207/q2fy26pr.htm

9. NVIDIA, Form 10-Q for the quarter ended July 26, 2026, Aug 2026. https://www.sec.gov/Archives/edgar/data/0001045810/000104581026000075/nvda-20260726.htm

10. NVIDIA, Form 8-K on the SB Energy residual value guaranties, Aug 17, 2026. https://www.sec.gov/Archives/edgar/data/1045810/000104581026000069/nvda-20260817.htm

11. NVIDIA, press release: NVIDIA Guarantees SB Energy’s PORTS-Pike Technology Campus, Aug 17, 2026. https://nvidianews.nvidia.com/news/nvidia-guarantees-sb-energy-s-ports-pike-technology-campus-in-ohio-to-exclusively-host-nvidia-ai-compute

12. NVIDIA, press release: compute financing platforms with Apollo, BlackRock, Blackstone, Brookfield, Goldman Sachs and KKR, Aug 10, 2026. https://nvidianews.nvidia.com/news/nvidia-partners-with-apollo-blackrock-blackstone-brookfield-goldman-sachs-and-kkr-to-establish-ai-compute-infrastructure-financing-platforms-to-mobilize-over-500-billion-of-third-party-capital

13. NVIDIA, press release: Vera Rubin Ramps Into Full Production, May 31, 2026. https://nvidianews.nvidia.com/news/vera-rubin-full-production-agentic-ai-factory

14. NVIDIA, Vera Rubin NVL72 product page and specifications table, updated Sep 11, 2026, accessed Sep 16, 2026. https://www.nvidia.com/en-us/data-center/vera-rubin-nvl72

15. NVIDIA Technical Blog, Inside the NVIDIA Vera Rubin Platform, Jan 5, 2026. https://developer.nvidia.com/blog/inside-the-nvidia-rubin-platform-six-new-chips-one-ai-supercomputer/

16. NVIDIA, press release: NVIDIA DSX Gives Infrastructure Builders the Playbook for AI Factories, May 31, 2026. https://nvidianews.nvidia.com/news/dsx-infrastructure-ai-factory

17. NVIDIA Technical Blog, NVIDIA 800 VDC Architecture Will Power the Next Generation of AI Factories, updated Jan 14, 2026. https://developer.nvidia.com/blog/nvidia-800-v-hvdc-architecture-will-power-the-next-generation-of-ai-factories/

18. Tom’s Hardware, Nvidia’s Kyber rack for Rubin Ultra reportedly delayed to 2028 (reporting a SemiAnalysis thread; includes NVIDIA’s statement), Jul 6, 2026. https://www.tomshardware.com/pc-components/gpus/nvidias-kyber-rack-for-rubin-ultra-slips-to-2028

19. DCD, Nvidia’s Rubin Ultra NVL576 rack expected to be 600kW, GTC 2025 remarks. https://www.datacenterdynamics.com/en/news/nvidias-rubin-ultra-nvl576-rack-expected-to-be-600kw-coming-second-half-of-2027/

20. NVIDIA, press release: Expands AI Infrastructure Capacity in Partnership With Australia’s Data Center Ecosystem, Sep 9, 2026. https://nvidianews.nvidia.com/news/nvidia-expands-ai-infrastructure-capacity-in-partnership-with-australias-data-center-ecosystem

21. Investing.com, NVIDIA at Goldman Sachs Communacopia + Technology Conference (transcript summary), Sep 10, 2026. https://www.investing.com/news/transcripts/nvidia-at-goldman-sachs-conference-huang-sees-ai-buildout-still-early-93CH-4896555

22. Data Center Frontier, PORTS-Pike Takes Shape as an 8-GW AI Infrastructure Model, Aug 2026. https://www.datacenterfrontier.com/hyperscale/article/55398883/ports-pike-takes-shape-as-an-8-gw-ai-infrastructure-model

23. Tom’s Hardware, Nvidia shows off Rubin Ultra with 600,000-Watt Kyber racks (GTC 2025 coverage), Mar 19, 2025. https://www.tomshardware.com/pc-components/gpus/nvidia-shows-off-rubin-ultra-with-600-000-watt-kyber-racks-and-infrastructure-coming-in-2027

24. NVIDIA, Groq 3 LPX product page, updated Aug 24, 2026, accessed Sep 16, 2026. https://www.nvidia.com/en-us/data-center/lpx/

25. Spheron, NVIDIA Vera Rubin NVL72 guide (quotes the NVIDIA specification table as retrieved on Aug 16, 2026), Sep 2026. https://www.spheron.network/blog/nvidia-vera-rubin-nvl72-guide/

26. VideoCardz, NVIDIA Vera Rubin NVL72 Detailed (CES 2026 specifications), Jan 6, 2026. https://videocardz.com/newz/nvidia-vera-rubin-nvl72-detailed-72-gpus-36-cpus-260-tb-s-scale-up-bandwidth

27. Fierce Network, Supercomputers can stay chill with hot water says Nvidia (CES 2026 keynote remarks), Jan 2026. https://www.fierce-network.com/cloud/nvidia-has-no-chill

28. SiliconReport, Nvidia Vera Rubin Explained (earlier scale-out figure and NVL144 naming history), Jul 3, 2026. https://www.siliconreport.com/nvidia-vera-rubin-everything-we-know-33727d4d

29. Thunder Compute, Nvidia Rubin Architecture (assembly time; quotes the earlier bandwidth figures), Sep 2026. https://www.thundercompute.com/blog/nvidia-rubin-architecture

30. NVIDIA, NVLink and NVLink Switch page (describes the NVLink figure as bidirectional), accessed Sep 16, 2026. https://www.nvidia.com/en-eu/data-center/nvlink

31. NVIDIA, Rubin platform page (Vera CPU memory bandwidth; cable-free trays, 18x faster assembly than Blackwell), accessed Sep 16, 2026. https://www.nvidia.com/en-eu/data-center/technologies/rubin/

Subscribe now

Claim dossier

A: filed or official financial document. B: company statement or product claim, not independently verified. C: third-party reporting. M: computed here from the cited inputs, with the formula or method given. Source numbers refer to the Sources list above.

Results and guidance

• Q2 FY27 revenue. $96,221M, up 18% q/q and 106% y/y. Source 1. Tier A.

• Data Center revenue. $89,023M, up 18% q/q and 117% y/y. Source 1. Tier A.

• Data Center share of revenue. 92.5% = 89,023 / 96,221; Q1 92.2%. Sources 1, 4. Tier M.

• Edge Computing revenue. $7,198M, up 13% q/q and 27% y/y. Source 1. Tier A.

• Revenue growth, y/y. Q1 FY26 69.2%, Q2 FY26 55.6%, Q3 FY26 62.5%, Q4 FY26 73.2%, Q1 FY27 85.2%, Q2 FY27 105.9%. Sources 1, 4, 5, 6, 7, 8. Tier M.

• Q3 FY27 guidance. $108.0B plus or minus 2%; GAAP gross margin 74.0% plus or minus 50 bp. Source 1. Tier A.

• Growth at the guide. 89.5% at midpoint, 93.2% at the top = guide / 57,006. Sources 1, 6. Tier M.

• Revenue needed to keep accelerating. $117.3B = 57,006 x 96,221 / 46,743; 8.7% above midpoint. Sources 1, 6. Tier M.

• Beats against guide midpoint. Q1 4.6% (guide $78.0B), Q2 5.7% (guide $91.0B). Sources 1, 4, 5. Tier M.

• Fiscal 2028 outlook. About 70% revenue growth, supply-constrained. Source 3. Tier B.

• China. Hopper shipments to China below 1% of Data Center revenue in Q2 (call); no China Data Center compute revenue in the Q3 outlook; $4.6B of H20 in Q1 FY26. Sources 2, 3, 7. Tiers A, B.

Customers

• Hyperscale and ACIE, Q2 FY27. $48,710M and $40,313M. Source 1. Tier A.

• Hyperscale and ACIE, Q1 FY27 as reported. $37,869M and $37,377M. Source 4. Tier A.

• Hyperscale and ACIE, Q1 FY27 recast. $43,050M and $32,196M. Source 1. Tier A.

• Revenue moved by the recast. $5,181M = 43,050 - 37,869; 6.9% of Q1 Data Center. Sources 1, 4. Tier M.

• ACIE share of Data Center. 49.7% (Q1 reported), 42.8% (Q1 recast), 45.3% (Q2). Sources 1, 4. Tier M.

• Neocloud capacity. About 3 GW at end of 2025; 8 GW expected at end of 2026. Source 3. Tier B.

• Hyperscaler capital expenditure. Nearly $800B in 2026 and $1.3T in 2027 (top five); backlog above $2T. Source 3. Tier B.

• AWS. Additional 2 million GPUs through Q2 FY2029. Source 3. Tier B.

Vera Rubin and the rack

• Vera Rubin NVL72 specification (Sep 11 revision). 72 GPUs, 36 CPUs; 288 GB and 19.2 TB/s per GPU; 20.7 TB and 1,400 TB/s per rack; NVLink 3 TB/s and 216 TB/s; C2C 1.8 TB/s; 88 cores and 176 threads per CPU; up to 1.5 TB LPDDR5X per CPU, 54 TB per rack; scale-out 0.45 and 32.4 TB/s bidirectional; 1,296 NVIDIA and HBM4 chips; inlet 45 C. Source 14. Tier B.

• Ratings per GPU. 50 PFLOPS NVFP4 inference (sparse); 35 NVFP4 training, 17.5 FP8/FP6, 4 BF16, 2 TF32 (dense); 130 TFLOPS FP32; 33 TFLOPS FP64. Source 14. Tier B.

• Specification ratios. 12.5 = 3,600 / 288; 15.6% = 3 / 19.2; span 1,515 = 50,000 / 33; FP8/BF16 4.375; TF32/FP32 15.4; sparse/dense NVFP4 1.43. Source 14. Tier M.

• HBM4 and memory bandwidth. HBM4 doubles HBM3e interface width; nearly 3x Blackwell memory bandwidth. Source 15. Tier B.

• Vera Rubin shipments. Production shipments began in August; about 20% of Q3 Data Center revenue. Source 3. Tier B.

• Vera Rubin revenue in Q3. About $20.0B = 108.0 x 0.925 x 0.20; Data Center about $99.9B. Sources 1, 3. Tier M.

• Vera Rubin performance claims. Product page: up to 10x tokens per MW and one tenth cost per million tokens against GB200 NVL72 (Kimi-K2-Thinking 32K/8K), a quarter of the GPUs for a 10T MoE on 100T tokens, up to 35x per MW with LPX; call: 30x per MW and 35x lower token cost against Grace Blackwell Ultra; May: 10x agent throughput against Grace Blackwell. Sources 3, 13, 14. Tier B.

• Grace Blackwell Ultra. GB300, the rack generation after GB200. Source 28. Tier C.

• CPUs. Grace above $5B trailing twelve months; CPU revenue more than double in FY28. Source 3. Tier B.

• Assembly and partners. Up to 18x faster assembly than Blackwell (January blog, Rubin page); over 1.5 hours to about 5 minutes (Thunder Compute); cable-free trays; more than 80 MGX partners. Sources 14, 15, 29, 31. Tiers B, C.

• CES 2026. Vera Rubin described as in full production. Sources 26, 27. Tier C.

The revised table

• Values before the revision. 22 TB/s and 1,580 TB/s HBM4; 260 TB/s NVLink per rack (page as retrieved Aug 16); 3.6 TB/s NVLink 6 per GPU (January blog); 28.8 TB/s scale-out per rack (July); compute and capacity as in January. Sources 15, 25, 26, 28. Tiers B, C.

• Revision changes. HBM per GPU -12.7%; per rack -11.4%; NVLink per GPU -16.7%; per rack -16.9%; scale-out +12.5%. Sources 14, 25, 28. Tier M.

• 100 MW factory table. 40,000 GPUs with MaxLPS; 2 ZFLOPS NVFP4 inference; 1.4 ZFLOPS NVFP4 training; 700, 160, 80 EFLOPS; 12 PB HBM4 at 800 PB/s; up to 30 PB LPDDR5X; 42 PB fast memory. Source 14. Tier B.

• Factory table check. 40,000 x 19.2 = 768 PB/s (listed 800); 40,000 x 22 = 880; rack product 1,382.4 against 1,400 (1.3%). Source 14. Tier M.

• Critical batch B*. New: 455.7 (NVFP4, FP8 training), 208.3 (BF16, TF32); old: 397.7, 181.8; change 14.6%; sparse row refused (651 if forced). Sources 14, 25. Tier M.

Memory, bandwidth and SRAM

• Bandwidth ladder. HBM4 / NVLink 6.4; / C2C 10.7; / Vera 16; / scale-out 42.7; NVLink / scale-out 6.67; BlueField-4 800 Gb/s = 0.1 TB/s. Sources 13, 14, 26. Tier M.

• Vera memory bandwidth. Up to 1.2 TB/s (Rubin platform page; also reported from CES). Sources 26, 31. Tier B.

• Transfer times for 100 GB. NVLink 67 ms, C2C 111 ms, scale-out 444 ms (figures halved as bidirectional, per the product page and the NVLink page); BlueField-4 1 s; HBM4 read 5.2 ms. Sources 13, 14. Tier M.

• Fast memory per rack. 20.7 + 54 = 74.7 TB. Source 14. Tier M.

• Groq 3 LPX. 256 LPUs; 500 MB SRAM, 150 TB/s SRAM bandwidth, 2.5 TB/s scale-up per LPU; 128 GB SRAM, 12 TB DDR5, 40 PB/s, 640 TB/s per rack. Source 24. Tier B.

• LPX rack checks. 256 x 0.5 = 128 GB; 256 x 2.5 = 640 TB/s; 256 x 150 = 38.4 PB/s against 40 listed. Source 24. Tier M.

• Sweep rates. Rubin 66.7/s; LPU 300,000/s; ratio 4,500; Vera 0.8/s. Sources 14, 24, 26. Tier M.

• LPX against NVL72. Capacity 161.7x lower (20,700 / 128 GB); bandwidth 27.8x higher (per-chip products; 28.6x with rounded rack figures). Sources 14, 24. Tier M.

Networking and Kyber

• Networking revenue. Q3 FY25 $3.1B, Q4 FY25 $3.0B, Q1 FY26 $5.0B, Q2 FY26 $7.3B, Q3 FY26 $8.2B, Q4 FY26 $11.0B, Q1 FY27 $14.8B. Sources 4, 5, 6, 7. Tier A.

• Networking share and growth. 8.5% to 19.7%; 4.89x from Q4 FY25 to Q1 FY27; FY26 $31.4B, up 142%. Sources 4, 5, 7. Tier M.

• Q2 FY27 networking estimate. About $17.5B = 14.8 x 1.18. Sources 3, 4. Tier M.

• Co-packaged optics claims. 200 Gb/s SerDes; 5x power efficiency, 5x uptime, 1.3x faster deployment. Source 13. Tier B.

• Scale-out fabrics. Quantum-X800 InfiniBand and Spectrum-X Ethernet; ConnectX-9 SuperNICs and BlueField-4 DPUs in the rack. Source 14. Tier B.

• Kyber. 600 kW per rack (GTC, March 2025); reported delay to 2028; 78-layer midplane; 144 against 72 packages; NVIDIA statement. Sources 18, 19, 23. Tier C.

Gigawatts, power and cooling

• PORTS-Pike site. 4.25 IT-GW initial; option 3.75 IT-GW (press release), about 3.8 GW (CFO commentary); 8 IT-GW for OpenAI; 10 GW generation; $4.2B grid; $1.5B NVIDIA investment; phases from 2028. Sources 1, 11. Tier A.

• NVIDIA revenue per generation at the site. About 1.5 million GPUs; $150B to $200B. Source 1. Tier B.

• Per-gigawatt arithmetic. 35.3 to 47.1 $B/GW; 40 x 4.25 = 170; 352,941 GPUs/GW; 2.83 kW/GPU; $100,000 to $133,333 per GPU. Source 1. Tier M.

• Revenue opportunity per gigawatt. Hopper $18B, Grace Blackwell $25B, Vera Rubin $40B. Source 3. Tier B.

• Total investment per gigawatt. About $30B five years ago, about $60B today; 40 / 60 = 66.7%. Source 3. Tiers B, M.

• Watts per GPU. Factory 2.50 kW (400,000 GPUs/GW); without full MaxLPS gain 3.50 kW; Ohio 2.83 kW, 88.2% of factory density. Sources 1, 14, 16. Tier M.

• 800 V DC. 54 V today; up to eight shelves; up to 64U; up to 200 kg; 85% more power; 45% less copper; up to 5% efficiency; timing tied to Kyber. Source 17. Tier B.

• Current and loss arithmetic. 11,111 A at 54 V; 750 A at 800 V; 14.81x; 219.5x. Sources 17, 19. Tier M.

• DSX MaxLPS. 45 C cooling; up to 40% more GPUs; break-even loss 28.6% = 1 - 1/1.4. Source 16. Tiers B, M.

• 45 C water and chillers. Rack inlet 45 C (product page); Huang at CES: twice the power of Grace Blackwell, same 45 C water, no chillers needed. Sources 14, 27. Tiers B, C.

• Water flow. 100 kW / (4.18 x 10 K) at 0.990 kg/L = 145.0 L/min; at 20 K 72.5 L/min. Source: textbook constants. Tier M.

• Australia. Up to 2 GW by 2027 with eight operators. Source 20. Tier B.

• AI infrastructure by 2030. $3T to $4T, repeated at Goldman Sachs conference. Source 21. Tier C.

Supply, memory costs and margins

• Supply-related commitments. Q3 FY26 $50.3B, Q4 FY26 $95.2B, Q1 FY27 $119B, Q2 FY27 $279B. Sources 1, 4, 5, 6. Tier A.

• Supply commitments due by FY2029. $267B of $279B (95.7%). Source 1. Tier M.

• Cancelability of supply agreements. Some agreements may be cancelable, rescheduled or adjusted before firm orders. Source 9. Tier A.

• Cost of revenue at the Q3 guide. $28.08B = 108.0 x (1 - 0.74); 92 / 28.08 = 3.28 quarters; two quarters 56.16. Source 1. Tier M.

• Inventory. Q3 FY26 $19.8B, Q4 FY26 $21.4B, Q1 FY27 $25.8B, Q2 FY27 $31.6B. Sources 1, 4, 5, 6. Tier A.

• Gross margin path. Q4 FY27 trough 71% to 72%; FY28 72% to 73%. Source 3. Tier B.

• GAAP gross margin history. Q3 FY25 74.6%, Q4 FY25 73.0%, Q1 FY26 60.5%, Q2 FY26 72.4%, Q3 FY26 73.4%, Q4 FY26 75.0%, Q1 FY27 74.9%, Q2 FY27 75.0%. Sources 1, 4, 5, 6, 7, 8. Tier A.

• Cost per $100 of revenue. 25.00, 26.00, 28.50, 27.50; up 4%, 14%, 10%. Sources 1, 3. Tier M.

Commitments, guarantees and financing

• Commitments. 279 + 29 + 25 + 25 + 8 = $366B. Source 1. Tier A.

• Additional commitments. 36 + 20 = $56B. Source 1. Tier A.

• Guarantees. 3.5 + 105.0 = $108.5B. Source 1. Tier A.

• Total against liabilities and revenue. $530.5B; 5.81x of $91.3B; 1.75x of $303.0B trailing revenue. Sources 1, 4, 5, 6. Tier M.

• Residual value guaranties. Cap $105B; effective at lease commencement; ready-for-service expected from 2028; triggers and remedies. Source 10. Tier A.

• Guarantee cap per gigawatt. $24.7B = 105 / 4.25; 52.5% to 70.0% of one generation’s revenue. Source 1. Tier M.

• Lab exposure. Nearly $50B invested; about 12 GW for OpenAI; nearly 2 GW credit enhancement for another lab; labs about a quarter of next year’s business. Source 3. Tier B.

• Financing platforms. More than $500B of third-party capital; MOUs subject to final agreements. Source 12. Tier B.

• AI cloud agreements. Upfront sale plus revenue share above a take-or-pay floor. Sources 1, 3. Tier A.

Cash and balance sheet

• Cash flow. Q2 net income $59,688M, OCF $24,077M, FCF $21,341M; OCF / net income 40.3% against 86.3% in Q1. Sources 1, 4. Tiers A, M.

• Working capital in Q2. Receivables $22,346M, inventories $5,784M, prepaid and other $5,497M; equity gains $7,771M. Source 2. Tier A.

• Days sales outstanding. 60 against 45; check 63,059 / 96,221 x 91 = 59.6. Source 1. Tiers A, M.

• Marketable equity and non-marketable securities. 42,783 + 51,157 = $93,940M against $35,137M at Jan 25; cash and marketable debt $56,586M. Source 2. Tier M.

• Equity activity, first half. Purchases $42,404M; net gains $23,707M. Source 2. Tier A.

• Debt. $33,366M against $8,468M; $25.0B notes issued in Q2. Sources 1, 2. Tier A.

• Groq payment. $2,944M in financing activities. Source 2. Tier A.

• Capital returns. 19,732 + 6,047 = $25,779M; dividend $0.01 to $0.25 (May 18); $99.0B authorization remaining. Sources 2, 4. Tier A.

• Buyback authorization. $80.0B added on May 18. Source 4. Tier A.

Dates

• Dates. Q3 call November 17; GTC Berlin October 21. Source 3. Tier B.

In May 2026, on the eve of IWOCL in Heilbronn, the Khronos OpenCL Working Group released OpenCL 3.1. It is the first version bump in almost six years, and the first since 2017 that makes the specification bigger rather than more permissive.

Two months later the first conformant 3.1 implementation appeared on the Khronos list: Apple M1 and M2 graphics, running Linux, driven by Mesa’s Rusticl.

Not Apple’s driver: Apple deprecated OpenCL in 2018 and never looked back. A reverse-engineered kernel driver plus a Rust userspace led by one engineer at Red Hat.

That is a reasonable summary of what OpenCL has become. It is not the thing that lost to CUDA and quietly died. It is a specification with a governance process that keeps grinding forward, implemented today mostly by people who are not the vendors that originally wrote it, running the mobile inference stacks of two of the largest silicon companies on earth, and serving as the portability substrate underneath SYCL, chipStar, and a lot of code that will never mention OpenCL in its README.

There is one constraint that explains almost everything about OpenCL, and it is worth putting up front because the rest of this article is evidence for it.

OpenCL is the only compute API that had to describe hardware it could not see. CUDA describes one company’s roadmap, and that company knows what silicon is coming. Vulkan compute describes graphics hardware. Metal describes Apple’s.

OpenCL signed a contract covering CPUs, DSPs, FPGAs, GPUs from four vendors, and processors that did not exist when the contract was written. The price of that contract is that every guarantee has to be weak enough for the weakest member, or optional.

Once you see that, the history stops looking like a series of unforced errors. OpenCL 2.0 tried to make strong guarantees universal, and half the industry declined to implement them. OpenCL 3.0 stopped pretending and made almost everything optional, which read as capitulation and was actually an accurate description of what already existed.

OpenCL 3.1, in May 2026, is the first release in the standard’s history that mostly ratifies what had already shipped rather than predicting what ought to.

The performance-portability problem is the same constraint wearing different clothes: a kernel is a description of a memory hierarchy, and OpenCL cannot tell you which one you have.

I’ve tried to make this concrete rather than rhetorical. Where the article makes a claim about behaviour, I actually ran it: the kernels here were compiled and executed, the accuracy bounds were measured against what an implementation actually delivers, and the compile-time and dispatch-overhead arguments have numbers attached.

Those measurements come from one CPU device and are labelled as such: they demonstrate mechanisms, not hardware.

I’ve also been fussy about versions and dates, because a large fraction of what is written about OpenCL online describes the world of 2015 and says so nowhere.

Subscribe now

The origins

The immediate ancestors of OpenCL are worth naming because they explain its shape.

Between 2006 and 2008 there were at least four serious attempts to make GPUs programmable for non-graphics work: Stanford’s BrookGPU and its commercial descendant at AMD, ATI’s Close to Metal, NVIDIA’s CUDA (released February 2007), and IBM’s toolchain for the Cell Broadband Engine. Each was tied to one vendor’s hardware.

Apple, which at the time shipped machines with GPUs from both NVIDIA and ATI and had a strategic interest in not being locked to either, wanted one API that worked on both plus the CPU.

Apple developed an initial proposal internally, with Aaftab Munshi, who had edited the OpenGL ES 1.1 and 2.0 specifications, as spec editor, and refined it with technical teams at AMD, IBM, Qualcomm, Intel, and NVIDIA before submitting it to Khronos.

The Khronos Compute Working Group was formed on 16 June 2008. It finished the technical content of OpenCL 1.0 on 18 November 2008, and the specification was approved for public release on 8 December 2008.

Five months from working group formation to a ratified cross-vendor standard is fast by any standards-body measure, and it shows in the result: OpenCL 1.0 has the feel of a design that was mostly finished before the committee got involved.

Apple demonstrated a beta at WWDC in June 2008, a grid of 64 emulated Apple II machines running on the CPU to show task parallelism, and an N-body simulation on a Mac Pro’s GPU to show data parallelism, and shipped the first production implementation in Mac OS X 10.6 Snow Leopard on 28 August 2009.

AMD abandoned Close to Metal and backed OpenCL. NVIDIA announced OpenCL support for its GPU Computing Toolkit the day after ratification and shipped drivers in September 2009. IBM shipped an implementation for POWER through its XL compilers in October 2009. For about eighteen months, OpenCL looked like it was going to be the way GPUs got programmed.

The 1.x line then filled in the obvious gaps:

OpenCL 1.1 (14 June 2010) added three-component vector types, sub-buffers, rectangular region read/write/copy for buffers, user events, async_work_group_strided_copy, and tighter OpenGL interop through event linking. It also allowed API calls from multiple host threads, which 1.0 had not required.

OpenCL 1.2 (15 November 2011) is the version that still matters most, because it became the mandatory baseline of OpenCL 3.0 and is therefore the safe target for portable code even today.

It added device partitioning (splitting a device into sub-devices along compute-unit or cache-hierarchy boundaries), separate compilation and linking of programs, 1D images and 1D/2D image arrays, built-in kernels and custom devices, clEnqueueMigrateMemObjects, clEnqueueFillBuffer/clEnqueueFillImage, DirectX 9 media surface and DirectX 11 sharing, and -cl-fp32-correctly-rounded-divide-sqrt for code that needs IEEE 754 semantics on single-precision division and square root rather than the looser default.

By the end of 1.2 the API had a coherent shape. What happened next is the interesting part, and we’ll come back to it.

Subscribe now

The four models

The specification organises itself around four models, and it is worth internalising them in the order the spec gives them, because most confusion about OpenCL comes from conflating two of them.

Platform model

A host is connected to one or more compute devices. Each device has one or more compute units, each of which has one or more processing elements. That is the entire hardware abstraction, and it is deliberately vague.

The mapping to real silicon is vendor-defined and it’s not always intuitive: on an NVIDIA GPU a compute unit is a streaming multiprocessor; on AMD GCN it is a compute unit containing four 16-wide SIMDs; on a CPU it is typically a hardware thread; on Adreno it is a shader processor.

CL_DEVICE_MAX_COMPUTE_UNITS therefore does not mean the same thing across vendors, and comparing it across vendors is meaningless. The number vendors quote in marketing material, “5888 cores”, usually counts processing elements or SIMD lanes, not compute units.

Devices are typed: CL_DEVICE_TYPE_CPU, CL_DEVICE_TYPE_GPU, CL_DEVICE_TYPE_ACCELERATOR, CL_DEVICE_TYPE_CUSTOM (1.2+), plus CL_DEVICE_TYPE_DEFAULT and CL_DEVICE_TYPE_ALL.

Custom devices are the interesting outlier: they are devices that do not support the OpenCL C programming language at all and expose only built-in kernels, fixed-function or firmware-defined entry points queried through CL_DEVICE_BUILT_IN_KERNELS. This is how OpenCL accommodates video encoders, ISPs, and fixed-function DSP blocks.

There is a trap here worth naming, because it caught me. Through OpenCL 1.2 and the 2.x line the specification defined CL_DEVICE_TYPE_ALL as every device except custom ones, and that is still what most tutorials say.

The current unified specification does not: CL_DEVICE_TYPE_ALL is now simply “all OpenCL devices in the platform”. The custom-device constraint that survives applies to CL_DEVICE_TYPE_DEFAULT, which must not be a custom device unless it is the only device in the platform.

If you are carrying a decade-old mental model of that enum, it is out of date.

Profiles

Cutting across all of this is a distinction the article has so far skipped and most desktop developers never meet. Every platform and every device reports either FULL_PROFILE or EMBEDDED_PROFILE through CL_PLATFORM_PROFILE and CL_DEVICE_PROFILE.

The embedded profile is a formally specified relaxation of the full one, written for hardware that cannot afford the full set of guarantees: it permits round-to-zero instead of round-to-nearest as the default rounding mode for single precision, allows denormals and infinities to be handled more loosely, relaxes the accuracy requirements on several built-ins, drops 64-bit integers to optional, and lowers the minimum image dimensions and object counts an implementation must support.

This matters because “OpenCL 3.0 conformant” on a phone or an automotive SoC does not automatically mean the same arithmetic as OpenCL 3.0 conformant on a workstation.

It is also why Qualcomm’s statement that current Snapdragon platforms support OpenCL 3.0 full profile is a specific and meaningful claim rather than marketing throat-clearing. Check CL_DEVICE_PROFILE before you assume anything about numerics on embedded hardware.

Devices are grouped under platforms. A platform corresponds roughly to one vendor’s implementation. A single machine routinely exposes three or four: an Intel GPU platform, an NVIDIA platform, a PoCL CPU platform, a Rusticl platform. The mechanism that lets them coexist is the ICD, described in section 4.

Execution model

A kernel is executed over an NDRange: a 1-, 2-, or 3-dimensional index space. Each point in the index space is a work-item, and every work-item runs the same kernel body with a different global ID.

Work-items are grouped into work-groups of a size the application chooses (or leaves to the implementation).

Work-items in a work-group can synchronise with a barrier and share local memory; work-items in different work-groups cannot synchronise at all, and the specification gives no guarantee about the order in which work-groups execute or whether they execute concurrently.

That last point is the single most important constraint in the execution model and it is routinely violated by people porting from CPU code. There is no legal way to spin-wait in one work-group for another work-group to make progress. On many implementations it will appear to work and then deadlock on a device with fewer compute units or a different scheduler.

Since OpenCL 2.1 (and via cl_khr_subgroups before that) there is a third level: the sub-group, a subdivision of a work-group that maps onto the hardware’s SIMD execution width, a warp on NVIDIA, a wavefront on AMD, a subgroup on Intel and Arm and Qualcomm.

Sub-groups are guaranteed to make forward progress independently of one another within a work-group, and they support collective operations (broadcast, reduce, scan, ballot, shuffle) that are dramatically cheaper than the local-memory equivalents, and which became core in OpenCL 3.1.

Work is submitted through a command queue attached to one context and one device. Commands are kernel executions, memory transfers, map/unmap operations, markers, and barriers.

Each returns an event, and events form a dependency graph: any enqueue can take a wait-list of events that must complete first. Queues are in-order by default; CL_QUEUE_OUT_OF_ORDER_EXEC_MODE_ENABLE makes them out-of-order, at which point the event graph is the only thing constraining execution order.

The context is the object that ties devices, memory objects, programs, and queues together. Memory objects belong to a context, not a device, which is how the runtime knows it is allowed to migrate a buffer between devices in the same context.

Memory model

Four address spaces, in decreasing scope and increasing speed:

SpaceQualifierScopeTypical hardwareGlobal__globalAll work-items, all work-groups, host-visibleDevice DRAMConstant__constantRead-only, all work-itemsConstant cache / DRAMLocal__localOne work-groupScratchpad (LDS / shared memory)Private__privateOne work-itemRegisters, spilling to scratch

Since OpenCL 2.0 there is optionally a fifth: the generic address space, which lets a pointer be resolved to global, local, or private at runtime, so you can write one function that operates on all three.

Consistency is relaxed and explicit. Within a work-item, memory is consistent in program order. Across work-items in a work-group, memory is consistent only at a barrier(). Through work-groups, there is no consistency guarantee during a kernel’s execution at all, only at kernel boundaries and at explicit synchronisation points defined by the host API.

OpenCL 2.0 layered a formal memory model on top of this, adapted from C11, with atomic operations parameterised by memory order and memory scope (memory_scope_work_item, _work_group, _device, _all_svm_devices), which section 6 covers in detail.

Programming model

Two are named in the spec: data parallel (the NDRange) and task parallel (a kernel enqueued with a single work-item, or independent kernels in an out-of-order queue).

The task-parallel model is largely vestigial on GPUs; it mattered for the Cell processor and for CPU devices, and clEnqueueTask was deprecated in 2.0 in favour of just enqueuing a one-element NDRange.

Subscribe now

The API, and what it feels like to use

OpenCL is a C API with an object model built on opaque handles and manual reference counting. Every object type has clRetain* and clRelease*.

Objects are freed when their count reaches zero, but the specification is careful to say that the implementation may keep an object alive as long as it is referenced by a queued command, so releasing a buffer that a running kernel still uses is legal.

The canonical bring-up sequence is longer than people expect:

#define CL_TARGET_OPENCL_VERSION 300
#include <CL/cl.h>

cl_platform_id   platform;
cl_device_id     device;
cl_int           err;

clGetPlatformIDs(1, &platform, NULL);
clGetDeviceIDs(platform, CL_DEVICE_TYPE_GPU, 1, &device, NULL);

cl_context context = clCreateContext(NULL, 1, &device, NULL, NULL, &err);

/* clCreateCommandQueue is deprecated since 1.2; this is the 2.0+ form */
cl_queue_properties qprops[] = { CL_QUEUE_PROPERTIES,
                                 CL_QUEUE_PROFILING_ENABLE, 0 };
cl_command_queue queue =
    clCreateCommandQueueWithProperties(context, device, qprops, &err);

const char *src = "__kernel void saxpy(float a,                      \n"
                  "                    __global const float *x,       \n"
                  "                    __global float *y) {           \n"
                  "    size_t i = get_global_id(0);                   \n"
                  "    y[i] = fma(a, x[i], y[i]);                     \n"
                  "}                                                  \n";

cl_program program = clCreateProgramWithSource(context, 1, &src, NULL, &err);
err = clBuildProgram(program, 1, &device, "-cl-std=CL1.2", NULL, NULL);
if (err != CL_SUCCESS) {
    size_t log_size;
    clGetProgramBuildInfo(program, device, CL_PROGRAM_BUILD_LOG,
                          0, NULL, &log_size);
    char *log = malloc(log_size);
    clGetProgramBuildInfo(program, device, CL_PROGRAM_BUILD_LOG,
                          log_size, log, NULL);
    fprintf(stderr, "%s\n", log);
}

cl_kernel kernel = clCreateKernel(program, "saxpy", &err);

cl_mem xbuf = clCreateBuffer(context, CL_MEM_READ_ONLY  | CL_MEM_COPY_HOST_PTR,
                             n * sizeof(float), hx, &err);
cl_mem ybuf = clCreateBuffer(context, CL_MEM_READ_WRITE | CL_MEM_COPY_HOST_PTR,
                             n * sizeof(float), hy, &err);

float a = 2.0f;
clSetKernelArg(kernel, 0, sizeof(float),  &a);
clSetKernelArg(kernel, 1, sizeof(cl_mem), &xbuf);
clSetKernelArg(kernel, 2, sizeof(cl_mem), &ybuf);

size_t global = n, local = 256;
cl_event ev;
clEnqueueNDRangeKernel(queue, kernel, 1, NULL, &global, &local, 0, NULL, &ev);
clEnqueueReadBuffer(queue, ybuf, CL_TRUE, 0, n * sizeof(float), hy, 1, &ev, NULL);

The compiler is the core part of the runtime here: kernels are shipped as source strings and built at application startup, which is a design decision with large consequences.

And clSetKernelArg is stateful and untyped: the argument index and size are passed by hand, and getting either wrong is a runtime error at best and silent corruption at worst. This is the single biggest ergonomic difference from CUDA’s <<<>>> syntax, which type-checks arguments at compile time because the kernel and the launch site are in the same translation unit.

The C++ bindings (CL/opencl.hpp, formerly cl2.hpp, formerly cl.hpp) fix most of this with RAII wrappers and a variadic KernelFunctor. They are a Khronos-maintained header-only library, not part of the core spec, and they are what you should actually use from C++.

Error handling deserves a note. Every function returns or writes a cl_int, and there are around sixty error codes. Two behaviours cause most of the confusion:

• Asynchronous errors don’t surface where you expect. A kernel that reads out of bounds does not produce an error from clEnqueueNDRangeKernel; it produces CL_OUT_OF_RESOURCES or a device reset from a later clFinish, or nothing at all. Attach a context error callback (the pfn_notify parameter of clCreateContext): most implementations report useful diagnostics through it and almost nobody sets it.

• CL_INVALID_WORK_GROUP_SIZE has many causes. The local size must divide the global size (unless the device supports non-uniform work-groups, an OpenCL 2.0 feature that is optional in 3.0 and queried via CL_DEVICE_NON_UNIFORM_WORK_GROUP_SUPPORT), must not exceed CL_KERNEL_WORK_GROUP_SIZE for that specific kernel on that device, and must match any reqd_work_group_size attribute in the kernel source.

Deprecated APIs remain callable. The headers hide 1.2-deprecated entry points unless you define CL_USE_DEPRECATED_OPENCL_1_2_APIS, and similarly for earlier versions.

The CL_TARGET_OPENCL_VERSION macro (a three-digit value: 120, 200, 300) controls which version’s prototypes the unified headers expose, and setting it deliberately is good hygiene: it turns “this device doesn’t support that” into the worst compile error.

Subscribe now

The ICD, and how a call reaches a driver

Nothing in the OpenCL specification requires that multiple vendors’ implementations coexist on one machine, but in practice they must, and the mechanism is the Installable Client Driver loader.

An application links against libOpenCL.so (or OpenCL.dll, or libOpenCL.dylib). That library is not a driver. It is a dispatcher, maintained by Khronos in the OpenCL-ICD-Loader repository, which enumerates the vendor implementations installed on the system and forwards calls to the right one.

On Linux it does this by reading /etc/OpenCL/vendors/*.icd, each a one-line text file containing the path of a vendor’s shared object; on Windows it reads HKEY_LOCAL_MACHINE\SOFTWARE\Khronos\OpenCL\Vendors.

Two environment variables control the search: OCL_ICD_VENDORS replaces the default directory, and OCL_ICD_FILENAMES takes a separator-delimited list of extra ICDs to load, which are enumerated before anything found by the default mechanism.

Both are invaluable when you want a test suite to hit one specific implementation on a machine that has four. Layers are selected separately, through OPENCL_LAYERS.

Mechanically, each vendor library exports clGetExtensionFunctionAddress and (per cl_khr_icd) clIcdGetPlatformIDsKHR. Every cl_platform_id the vendor returns points to a struct whose first member is a pointer to a dispatch table.

Every other OpenCL object (contexts, devices, queues, buffers) is likewise required to begin with a pointer to the dispatch table of the implementation that created it. So clEnqueueNDRangeKernel(queue, ...) in the loader is one indirect call through queue->dispatch->clEnqueueNDRangeKernel.

This is why passing objects from one platform into a call on another platform is undefined behaviour rather than a clean error: the loader has already dispatched before anyone checks.

Version 2.0.0 of cl_khr_icd added clIcdGetFunctionAddressForPlatformKHR and related entry points to handle extension functions cleanly, which the older scheme did badly.

The loader also supports layers, a mechanism borrowed from Vulkan. A layer is a shared object that sits between the application and the ICD and can intercept, log, modify, or synthesise OpenCL calls. This is the foundation of several genuinely useful tools:

• OpenCL Intercept Layer (Intel, open source, version 3.0.4 as of 2026): call tracing, timing, kernel dumping, device-side printf capture, injection of modified kernel source without rebuilding the application, and USM validity checking. Runs on Windows, Linux, macOS, Android, and FreeBSD.

• CLVizulayer (StreamHPC, presented at IWOCL 2026): emits the directed acyclic graph of device submissions in Graphviz DOT format. Unlike a timeline trace, this shows the constraints rather than the observed order, which is how you find the case where two commands ran sequentially because the implementation felt like it rather than because you asked for it. It has been used to inspect llama.cpp, Leela Chess Zero, GROMACS and LAMMPS.

Layers are underused. If you are debugging a portability problem across four implementations, the ability to insert instrumentation without touching either the application or the driver is worth a great deal.

Subscribe now

The execution model in depth

NDRange geometry

clEnqueueNDRangeKernel takes work_dim (1–3), a global_work_offset (added in 1.1; usually NULL), global_work_size, and local_work_size.

Inside the kernel:

size_t gid   = get_global_id(0);      // global index, includes offset
size_t lid   = get_local_id(0);       // index within the work-group
size_t grp   = get_group_id(0);       // work-group index
size_t gsz   = get_global_size(0);
size_t lsz   = get_local_size(0);     // actual, may differ from enqueued
size_t ngrp  = get_num_groups(0);
size_t off   = get_global_offset(0);  // 1.1+
uint   dim   = get_work_dim();

Passing NULL for local_work_size lets the implementation choose. How much this matters is easy to measure, and the answer has two halves.

A 6.8× spread between the worst and best explicit choice. This is not a knob you can leave unconsidered; it is frequently the largest single factor in a kernel’s performance, ahead of most things people spend their time on.

And yet the implementation’s own choice beat every value I picked by hand. I’d written flatly that passing NULL is wrong for code you care about, and on this device that advice is (sorry) simply false: the runtime knows the vectorisation width its own compiler chose and I do not.

The honest version is much narrower: the parameter matters enormously, NULL is a real candidate rather than a lazy default, and the only way to know is to measure both on the hardware you ship to.

Where you do choose by hand, the size is bounded by three things you should query:

size_t max_wg, pref_mult;
cl_ulong local_used, private_used;
clGetKernelWorkGroupInfo(kernel, device, CL_KERNEL_WORK_GROUP_SIZE,
                         sizeof(max_wg), &max_wg, NULL);
clGetKernelWorkGroupInfo(kernel, device,
                         CL_KERNEL_PREFERRED_WORK_GROUP_SIZE_MULTIPLE,
                         sizeof(pref_mult), &pref_mult, NULL);
clGetKernelWorkGroupInfo(kernel, device, CL_KERNEL_LOCAL_MEM_SIZE,
                         sizeof(local_used), &local_used, NULL);
clGetKernelWorkGroupInfo(kernel, device, CL_KERNEL_PRIVATE_MEM_SIZE,
                         sizeof(private_used), &private_used, NULL);

CL_KERNEL_WORK_GROUP_SIZE is per-kernel, not per-device: it accounts for the register and local-memory pressure of this kernel and is therefore often much smaller than CL_DEVICE_MAX_WORK_GROUP_SIZE.

CL_KERNEL_PREFERRED_WORK_GROUP_SIZE_MULTIPLE is the closest thing OpenCL 1.x has to a warp-size query: in practice it is 32 on NVIDIA, 64 on GCN and typically 32 on RDNA, 8, 16 or 32 on Intel depending on the SIMD width the compiler chose, and commonly 64 or 128 on Adreno.

Those are the values you will usually see, not values the specification promises; query it rather than hard-coding it, and a local size that is not a multiple of it wastes lanes.

CL_KERNEL_PRIVATE_MEM_SIZE is the register-spill indicator and the most useful single number for diagnosing a slow kernel. If it is non-zero and large, you are spilling to scratch memory and no amount of tuning the local size will help.

OpenCL 3.1 finally standardises what everyone had been approximating: a suggested local work-group size query, promoted to core from cl_khr_suggested_local_work_size. The runtime tells you what it thinks the right work-group size is for a given kernel, global size, and device. It is a hint, not an oracle, but it removes a class of per-device tuning tables from application code.

Sub-groups

uint sg_size  = get_sub_group_size();
uint sg_id    = get_sub_group_id();
uint sg_local = get_sub_group_local_id();
uint n_sg     = get_num_sub_groups();

float total   = sub_group_reduce_add(x);
float bcast   = sub_group_broadcast(x, 0);
int   any     = sub_group_any(pred);
float shuf    = sub_group_shuffle(x, src_lane);       // cl_khr_subgroup_shuffle
float rot     = sub_group_rotate(x, delta);           // cl_khr_subgroup_rotate
uint4 mask    = sub_group_ballot(pred);               // cl_khr_subgroup_ballot

Sub-group operations avoid the round trip through local memory and the barrier that a work-group reduction requires. On a modern GPU a sub_group_reduce_add over 32 lanes is a handful of shuffle-and-add instructions; the local-memory equivalent is a store, a barrier, log₂(n) rounds of load-add-store with a barrier each, and a load.

For the reduction-heavy inner loops in attention and normalisation kernels this is the difference between competitive and not.

Before 3.1 this was a portability minefield: cl_khr_subgroups was optional, the extended type support was a separate extension (cl_khr_subgroup_extended_types), shuffles were another (cl_khr_subgroup_shuffle, cl_khr_subgroup_shuffle_relative), ballots another, and Intel had its own pre-standard cl_intel_subgroups.

The llama.cpp OpenCL backend requires subgroup support and therefore requires OpenCL 2.x or a 3.0 implementation that opted in. OpenCL 3.1 makes sub-groups core, including shuffles, rotations, and an expanded set of supported data types, which is the change most likely to simplify real kernel code.

You can pin the sub-group size with __attribute__((intel_reqd_sub_group_size(N))) on Intel, or query CL_KERNEL_MAX_SUB_GROUP_SIZE_FOR_NDRANGE and CL_KERNEL_SUB_GROUP_COUNT_FOR_NDRANGE via clGetKernelSubGroupInfo (2.1+).

Queues, events, and the flush trap

An event has five states: CL_QUEUED, CL_SUBMITTED, CL_RUNNING, CL_COMPLETE, or a negative value indicating abnormal termination. With CL_QUEUE_PROFILING_ENABLE set, clGetEventProfilingInfo returns nanosecond timestamps for CL_PROFILING_COMMAND_QUEUED, _SUBMIT, _START, _END, and (2.0+) _COMPLETE. END - START is device execution time; START - SUBMIT is queue latency; SUBMIT - QUEUED is host-side runtime overhead.

All three are worth looking at separately, a kernel that is fast but submitted late is a different problem from a kernel that is slow. The three intervals are easy to see. Timing a trivial kernel over 65,536 work-items with the queue warm:

Kernel work QUEUED→SUBMIT SUBMIT→START START → END

1 iteration 0.07 µs 1.75 µs 63.9 µs

50 iterations 0.13 µs 2.66 µs 3.6 ms

1000 iterations 0.37 µs 13.2 µs 89.0 ms

Host-side runtime cost is sub-microsecond, queue latency is single-digit microseconds, and everything else is the device. That ordering is what you want, and it is why the interesting overhead is not inside a single enqueue but in how many of them you make and how you make them.

Which is measurable too. Dispatching an empty kernel repeatedly, comparing a round trip per dispatch against submitting a batch and finishing once:

Dispatches Round-trip Batched Ratio

200 7.28 µs each 2.00 µs each 3.6×

1000 7.23 µs each 1.91 µs each 3.8×

5000 7.22 µs each 1.85 µs each 3.9×

PoCL 3.0, CPU device.

Nearly 4× on the same work, from nothing but submission discipline, and stable across three orders of magnitude of batch size. Note that this is measured against a CPU device with no PCIe bus and no kernel-mode driver round trip; on a discrete GPU the round-trip figure is worse, not better.

Around 2 µs of irreducible per-dispatch cost is also the number that makes command buffers interesting: a transformer decode step issuing three hundred kernels per token is paying roughly 600 µs of pure submission before any arithmetic happens, every token.

The classic OpenCL bug is forgetting clFlush. Enqueuing a command does not guarantee it is submitted to the device. If you enqueue work and then block on an event with clWaitForEvents without flushing the queue, some implementations will hang, because the command that would signal the event was never sent. The rules:

• clFlush guarantees all previously enqueued commands are submitted to the device. It does not wait.

• clFinish blocks until all previously enqueued commands have completed. It implies a flush.

• Blocking enqueue calls on a queue (clEnqueueReadBuffer with blocking_read = CL_TRUE) imply a flush of that queue. clWaitForEvents does not: the specification says the behaviour is undefined if you wait on events from commands that have not been flushed. This bites hardest across queues, since a blocking call on queue A flushes nothing in queue B. Flush explicitly.

OpenCL 3.1 fixes a genuinely subtle related hazard. Previously, polling an event’s status with clGetEventInfo and observing CL_COMPLETE did not by itself establish the memory ordering needed to safely read the results; you were expected to call a waiting function.

In practice a great deal of code polled and then read, and it mostly worked. In 3.1, observing that an event has reached CL_COMPLETE is itself a synchronisation point.

That is a spec change that legalises what people were already doing, which is the right call, but it means code written against 3.1 semantics can be subtly broken on a 3.0 driver.

Device-side enqueue, and why it did not take

OpenCL 2.0 introduced device-side enqueue: a kernel could enqueue further kernels onto a device-side queue without host involvement, using blocks (the Clang/Apple ^{} extension) as the payload.

It was the answer to CUDA Dynamic Parallelism, and it was intended for irregular workloads: adaptive mesh refinement, tree traversal, anything where the amount of work is data-dependent.

It never got broad adoption, unfortunately due to a lot of reasons. The implementation burden was high (it requires a device-side scheduler), the syntax was unfamiliar, and the performance on the implementations that did support it was frequently worse than doing multiple host-side dispatches.

In OpenCL 3.0 it became optional, queried through CL_DEVICE_DEVICE_ENQUEUE_CAPABILITIES, and many implementations report zero.

It’s the clearest example of a 2.0 feature that was standardised before it was proven, which is exactly the mistake the working group says its current extension-first process is designed to avoid.

Command buffers

The replacement for a different problem, per-submission host overhead, is cl_khr_command_buffer, released provisionally in November 2021 as part of OpenCL 3.0.10 and developed largely by Codeplay with Qualcomm, Arm, Intel, Tampere University, NVIDIA and Google.

The idea is the one CUDA Graphs, Vulkan command buffers, and Level Zero command lists all landed on independently: record a sequence of commands once, finalise it, then dispatch the whole thing repeatedly with a single API call.

For inference serving, where the same graph of thirty or three hundred kernels is executed per token, the per-enqueue host cost dominates at small batch sizes.

cl_command_buffer_khr cb =
    clCreateCommandBufferKHR(1, &queue, NULL, &err);

clCommandNDRangeKernelKHR(cb, NULL, NULL, k0, 1, NULL, &g0, &l0,
                          0, NULL, NULL, NULL);
clCommandNDRangeKernelKHR(cb, NULL, NULL, k1, 1, NULL, &g1, &l1,
                          0, NULL, NULL, NULL);
clFinalizeCommandBufferKHR(cb);

for (int i = 0; i < steps; i++)
    clEnqueueCommandBufferKHR(0, NULL, cb, 0, NULL, NULL);

Note that a command buffer is internally out-of-order regardless of the queue it targets; ordering comes from the sync-point dependencies you declare when recording.

cl_khr_command_buffer_mutable_dispatch (OpenCL 3.0.12, September 2022) relaxes the immutability constraint so that kernel arguments, global size, local size and offsets of a recorded dispatch can be updated between replays via clUpdateMutableCommandsKHR, which is what you need for anything with a changing sequence length.

A further extension allows a command buffer to span multiple queues and devices (added in 3.0.14).

As of OpenCL 3.1 command buffers are still an extension, but Khronos names them explicitly as one of the features in the pipeline for a future core release.

Subscribe now

The memory model in depth

Buffers, images, pipes

A buffer (cl_mem created by clCreateBuffer) is a linear byte range. An image (clCreateImage) is an opaque object with a channel order, channel data type, and dimensionality, accessed through read_imagef / write_imagef and friends, optionally through a sampler that performs normalised coordinates, addressing modes (clamp, repeat, mirror), and linear filtering in fixed-function hardware.

Pipes (2.0, optional in 3.0 via CL_DEVICE_PIPE_SUPPORT) are FIFO objects for producer-consumer patterns between kernels; they were designed with FPGAs in mind and are rarely used on GPUs.

The buffer-versus-image decision is more consequential than it looks. On desktop GPUs, buffer loads go through the general L1/L2 path; image reads go through the texture path, which on many architectures has a separate cache, hardware address computation, hardware boundary handling, and format conversion for free.

On mobile GPUs, Adreno in particular, the texture path is substantially faster for the read patterns typical of convolution and GEMM. This is why Qualcomm’s TVM and MLC work for Adreno has a dedicated “texture path” and specialised layouts, and why the llama.cpp Adreno kernels are written the way they are. On CPU devices the distinction mostly evaporates and images are usually slower.

Image support is itself optional: CL_DEVICE_IMAGE_SUPPORT can be CL_FALSE. It commonly is on custom devices and on some early open-source stacks, the original Mesa Clover never supported images, which is precisely why it could not run darktable, and why Rusticl’s image support was the thing that made it useful.

Image formats and samplers

An image is not a typed buffer; it is a channel order plus a channel data type, and only some combinations are required. The channel orders in the specification are CL_R, CL_A, CL_RG, CL_RA, CL_RGB, CL_RGBA, CL_BGRA, CL_ARGB, CL_INTENSITY, CL_LUMINANCE, CL_DEPTH and CL_sRGBA, among others.

The data types run from CL_SNORM_INT8 and CL_UNORM_INT8 through the packed short formats (CL_UNORM_SHORT_565, CL_UNORM_SHORT_555, CL_UNORM_INT_101010) to CL_SIGNED_INT8/16/32, CL_UNSIGNED_INT8/16/32, CL_HALF_FLOAT and CL_FLOAT, with CL_UNORM_INT10, INT12 and INT14 added for higher-precision sensor data.

The normalised types do conversion in hardware: reading a CL_UNORM_INT8 image with read_imagef returns floats in [0, 1] with no instruction spent on the divide. That is free range conversion, and a real reason to prefer images for pixel data.

Only a small subset of order and type combinations is guaranteed, though; everything else must be checked with clGetSupportedImageFormats against the specific device, memory flags and image type. Assuming a format exists because it is in the enum is one of the more common portability failures.

Samplers carry three orthogonal settings: normalised or unnormalised coordinates, an addressing mode for out-of-range coordinates (CLK_ADDRESS_NONE, CLAMP, CLAMP_TO_EDGE, REPEAT, MIRRORED_REPEAT), and a filter mode (CLK_FILTER_NEAREST or CLK_FILTER_LINEAR). All of it is fixed-function on GPUs.

A bilinear tap that would cost four loads and three lerps in a buffer kernel is one read_imagef with CLK_FILTER_LINEAR, and boundary clamping that would cost a branch per axis is free.

Samplers can be declared in the kernel as a const sampler_t constant or created on the host and passed in.

Allocation flags and the map path

CL_MEM_READ_WRITE | CL_MEM_WRITE_ONLY | CL_MEM_READ_ONLY   /* access, from the kernel's view */
CL_MEM_USE_HOST_PTR                                        /* use this host allocation */
CL_MEM_ALLOC_HOST_PTR                                      /* allocate host-accessible memory */
CL_MEM_COPY_HOST_PTR                                       /* allocate device memory, copy in */
CL_MEM_HOST_WRITE_ONLY | CL_MEM_HOST_READ_ONLY | CL_MEM_HOST_NO_ACCESS  /* 1.2 */

The semantics people get wrong: CL_MEM_USE_HOST_PTR doesn’t mean “the device will read your pointer directly”.

It means the implementation may cache the contents in device memory and is required to keep the host pointer as the backing store; you must map/unmap to access it safely from the host.

CL_MEM_ALLOC_HOST_PTR is the closest OpenCL comes to CUDA’s pinned memory: it asks the implementation to allocate memory the host can access efficiently, which on a discrete GPU usually means page-locked system memory suitable for DMA, and on an integrated GPU usually means memory both processors can access without a copy at all.

It is worth being precise about what the flag buys, because it is easy to assume it makes transfers faster on its own. Writing 64 MB into a buffer three ways on the same device:

Path Time Eff. Rate clEnqueueWriteBuffer into a plain buffer 14.9 ms 4.5 GB/s

clEnqueueWriteBuffer into an ALLOC_HOST_PTR buffer 29.1 ms 2.3 GB

/smap / write / unmap on the ALLOC_HOST_PTR buffer 5.2 ms 12.9 GB/s

PoCL 3.0, CPU device, mean of five.

Adding CL_MEM_ALLOC_HOST_PTR and changing nothing else made the copy twice as slow. The 2.9× win only appears when the access pattern changes to match the allocation: when you stop copying and start writing directly into mapped memory.

The flag is not an optimisation; it is a request for memory with different properties, and it pays only if you then use those properties. This generalises: allocation hints in OpenCL are contracts about placement, and a contract you do not exercise is overhead.

The idiomatic zero-copy pattern on integrated hardware is therefore CL_MEM_ALLOC_HOST_PTR plus clEnqueueMapBuffer:

cl_mem b = clCreateBuffer(ctx, CL_MEM_READ_WRITE | CL_MEM_ALLOC_HOST_PTR,
                          n, NULL, &err);
float *p = clEnqueueMapBuffer(queue, b, CL_TRUE, CL_MAP_WRITE,
                              0, n, 0, NULL, NULL, &err);
/* fill p */
clEnqueueUnmapMemObject(queue, b, p, 0, NULL, NULL);

On a system where host and device share physical memory this can be a genuine no-copy path. On a discrete GPU it still copies, but from pinned memory, at full PCIe bandwidth rather than the roughly half you get from pageable memory.

OpenCL 3.1 clarifies the semantics of CL_DEVICE_HOST_UNIFIED_MEMORY so that it can be used reliably to distinguish integrated from discrete devices.

That sounds trivial, but it’s not: for a decade the flag was defined loosely enough that implementations disagreed about it, and every serious application ended up with its own heuristic (usually string-matching the device name) to decide whether to take the map path or the copy path, so having one query that means one thing removes real code from real applications.

clEnqueueMigrateMemObjects (1.2) lets you explicitly move a memory object to a device, or to the host with CL_MIGRATE_MEM_OBJECT_HOST, ahead of the kernel that will use it. In multi-device contexts this is how you avoid a fault-driven migration in the middle of a dispatch.

The 2.0 memory model

OpenCL 1.x had no formal memory model, just “relaxed consistency“ plus barriers and fences, described in prose. OpenCL 2.0 replaced this with a model derived from C11’s, which was the right decision and made OpenCL one of the first GPU APIs with a mathematically specified memory model.

Atomics are typed (atomic_int, atomic_float, atomic_uintptr_t, and atomic_long/atomic_double if supported) and take a memory order and a memory scope:

atomic_fetch_add_explicit(&counter, 1,
                          memory_order_relaxed,
                          memory_scope_work_group);

atomic_store_explicit(&flag, 1,
                      memory_order_release,
                      memory_scope_device);

Orders are relaxed, acquire, release, acq_rel, seq_cst. Scopes are work_item, sub_group, work_group, device, and all_svm_devices. The scope is the part with no C11 analogue and it is what makes the model usable on GPUs: an atomic scoped to a work-group can be implemented in the scratchpad with no cache-coherence traffic, while an atomic scoped to all_svm_devices may require flushing through to system memory.

The historical wrinkle is the inclusive scope rule. Until 2026, for two atomic operations to synchronise with each other, their scopes had to match, or more precisely, both had to include each other’s work-items.

This essentially meant that a release at memory_scope_work_group did not necessarily synchronise with an acquire at memory_scope_device, even though the device scope is strictly larger.

It is a rule that made implementations easier and reasoning harder, and it produced code that specified everything at device scope out of caution, giving up the performance that scoped atomics exist to provide.

OpenCL 3.1 relaxes it: scopes no longer have to match exactly, and a finer-grained scope can satisfy a coarser-grained synchronisation requirement.

Combined with sub-groups becoming core, this makes the fine-grained synchronisation patterns used in modern GPU kernels expressible without the previous defensive over-specification.

Shared Virtual Memory, and the USM successor

OpenCL 2.0 added SVM in three tiers, queried through CL_DEVICE_SVM_CAPABILITIES:

Coarse-grained buffer: CL_DEVICE_SVM_COARSE_GRAIN_BUFFER. Pointers are valid on both sides; consistency at map/unmap and kernel boundaries.

Fine-grained buffer: CL_DEVICE_SVM_FINE_GRAIN_BUFFER. Concurrent host/device access to the same allocation, no map needed.

Fine-grained system: CL_DEVICE_SVM_FINE_GRAIN_SYSTEM. Any malloc pointer is usable on the device.

Atomics: CL_DEVICE_SVM_ATOMICS. Cross-side atomics on SVM allocations.

Coarse-grained buffer SVM is widely supported. Fine-grained system SVM is rare and requires real hardware support for demand paging and coherent access to host page tables.

The problem with SVM is not the concept but the API. clSVMAlloc returns a pointer, but you still set it as a kernel argument through clSetKernelArgSVMPointer, you still have to declare indirect accesses with clSetKernelExecInfo and CL_KERNEL_EXEC_INFO_SVM_PTRS, and the capability matrix is coarse.

Intel’s answer, cl_intel_unified_shared_memory (added to the registry alongside OpenCL 3.0.10), is a cleaner design borrowed from what became SYCL 2020’s USM: three allocation kinds (host, device, and shared) with explicit control over placement and migration, no map/unmap, allocations associated with both a device and a context, and a richer capability query.

It is the model that oneAPI, SYCL, and by extension a lot of HPC code actually use.

The working group has been standardising this as cl_khr_unified_svm, and Khronos lists Unified Shared Memory as one of the extensions in flight for a future core version.

For anyone writing new OpenCL against Intel or Level Zero-adjacent stacks, USM is already the pragmatic choice; for portable code, SVM coarse-grained remains the lowest common denominator, and plain buffers remain the actual lowest common denominator.

Subscribe now

OpenCL C

OpenCL C is C99 with removals and additions. Removed: function pointers, recursion, variable-length arrays, bit fields, most of the standard library, goto into blocks, and (before 2.0) program-scope variables in the global address space.

Added: address space qualifiers, vector types, a large built-in function library, work-item query functions, and a set of type qualifiers and attributes.

Vector types exist in widths 2, 3, 4, 8, and 16 for all scalar types: float4, int8, uchar16, double2. They support arithmetic elementwise, and component access through several syntaxes:

float4 v = (float4)(1.0f, 2.0f, 3.0f, 4.0f);
float  a = v.x;          /* also .y .z .w; and .r .g .b .a since OpenCL 3.0 */
float2 b = v.xy;
float4 c = v.wzyx;       /* arbitrary swizzle, including repeats */
float  d = v.s3;         /* hex-index form, required for width 8 and 16 */
float8 e = (float8)(v, v);
float2 lo = v.lo, hi = v.hi;   /* halves */
float2 ev = v.even, od = v.odd;

Three-component vectors have size 16 bytes, not 12: sizeof(float3) == sizeof(float4). This surprises people writing structs that cross the host/device boundary.

Whether vector types help performance is architecture-dependent, and the answer has inverted over time. On AMD’s pre-GCN VLIW architectures (TeraScale) and on CPU devices, vector code was essential because the compiler could not always find the parallelism itself.

On GCN, RDNA, NVIDIA, and modern Intel GPUs, the hardware is scalar-per-lane and the vector types are mostly a way to express wide loads and stores. That is still worth something: a float4 load is one 128-bit memory instruction instead of four 32-bit ones, which matters for memory-bound kernels.

On Adreno and Mali, vector width still maps to real SIMD capability and choosing it correctly matters more.

Precision, and the parts nobody reads

The specification contains a table of maximum error in ULP for every math built-in, and it is the most under-appreciated part of the document. Some entries:

Function Max error

x + y, x * y, fma correctly rounded

1.0/x, x / y ≤ 2.5 ULP; correctly rounded with -cl-fp32-correctly-rounded-divide-sqrt

sqrt ≤ 3 ULP; correctly rounded with the same flag

rsqrt, cbrt, log1p ≤ 2 ULP

exp, exp2, exp10,

log, log2, log10 ≤ 3 ULP

sin, cos, sinpi,

cospi, hypot ≤ 4 ULP

tan, tanh, atan, atanh ≤ 5 ULP

pow, pown, powr, rootn,

erf, erfc, tgamma ≤ 16 ULP

mad unbounded: any value is conforming

native_* variants implementation-defined, no bound

half_* variants ≤ 8192 ULP

Two entries deserve attention. mad has no accuracy requirement at all: the specification permits any result, because it exists to let the implementation pick whatever multiply-add the hardware has, fused or not.

If you want a fused multiply-add with defined semantics, write fma. And in double precision, division, reciprocal and square root are all correctly rounded, the loose bounds above are a single-precision phenomenon.

What the table does not tell you is what you will actually get, and the gap between the two is larger than most people assume. Measuring the observed error of a conformant implementation against a double-precision reference over 65,536 points per function.

Two results are worth sitting with. native_sqrt, native_exp and native_log returned bit-identical values to their accurate counterparts: zero of 65,536 samples differed in any bit, so this implementation simply aliases them.

native_sin is the exception, differing on 21% of samples. half_exp, permitted to be wrong by 8192 ULP, was accurate to under one. So on this implementation, every fast-math variant is free accuracy, and code written to tolerate 8192 ULP is running on results good to 1.

That is not a reassuring finding. It means you cannot learn anything about native_ behaviour by testing it, because the next implementation is equally entitled to return something wildly different and still be conformant.

This is the article’s opening constraint in miniature: a standard spanning unknown hardware can specify floors and nothing else, and a floor tells you almost nothing about the room. Test on the implementation you ship against, or use the accurate functions.

The native_ family (native_sin, native_exp2, native_recip, native_rsqrt, native_divide) maps directly to whatever hardware instruction exists and gives no accuracy guarantee at all.

On most GPUs these are single-instruction and roughly an order of magnitude faster than the accurate versions. -cl-fast-relaxed-math implies -cl-finite-math-only, -cl-unsafe-math-optimizations, and permits the compiler to substitute native_ variants globally, which is why enabling it can change results by several ULP and can turn a NaN check into dead code.

Denormal handling is implementation-defined for single precision unless the device reports CL_FP_DENORM in CL_DEVICE_SINGLE_FP_CONFIG. -cl-denorms-are-zero explicitly permits flush-to-zero.

Most GPUs flush single-precision denormals by default and handle double-precision denormals correctly, which is the opposite of what people assume.

Double precision requires cl_khr_fp64 (or __opencl_c_fp64 in OpenCL C 3.0), half precision requires cl_khr_fp16. Neither is guaranteed. half as a storage type, via vload_half / vstore_half, which convert to and from float, is available without cl_khr_fp16; only arithmetic on half requires the extension.

This is a useful distinction for anyone storing FP16 weights and computing in FP32.

Attributes and hints

__attribute__((reqd_work_group_size(16, 16, 1)))
__attribute__((work_group_size_hint(64, 1, 1)))
__attribute__((vec_type_hint(float4)))
__attribute__((intel_reqd_sub_group_size(16)))    /* vendor */

reqd_work_group_size is a contract: enqueue with any other local size and you get CL_INVALID_WORK_GROUP_SIZE.

It is worth using, because it lets the compiler size local arrays statically, unroll fully, and allocate registers knowing the occupancy, which frequently produces measurably better code than the hint version.

OpenCL C 3.0 and feature macros

OpenCL 3.0 made almost everything from the 2.x line optional. In the language, that optionality is expressed through predefined feature-test macros named __opencl_c_<feature>, which the compiler defines with value 1 when the feature is present:

__opencl_c_3d_image_writes
__opencl_c_atomic_order_acq_rel
__opencl_c_atomic_order_seq_cst
__opencl_c_atomic_scope_device
__opencl_c_atomic_scope_all_devices
__opencl_c_device_enqueue
__opencl_c_fp64
__opencl_c_generic_address_space
__opencl_c_images
__opencl_c_int64
__opencl_c_pipes
__opencl_c_program_scope_global_variables
__opencl_c_read_write_images
__opencl_c_subgroups
__opencl_c_work_group_collective_functions

So portable OpenCL C 3.0 looks like this:

#if defined(__opencl_c_subgroups)
    float total = sub_group_reduce_add(partial);
#elif defined(__opencl_c_work_group_collective_functions)
    float total = work_group_reduce_add(partial);
#else
    /* hand-rolled local memory tree reduction */
#endif

On the host side there is a matching set of device queries: CL_DEVICE_ATOMIC_MEMORY_CAPABILITIES, CL_DEVICE_ATOMIC_FENCE_CAPABILITIES, CL_DEVICE_DEVICE_ENQUEUE_CAPABILITIES, CL_DEVICE_PIPE_SUPPORT, CL_DEVICE_GENERIC_ADDRESS_SPACE_SUPPORT, CL_DEVICE_WORK_GROUP_COLLECTIVE_FUNCTIONS_SUPPORT, CL_DEVICE_NON_UNIFORM_WORK_GROUP_SUPPORT, and CL_DEVICE_OPENCL_C_ALL_VERSIONS / CL_DEVICE_OPENCL_C_FEATURES, so an application can decide which kernel variant to build before it builds anything. clang exposes the same switches for offline compilation through -cl-ext, e.g. -cl-std=CL3.0 -cl-ext=+cl_khr_fp64,+__opencl_c_fp64.

C++, twice

OpenCL 2.2 (May 2017) introduced OpenCL C++, a static subset of C++14 as a kernel language, defined alongside SPIR-V ingestion. It is a carefully written specification. It appears never to have shipped in a production driver, there is no extension defined to detect support for it, and it was deprecated in OpenCL 3.0.

The replacement is C++ for OpenCL, which is a different thing: not a Khronos-ratified specification but a community language, documented in the OpenCL-Docs repository and implemented in upstream Clang since release 9.

It is C++17 layered over OpenCL C, backward compatible with OpenCL C source, and it works with any implementation that ingests SPIR-V, which, as of OpenCL 3.1, is all of them. Version 1.0 was published in December 2020 (compatible with OpenCL 2.0); the 2021 revision (December 2021) is compatible with OpenCL 3.0.

What it doesn’t support: virtual functions, dynamic_cast, non-placement new/delete, exceptions, pointers to member functions, references to functions, and the C++ standard library.

What it does support is templates, classes, operator overloading, lambdas, and auto, which is enough for the thing people actually want, namely writing one templated kernel body instead of four macro-expanded copies.

Address spaces are extended across C++ constructs: functional casts, templates, class members, references, lambdas, and operators.

template<typename T>
class complex_t {
    T re, im;
public:
    complex_t(T r, T i) : re{r}, im{i} {}
    complex_t operator*(const complex_t &o) const {
        return { re*o.re - im*o.im, re*o.im + im*o.re };
    }
    T real() const { return re; }
    T imag() const { return im; }
};

__kernel void mul_sp(__global float *in, __global float *out) {
    auto i = get_global_id(0);
    auto r = complex_t{in[4*i], in[4*i+1]} * complex_t{in[4*i+2], in[4*i+3]};
    out[2*i] = r.real(); out[2*i+1] = r.imag();
}

Online compilation of C++ for OpenCL requires the cl_ext_cxx_for_opencl extension and -cl-std=CLC++; Arm announced support in December 2020. In practice most people compile it offline with Clang to SPIR-V, which is the flow the language was designed for.

Subscribe now

Compilation: source, IR, and machine code

The four ways to make a program object

clCreateProgramWithSource(ctx, count, strings, lengths, &err);
clCreateProgramWithIL(ctx, il, length, &err);              /* 2.1+, SPIR-V */
clCreateProgramWithBinary(ctx, n, devs, sizes, bins, st, &err);
clCreateProgramWithBuiltInKernels(ctx, n, devs, names, &err);  /* 1.2+ */

Source is the original path and still the most common. It means the OpenCL driver contains a full C compiler, that compiler runs at application startup, and compile time is user-visible. A large kernel library (llama.cpp’s OpenCL backend, a serious image processing pipeline) can take seconds to build.

It is worth knowing the size of that cost rather than assuming it. Building a synthetic kernel library on PoCL’s CPU device, then rebuilding the same library from the binaries the first build produced:

Kernels in the librarySource sizeclCreateProgramWithSource + buildclCreateProgramWithBinary + buildRatio83.5 KB158.0 ms2.2 ms72×3214.1 KB190.4 ms5.2 ms37×6428.2 KB276.4 ms13.8 ms20×

The absolute numbers are one implementation’s, and PoCL is running a full Clang and LLVM pipeline where a vendor driver would be more streamlined.

The shape is the point: compiling from source is 20× to 72× more expensive than loading a binary here, and the gap narrows only slowly as the library grows because a large part of the source cost is fixed compiler startup.

For an application with a hundred kernels this is the difference between an instant launch and a visible pause, which is exactly why every serious OpenCL application ends up building a cache, and exactly what the OpenCL 3.1 SPIR-V mandate is for.

Binaries via clCreateProgramWithBinary are the obvious fix and come with a hard constraint: program binaries are not portable. They are opaque, implementation-defined blobs, valid only for the device, driver version, and build options that produced them. CL_PROGRAM_BINARY_TYPE tells you whether a binary is an executable, a compiled object, or a library.

The correct use is a persistent cache keyed on device name, driver version, platform version, kernel source hash, and build options, exactly what llama.cpp does with GGML_OPENCL_KERNEL_CACHE_DIR, defaulting to %LOCALAPPDATA%\llama.cpp\cl-cache on Windows and ~/Library/Caches/llama.cpp/cl-cache on macOS, and invalidating on any of those inputs changing.

Separate compilation and linking arrived in 1.2:

clCompileProgram(prog, n, devs, opts, n_hdrs, hdrs, hdr_names, cb, data);
cl_program lib = clLinkProgram(ctx, n, devs, "-create-library",
                               n_in, inputs, cb, data, &err);

This lets you build a device-side library once and link it into several programs, and it lets you pass headers by name rather than concatenating strings.

Build options worth knowing:

-cl-std=CL1.2 | CL2.0 | CL3.0 | CLC++ | CLC++2021
-D name=value                  -I dir
-cl-single-precision-constant  -cl-denorms-are-zero
-cl-opt-disable                -cl-mad-enable
-cl-no-signed-zeros            -cl-unsafe-math-optimizations
-cl-finite-math-only           -cl-fast-relaxed-math
-cl-fp32-correctly-rounded-divide-sqrt
-cl-uniform-work-group-size    -cl-kernel-arg-info
-w  -Werror

-cl-kernel-arg-info is the one people forget: without it, clGetKernelArgInfo cannot tell you argument names, types, or address space qualifiers, which breaks any tooling that wants to bind arguments by name.

SPIR, SPIR-V, and the 2026 mandate

The first attempt at a portable IR was SPIR (2012, versions 1.2 and 2.0), which was LLVM IR with an OpenCL-specific metadata layer. It inherited LLVM’s problem: LLVM IR is not a stable format and its semantics are defined by the LLVM version that produced it. Consuming SPIR meant, in practice, having a compatible LLVM inside the driver.

SPIR-V, released with OpenCL 2.1 in November 2015 and adopted by Vulkan 1.0 three months later, replaced it with a purpose-built, versioned, SSA-form binary IR that is not tied to any compiler. It is shared with Vulkan, and a SPIR-V module declares which execution environment it targets, the OpenCL SPIR-V Environment Specification defines what an OpenCL implementation must accept.

The OpenCL flavour uses the OpenCL memory model and the Kernel execution model, and calls into the OpenCL Extended Instruction Set for SPIR-V for math built-ins.

There are now several ways to produce it:

• Clang’s SPIR-V target. clang -target spirv64 -c kernel.cl -o kernel.spv, using the SPIR-V backend that has been maturing in upstream LLVM. This is the path that no longer requires a translator.

• SPIRV-LLVM-Translator, the older, still widely used tool that converts LLVM IR to SPIR-V; the standard route for clang -cl-std=CL3.0 -emit-llvm output and for oneAPI’s toolchain.

• clspv. Google’s compiler from a subset of OpenCL C to Vulkan compute shaders, which is a different SPIR-V dialect. Paired with clvk, a runtime that implements the OpenCL API on top of Vulkan.

OpenCL 3.1’s headline change is that SPIR-V ingestion is mandatory. Every conformant OpenCL 3.1 implementation must accept SPIR-V kernels through clCreateProgramWithIL, and must additionally support the SPIR-V query extension so applications can enumerate which SPIR-V capabilities, extensions, and versions a device handles.

This is more consequential than it sounds, and it is worth being precise about why. Before 3.1, clCreateProgramWithIL was core in 2.1 but optional in 3.0, so a 3.0 implementation could legally accept only source. Any tool that wanted to target OpenCL as a backend therefore had to either ship an OpenCL C source generator or accept that it would not run everywhere.

That is a real tax on SYCL implementations, on chipStar (which compiles CUDA and HIP to SPIR-V), on Julia’s and Rust’s GPU backends, and on every domain-specific compiler. Making ingestion mandatory turns OpenCL from “an API you can target if the driver cooperates” into a guaranteed compilation target.

Neil Trevett, who chairs the working group, called it the most consequential change in 3.1, and on the evidence that is not marketing.

The secondary benefits are the ones that matter operationally: kernels can ship pre-compiled and pre-optimised rather than as source, which removes startup compile cost, allows ahead-of-time specialisation, and means you are not shipping your kernel source to customers.

What the vendor stacks actually do

ImplementationFront endIRBack endNVIDIAClang-derivedNVVM (LLVM IR)PTX → SASS via ptxasIntel (NEO / compute-runtime)ClangSPIR-VIGC → Gen ISAAMD (ROCm CLR)ClangLLVM IRAMDGPU back end → GCN/RDNA ISAArm MaliClang-basedvendor IRMali ISAQualcomm AdrenoLLVM-basedvendor IRAdreno ISAPoCLClangLLVM IR / SPIR-VLLVM target back ends, plus CUDA / Level Zero / remote driversRusticlvia SPIRV-Tools / MesaSPIR-V → NIRGallium driver back endsclvk / clspvclspv (Clang-based)Vulkan-flavour SPIR-Vwhatever the Vulkan driver does

Apple’s implementation, NVIDIA’s, AMD’s, RapidMind’s and Gallium’s have all been LLVM-based from early on.

The practical implication is that “OpenCL C” in the field means “whatever Clang’s OpenCL front end accepts, plus vendor quirks” more than it means the paper specification, which is mostly good, because Clang’s OpenCL support is well maintained and tracks the spec closely, and occasionally bad, because a kernel that builds on four Clang-derived stacks may still fail on a non-LLVM one.

Subscribe now

Performance engineering

The mental model

The abstraction says work-items are independent. The hardware executes them in lockstep groups. Everything about OpenCL performance follows from taking the second statement seriously while writing code that satisfies the first.

Concretely: a work-group is scheduled onto one compute unit and stays there. It is subdivided into sub-groups, which are the actual scheduling and execution unit. Divergent control flow within a sub-group is executed by predicating both sides.

Memory requests from a sub-group are coalesced by the hardware into as few transactions as possible; consecutive lanes reading consecutive addresses is one transaction, consecutive lanes reading strided addresses is many.

The occupancy story is the same as CUDA’s without the vocabulary: a compute unit has a fixed register file and a fixed scratchpad, and the number of work-groups it can host concurrently is bounded by whichever runs out first.

More concurrent work-groups means more latency hiding. CL_KERNEL_LOCAL_MEM_SIZE and CL_KERNEL_PRIVATE_MEM_SIZE are how you see the pressure; CL_DEVICE_LOCAL_MEM_SIZE and CL_DEVICE_MAX_WORK_GROUP_SIZE are the budget.

A worked GEMM

The single-work-item-per-output-element version is the reference, and it is memory bound at an arithmetic intensity of about 1 FLOP per byte:

__kernel void sgemm_naive(const int M, const int N, const int K,
                          __global const float *A,
                          __global const float *B,
                          __global float *C)
{
    const int col = get_global_id(0);
    const int row = get_global_id(1);
    float acc = 0.0f;
    for (int k = 0; k < K; k++)
        acc += A[row*K + k] * B[k*N + col];
    C[row*N + col] = acc;
}

Tiling through local memory raises intensity by the tile width. Each work-group cooperatively stages a TS × TS tile of A and of B, barriers, then every work-item does TS multiply-accumulates out of the scratchpad:

#define TS 16

__attribute__((reqd_work_group_size(TS, TS, 1)))
__kernel void sgemm_tiled(const int M, const int N, const int K,
                          __global const float *A,
                          __global const float *B,
                          __global float *C)
{
    const int lx = get_local_id(0), ly = get_local_id(1);
    const int col = get_group_id(0)*TS + lx;
    const int row = get_group_id(1)*TS + ly;

    __local float Asub[TS][TS];
    __local float Bsub[TS][TS];

    float acc = 0.0f;
    for (int t = 0; t < K/TS; t++) {
        Asub[ly][lx] = A[row*K + (t*TS + lx)];
        Bsub[ly][lx] = B[(t*TS + ly)*N + col];
        barrier(CLK_LOCAL_MEM_FENCE);

        #pragma unroll
        for (int k = 0; k < TS; k++)
            acc = fma(Asub[ly][k], Bsub[k][lx], acc);

        barrier(CLK_LOCAL_MEM_FENCE);
    }
    C[row*N + col] = acc;
}

TS is 16 here rather than the textbook 32 for a reason: a 32×32 work-group is 1024 work-items, which exceeds CL_DEVICE_MAX_WORK_GROUP_SIZE on a great deal of mobile and embedded hardware and sits at the ceiling on most desktop GPUs. Both kernels here also assume M, N and K are multiples of TS; real code needs edge handling or padded allocations.

Both barriers are required. The second one, after the inner loop and before the next tile overwrites the scratchpad, is the one people omit, and it produces a race that is invisible on any implementation where a work-group happens to be one sub-group wide.

The tiled version is usually still short of peak, because each work-item does one FMA per two scratchpad reads. The next step is register tiling: give each work-item a WPT × WPT block of outputs so operands loaded into registers are reused across several accumulations.

#define TS  32     /* tile size            */
#define WPT 4      /* outputs per work-item per dimension */
#define RTS (TS/WPT)

__attribute__((reqd_work_group_size(RTS, RTS, 1)))
__kernel void sgemm_regtiled(const int M, const int N, const int K,
                             __global const float *A,
                             __global const float *B,
                             __global float *C)
{
    const int lx = get_local_id(0), ly = get_local_id(1);
    const int gx = get_group_id(0)*TS, gy = get_group_id(1)*TS;

    __local float Asub[TS][TS];
    __local float Bsub[TS][TS];

    float acc[WPT][WPT];
    #pragma unroll
    for (int a = 0; a < WPT; a++)
        #pragma unroll
        for (int b = 0; b < WPT; b++) acc[a][b] = 0.0f;

    for (int t = 0; t < K/TS; t++) {
        #pragma unroll
        for (int a = 0; a < WPT; a++)
            #pragma unroll
            for (int b = 0; b < WPT; b++) {
                const int r = ly*WPT + a, c = lx*WPT + b;
                Asub[r][c] = A[(gy + r)*K + t*TS + c];
                Bsub[r][c] = B[(t*TS + r)*N + gx + c];
            }
        barrier(CLK_LOCAL_MEM_FENCE);

        #pragma unroll
        for (int k = 0; k < TS; k++) {
            float av[WPT], bv[WPT];
            #pragma unroll
            for (int a = 0; a < WPT; a++) av[a] = Asub[ly*WPT + a][k];
            #pragma unroll
            for (int b = 0; b < WPT; b++) bv[b] = Bsub[k][lx*WPT + b];
            #pragma unroll
            for (int a = 0; a < WPT; a++)
                #pragma unroll
                for (int b = 0; b < WPT; b++)
                    acc[a][b] = fma(av[a], bv[b], acc[a][b]);
        }
        barrier(CLK_LOCAL_MEM_FENCE);
    }

    #pragma unroll
    for (int a = 0; a < WPT; a++)
        #pragma unroll
        for (int b = 0; b < WPT; b++)
            C[(gy + ly*WPT + a)*N + gx + lx*WPT + b] = acc[a][b];
}

Now WPT loads of A and WPT of B feed WPT² FMAs. With WPT = 4 that is 16 FMAs per 8 scratchpad reads.

All three kernels above were compiled and run before publication, on PoCL 3.0 targeting a Xeon CPU device, and checked against a NumPy reference at 256×256: all three match to zero relative error. That is a correctness check, not a performance claim, and the CPU result below is the reason to keep those two things apart.

There is no single correct TS and WPT. The optimum depends on scratchpad size, register file size, sub-group width, cache line size, and whether the compiler decides to spill. This is exactly what CLBlast does, a tuned BLAS in OpenCL that ships per-architecture parameter sets and can autotune on unseen hardware, and what CLTune and KTT exist to automate.

The honest position is that hand-written OpenCL GEMM is a tuning problem, not a coding problem, and that any claim about “OpenCL performance” that does not say which parameters were used is not a measurement.

Scratchpad bank conflicts are the other thing to watch. Local memory is divided into banks (typically 32 words wide); simultaneous accesses by different lanes to different addresses in the same bank serialise.

Declaring __local float Asub[TS][TS+1], padding the row stride by one, breaks the conflicting stride pattern at a cost of a few hundred bytes.

Everything that isn’t the kernel

For inference workloads the kernel is frequently not the bottleneck. Intel’s IWOCL 2026 talk on optimising AI workloads on their GPUs spent most of its time on three things that are not kernel code: queue model design and how to batch versus submit immediately, Ultra Low Latency Submission to cut dispatch overhead for latency-sensitive inference, and driver-level memory pooling and resource recycling to avoid the allocation churn that dominates when a model allocates and frees repeatedly.

They reported allocation overhead reductions of orders of magnitude from pooling alone.

That matches what anyone who has profiled a token-by-token decode loop finds: at batch size 1, per-dispatch host cost and allocator behaviour can be a larger share of wall time than the arithmetic. It is why command buffers exist, and why they are the extension most worth watching.

Asynchronous copies

For embedded and DSP targets where the scratchpad is filled by an explicit DMA engine rather than by ordinary loads, OpenCL C has:

event_t e = async_work_group_copy(dst_local, src_global, n, 0);
event_t f = async_work_group_strided_copy(dst_local, src_global, n, stride, e);
wait_group_events(1, &f);
prefetch(ptr, n);

cl_khr_extended_async_copies and cl_khr_async_copy_fence, added in OpenCL 3.0.10, extend this with 2D and 3D copy patterns and finer-grained fencing.

These were introduced specifically for the class of embedded processors that motivated much of the 3.0 work, and on a GPU they are usually implemented as ordinary loads.

Performance portability, honestly

The two studies everyone quotes are worth reading rather than citing, because the numbers travel without their conditions.

Karimi, Dickson and Hamze at D-Wave measured a quantum Monte Carlo kernel in both languages and reported the OpenCL kernel between about 13% and 63% slower, with end-to-end times 16% to 67% slower. Those figures are exact, and they are also from 2010, on a GeForce GTX-260, with the CUDA and OpenCL toolkits both at version 2.3.

More importantly, the paper is explicit that its OpenCL kernel is a near-identical port of the CUDA one, changed only where OpenCL forced a change: __shared__ to __local, threadIdx to get_local_id(), __syncthreads() to barrier(), and one array-indexing rewrite.

It measures what a direct translation costs, which is a real and useful thing to know, and it doesn’t measure what tuned OpenCL costs. The spread also narrows as problems get larger, which the authors attribute to the kernel’s share of total runtime rising.

The 2011 Delft comparison is usually summarised as CUDA leading a straightforward OpenCL translation by at most 30% on NVIDIA hardware, with the gap attributed to programming-model differences and to NVIDIA having invested more in its CUDA compiler than its OpenCL one.

I have not read that paper in full and am repeating the published summary, which is exactly the sort of secondhand number this section is warning about.

The modern version of this result is more encouraging and comes from chipStar, which compiles unmodified CUDA and HIP to OpenCL and SPIR-V.

Its IWOCL 2026 keynote reported performance competitive with vendor-native toolchains across Intel discrete and integrated GPUs, AMD and NVIDIA GPUs through Rusticl, Arm Mali-G52, RISC-V systems with PowerVR graphics, and x86 and Arm CPUs, with overhead negligible on some platforms and a reasonable trade on others.

It was validated on real codes, including a quantum chemistry package with more than 20,000 lines of GPU kernels and the libCEED finite-element library.

The cheapest demonstration of the gap is the one sitting in this article. Running the three GEMM kernels above on a CPU device (PoCL’s pthread driver on a Xeon, where __local memory is ordinary RAM and there is no scratchpad to tile into), the ranking inverts completely.

Every optimisation that would win on a GPU loses here, consistently: the tiled kernel by 1.45× to 1.78×, the register-tiled one by 1.14× to 1.35×, at every size tested.

Staging a tile into __local is a copy the CPU did not need, and the barriers are pure overhead when a work-group is executed by one thread.

This is one device and it proves nothing about any GPU. What it does show is that the tuning is not incidental to the kernel: it is the kernel, and it is aimed at a memory hierarchy the target may not have.

So: OpenCL gives you functional portability for free and performance portability only if you do the work. A kernel tuned for a 64-wide wavefront and a 64 KB scratchpad will run correctly and badly on a device with a 16-wide sub-group and 32 KB, or on a CPU with neither.

Share The Software Frontier

The 2.x detour and the 3.0 reset

OpenCL 2.0 (18 November 2013) was an ambitious release. It added shared virtual memory, device-side enqueue, the C11-derived memory model with scoped atomics, the generic address space, pipes, program-scope global variables, and read-write images.

Every one of these was a reasonable answer to a real limitation. Together they were too much to ask of every implementer at once.

What followed is the part that shaped the next decade. AMD and Intel implemented 2.0. NVIDIA did not: it stayed on OpenCL 1.2 as its conformance level from 2015 until 2021, offering a partial 2.0 evaluation driver from February 2017 that was never conformant. Mobile vendors implemented selectively.

So a developer writing OpenCL faced a choice between targeting 1.2 and reaching everyone, or targeting 2.0 and excluding the largest installed base of discrete GPUs.

Almost everyone chose 1.2, which meant SVM, device-side enqueue, and the new memory model went largely unused, which meant there was little pressure on anyone to implement them.

OpenCL 2.1 (16 November 2015) added SPIR-V ingestion, sub-groups, clCloneKernel, low-latency device timer queries, and the OpenCL C++ kernel language. OpenCL 2.2 (16 May 2017) brought OpenCL C++ into core along with SPIR-V 1.2 and pipe storage. Both landed on an ecosystem that had not adopted 2.0, and neither changed that.

At SIGGRAPH 2017 Khronos announced that OpenCL would converge with Vulkan. This was widely reported as OpenCL being merged into Vulkan and discontinued, which was not quite what was said but was a reasonable reading of the slides.

The actual outcome was different and, in retrospect, better: the convergence happened at the IR level, through SPIR-V, and produced clspv and clvk, a compiler and runtime that let OpenCL C kernels execute on Vulkan drivers. Adobe used exactly this to ship Premiere Rush on Android. Meanwhile OpenCL kept its own roadmap under the working title “OpenCL Next”.

That became OpenCL 3.0, provisional on 27 April 2020 and final on 30 September 2020. Its central move was to invert the compatibility model: OpenCL 1.2 becomes the mandatory baseline, and every 2.x feature becomes optional and queryable.

The immediate effect was that vendors who had been stuck could ship a “current” version. NVIDIA became OpenCL 3.0 conformant on Windows and Linux with the R465 driver in April 2021, covering Maxwell and later, and exposing a specific set of optional pieces: RGBA vector component naming, the pragma unroll hint, opencl_3d_image_writes, the clCreate*WithProperties entry points, clSetContextDestructorCallback, clCloneKernel, and clEnqueueSVMMigrateMem.

Intel’s NEO compute-runtime had shipped OpenCL 3.0 for Tiger Lake from version 20.41 in October 2020, including complete 2.0 and 2.1 optional functionality and parts of 2.2.

The so called criticism of 3.0, that “everything is optional” means the standard guarantees nothing, is half right. It is true that a conformant OpenCL 3.0 device may expose exactly the OpenCL 1.2 feature set and nothing more. It is also true that this describes what the ecosystem already was; 3.0 made the situation legible instead of pretending otherwise.

The query surface it introduced is genuinely usable, and the alternative, a specification that most vendors ignore, is worse.

There are real backward-compatibility traps, though, and they are not always obvious. Program-scope global variables are an example: they were introduced in OpenCL C 2.0, and under 3.0 they are optional (__opencl_c_program_scope_global_variables).

Code that built fine against a vendor’s OpenCL 2.0 compiler can silently produce wrong results, not a build error, on the same vendor’s 3.0 driver if the driver drops the feature and the compiler’s handling of the declaration changes.

Users of PyOpenCL’s ElementwiseKernel hit precisely this on NVIDIA’s R465 driver. “OpenCL 1.2 applications run unchanged on OpenCL 3.0” is true. “OpenCL 2.x applications run unchanged” is true only if the driver still supports every 2.x feature they use, and you must check.

Leave a comment

What OpenCL 3.1 changed

Released 4–5 May 2026; specification revision 3.1.1 dated 22 May 2026. The design philosophy is stated explicitly by the working group and is worth quoting in substance: features are proven in the field as extensions first, watched across multiple implementations, refined on developer feedback, and only then promoted to core. Everything mandated in 3.1 was already shipping somewhere.

Mandatory SPIR-V ingestion, plus mandatory support for the SPIR-V query extension. Covered in section 8; this is the change with the largest downstream effect.

Sub-groups in core, including shuffles, rotations, and an expanded set of supported data types. Applications no longer need extension guards or fallback paths for the collective operations that tuned reductions, scans, and matrix kernels are built from.

Integer dot products in core, including saturating and accumulating variants, together with extended bit operations. Both map to dedicated instructions on a wide range of modern silicon, the dp4a-class instructions, and both are the arithmetic primitives underneath INT8 inference. Promoted from cl_khr_integer_dot_product and cl_khr_extended_bit_ops.

A suggested local work-group size query in core, from cl_khr_suggested_local_work_size.

A standard device UUID query in core, matching Vulkan’s VkPhysicalDeviceIDProperties::deviceUUID. This lets an application correlate the same physical device across OpenCL and Vulkan, which is required for external memory sharing and for any sane device selection policy on a multi-GPU box. Promoted from cl_khr_device_uuid.

printf gains z and t length modifiers, for size_t and ptrdiff_t. Device-side printf could not previously format pointer-sized values without casts or format-string tricks, a small thing that shows up constantly in debugging.

CL_DEVICE_HOST_UNIFIED_MEMORY semantics clarified so it reliably distinguishes integrated from discrete GPUs.

Local memory kernel arguments may be set to zero, meaning “no local memory needed”. Kernels that opportunistically use the scratchpad no longer need a separate code path for the configuration where they don’t.

Observing CL_COMPLETE is now a synchronisation point.

The inclusive scopes rule is relaxed in the memory model, so a finer-grained scope can satisfy a coarser-grained requirement.

Implementations were in flight at release from Arm, Imagination, Intel and Qualcomm, plus Rusticl, PoCL and clvk. The first conformant listing arrived on 14 July 2026: Rusticl on Apple M1/M2 hardware under Asahi Linux, with radeonsi and Zink submissions in progress.

Mesa 26.2, released 5 August 2026, ships OpenCL 3.1 support for Rusticl on Asahi, Iris, radeonsi, llvmpipe and Zink, along with cl_khr_subgroup_rotate on radeonsi and Iris and other subgroup extensions.

The forward roadmap named by Khronos: command buffers, unified shared memory, cooperative matrix operations, low-precision AI data types including int4 and fp8, improvements to external memory sharing, and image tiling controls.

Beyond extensions, the group says it is exploring OpenCL’s role as a substrate for higher-level programming models, in safety-critical markets, and on NPUs and RISC-V accelerators.

Extensions

Extensions are named cl_khr_* (ratified, cross-vendor), cl_ext_* (multi-vendor but not ratified), or cl_<vendor>_*. A device advertises them in CL_DEVICE_EXTENSIONS as a space-separated string, or since 3.0 in the structured CL_DEVICE_EXTENSIONS_WITH_VERSION. Language-level extensions are enabled in the kernel with #pragma OPENCL EXTENSION cl_khr_fp16 : enable.

Provisional extensions carry version numbers below 1.0 and are subject to change; cl_khr_command_buffer spent years at 0.9.x. Some are shipped behind experimental headers.

That is a feature of the process: it is how features get tested before they are mandated. But it means “supports cl_khr_command_buffer“ is not a stable claim without a version.

The families worth knowing:

Numeric and language. cl_khr_fp16, cl_khr_fp64, cl_khr_int64_base_atomics, cl_khr_int64_extended_atomics, cl_khr_3d_image_writes, cl_khr_integer_dot_product, cl_khr_extended_bit_ops, cl_khr_expect_assume (compiler hints), cl_khr_kernel_clock (added provisionally in OpenCL 3.0.16, April 2024, for in-kernel profiling; developed by Arm, Imagination, Intel and Qualcomm).

Sub-groups. cl_khr_subgroups plus _extended_types, _non_uniform_vote, _ballot, _non_uniform_arithmetic, _shuffle, _shuffle_relative, _clustered_reduce, _rotate. Mostly subsumed by 3.1 core.

Execution. cl_khr_command_buffer, cl_khr_command_buffer_mutable_dispatch, cl_khr_command_buffer_multi_device, cl_khr_suggested_local_work_size, cl_khr_priority_hints, cl_khr_throttle_hints.

Memory. cl_khr_unified_svm, cl_intel_unified_shared_memory, cl_khr_extended_async_copies, cl_khr_async_copy_fence, cl_khr_device_uuid.

Interop. cl_khr_gl_sharing and cl_khr_gl_event for OpenGL; cl_khr_egl_image and cl_khr_egl_event for EGL; cl_khr_d3d10_sharing, cl_khr_d3d11_sharing, cl_khr_dx9_media_sharing on Windows; and the modern replacements: cl_khr_semaphore, cl_khr_external_semaphore with its opaque_fd and sync_fd variants, and cl_khr_external_memory with dma_buf, opaque_fd and win32 variants.

All eight were finalised together in OpenCL 3.0.16 and are the correct way to share memory and synchronisation primitives with Vulkan. NVIDIA collaborated on these specifically for Vulkan interop and ships samples using them.

Machine learning. OpenCLML on Adreno, exposed through Qualcomm’s OpenCL ML SDK. It does not appear in the Khronos registry (the cl_qcom_* extensions registered there cover host pointers, ION and Android native buffers, and performance hints), so treat it as SDK-delivered rather than as a registry extension.

This is a vendor extension that provides accelerated neural network operations, and Qualcomm reported at IWOCL 2026 that adding new accelerated ops in OpenCLML extension version 5 doubled prefill performance for their generative AI models in TVM’s Relax pipeline.

Cooperative matrix, the newest and the one that matters most for inference. On 29 April 2026 the working group published a draft cl_khr_cooperative_matrix, developed with Arm, Intel and Qualcomm. It lets an OpenCL implementation accept SPIR-V modules using SPV_KHR_cooperative_matrix, the same extension Vulkan standardised, providing cooperative load, store and multiply-add at sub-group scope, with supported matrix shapes, component types and saturation behaviours queried through a new clGetDeviceCooperativeMatrixInfoKHR.

A companion extension to expose the same capability directly in OpenCL C is in progress, with an RFC on LLVM Discourse proposing the Clang front-end changes: a cooperative matrix type attribute, built-in load/store/MAD functions, and lowering to SPIR-V-friendly LLVM IR through target extension types.

The absence of this has been a concrete, measurable limitation. Qualcomm’s IWOCL 2026 paper on llama.cpp says so plainly: unlike APIs with native cooperative matrix or tensor core abstractions, OpenCL has no standardised interface for them, so their team had to engineer portable GEMM implementations for dense and mixture-of-experts workloads by hand, using adaptive tiling, subgroup-aware parallelisation and device-specific kernel variants, to reach high utilisation without dedicated matrix instructions.

Every generation of hardware that adds matrix units widens the gap that this extension is meant to close.

Subscribe now

The implementation landscape in 2026

NVIDIA. OpenCL 3.0 conformant since R465 (April 2021), Maxwell and later, x86/x86-64 Linux and Windows only. Functional and maintained; not where NVIDIA’s effort goes. External memory and semaphore extensions are supported for Vulkan interop.

AMD. OpenCL ships as part of CLR (Compute Language Runtimes), the repository that also contains the HIP runtime, sharing the ROCclr device layer. Development moved there from ROCm-OpenCL-Runtime at ROCm 5.6, and kernel compilation goes through the same Clang/LLVM AMDGPU path as HIP. AMD’s own effort is overwhelmingly directed at HIP, and OpenCL rides along on the shared runtime rather than being driven forward on its own; on Linux, Rusticl has at times outperformed the ROCm OpenCL stack on the same hardware.

Intel. The most complete implementation. NEO / intel/compute-runtime on Linux and Windows, OpenCL 3.0 with the full 2.0 and 2.1 optional feature set plus parts of 2.2, USM, and an unusually large set of vendor extensions. Intel also maintains the OpenCL Intercept Layer and contributes heavily to the working group: Ben Ashbaugh of Intel gave the OpenCL state-of-the-union at IWOCL 2026.

Arm. Mali GPUs, OpenCL 3.0 conformant from Mali-G78, G310, G510, G610, G710 and G78AE onward, with G615 and G715-Immortalis listed in October 2022. Arm implemented cl_ext_cxx_for_opencl and is a co-author of the cooperative matrix work.

Qualcomm. Adreno, OpenCL 3.0 full profile on current Snapdragon platforms, plus an OpenCL SDK, the OpenCL ML SDK, the Snapdragon Profiler, and a published Adreno OpenCL best-practices guide. Qualcomm is, on the evidence of the last two years, the most active silicon vendor in OpenCL: the llama.cpp backend, the TVM and MLC work, OpenCLML, and co-authorship of command buffers and cooperative matrix.

Imagination, VeriSilicon (Vivante GPU IP, OpenCL 3.0 and 1.2 full profile on the VIP9000 series for automotive and edge AI), Texas Instruments (DSP platforms), Samsung, and Cadence all appear on the Khronos conformant list.

PoCL. Portable Computing Language, MIT licensed, fifteen years old as of IWOCL 2026, developed largely at Tampere University. Conformant for CPU and Level Zero GPU targets.

PoCL 7.2-RC1 (August 2026) achieved OpenCL 3.0 conformance for CPU devices on both x86-64 (submitted on a Ryzen 9 9900X) and RISC-V (on the Star64 board and Milk-V Jupiter), and added cl_khr_extended_bit_ops, cl_khr_device_uuid, cl_khr_suggested_local_work_size, cl_khr_integer_dot_product and cl_khr_kernel_clock, with LLVM 22 support for CUDA and Level Zero back ends and LLVM 22/23 for the CPU back end.

PoCL also has a remote back end that transparently offloads OpenCL work to other machines over the network, with real memory management and distributed command scheduling rather than naive call forwarding.

Rusticl. Mesa’s OpenCL implementation on top of Gallium, written in Rust, led by Karol Herbst at Red Hat. Merged into mainline Mesa in September 2022 and shipped in Mesa 22.3, conformant with OpenCL 3.0 in November 2022 on 12th-generation Intel graphics via the Iris driver, and the first conformant OpenCL 3.1 implementation in July 2026.

It replaces Clover, which never supported images and was effectively abandoned. It requires RUSTICL_ENABLE to advertise devices for most drivers, since enabling by default is a per-driver opt-in.

Layered implementations, the strategic development of the last five years. clvk (Kévin Petit) implements the OpenCL API on Vulkan, using clspv to compile kernels; OpenCLOn12 layers OpenCL over Direct3D 12 through Mesa Gallium, which is how Windows on Arm got OpenCL; Ancle and Rusticl-over-Zink cover further combinations.

The consequence is that “does this platform have OpenCL?” increasingly means “does this platform have Vulkan or D3D12?”, and both are close to universal. chipStar’s keynote makes the point concretely: through clvk and Rusticl, CUDA code compiled to OpenCL can reach macOS GPUs by way of MoltenVK and Metal, and mobile GPUs on Android and iOS.

Conformance itself is worth explaining. The Khronos Conformance Test Suite has been open source on GitHub since 2017. Passing it and submitting the results to the Adopters Program is what entitles a product to be listed as conformant and to use the trademark; you do not need to be a Khronos member to become an adopter.

This is a meaningfully stronger regime than “we support OpenCL”: a listing on the conformant products page is a dated, versioned claim about a specific product on specific hardware.

Subscribe now

Where OpenCL actually runs today

Mobile and edge LLM inference. llama.cpp has had an OpenCL backend since late 2024, contributed by Qualcomm and built for Adreno first. It requires sub-group support. It is tuned for Q4_0, with --pure quantisation giving the best results, and supports Q6_K and others.

For Snapdragon X2 SoCs there is a prebuilt binary kernel library covering MUL_MAT_ID with Q4_0, Q4_1, Q4_K and MXFP4, distributed through Qualcomm’s software centre, plus an on-disk compiled-program cache.

At IWOCL 2026 the Qualcomm team reported a more than fourfold prefill speedup on GPT-OSS-20B with mixture-of-experts on Snapdragon X2 Elite, from roughly 120 tokens/s to over 500, through kernel restructuring, memory-access tuning and expert load balancing.

Their roadmap names FlashAttention-style kernels, more quantisation schemes, expanded INT8 paths, and auto-tuning across vendors. That paper won the IWOCL 2026 outstanding short paper award.

Deep learning compilers. TVM and MLC have long-standing Adreno OpenCL support, and Qualcomm has upstreamed the Adreno enhancements (texture paths, specialised layouts, memory management, OpenCLML integration) into TVM’s Relax pipeline after the community deprecated Relay. Notably, they are now adding a Vulkan backend in parallel, reusing more than 90% of the target-independent optimisations, specifically to get access to Vulkan’s cooperative matmul. That is a clear signal about what the missing cooperative matrix extension costs OpenCL.

CUDA portability. chipStar compiles unmodified CUDA and HIP into fat binaries built on OpenCL and SPIR-V. Unlike source-to-source translators, it preserves the programming model and produces binaries that run without recompilation across Intel, AMD, NVIDIA, Mali, PowerVR-on-RISC-V, and CPUs.

SYCL. SYCL began as a single-source C++ layer over OpenCL. SYCL 2020 generalised to multiple backends (Level Zero, CUDA, HIP) but OpenCL remains a first-class one, and it is the backend that gives SYCL reach onto hardware without a vendor SYCL implementation. Intel’s “SYCL Everywhere” work at IWOCL 2026 described exactly this: LLVM evolving for SYCL compilation, SPIR-V as the IR, OpenCL as the portability layer for accelerator offload, and PoCL and Mesa quietly extending device coverage. There is a nice concrete instance of the dependency in the other direction: the FunGT graphics engine uses SYCL’s OpenCL backend specifically to reach cl_khr_gl_sharing, because SYCL has no native OpenGL interop.

FPGAs. Intel’s FPGA SDK for OpenCL and, historically, Xilinx SDAccel (Khronos-conformant since January 2015, later folded into Vitis) used OpenCL as an HLS front end. Pipes and pipe storage exist in the specification largely because of this constituency. The centre of gravity here has moved toward vendor HLS flows and oneAPI, but OpenCL-derived tooling persists in shipping products.

Safety-critical and the road not taken. Khronos says it is exploring OpenCL’s role in safety-critical markets, and there is precedent in the neighbourhood: Vulkan SC exists, SYCL SC is in development, and IWOCL 2026 carried a paper on functional-safety-oriented GPU development that migrates CUDA to SYCL and then applies static analysis to strip constructs unsafe under IEC 61508 and ISO 26262. There is no OpenCL SC. Whether one appears is a reasonable proxy for how seriously the automotive and industrial constituency takes the standard.

WebCL is the road that was not taken. Khronos formed the working group in March 2011 and published a 1.0 specification in March 2014, aiming to expose OpenCL to JavaScript. No browser shipped it; the security surface of arbitrary compute kernels in a web page proved unattractive, and Khronos now lists it among its inactive standards. The niche it aimed at is being filled by WebGPU instead.

The long tail. darktable, GIMP, Blender’s historical Cycles OpenCL backend, LuxMark, Leela Chess Zero, LAMMPS, GROMACS, ffmpeg filters, Arm Compute Library, ncnn, MNN, and the TensorFlow Lite GPU delegate. This is not glamorous work and it is a great deal of deployed code.

Subscribe now

Why CUDA really won

It is worth being accurate about this rather than reaching for “NVIDIA had better marketing.”

Single-source versus split-source. In CUDA, host and device code live in the same translation unit, compiled by one compiler that type-checks kernel launches. In OpenCL, kernels are strings compiled by a different compiler at runtime, with arguments bound by index and size. The gap is not aesthetic; it is a difference in how many errors the type system catches. Templates work across the boundary in CUDA and did not in OpenCL until C++ for OpenCL, which arrived in 2020 and is not ratified.

Libraries. cuBLAS, cuDNN, cuFFT, cuSPARSE, Thrust, CUB, NCCL: vendor-maintained, aggressively tuned, and free. The OpenCL equivalents were clBLAS (deprecated), CLBlast (excellent, and maintained by a far smaller group), clFFT, VexCL, ArrayFire, Boost.Compute. Building a GPU application on CUDA meant assembling tuned components; on OpenCL it frequently meant writing them.

Asymmetric investment, for structural reasons. NVIDIA’s compiler team optimises one language for one family of architectures. An OpenCL implementation is a compiler plus a runtime that must be correct across a specification designed to accommodate CPUs, DSPs, FPGAs and fixed-function accelerators. Vendors were also, understandably, not motivated to make the vendor-neutral API as fast as their proprietary one. A compiler team’s attention is a budget, and it goes where the strategic return is.

The 2.x fragmentation. Covered above. Between 2013 and 2020 the answer to “which OpenCL version can I target?” was 1.2, which meant OpenCL was frozen at a 2011 feature set during precisely the years when GPU compute exploded.

Tooling. Nsight Compute and Nsight Systems have no OpenCL equivalent. There are good OpenCL tools, the Intercept Layer, clinfo, clpeak, Oclgrind for memory error detection, Snapdragon Profiler, Intel VTune, now CLVizulayer, but they are assembled from several projects rather than shipped as one supported product.

Apple’s exit. Apple invented OpenCL, held the trademark, and deprecated it in macOS 10.14 Mojave in 2018 in favour of Metal, along with OpenGL, telling developers to move computational work to Metal Performance Shaders. Losing your originator is a bad look regardless of installed base.

Blender’s retirement of the OpenCL path in Cycles is the compact version of the whole story. The stated reasons were a limited kernel implementation, driver bugs, and a stalled standard, and the combination made maintenance untenable. Not one of those is a flaw in the specification. All three are consequences of the specification being implemented by people who were investing elsewhere.

What is also true, and less often said: a striking amount of what OpenCL specified early has since become the industry consensus. SPIR-V is now the IR for Vulkan, OpenCL, and SYCL, and is a target for Slang, clspv, and a growing set of DSLs. Sub-groups are warps by another name and every API now exposes them.

Command buffers are CUDA Graphs and Level Zero command lists. Unified shared memory is CUDA’s unified memory with a better capability model. Cooperative matrix is the same abstraction in Vulkan and OpenCL, standardised jointly. The ideas held up; what OpenCL never got was the distribution to go with them.

Subscribe now

Writing OpenCL in 2026

Practical positions, offered as opinion rather than doctrine:

Target OpenCL 1.2 plus queried features, or 3.0 with capability checks. Do not target 2.x. Query at startup, build the kernel variant the device can run, and keep the fallback paths honest by testing them: PoCL and Rusticl on a laptop will exercise a very different feature set from a vendor GPU driver.

Ship SPIR-V, once 3.1 drivers are common. Until then, ship source with a persistent binary cache keyed on device name, driver version, source hash and build options. The startup compile cost is real and users notice it.

Concretely, the startup path that makes all of this work is short enough to paste:

/* Select a device, discover what it can actually do, and build the
   matching kernel variant. This is the whole portability story. */
cl_device_id dev = pick_device();

char prof[64], ver[128];
clGetDeviceInfo(dev, CL_DEVICE_PROFILE, sizeof(prof), prof, NULL);
clGetDeviceInfo(dev, CL_DEVICE_VERSION, sizeof(ver),  ver,  NULL);
int embedded = (strcmp(prof, "EMBEDDED_PROFILE") == 0);
int major = ver[7] - '0';                 /* "OpenCL M.m ..."         */
int minor = ver[9] - '0';

/* OpenCL 3.0+ optionality queries; on 1.2 they simply fail, which is
   the answer.  Treat a failed query as "unsupported", never as fatal. */
cl_device_atomic_capabilities atomics = 0;
cl_bool generic_as = CL_FALSE, images = CL_FALSE;
cl_device_device_enqueue_capabilities dq = 0;
clGetDeviceInfo(dev, CL_DEVICE_ATOMIC_MEMORY_CAPABILITIES,
                sizeof(atomics), &atomics, NULL);
clGetDeviceInfo(dev, CL_DEVICE_GENERIC_ADDRESS_SPACE_SUPPORT,
                sizeof(generic_as), &generic_as, NULL);
clGetDeviceInfo(dev, CL_DEVICE_DEVICE_ENQUEUE_CAPABILITIES,
                sizeof(dq), &dq, NULL);
clGetDeviceInfo(dev, CL_DEVICE_IMAGE_SUPPORT, sizeof(images), &images, NULL);

size_t next; clGetDeviceInfo(dev, CL_DEVICE_EXTENSIONS, 0, NULL, &next);
char *ext = malloc(next);
clGetDeviceInfo(dev, CL_DEVICE_EXTENSIONS, next, ext, NULL);
int subgroups = strstr(ext, "cl_khr_subgroups") != NULL || (major > 3)
             || (major == 3 && minor >= 1);
int fp16      = strstr(ext, "cl_khr_fp16")      != NULL;
int fp64      = strstr(ext, "cl_khr_fp64")      != NULL;

/* Build options carry the decisions into the kernel. */
char opts[512];
snprintf(opts, sizeof opts,
         "-cl-std=%s -DSUBGROUPS=%d -DFP16=%d -DFP64=%d -DIMAGES=%d "
         "-DEMBEDDED=%d -cl-kernel-arg-info",
         (major >= 3) ? "CL3.0" : "CL1.2",
         subgroups, fp16, fp64, images == CL_TRUE, embedded);

cl_program p = build_from_cache_or_source(ctx, dev, opts);   /* see §8 */

and the kernel side picks its own path from the same flags, preferring the mechanism the device actually has:

inline float block_sum(float v, __local float *scratch) {
#if SUBGROUPS && (defined(__opencl_c_subgroups) || defined(cl_khr_subgroups))
#  if defined(cl_khr_subgroups) && !defined(__opencl_c_subgroups)
#    pragma OPENCL EXTENSION cl_khr_subgroups : enable   /* needed on 2.x */
#  endif
    return sub_group_reduce_add(v);
#elif defined(__opencl_c_work_group_collective_functions)
    return work_group_reduce_add(v);
#else
    size_t l = get_local_id(0), n = get_local_size(0);
    scratch[l] = v;
    barrier(CLK_LOCAL_MEM_FENCE);
    for (size_t s = n >> 1; s > 0; s >>= 1) {
        if (l < s) scratch[l] += scratch[l + s];
        barrier(CLK_LOCAL_MEM_FENCE);
    }
    return scratch[0];
#endif
}

Three variants of one reduction is not elegant, and it is the actual cost of the portability the standard sells. The fallback path is the one to test hardest, because it is the one that runs on the hardware you did not anticipate.

Use -cl-kernel-arg-info, set a context error callback, and check CL_KERNEL_PRIVATE_MEM_SIZE. These three cost nothing and catch a disproportionate share of problems.

Treat the local work size as a measured parameter, not a guess or a default. A 6.8× spread across plausible values is normal. Benchmark NULL against a handful of multiples of CL_KERNEL_PREFERRED_WORK_GROUP_SIZE_MULTIPLE, bounded by CL_KERNEL_WORK_GROUP_SIZE, or use the 3.1 suggested-size query. Sometimes the runtime wins; assume neither.

Assume nothing about work-group execution order. No inter-work-group spin-waiting, ever.

Treat tuning parameters as data, not code. Tile sizes, work-per-thread, vector width and unroll factors should be build-time defines selected from a per-architecture table, with autotuning as the fallback. This is what CLBlast does and it is the only approach that survives contact with five vendors.

Reach for the ecosystem before writing kernels. CLBlast for BLAS, VkFFT or clFFT for transforms, Arm Compute Library on Mali, OpenCLML on Adreno. And if the actual goal is portable C++ rather than portable kernels, SYCL over an OpenCL backend is very likely the better answer: OpenCL is a good compilation target and a mediocre application-level programming model, and the working group’s own framing now treats it that way.

Debug with the layer mechanism. The Intercept Layer will tell you what your application is really doing to the driver, without a rebuild.

Subscribe now

Where this goes

The OpenCL of 2026 has settled into a role nobody planned for it in 2008. It is not the API most application developers write. It is the layer that other things stand on: SYCL implementations, chipStar, TVM, MLC, llama.cpp’s mobile path, the Mesa and PoCL stacks that give OpenCL, and therefore everything above it, reach onto hardware whose vendors have shipped nothing.

The working group’s own language has shifted to match, talking about OpenCL as a substrate for higher-level programming models rather than as the model itself.

The most urgent open question is cooperative matrix. Hardware matrix units are where the FLOPs are, Vulkan standardised access to them years ago, and until the OpenCL extension is final and shipping, every inference backend on OpenCL is hand-rolling GEMM against silicon it cannot fully address.

Qualcomm adding a Vulkan path to TVM alongside the OpenCL one, explicitly to get cooperative matmul, is the warning shot. The low-precision data types are the same problem one layer down: int4 and fp8 are on the roadmap, and the workloads that need cooperative matrix need those too.

Underneath both sits the question of whether the 3.1 SPIR-V mandate converts into broad driver support quickly enough to matter to the compiler authors it was written for.

The encouraging signal is that the working group has stopped repeating the 2.0 mistake. Everything mandated in 3.1 was already deployed. The extension pipeline is public, the drafts are on GitHub, the Clang RFCs are on Discourse, and the conformance suite is open source.

Whether that process moves fast enough against a competitor with a decade’s head start and no committee is a genuinely open question, and I do not think anyone should be confident either way.

What I’d say with more confidence is that the obituaries were premature by about a decade, and that if you are building anything that has to run on hardware you do not control (phones, embedded silicon, whatever a customer happens to own), OpenCL is currently the only thing in the category that is both open and actually there.

Which returns to where this started. A standard that has to describe hardware it cannot see can only ever specify floors. The measurements in this article are all instances of that: an accuracy bound four orders of magnitude looser than what an implementation delivers, a work-group size the specification will not choose for you and that costs 6.8× to get wrong, an allocation flag that means nothing until you change how you use it.

Every one of those is the standard declining to promise something it cannot promise for every device.

That is the failure mode and it is also the whole value. CUDA can tell you what your hardware does because NVIDIA built it. OpenCL cannot, and in exchange it is the reason a kernel written in 2011 runs today on a phone, an FPGA, a RISC-V board, and an Apple GPU under a driver Apple did not write and does not support.

The interesting question was never whether that trade was worth it in general.

It is whether it is worth it for the thing you are building, and the honest answer is that for most people writing CUDA in 2026 it is not, and for the people shipping software onto hardware they will never see it is the only trade on offer.

Subscribe now

Appendix A: version timeline

Subscribe now

Appendix B: queries worth calling at startup

/* Identity and version */
CL_PLATFORM_VERSION            CL_DEVICE_VERSION
CL_DEVICE_NAME                 CL_DEVICE_VENDOR
CL_DRIVER_VERSION              CL_DEVICE_OPENCL_C_ALL_VERSIONS   /* 3.0 */
CL_DEVICE_UUID_KHR                                               /* core in 3.1 */

/* Shape of the machine */
CL_DEVICE_MAX_COMPUTE_UNITS    CL_DEVICE_MAX_WORK_GROUP_SIZE
CL_DEVICE_MAX_WORK_ITEM_SIZES  CL_DEVICE_LOCAL_MEM_SIZE
CL_DEVICE_LOCAL_MEM_TYPE       CL_DEVICE_GLOBAL_MEM_CACHELINE_SIZE
CL_DEVICE_MAX_MEM_ALLOC_SIZE   CL_DEVICE_GLOBAL_MEM_SIZE
CL_DEVICE_HOST_UNIFIED_MEMORY  /* meaningful as of 3.1 */

/* Capabilities */
CL_DEVICE_EXTENSIONS_WITH_VERSION
CL_DEVICE_SVM_CAPABILITIES
CL_DEVICE_ATOMIC_MEMORY_CAPABILITIES
CL_DEVICE_ATOMIC_FENCE_CAPABILITIES
CL_DEVICE_DEVICE_ENQUEUE_CAPABILITIES
CL_DEVICE_PIPE_SUPPORT
CL_DEVICE_GENERIC_ADDRESS_SPACE_SUPPORT
CL_DEVICE_WORK_GROUP_COLLECTIVE_FUNCTIONS_SUPPORT
CL_DEVICE_NON_UNIFORM_WORK_GROUP_SUPPORT
CL_DEVICE_IMAGE_SUPPORT
CL_DEVICE_SINGLE_FP_CONFIG     CL_DEVICE_DOUBLE_FP_CONFIG

/* Per kernel, after build */
CL_KERNEL_WORK_GROUP_SIZE
CL_KERNEL_PREFERRED_WORK_GROUP_SIZE_MULTIPLE
CL_KERNEL_LOCAL_MEM_SIZE
CL_KERNEL_PRIVATE_MEM_SIZE
CL_KERNEL_COMPILE_WORK_GROUP_SIZE

Subscribe now

Appendix C: the same concept in seven APIs

Every modern compute API converged on the same execution model with different nouns. This is the translation table, and it is most of what porting between them consists of.

Two structural observations fall out of the table. The rows are almost perfectly aligned, which is why source-to-source translation between these APIs works as well as it does and why chipStar can compile CUDA onto OpenCL at all. And the one row where OpenCL is behind, the last one, is the row where the FLOPs are.

The source model row is the difference that decided the market. Single-source means host and device code are compiled together by one compiler, so a kernel launch is type-checked and templates cross the boundary.

Split-source means kernels are strings or binaries bound by index at runtime. The pattern is not simply proprietary-versus-open, and Metal is the counterexample that shows it: Apple controls its compiler completely and still chose split-source, because MSL is its own language in its own files.

What actually predicts the choice is whether one compiler is expected to consume both halves. CUDA, HIP and SYCL say yes and get type-checked launches and templates across the boundary.

OpenCL, Vulkan, Metal and Level Zero say no, and pay for it at every call site where an argument is bound by index instead of by name.

Subscribe now

Appendix D: sources

Khronos OpenCL Registry and the unified API, C, and SPIR-V Environment specifications (rev. 3.1.1, 22 May 2026); the Khronos blog posts announcing OpenCL 3.1 (4 May 2026) and the cooperative matrix extensions (29 April 2026);

the OpenCL-Docs, OpenCL-Headers, OpenCL-ICD-Loader and OpenCL-CTS repositories; the IWOCL 2026 program and papers, particularly the Qualcomm llama.cpp paper, the chipStar keynote, Intel’s AI-workload talk, and the CLVizulayer presentation;

NVIDIA’s OpenCL 3.0 conformance announcement (April 2021); llama.cpp’s OpenCL backend documentation; PoCL and Mesa release notes; Clang’s OpenCL support documentation; and Phoronix’s reporting on the 3.0.x and 3.1 releases and on conformance submissions.

Introduction

As many of you know, even if you’re not in the financial side of technical companies, NVIDIA published two numbers twelve hours apart this week. On Monday its engineers were at Hot Chips and they described a rack of 256 chips that holds 128 gibibytes of SRAM.

On Wednesday its CFO reported that the company has committed 279 billion dollars, primarily to buying DRAM: those are not separate stories. They are the same thing said twice, once by the engineering organisation and once by finance, and what connects them is a quantity that the standard formula for it gets wrong.

Everyone who deals with inference hardware knows the critical batch: the point where a decode step stops being memory bound and starts being compute bound, B* equals peak arithmetic times bytes per element over twice the memory bandwidth. On a Rubin package that is 398, whereas on the LPX rack it is 3.94. Those are the numbers I expected to build this piece on.

They describe a machine serving no context at all. The formula is derived from a decode step that reads only weights, and once you put the KV cache back into it the expression acquires a pole.

After a certain context length the denominator goes negative and no batch size makes the machine compute bound, because each additional sequence adds more memory traffic than amortisable arithmetic.

For Gemma 4 31B, the model NVIDIA benchmarked, a Rubin package reaches that point at 84 tokens. The LPX rack holds out to roughly 68,000. That precise gap, and not the 101 times in the spec sheets, is what NVIDIA spent 20 billion dollars to buy.

What follows is the derivation, the money behind it, and the places where I got it wrong first.

The short version

I want to introduce this section in this research post, so that you are able to take the core concepts and principles in the short form. I do really think that it will help, let me know if that’s the correct approach.

1. The critical batch formula in common use drops the KV term. Restored, it has a pole at the saturation context: 84 tokens on Rubin, 68,000 on the LPX rack, a ratio of 811 rather than 101.

2. Above saturation, raising batch buys capacity utilisation, not arithmetic utilisation. Throughput ceilings at memory bandwidth over cache bytes per sequence. For this model on an NVL72 that is about 143,000 tokens per second and no scheduler setting beats it.

3. At 100K of context the NVL72 makes 13 times the rack tokens. The LPX makes each user’s tokens arrive 45 times faster. That is the entire trade, measured at one context length.

4. The LPX is a latency product, not a bandwidth product. It runs 279 times below its own bandwidth roofline, and streaming plus arithmetic account for 0.43 percent of a token’s time budget.

5. SRAM is 1.4 to 6.4 times dearer per byte than HBM4 and 642 to 2,924 times cheaper per byte per second. Every design decision in the rack follows from that one ratio.

6. NVIDIA’s 279 billion dollar commitment, read against its own revenue guide, implies 28.2 dollars per gigabyte of HBM4, which lands within 11 percent of the price Korean and Taiwanese analysts publish. Two methods sharing no inputs.

Subscribe now

The ledger

Let’s start with the disclosure, because it is unambiguous and it is filed directly with the SEC. NVIDIA reported revenue of 96,221 million dollars for the quarter ended 26 July 2026, of which 89,023 million was the Data Center section.

Gross margin was 75.0 percent on both a GAAP and a non-GAAP basis. Cost of revenue was precisely at 24,079 million.

Three things in the same document deserve more attention than this short description.

The first one is the commitment table. In the CFO commentary, Colette Kress wrote that supply commitments rose from 119 billion dollars a quarter ago to 279 billion, and attributed the increase to the procurement of memory.

The table below breaks it out by fiscal year: 92 billion for the remainder of FY2027, 87 for FY2028, 88 for FY2029, then 6, then 5, then 1.

Ninety six percent of that money, 267 billion of the 279, lands inside three fiscal years. This isn’t a long-dated strategic reserve: it is a three year buy with a cliff at the end of it, which is what a purchase schedule looks like when you are securing a component whose supply is allocated rather than traded.

The second one is the guide, and it runs way further than the headline. NVIDIA guided Q3 revenue to 108 billion dollars and gross margin to 74.0 percent, a hundred basis points below the quarter it just reported. On the call Kress said that margin falls to 71 or 72 percent by the January quarter before recovering to 72 or 73 next year.

That is a three hundred to four hundred basis point trough, not a one hundred point wobble, and it is scheduled. On 108 billion, a hundred basis points is 1,080 million dollars of additional cost of revenue in a single quarter.

If memory is roughly 30 to 40 percent of accelerator manufacturing cost, which is the range the trade estimates converge on and which is consistent with the roughly 3,250 dollars of HBM3e against 850 dollars of logic die in a B200, then the entire memory bill in that quarter is somewhere between 8.4 and 11.2 billion dollars. The guide-down is about eleven percent of it.

The third is the working capital. Days sales outstanding went from 45 to 60, attributed to extended payment terms on multi-quarter agreements. Inventory went from 25.8 to 31.6 billion.

The company issued 25 billion dollars of senior unsecured notes in a quarter in which it generated 21.3 billion of free cash flow and returned 26.0 billion to shareholders.

Separately it disclosed guarantees with a maximum gross exposure of 108.5 billion, of which 105 billion is credit support for 4.25 gigawatts at SB Energy’s PORTS-Pike campus in Ohio, leased to OpenAI for twenty years.

I’m not going to make this post about the financing structure, which is clearly outside what I can measure and out of this niche. We just want the memory number, and this is checkable against something else NVIDIA said.

Subscribe now

Solving for the price of a gigabyte

The commitment schedule and the revenue guide are two independent statements about the same physical object count. Force them to agree and the effective memory price falls out.

Take the revenue path first of all.

Q1 came in at 81.6 billion, Q2 at 96.2, Q3 is guided to 108.0. If Q4 grows at the same sequential rate as the Q3 guide, FY2027 lands near 407 billion. Management gave preliminary FY2028 guidance of roughly 70 percent growth, supply constrained, which puts FY2028 near 692 billion. Decay that growth by half for FY2029 and the three year total is about 2.03 trillion dollars. The 279 billion commitment is 13.7 percent of it.

Now let’s try to convert revenue to units.

Data Center was 92.5 percent of revenue this quarter. Not all of that is GPU: there are CPUs, switches, NICs and complete systems in the number, and the CFO commentary breaks Data Center out by customer type rather than by product, so the GPU share is not disclosed. Take a band of 55 to 70 percent of Data Center revenue as GPU, and an average selling price of 40,000 to 50,000 dollars per package. The three year revenue path then implies somewhere between 20.7 and 32.9 million GPUs.

Each Rubin package carries 288 gigabytes of HBM4. If 60 to 90 percent of the 279 billion is memory, dividing through gives an effective price of 17.7 to 42.1 dollars per gigabyte, with a simple average midpoint of 28.2.

What’s worth noting is that two different HBM4 prices are in wide circulation and they are not comparable, which is sort of a trap we walked into in the first edit of this piece.

The factory gate figure is a component price: it’s circa 550 dollars for a 36 gigabyte twelve-high stack, or 15.28 dollars per gigabyte. The contract figure is what NVIDIA is projected to actually pay under long term agreements.

Seoul Economic Daily reported in July that HBM4 was moving from about 2 dollars per gigabit to 4 or 5, which is 32 to 40 dollars per gigabyte. Various analyst projections we’ve seen during the research process reported this month put it at 31 to 32 dollars per gigabyte for NVIDIA specifically and 35 to 36 for other buyers.

As stated before, our reconciled figure is 28.2 dollars per gigabyte with a band of 17.7 to 42.1. That lands within 11 percent of the 31 to 32 the Korean and Taiwanese analysts are publishing, from a completely independent direction: a commitment table and a revenue guide, with no memory market data used anywhere in the derivation.

The strongest result in this analysis is that two completely independent methods converge to within a tenth of each other. That agreement provides the strongest evidence that the underlying financial calculations are in some ways, real.

Let’s call it 30 dollars a gigabyte, then. At 288 gigabytes per package, the memory in a single Rubin GPU costs on the order of 8,600 dollars, and the memory in a Vera Rubin NVL72 rack costs on the order of 620,000.

On the factory gate basis it would be 4,400 and 317,000, which is the number to use if you want a component cost and the wrong number to use if you want NVIDIA’s bill. Either way, memory is the largest single line, and it is the line whose price is set by three suppliers in a market where HBM has gone from 8 percent of DRAM wafer output in 2024 to 23 percent in 2026.

That is the position NVIDIA is buying its way through. The interesting question is what it is building to avoid it.

Subscribe now

The machine

NVIDIA licensed Groq’s LPU architecture and hired most of the company in a 20 billion dollar transaction struck at the end of December 2025. The first silicon, the LP30, was shown at GTC in March and went into full production this month.

Igor Arsovski, who came over from Groq, presented the rack at Hot Chips precisely on Monday. Nebius is the named first customer. The cash flow statement carries a 2,944 million dollar line item labelled simply “Groq, Inc.” under financing activities.

The published rack specification is 256 LP30 accelerators, 128 GB of SRAM, 40 PB/s of aggregate SRAM bandwidth, 315 PFLOPS of FP8, and 96 chip-to-chip links per die running at 112 Gbps each. The die is Samsung 4nm and the trade press puts 512 MB of SRAM and 150 TB/s on it.

Those figures do not quite close, and the way they fail to do it so is merely informative. 256 dies at 512 MB each is 131.07 gigabytes in decimal units, not 128, which is exactly 128 gibibytes. So the rack figure is binary and the die figure is decimal, and the only reading under which both are correct is 512 mebibytes per die.

Similarly, 40 PB/s across 256 dies is 156.25 TB/s each, not 150; and 72 Rubin GPUs at 22 TB/s is 1.584 PB/s, not the 1.6 that gets quoted. I use the exact values throughout and note where they differ from the round ones.

Subscribe now

One ratio we want you to think

An LP30 contains 512 MiB of SRAM. That lets us estimate its cost within a reasonable range. A high-density six-transistor SRAM cell measures about 0.021 µm² on TSMC N5, a figure that remains broadly unchanged at N3E. The LP30, however, is built on Samsung 4 nm, where 4LPP uses a 54 nm contacted gate pitch versus 51 nm for N5, putting the equivalent cell closer to 0.026 µm².

At 512 MiB, the LP30 contains 4,294,967,296 bits. At 0.021–0.026 µm² per bit, that corresponds to roughly 90–112 mm² of raw SRAM cell area. Once decoders, sense amplifiers, column multiplexers, and other array overhead are included, assuming 60–75% array efficiency, the SRAM array itself comes to approximately 120–186 mm².

Wafer price cuts the other way. TSMC N5 and N4 wafers run at roughly $18,500, while Samsung typically prices 15–30 percent below TSMC at comparable nodes, putting a Samsung wafer at approximately $12,950–$15,725. Spread across 70,686 usable square millimetres, that gives roughly $22–$49 of silicon cost per die, or $44–$97 per gigabyte of SRAM, with a central case around $64.

The two corrections partly cancel. Using a TSMC SRAM cell size together with a TSMC wafer price for a Samsung-built part understates the die area by roughly a quarter while overstating the wafer cost by a similar amount.

Those errors happen to push the final estimate in opposite directions, leaving the original point estimate of $66 per gigabyte surprisingly close to the corrected central case of $64. The agreement is therefore accidental rather than methodological. The defensible result is a range, not a single number.

Against HBM4 at roughly $15.28 per gigabyte as a component and $31.50 under contract, SRAM comes out somewhere between 1.4 and 6.4 times more expensive per byte. And even that comparison is generous to SRAM: the calculation covers essentially bare array area, with no allowance for yield loss, peripheral circuitry, testing, packaging, or margin.

On these assumptions, the conclusion is difficult to avoid: SRAM is extraordinarily expensive memory.

Now divide by bandwidth instead of by capacity.

The 512 mebibyte array delivers 156 TB/s, so the silicon under a terabyte per second of SRAM bandwidth costs 14 to 31 cents, centrally 21. The 288 gigabytes of HBM4 on a Rubin package delivers 22 TB/s, so the memory under a terabyte per second of HBM bandwidth costs about 200 dollars as a component and 412 under contract.

Take the corner least favourable to my argument, the dearest SRAM against the cheapest HBM, and the ratio is 642. Take the central cases and it is 974 against the component price and 2,009 against the contract price. Take the corner most favourable and it is 2,924.

There is no assumption inside any of these ranges under which the answer is less than two and a half orders of magnitude, which is the only precision the conclusion needs.

Stated the other way: the SRAM in an entire LPX rack is 5,600 to 12,500 dollars of silicon area, centrally about 8,200. The HBM4 in a single Vera Rubin NVL72 rack is 317,000 dollars as a component and about 653,000 at the price NVIDIA is projected to pay. The LPX rack has twenty five times the aggregate bandwidth.

Every design decision in the LPX falls out of that one ratio. Bandwidth per byte is 3,810 times higher on the SRAM machine, which means capacity per unit of bandwidth is 3,810 times lower. A Vera Rubin NVL72 holds 20.7 terabytes. The LPX holds 128 gibibytes, which is 151 times less.

You cannot put a large model in it, you cannot put a long KV cache in it, and the only workload it can run alone is one whose entire working set is under about 137 gigabytes.

Subscribe now

The knee

In a piece earlier this month I derived the batch size at which a decode step stops being memory bound. A decode step reads W bytes of weights and performs 2NB floating point operations, where N is the parameter count, W equals N times b bytes per element, and B is the batch. Setting compute time equal to memory time gives

B* = P · b / (2 · Bmem)

and because peak arithmetic P scales as 1/b on tensor cores, the product P·b is a generational constant. B* is invariant to precision. Running the same model in FP4 instead of FP8 doubles your token rate and moves the knee not at all.

Both sides of that ratio have to be on the same basis, which is the part that is easy to get wrong and which we both got wrong in the earlier drafts of this piece. A sparse or compressed peak divided by a real bandwidth inflates B* by the compression ratio. Every figure below is dense.

The invariance held well for two generations. H100 sits at 295, H200 at 206, HGX B200 at 292, GB200 and MI355X at 312. All five re-derive here from datasheet FP8 dense throughput and published HBM bandwidth, so the plateau is a result rather than a citation.

Rubin is where the basis matters. NVIDIA markets the R100 at 50 PFLOPS of NVFP4 inference, and that figure is not dense. SemiAnalysis reports that NVIDIA brands 50 PFLOPS as the inference number while 35 PFLOPS of NVFP4 is the dense one, and that the five times over Blackwell claim compares compressed FP4 against dense FP4. Their Rubin CPX analysis puts the same part at 50 sparse and 33.3 dense on a three to two ratio.

NVIDIA’s own rack table settles it. The NVL72 is published at 3,600 PFLOPS of NVFP4 inference, 2,520 of dense NVFP4, and 1,260 of dense FP8.

Divide by 72 and the per package figures are 50 marketed, 35 dense NVFP4, and 17.5 dense FP8. The ratio between the marketed and the dense number is 3,600 over 2,520, which is ten to seven.

Both dense figures give the same answer, as they must, since P scales as 1 over b: 35 PFLOPS at half a byte and 17.5 at one byte are the same product. Against 22 TB/s, B* is 398. The SemiAnalysis route through the three to two tensor core ratio gives 378. Had I used the marketed 50, I would have got 568, which is exactly the ten to seven too high.

So the knee moved 1.36 times over HGX B200, and not 1.94. That is a smaller claim than the one I started with and it is the one the arithmetic supports. To run a Rubin package at the batch where its arithmetic is fully occupied you need 398 concurrent sequences resident on it, against 292 on Blackwell.

One refinement falls out of the same correction. The invariance is not across all precisions on this part. SemiAnalysis reports that the third generation Transformer Engine doubles tensor core width for FP4 and FP8 only, leaving BF16 and TF32 to scale about 1.6 times. P·b is therefore no longer one constant: B* is 398 at FP8 and at NVFP4, and near 164 at BF16. On Blackwell it was 292 at all three. Anyone still serving in BF16 is on a machine with a very different knee from the one on the slide.

An LPX rack is rated at 315 PFLOPS of FP8 across 40 PB/s of SRAM. B* equals 3.94. Per chip the answer is identical, because the ratio is scale invariant: 1.23 PFLOPS against 156 TB/s gives the same 3.94.

The ratio between the two machines is 101.

Subscribe now

The benchmark model is not incidental

Gemma 4 31B was released by Google DeepMind on 2 April 2026 under Apache 2.0 license. Its configuration is public: 60 layers, of which 50 are sliding window attention with a window of 1,024 and 10 are full global attention, 32 query heads, 16 KV heads, a head dimension of 256 in the sliding layers and 512 in the global ones.

The Gemma 4 technical report describes two memory optimisations on the global layers. Keys are re-used as values, and position is encoded with p-RoPE at p equal to 0.25. It then states that this reduces the global KV cache by 37.5 percent, without saying how.

That constant is derivable, and deriving it tells you what is actually stored. If you keep K and V separately you store 2d bytes per head per token. If values equal keys you would store d, a 50 percent cut.

But K carries rotary position and V must not, so the shared tensor can only be the unrotated part. With p-RoPE only a fraction p of the head dimension is rotated, so what you keep is the full unrotated tensor plus a rotated copy of the p fraction:

stored / baseline = d(1 + p) / 2d = (1 + 0.25) / 2 = 0.625

which is a 37.5 percent reduction, exactly. The residual against the published figure is zero. So the global layers store 1.25 tensors where a conventional model stores 2.

One step here is entirely from me and Lorenzo Tettamanti and not the config’s. The published file gives 16 KV heads, a head dimension of 256 and a global head dimension of 512, and does not state a separate global head count.

I take it to be 8, on the reasoning that 8 heads at 512 is the same 4,096 element width as 16 at 256, which is the pattern the 26B variant follows and the only one under which the report’s 37.5 percent lands on a round byte figure. The whole cache number below rests on that inference, so it is marked as an inference in the dossier rather than buried here.

I flag one disagreement. Several community analyses of this model use 8,192 bytes per token per global layer, which corresponds to a clean 50 percent reduction and omits the rotated copy. The report’s own 37.5 percent implies 10,240. I use the report.

Run the cache arithmetic at 100K of context and the sliding layers contribute 0.84 gigabytes, capped by their 1,024 token window and therefore constant in context length, while the global layers contribute 10.24. The total is 11.1 gigabytes. A conventional model with the same head count and 60 full attention layers would need 98.3. The hybrid saves 8.87 times at 100K and 9.31 times at the model’s full 262,144.

Weights plus cache at 100K in FP8 is 41.8 gigabytes, which is 30.4 percent of the LPX rack’s 137.4 gigabytes. In BF16 it is 52.7 percent. The benchmark fits with room to spare, and it fits because Gemma 4 is close to the best case a 128 gibibyte SRAM machine could ask for: a dense model small enough to shard 256 ways, with an attention design that caps its own cache growth.

Work out what would not fit. With 100K of its own cache resident, the largest dense model an LPX rack holds is 126 billion parameters at FP8 or 253 billion at FP4.

The two trillion parameter figure NVIDIA quotes is 1,000 gigabytes at FP4, or 7.3 racks of SRAM, and the figure in their blog that carries it is labelled a projection of a scaled up GPT-OSS, not a measurement. That is honest labelling on their part and I want to repeat it rather than let it blur.

Subscribe now

The term the formula drops

Here is where we have to correct our own framing, because the formula above is incomplete and the incompleteness is not small.

B* comes from a decode step that reads W bytes of weights and performs 2NB flops. That accounting is right only if the KV cache is negligible. It is not.

At batch B and context S a decode step reads W plus B times KV(S) bytes, because every resident sequence’s cache is read once per token, while the flops that batching amortises are still only 2NB. Attention’s own flops scale with context exactly as its bytes do, so attention has fixed arithmetic intensity and never becomes compute bound at any batch.

Restore the term and set compute time equal to memory time:

2NB / P = (W + B·KV) / Bmem
B (2N·Bmem/P − KV) = W
B* = b / (2·Bmem/P − KV/N)

This reduces to the weights-only form when KV goes to zero. It also has a pole. When KV per sequence reaches 2·B_mem/P per parameter, the denominator vanishes and B* goes to infinity: past that context, no batch size makes the machine compute bound, because each additional sequence adds more memory traffic than it adds amortisable arithmetic.

Call that the saturation context. It is the number I should have led with.

For a Rubin package the headroom is 2.51 millibytes of KV per parameter, which on a 30.7 billion parameter model is 77 megabytes. Gemma 4’s cache passes 77 megabytes at 84 tokens of context. Eighty four. Past that, a Rubin GPU serving this model is memory bound at every batch size there is.

For the LPX rack the headroom is 254 millibytes per parameter, or 7.8 gigabytes of cache, which Gemma 4 reaches at about 68,000 tokens. The ratio between the two saturation contexts is 811, not the 101 that the weights-only knees give, because the sliding window caps 50 of the 60 layers and makes the relationship nonlinear.

One input to that carries more weight than I would like. The published config gives 16 KV heads and a 512 dimension global head, but does not state a separate global head count, and I read it as 8 on the constant width argument above.

Take the literal 16 instead and the cache doubles to 21.3 gigabytes at 100K, Rubin’s saturation moves only from 84 tokens to 75, and the LPX’s halves from 68,000 to 34,000. The ratio becomes 451 rather than 811. Every qualitative claim survives that swing and the LPX figure is good to within a factor of two, which is the honest precision on it.

Run it across every machine in the table and the picture is worse than a two way comparison suggests.