I’ll show you seven numbers this piece derives

I didn’t set out to write about DeepSeek V4-Flash, but i just wanted to check a precise number. The technical report gives the model’s constants in a paragraph on page 25 and its headline size, 284B total and 13B activated, on page 4, and I wanted to know whether the two agreed before I trusted anything else in the document.

They agree to seven hundredths of one percent, but only once you notice that the headline quietly leaves out the multi-token prediction module and that Heavily Compressed Attention carries half the key-value projections that Compressed Sparse Attention does.

Doing the raw math anyway shows something that the report never states: attention is 1.7 percent of this model’s weights and 39 percent of the weights it touches per token.

That is the kind of fact that changes what you build. It means the sparsity everyone is now discussing on reddit, lives entirely in the expert bank, that the attention stack is as dense as it has ever been, and that there is a hard floor under how cheap a model in this family can get. It also doesnt appear in any of the two dozen write-ups of this model I read before starting.

So this is more a reconstruction rather than a summary. Every architectural number below was just rebuilt from the published constants and checked against something DeepSeek or the vLLM team published independently.

Where the reconstruction disagrees with what is currently written about V4-Flash on the open web, I say so. Where it disagrees with the report, I say that too.

Subscribe now

What shipped, and what the number on the card means

As of now, there are two DeepSeek-V4-Flash releases and they are the same weights. The preview landed on 24 April 2026 with the 58 page technical report and DeepSeek-V4-Pro.

The official release, tagged 0731, landed on 31 July 2026 as a public beta of the API. DeepSeek’s changelog states that DeepSeek-V4-Flash-0731 has the same model structure and the same size as the preview and that only the post-training was rerun. I’ve not yet seen a new architecture, or a new parameter count; even the price hasn’t changed.

That is convenient for a piece like this, because every architectural fact in the April report still describes the model you can call today. It also means the agent scores DeepSeek published with the 0731 release, Terminal Bench 2.1 at 82.7 and DeepSWE at 54.4, are alignment results rather than efficiency results.

They were measured with DeepSeek’s own harness in what the changelog calls minimal mode at the max reasoning tier, temperature 1.0, top_p 0.95. The harness has not shipped.

Agent scores move by ten points on harness changes. Just treat them as a statement of intent.

DeepSeek-V4-Flash, published configuration. Report section 4.2.1, stated directly.

I want to focus on one correction, before we go further and deeper. A widely cited third-party guide states that DeepSeek publishes the layer arrangement for Pro but not for Flash, and advises readers to treat Flash’s layer count as unconfirmed.

That is totally wrong. The report gives Flash 43 layers and hidden dimension 4096 in the same paragraph that gives it 284B parameters. Pro gets 61 layers and 7168.

The arrangements differ in one way that matters: Pro’s first two layers are HCA, Flash’s first two are pure sliding window attention with no compression at all.

Rebuilding the model from its constants

Every expert is a SwiGLU block, so three matrices of 4096 by 2048, which is precisely 25.17M parameters. With 256 routed experts plus one shared expert in all 43 blocks, the expert bank alone is 278.108B.

That is 97.8 percent of the model and it takes one line of arithmetic. Everything interesting is (in my opinion) in the remaining 2.2 percent.

The attention shapes are not what you would guess from a normal transformer, and CSA and HCA are not the same size. CSA computes two independent key-value streams, so four projection matrices from equations 9 and 10 of the report.

HCA computes one, so two matrices, from equations 20 and 21. Getting this wrong is the difference between a reconstruction that lands and one that does not.

CSA layer
  W^aKV, W^bKV, W^aZ, W^bZ    4 x (4096 x 512)   =  8.389 M   two overlapped KV streams
  B^a, B^b                     2 x (4 x 512)      =  0.004 M   learnable positional bias
  W^DQ                         4096 x 1024        =  4.194 M   query down-projection
  W^UQ                         1024 x (512 x 64)  = 33.554 M   query up-projection
  W^IUQ                        1024 x (128 x 64)  =  8.389 M   indexer query up-projection
  W^w                          4096 x 64          =  0.262 M   per-head indexer gate
  grouped output, 8 groups     8 x (4096 x 1024)  = 33.554 M
  final output projection      8192 x 4096        = 33.554 M
  attention sink logits        64                 =  0.000 M
                                                    ---------
                                                    121.901 M

HCA layer   one KV stream, no indexer, bias over m’ = 128         109.117 M
SWA layer   uncompressed KV, no compression weights, no indexer   106.954 M

Manifold-constrained hyper-connections add 393K per residual junction. Router gates add 45M across the model. Embeddings, using the DeepSeek-V3 tokenizer with a handful of added context-construction tokens, add 530M on each side.

The report says the vocabulary remains 128K, which I read as the 129,280 entries V3 used.

Where 284 billion parameters actually sit. Reconstructed from the published constants; the residual against the published figure is 0.07 percent.

284.202B against a published 284B, a 0.07 percent residual on a reconstruction with no free parameters.

The MTP head is real, it is 6.6B, vLLM will use it for speculative decoding, and it is not in the number on the model card. DeepSeek did the same with V3, whose Hugging Face repository showed 685B against a stated 671B.

There is a second confirmation of the reconstruction hiding in an unlikely place. In the determinism section, discussing why they cannot use split-k for one particular GEMM, the report mentions in passing that “mHC involves a matrix multiplication with an output dimension of only 24.”

With n_hc = 4, the dynamic parameterisation generates A in R^4, B in R^{4x4} and C in R^4 from one flattened input. Four plus sixteen plus four is twenty-four.

The three mappings are produced by a single GEMM, and a throwaway sentence in a section about floating-point associativity confirms the shape.

The activated count follows. Seven experts fire per token, six routed plus the shared one, which is 7.575B. Everything else in the forward pass is dense: all 4.956B of attention, all of mHC, all of the router gates.

That is 12.610B excluding embeddings and 13.140B counting the output head, against a published 13B.

Attention weights are a rounding error in the checkpoint and a plurality of the arithmetic at decode. Reconstructed from the constants in section 4.2.1.

Attention is 1.74 percent of the weight bank and 39.3 percent of what gets touched per token. That’s most load-bearing fact about serving this model and it appears nowhere in the report, because the report presents attention as the thing being optimised away rather than as a fixed cost that survives the optimisation.

The sparsity everyone is discussing lives entirely in the expert bank. The attention stack is as dense as it has ever been.

The consequence is a floor. Push MoE sparsity as far as you like, drop from six routed experts to four, go from 256 experts to 512 at the same activation count, and the activated parameter count will not fall below roughly 5.0B, because the attention stack is dense and it is 4.956B of it.

Every decode step of every request reads all of it. At 12.6B activated you are already 40 percent of the way to that floor. A hypothetical V4-Flash-Nano with two routed experts would be a 10.1B-activated model, not a 4B one.

It also reframes what the hybrid attention is for. It is not there to make attention cheap in absolute terms. It is there to stop attention’s state from growing without bound, which is a memory problem rather than a FLOPs problem, and the FLOPs bill arrives anyway.

We will come back later to how large it gets.

Subscribe now

Compressed Sparse Attention

CSA is five separate ideas stacked, and it is often described as one. Taken apart, each of them is simple, and two of them are the sort of thing you only invent after you have been beaten by the alternative.

Key and value are the same vector

V4 stores one cached entry per compressed position, not two. Here, the key and the value are literally the same tensor, used in both roles by a multi-query attention. That is an immediate halving of the cache before any compression happens at all.

It should not work. Attention output is normally translation invariant: rotate the query by R(t) and the key by R(s) and the score depends on R(t-s), which is relative. Values carry no rotation, so the output carries none.

Share the key and the value, and the value inherits the key’s rotation, the output acquires an absolute position through R(s), and shifting the whole sequence changes the answer.

The fix is in section 2.3.3 and it is one line: apply RoPE with position minus i to the last 64 dimensions of each head’s output. Because R is orthogonal and R(t)^-1R(s) = R(s-t), rotating the output backwards restores relative positioning, and the contribution of each cached entry ends up depending on its distance from the query again.

The vLLM team’s write-up derives the same result from the other direction and calls it inverse RoPE. Their implementation fuses it into the FP8 quantisation ahead of the output projection, worth two to three times over doing the two separately.

Two times the cache, bought with one elementwise kernel. This is the cheapest trick in the model and it is the one that has attracted the least attention.

The compressor has two streams and they overlap

CSA does not average four tokens into one. It computes two independent projections of the hidden state, C^a and C^b, each with its own learned compression weights Z^a and Z^b, then takes a softmax across the concatenation of 2m weights and forms a weighted sum over both streams.

Compressed entry i draws C^a from positions [mi, m(i+1)] and C^b from positions [m(i-1), mi].

With m = 4 that makes every compressed entry a data-dependent weighted sum of eight consecutive tokens taken at a stride of four. vLLM names this attention type c4a and documents it as a weighted sum of 8 uncompressed tokens with a stride of 4, which is exactly equations 11 and 12.

The overlap is the point. A hard boundary every four tokens cuts arbitrary spans in arbitrary places and the model is blind across the seam. Overlapping means every token appears in two compressed entries under different weights, while the sequence still shrinks by exactly four because the stride is four. The receptive field of an entry is eight; the compression ratio is four.

The compression weights are per dimension. The softmax runs over 2m elements independently for each of the 512 channels, so each channel picks its own mixture of the eight tokens.

This is a learned per-channel pooling rather than a summarisation step, and describing it as “remembering the paragraph’s key point” undersells it by a wide margin.

The Lightning Indexer

After compression a one-million-token context still holds 262,144 entries in every CSA layer. Attending densely to a quarter of a million entries is not obviously better than attending densely to a million, so CSA runs DeepSeek Sparse Attention over the compressed stream: a cheap scoring pass picks the top 512, and the real attention runs only over those.

The scorer builds 64 low-rank query heads of dimension 128 from the same compressed query latent the main attention uses, computes a rectified dot product against a separately compressed indexer key for each block, and sums across heads with a learned per-head gate:

c^Q_t   = h_t · W^DQ                                   shared with the main attention queries
q^I_t   = c^Q_t · W^IUQ                                64 indexer heads of dimension 128
w^I_t   = h_t · W^w                                    one gate per indexer head, from the hidden state
I(t,s)  = sum_h  w^I(t,h) · ReLU( q^I(t,h) · K^IComp_s )

Two choices there are worth stopping on. ReLU rather than softmax leaves the score unnormalised, so a head can contribute nothing to a block rather than merely down-weighting it, and heads can veto. And the gate w is produced from the hidden state by a 4096 by 64 matrix, so the model decides per token which of its 64 scoring heads to trust.

That is a router in everything but name, sitting in front of the attention, and it is trained end to end with it.

Flash’s top-k is 512. Pro’s is 1024. V3.2’s was 2048. The report is direct about the reason: a smaller top-k greatly improves efficiency on short and medium texts, which is where the traffic is.

Grouped output projection

Sixty-four heads at head dimension 512 produce 32,768 values per token. A conventional output projection would be 32,768 by 4096, which is 134M parameters per layer, more than everything else in the attention block put together.

V4 splits the heads into g = 8 groups of eight, projects each group’s 4096 values to 1024, concatenates the eight results into 8192, and projects that to 4096. Total 67.1M, half the naive cost, and the bottleneck at 8192 acts as a constraint on how freely heads can mix.

Pro uses g = 16 at the same d_g = 1024, so a 16,384-wide concatenation from 128 heads. The report also applies RMSNorm per head on the queries and on the single head of the compressed KV entries just before the core attention, which is the same numerical hygiene MLA needed, for the same reason, and which turns out to matter for the optimiser.

Attention sink, and heads that abstain

Both CSA and HCA carry a set of learnable sink logits, one per head. For head h, Exp(z’_h) is added to the denominator of the softmax and to nothing else:

s(h,i,j) = Exp(z(h,i,j)) / ( sum_k Exp(z(h,i,k)) + Exp(z’_h) )

The report’s description of what this buys is unusually blunt. It allows each query head to make its total attention score not equal to one, “and even to be near 0.” A head with a large sink logit contributes almost nothing to the output no matter what is in the context.

Given that this model asks 64 heads per layer to attend to a top-512 selection out of a quarter of a million compressed blocks, giving heads a principled way to decline is not decoration.

It is what stops a head from being forced to spend its mass on whichever blocks the indexer happened to hand it.

The sliding window is not an optimisation

Every compressed layer also keeps 128 uncompressed tokens in a window, concatenated with the selected compressed entries before the softmax. This is a correctness requirement and the reason is causality.

A compressed entry i in c128a summarises positions 128i through 128(i+1)-1. A query at position t may only use information derived from positions at or before t, so it cannot use entry i unless 128(i+1)-1 is at or before t.

A query sitting anywhere inside the current block therefore has no compressed entry it is permitted to read. Without the window, tokens 1 through 127 of every block would attend to no local context whatsoever.

The window covers the distance between the query and the most recent legal compression boundary, and 128 is exactly m’ for that reason.

Subscribe now

Heavily Compressed Attention

HCA is the same compressor at m’ = 128, with two differences that both simplify it. There is one key-value stream instead of two, so no overlap and no seam handling. And there is no indexer, so no top-k. It attends densely over everything it holds.

The arithmetic explains the second choice. A one-million-token context under 128x compression yields 8,192 entries, and eight thousand keys is an ordinary attention problem, shorter than most models’ native context. Sparsity would buy nothing; vLLM implements it as a sparse-attention call with top-k set to 8192, a selection that selects everything, purely so one kernel serves both paths.

Dropping the overlap is defensible because HCA is not responsible for local detail. The sliding window handles that and the interleaved CSA layers handle the middle range. HCA’s job is coarse global memory, and boundary precision on a 128-token block does not matter much when the question is what the document was about.

What the pair produces is a genuinely two-rate memory. In Flash’s stack, two sliding-window layers are followed by 41 alternating layers, giving 21 CSA and 20 HCA.

Every token’s representation passes through 21 layers that can retrieve 512 four-token spans from anywhere in the history and 20 layers that see all 8,192 coarse summaries at once. Neither works alone.

Sparse selection over fine blocks has recall problems on diffuse queries; dense attention over coarse blocks cannot resolve a specific line of code.

Subscribe now

The router, and the three ways V4 changed it

Every write-up of this model spends its length on the attention and treats the mixture of experts as inherited furniture. It is not. Section 2.1 makes four changes to routing, and one of them removes a whole class of layer that every DeepSeek model before this one had.

Forty-three layers, two independent schedules. Reconstructed from report sections 2.1 and 4.2.1; the CSA and HCA assignment within the interleave is inferred, and the split is either 21 to 20 or the reverse.

There are no dense FFN layers. V3 kept ordinary dense feed-forward blocks in its first three Transformer layers, on the standard argument that early layers do generic work and routing them wastes capacity.

V4 replaces those with MoE layers that use hash routing: the target experts for a token are fixed by a hash of its token ID, following Roller’s Hash Layers work from 2021. The Hugging Face implementation makes the mechanism concrete.

Routing type is set per layer through mlp_layer_types, and a hash layer resolves its experts through a frozen tid2eid lookup shipped inside the checkpoint.

The details that make this more than a curiosity is that only the selection is static. The learned gate still produces the per-expert scores that weight the chosen experts. So a hash layer is not an un-routed layer; it is a layer where the router has been told which experts to consider and gets to decide how much to trust each one.

That converts the hardest part of early-layer routing, an unstable argmax over 256 options while the model knows nothing, into a fixed assignment with a learnable mixture on top.

It also does something useful for the infrastructure. A hash of the token ID is known before the forward pass reaches the layer, which means dispatch for the first three layers can be planned as soon as the tokens are known rather than after the previous block finishes.

three layers where the overlap is free

• The affinity function changed. V3 scored expert affinity with a sigmoid. V4 uses the square root of a softplus. Both are positive and monotone, so the ranking behaviour is similar, but the tails are not. A sigmoid saturates at one, so once an expert is clearly the best its score stops responding and the gradient through it vanishes. Softplus does not saturate, and the square root damps its growth to sublinear without ever flattening. The practical effect is that a strongly preferred expert keeps receiving gradient signal instead of going quiet, which is exactly the failure mode you would expect to precede the routing-driven loss spikes described two sections later.

• The routing target cap is gone. V3 constrained how many nodes a token’s experts could be spread across, through the n_group and topk_group parameters, because unconstrained routing means a token’s six experts can live on six different machines and the all-to-all cost is set by the worst case. V4 drops the constraint entirely and says the parallelism strategy was redesigned to pay for it. Which is the same trade appearing again in a different costume. The wave-partitioned mega-kernel in section 3.1 exists so that communication hides under computation. Once it does, capping communication to protect throughput stops being necessary, and the model gets its routing freedom back. Every efficiency result in this report buys an architectural freedom somewhere else, and the report never quite says so.

• Load balancing stays auxiliary-loss-free, with one addition. V4 keeps V3’s scheme, where a per-expert bias is added to the score for the purpose of top-k selection and excluded from the gating weight, so balance is enforced without a gradient term competing with the language modelling objective. The Hugging Face implementation keeps it as an e_score_correction_bias buffer that shifts the argmax without carrying gradients. On top of that V4 adds a mild sequence-wise balance loss whose only job is to stop a single sequence collapsing onto a handful of experts, with a weight of 0.0001 and a bias update speed of 0.001. Global balance from a bias, local balance from a loss, and the loss is small enough to be a guardrail rather than an objective.

Subscribe now

Manifold-Constrained Hyper-Connections

Hyper-connections widen the residual stream from one vector to n_hc vectors and let the model learn the mixing, which decouples residual width from hidden size.

The update is X_l+1 = B_lX_l + C_lF_l(A_lX_l), with A projecting the widened stream down to the layer input, C projecting the layer output back up, and B mixing the residual lanes among themselves.

DeepSeek’s stated problem with plain hyper-connections is that stacking them is numerically unstable. B is applied at every junction, so across 86 junctions the model computes a product of 86 learned matrices. If the spectral norm of B exceeds 1 by any margin, the product diverges. If it falls below 1, the signal dies.

The fix is to constrain B to the Birkhoff polytope, the set of doubly stochastic matrices: nonnegative, every row and column summing to one. Two properties make this the right set rather than a convenient one.

A doubly stochastic matrix has spectral norm exactly 1, so the residual mapping is non-expansive by construction and neither the forward nor the backward pass can blow up through it. And the set is closed under multiplication, so a product of 86 of them is still doubly stochastic. The stability is structura-l, not empirical.

Projection onto the polytope uses Sinkhorn-Knopp: exponentiate the raw matrix for positivity, then alternate row and column normalisation, twenty times. A and C get a sigmoid, with C scaled by two so it can express amplification up to a factor of two while staying nonnegative, which rules out lanes cancelling one another.

One implementation detail worth recording because the paper skips it: the widened stream has to collapse before the output. A final hyper-head folds the four residual lanes back into a single sequence just ahead of the model norm, so the language modelling head sees an ordinary hidden state and nothing downstream needs to know the residual was ever four vectors wide.

At n_hc = 4 this costs 393K parameters per junction and 34M across the model, about a hundredth of one percent of the weights. The cost is elsewhere. The residual stream is four times wider in activation memory, twenty Sinkhorn iterations sit on the critical path of every junction, and pipeline communication between stages grows.

DeepSeek’s answer is obviously fused kernels, a recomputation strategy that checkpoints most inter-layer hidden states and all normalised layer inputs while leaving compute-intensive operations alone, and an adjustment to the DualPipe 1F1B overlap so parts of mHC run concurrently with the pipeline.

The number they report for the whole apparatus is 6.7 percent of the overlapped 1F1B stage.

SGLang found the other end of the same problem at serving time. In low-latency decode the batch is small, the pre-GEMM that feeds the Sinkhorn normalisation has almost no parallelism, and it becomes the bottleneck. Their answer was to split the K dimension of that GEMM across CTAs.

Which is where the determinism section’s remark about an output dimension of 24 comes from: the GEMM is small enough that split-k is compulsory and split-k is non-deterministic, so they emit each split separately and reduce in a following kernel.

Subscribe now

Muon, and the thing it did not break

V4 trains with Muon for most parameters and AdamW for the embedding, the prediction head, the static biases and gating factors of mHC, and all RMSNorm weights. Momentum 0.95, weight decay 0.1, update RMS rescaled to 0.18 so the AdamW learning rate schedule could be reused unchanged.

The orthogonalisation is a hybrid Newton-Schulz, ten iterations in two stages. The first eight use coefficients (3.4445, -4.7750, 2.0315), which converge fast and overshoot; the final two use (2, -1.5, 0.5), which are gentler and settle the singular values precisely at one.

Splitting the schedule this way is the practical answer to a known tension in Newton-Schulz: aggressive coefficients reach the neighbourhood quickly and oscillate there, conservative ones land cleanly but slowly.

The detail worth flagging is a negative result. Muon has a documented pathology where orthogonalised updates keep singular values near uniform, query and key norms drift up together, and pre-softmax logits reach values low precision cannot hold. Moonshot’s answer in the Kimi work was QK-Clip.

DeepSeek states plainly that they do not use it, because the attention architecture already applies RMSNorm to the queries and the KV entries, which prevents the logits from exploding in the first place.

So the RMSNorm in section 2.3.3, which reads like routine hygiene, is load-bearing for the optimiser choice. That is a co-design decision presented as a footnote.

Subscribe now

How the sparsity was actually taught

You can’t train a top-k selector from scratch. If the indexer is random at initialisation, the attention sees 512 random blocks, the gradient signal for choosing better blocks is buried, and the model learns to ignore the compressed path entirely.

The schedule in section 4.2.2 solves this in stages, and it is worth reading as a recipe rather than a list. Flash starts at sequence length 4K and extends through 16K and 64K to 1M. Attention is dense for the first 1T tokens. Sparse attention is introduced at the 64K stage, and before it is switched on there is a short stage that warms up the lightning indexer alone. Then sparse attention runs for the rest of training, which is most of 32T tokens.

The ordering is the interesting part. Dense first so the model learns what to attend to; then an indexer warmup so the selector learns to imitate the dense attention’s choices; then sparsity, at a sequence length long enough that selection matters and short enough that dense supervision was still affordable to produce.

Pro gets a longer dense stage than Flash, which is what you would expect if the dense phase is the expensive part and the larger model needs more of it.

Two other schedule details are worth having. Batch size ramps to 75.5M tokens and stays there. Learning rate warms over 2000 steps to 2.7e-4, holds, then decays to 2.7e-5 on a cosine near the end.

MTP loss weight is 0.3 for most of training and drops to 0.1 when the learning rate starts decaying, which reads as a decision to stop letting the speculative head pull on the backbone once the model is being finished.

The instability, and the two things that fixed it

The report is unusually candid here. Training was unstable, rollbacks did not prevent recurrence, and the spikes were consistently traced to outliers in the MoE layers, with the routing mechanism itself appearing to make the outliers worse.

Two techniques fixed it and DeepSeek says openly that they do not have a theory for why.

Subscribe now

What the cache costs, checked four ways

The report gives ratios against DeepSeek-V3.2 and vLLM gives absolute figures for Pro. Nobody publishes the absolute figure for Flash. It is derivable, and the derivation can be validated before it is used.

Build a byte model from the published dimensions. Section 2.3.3 fixes the rotary dimension at exactly 64, and section 2.3.4 specifies bf16 for the RoPE dimensions and FP8 for the rest, so a shared key-value entry of head dimension 512 costs 64 times 2 plus 448, which is 576 bytes.

The indexer cache runs in FP4 under quantization-aware training, so a 128-dimension indexer key costs 64 bytes.

Two against figures vLLM published independently, two against ratios stated in the report itself.

V3.2 caches an MLA latent of 512 dimensions plus 64 RoPE and a 128-dimension indexer key. In bf16 that is 1,408 bytes per token per layer, which over 61 layers at 1,048,576 tokens is 83.88 GiB. vLLM publishes 83.9. V4 at Pro’s 30 CSA and 31 HCA layers gives 9.62 GiB in bf16. vLLM publishes 9.62. Applying the same model at production precision, V3.2 comes to 49.12 GB and Flash to 3.62 GB, a ratio of 13.57 to one, against the 13.7x the report prints on Figure 1.

And section 3.6.2 says the uncompressed sliding-window state would be roughly eight times the volume of the compressed state; the byte model says 7.2.

One 1,048,576-token sequence. The model reproduces vLLM’s published V3.2 and V4-Pro figures before being applied to Flash.

So: 3.372 GiB per one-million-token sequence, or 3,453 bytes for each token of context across the whole 43-layer stack. Against the baseline the report chooses, bf16 grouped-query attention with 8 KV heads at head dimension 128, which is 176,128 bytes per token and 172 GiB for the same sequence, Flash comes in at 1.96 percent.

The report claims approximately 2 percent. Derived independently, it holds.

Three multipliers produce that and only one of them is the headline. Fifty-one times from sequence-axis compression and the interleave, two times from sharing key and value, and slightly under two times from mixed FP8 and FP4 storage.

Top-k selection, the mechanism everyone names when they describe this model, saves no memory at all. It saves bandwidth at read time. The memory win is compression, sharing and precision, in that order.

Subscribe now

The indexer is O(n), and that is the ceiling

Here is what the efficiency section does not say. Top-k selection bounds the cost of attending. It does not bound the cost of selecting. To pick the best 512 of 262,144 compressed entries, the indexer scores all of them. That scan is linear in context length and it never becomes sparse.

The matmul side is flat by construction. The attention side is linear in context because of the indexer scan and the HCA dense pass.

At 4K of context, attention is 9 percent of per-token arithmetic and the model behaves like a cheap 13B. At 32K it is 18 percent. At 128K it is 39 percent. Around 213K tokens the attention overtakes the entire 284B expert bank, and at 1M it is 82 percent of the work.

The model is linear in context, not sub-linear. Compression changed the constant by roughly fifty. It did not change the exponent.

At 32K of context, half the attention arithmetic is spent deciding what to attend to. At 1M, four fifths of it is.

Half the attention FLOPs at 32K are the indexer. Four fifths at 1M. The thing that makes attention sparse is the dominant cost of the attention.

Four fifths of the cost of attention in this model is the cost of deciding what to attend to.Derived from the published dimensions

DeepSeek clearly knew. The indexer runs in FP4 while the main attention runs in FP8, which is the more aggressive precision going to the larger consumer. Flash’s top-k came down to 512 while Pro’s is 1024, because reducing k shrinks the attention pass and does nothing at all to the scan, and Flash needs the attention pass small relative to its smaller matmul side.

And the compression rate m = 4 is not primarily a memory optimisation: the scan runs over n/m keys, so compressing by four is a four times discount on selection, with the cache saving arriving as a side effect.

If someone finds a sub-linear selector, this architecture gets a second life. There is already a paper trying, from a group that fine-tuned V4-Flash with a Neural Memory Indexer that predicts and prefetches only the query-critical KV chunks, reporting comparable benchmark scores at 13.5 percent of the GPU memory.

Its authors are explicit that the work was constrained by resources and cut short, with the indexer trained on frozen keys and no end-to-end optimisation against the backbone. As a result it is a direction rather than a result. It is the right direction.

Subscribe now

FP4, down to the index scores

The so-called Quantization-aware training is applied during post-training to two things: the MoE expert weights, and the query-key path of the CSA indexer, where activations are cached, loaded and multiplied entirely in FP4.

The expert-weight scheme has a property worth stating because it explains why the whole thing was affordable. Master weights are held in FP32, quantised to MXFP4, then dequantised back to FP8 for the actual computation, and the FP4 to FP8 dequantisation is lossless.

FP8 in E4M3 has two more exponent bits than FP4 in E2M1, so as long as the ratio between the largest and smallest scale factors of the FP4 sub-blocks, which are 1 by 32 tiles, inside a given FP8 quantisation block, which is 128 by 128, stays under a threshold, the finer scale information is absorbed entirely by the wider dynamic range.

DeepSeek verified their weights satisfy the condition. The consequence is that the entire QAT pipeline reuses the existing FP8 training framework without modification, with a straight-through estimator carrying gradients back to the FP32 masters, and no need to requantise transposed weights.

Then there is one number in that section that deserves its own paragraph. They also quantise the index scores themselves, the output of the lightning indexer, from FP32 to BF16.

That gives a two times speedup on the top-k selector while preserving a 99.7 percent recall rate of KV entries. Given that the selector is the dominant term in attention cost at long context, halving it for three tenths of a percent of recall is the highest-leverage line in the report.

During rollouts and any inference-only forward pass, including teachers and reference models, real FP4 weights are used rather than simulated quantisation, so sampling behaviour during RL is identical to deployment behaviour.

That is a correctness argument dressed as an efficiency one, and it matters: a policy trained against a simulated-quantisation rollout is optimising a model that will never be served.

Subscribe now

The mega-kernel, and the balance point DeepSeek wants hardware to hit

Expert parallelism needs an all-to-all dispatch and an all-to-all combine per MoE layer, and the conventional implementation runs communication and computation as separate serial kernels, which leaves both the interconnect and the SMs idle half the time.

DeepSeek’s answer fuses them into one pipelined kernel and then partitions the experts into waves. As soon as every expert in a wave has its tokens, that wave computes, while the next wave’s tokens are still in flight and the previous wave’s results are being sent back. In steady state all three proceed at once.

The reported gains are 1.50 to 1.73 times against strong non-fused baselines for general inference, and up to 1.96 times for latency-sensitive work like RL rollouts and high-speed agent serving, where batches are small and long-tailed and the pipeline has the most idle time to recover.

The report’s own figure puts the theoretical ceiling of the wave scheme at 1.92 times against 1.42 for Comet, which overlaps dispatch with the first linear and the second linear with combine but not at wave granularity, and it evaluates both in the V4-Flash configuration specifically. The implementation is open, as MegaMoE inside DeepGEMM, and it was validated on both NVIDIA GPUs and Huawei Ascend NPUs.

Then comes the paragraph I think is the most economically consequential in the entire report, and it is addressed to hardware vendors rather than to users.

Communication hides under computation when C/B is at most V_comp/V_comm, where C is peak compute and B is interconnect bandwidth. For a DeepSeekMoE layer each token-expert pair costs 6hd_ff FLOPs across the gate, up and down projections, and 3h bytes of traffic, being h bytes of FP8 dispatch and 2h bytes of BF16 combine.

The h cancels. The condition collapses to:

C / B  <=  2 * d_ff

The report evaluates this for Pro, whose expert intermediate dimension is 3072, and gets 6144 FLOPs per byte, then observes that once bandwidth clears that threshold it stops being the bottleneck and further silicon spent on it brings diminishing returns. Their recommendation to hardware designers is to target the balance point rather than scale bandwidth unconditionally.

The report contains the equation that says the cheaper model is the harder one to host. It does not evaluate it for the cheaper model.Report section 3.1

Flash’s expert intermediate dimension is 2048. Run the same derivation and Flash’s balance point is 4096 FLOPs per byte, which is two thirds of Pro’s.

Derived from the report’s own condition at 4.5 PFLOP/s of dense FP8 per GPU. Smaller experts do less arithmetic per byte moved.

The smaller model is the harder one to interconnect. Flash needs 1.5 times more bandwidth per unit of compute than Pro does, because its experts do less arithmetic for the same number of bytes dispatched and combined.

At a B200’s dense FP8 throughput, Flash wants about 1.10 TB/s per GPU to hide its all-to-all and Pro wants about 0.73. NVLink 5 covers both comfortably. A PCIe Gen5 box does not cover either, and misses Flash by a factor of seventeen.

Anyone sizing a deployment on the assumption that the cheaper model is the easier one to host has the relationship backwards, and the report contains the equation that says so.

Three other proposals in that section are worth recording because they are a roadmap. DeepSeek asks for more power headroom, on the grounds that extreme fusion drives compute, memory and network to high load simultaneously and power throttling becomes the limiter, which is a real and underdiscussed consequence of fusing everything.

They use pull-based communication, where each GPU reads from remote GPUs, because fine-grained push carries too much notification latency, and they ask for lower-latency cross-GPU signalling so push becomes viable.

And they propose replacing SwiGLU with a cheap elementwise activation with no exponential and no division, because that lightens post-GEMM work and, under a fixed parameter budget, removing the gate projection lets d_ff grow, which pushes the balance point up and relaxes the bandwidth requirement further.

That last one is a description of the next model.

Subscribe now

TileLang, and a theorem prover in the compiler

Section 3.2 is the part of this report that a compiler person should read twice. DeepSeek’s architecture, written naively, decomposes into hundreds of fine-grained Torch ATen operators, and they replaced most of them with fused kernels written in TileLang, a tile-level DSL, rather than in CUDA.

They did two things to TileLang along the way are more interesting than the choice itself.

A production LLM shipped with an SMT solver inside its kernel compiler. This is the direction I have argued the field goes: the bottleneck in kernel performance is not the language, it is how much the compiler can prove, and buying proving power off the shelf is cheaper than hand-writing the kernel.

The numerics policy in the same section is equally deliberate. Fast-math is disabled at the compiler level by default, precision-affecting approximations are opt-in frontend operators, and IEEE-compliant intrinsics with explicit rounding modes are available when strict semantics are required.

They also align TileLang’s algebraic simplification and lowering rules with NVCC so that kernels can be validated bit-for-bit against hand-written CUDA baselines, with layout annotations available to pin down lowering decisions and hold accumulation order constant. Accuracy by default, speed by opt-in, which is the opposite of the usual arrangement.

Share The Software Frontier

Bitwise batch invariance, and why they wanted it

Batch invariance means a given token’s output is bitwise identical regardless of where it sits in a batch. Almost nobody ships this, because it costs performance, and DeepSeek’s reasoning for paying is that they wanted bitwise alignment across pre-training, post-training and inference, which makes loss spikes diagnosable and post-training behaviour consistent with what gets served.

Getting it required giving up two standard optimisations and then engineering the loss back out.

Attention. Split-KV, which spreads one sequence’s attention across many SMs to balance load, is not batch invariant. Abandoning it causes wave quantisation, where the final partially filled wave of thread blocks leaves most of the GPU idle. DeepSeek’s answer is a dual-kernel decode: a first kernel computes an entire sequence’s attention inside a single SM, giving throughput on fully occupied waves, and a second kernel spreads one sequence across multiple SMs to shorten the trailing partial wave. The two are engineered to have the same accumulation order so their outputs are bit-identical, and the second uses distributed shared memory within thread-block clusters to exchange partial results across SMs at speed. The reported overhead of batch-invariant decoding after this is negligible.

Matrix multiplication. cuBLAS cannot be made batch invariant, so it is replaced end to end by DeepGEMM. Split-k, which is how you get performance at very small batch, is also not batch invariant, so it is dropped in most scenarios and the resulting loss is recovered by other means.

Determinism is a separate problem from batch invariance and it comes from accumulation order, usually via atomic addition in the backward pass.

Sparse attention backward normally uses atomicAdd to accumulate KV gradients, so they allocate a separate accumulation buffer per SM and do a global deterministic summation afterwards. MoE backward is non-deterministic because SMs from different ranks negotiate write positions into the same receiving buffer, so they pre-process token order within each rank and isolate buffers across ranks.

And the mHC GEMM with its output dimension of 24 is small enough that split-k is unavoidable, so each split is emitted separately and reduced deterministically in a following kernel.

There is a payoff for this that shows up two sections later, in the rollout service. Because generation can be preempted at any time on their cluster, they keep a token-granular write-ahead log per request and resume from it.

The report explains why they cannot simply regenerate interrupted requests from scratch, and the argument is a good one: shorter responses are more likely to survive an interruption, so regenerating the survivors biases the training distribution towards short outputs.

They note that a batch-invariant deterministic stack could fix this instead by regenerating with a consistent sampler seed, but that this still costs a full re-decode, so the log wins. Batch invariance is what makes that alternative even expressible.

Subscribe now

Two cache hierarchies, and lcm(4, 128)

The hybrid attention breaks the assumption PagedAttention is built on, which is that every layer’s KV state has the same shape and the same eviction policy.

In V4 the compression ratios differ per layer, the indexer carries its own embedding size, the sliding-window layers have their own hit and eviction rules, and there is a rolling residual of tokens not yet numerous enough to compress that has to live somewhere.

DeepSeek splits it in two. A classical paged KV cache holds the compressed CSA and HCA entries. A separate state cache holds the sliding-window entries and the uncompressed tail, on the argument that both are a function only of the current position, which makes them a state-space model rather than a growing history, so a fixed-size pool can be pre-allocated and assigned per sequence.

The block geometry falls out of the two compression rates. A block has to cover a whole number of compressed entries in every layer, so it must span a multiple of the least common multiple of m and m’.

For Flash that is lcm(4, 128) = 128 original tokens, giving 32 CSA entries and exactly one HCA entry per block. vLLM independently chose 256 native positions, which is two of DeepSeek’s minimum blocks, giving 64 c4a entries and 2 c128a entries.

vLLM’s implementation notes are the best serving document published on this model, and their three decisions are worth having next to DeepSeek’s.

One logical block size in native token positions for every compressed layer, so slot mapping, scheduler accounting and prefix-hit detection use one unit instead of branching on the compression ratio. The compressor’s rolling residual registered as sliding-window KV with sliding_window set to the compression stride, rather than as a side buffer, so prefix caching lands on block boundaries and disaggregated prefill ships it through the existing SWA transfer path instead of a second one.

And a page-size argument: page size is block_size times compress_ratio times entry_size, all three are controllable, and chosen carefully the five cache kinds collapse into three buckets, each backed by one pool, sized once at load, with no runtime repartitioning and no cross-kind fragmentation.

On the kernel side vLLM fuses three groups: compressor with RMSNorm, RoPE and cache insertion, all elementwise, for 1.4 to 3 times; inverse RoPE with FP8 quantisation ahead of the output projection, for 2 to 3 times; and a horizontal fusion of query normalisation, KV RoPE and sliding-window key insertion using static warp-ID dispatch, each warp working independently on a query head or a key head with no cross-warp communication, for 10 to 20 times over the naive version.

The indexer then runs on its own CUDA stream alongside KV compression and window insertion, worth 5 to 6 percent end to end at low batch.

SGLang went at the FP4 weights instead, pairing MXFP8 activations with MXFP4 expert weights through FlashInfer’s TRTLLM-Gen fused MoE backend, and splitting K across CTAs in the mHC pre-GEMM for exactly the small-batch parallelism problem described earlier.

Share

What a cache hit actually costs

DeepSeek stores all compressed CSA and HCA entries to disk.

When a request hits a stored prefix, those entries are read back rather than recomputed, up to the last complete compression block; the tail of an incomplete block still has to be recomputed because uncompressed entries are not stored.

The sliding-window state is the problem. It is uncompressed and it exists in every layer, so storing it for every token would be roughly eight times the volume of everything else.

The report gives precisely three strategies with different trade-offs, and the third is the one that makes the economics work.

1. Full SWA caching stores everything, so a hit reads the last n_win tokens of the prefix and recomputes nothing. Zero redundancy, but only a sliver of what was written is ever read, which is a write-heavy unbalanced access pattern that SSDs handle badly.

2. Periodic checkpointing saves the window state every p tokens, loads the nearest checkpoint on a hit and recomputes the tail, with p tuning the storage against compute trade.

3. Zero SWA caching stores none of it. Here is the argument, and it is the neatest piece of reasoning in the report. Each token’s sliding-window entry in a given layer depends only on the window entries of the previous layer, which span n_win tokens. So the dependency cone going back through L layers is exactly n_win times L tokens wide. Recompute that many tokens and the entire window state is restored.

A million-token prefix comes back for the price of five and a half thousand.Report section 3.6.2, zero SWA caching

For Flash, n_win times L is 128 times 43, which is 5,504 tokens. Restoring the full sliding-window state of a one-million-token prefix costs a 5,504-token recompute, about 140 TFLOP, against the 26.6 PFLOP a cold prefill of that prefix would cost. That is a factor of 191, and it is why a cache hit can be sold for a fiftieth of a cache miss.

The whole thing only works because the compressed state is small. Storing 172 GiB per session on disk is possible, but reading it back at request time competes with the prefill it was meant to replace. At 3.62 GB it does not.

Compressing the sequence axis is what turns disk from a bad idea into the cheapest tier in the hierarchy, and the $0.0028 line on the rate card is the direct commercial expression of that.

Share

Post-training, and the parts of it that are cost mechanisms

The pipeline broadly follows DeepSeek-V3.2 with one substitution the report calls critical: the mixed reinforcement learning stage is replaced entirely by on-policy distillation.

Domain specialists are trained separately, each through supervised fine-tuning then GRPO against domain-specific rewards, and then more than ten of them are merged into one student by having the student sample its own trajectories and minimise reverse KL against the relevant teacher.

They use full-vocabulary logit distillation rather than the usual token-level KL estimate, on the grounds that the cheap estimator has high gradient variance and destabilises training. Making that affordable took two tricks.

Teacher weights live in centralised distributed storage and are loaded on demand with ZeRO-like sharding.

And rather than materialising logits for a vocabulary above 100k across more than ten teachers, they cache only the last-layer teacher hidden states and reconstruct logits on the fly through the prediction head at training time, ordering training samples by teacher index so exactly one teacher head is resident on device at a time. The KL itself is a TileLang kernel.

Three things from the post-training section have direct consequences for what you pay.

The reasoning ladder is a context ladder

Three modes, trained as separate RL configurations with distinct length penalties and context windows, then unified. Non-think evaluates at 8K of context, Think High at 128K, Think Max at 384K. Non-think emits an empty reasoning block and goes straight to the summary.

Think Max additionally prepends a system instruction, printed verbatim in the report, which tells the model that shortcuts are not permitted and that it must document every intermediate step, considered alternative and rejected hypothesis.

That instruction is why Artificial Analysis measures this model generating 210M output tokens across its index against a 100M median. The verbosity is not a training accident.

It is an instruction, in the system prompt, that DeepSeek wrote and that you are billed for at $0.28 per million. Anyone running Flash at max effort and complaining about token burn is paying for a behaviour they asked for by name.

The tool schema is XML on purpose

V4 introduces a tool-call format built on a dedicated |DSML| token with XML-shaped invocations rather than JSON.

String parameters go through as-is with an explicit string=”true” flag; everything else is JSON-encoded with the flag false. The stated reason is that XML mitigates escaping failures and reduces tool-call errors.

This is a small decision with a large downstream effect. Every escaping failure in a JSON tool call is a wasted turn, and a wasted turn in an agent loop costs a full round of prefill plus decode. Reducing tool-call error rate is a cost reduction that never appears on a rate card.

Interleaved thinking, and a warning inside it

V3.2 kept reasoning traces across tool-result rounds but discarded them when a new user message arrived. V4 keeps everything, across user message boundaries, for tool-calling conversations, so a long-horizon agent maintains one cumulative chain of thought instead of reconstructing its state each turn.

General conversation keeps the old discarding behaviour, on the reasonable grounds that persistent traces buy little there and cost context.

The warning is in the same paragraph and it is easy to miss. Agent frameworks that simulate tool interactions through user messages, and the report names Terminus, will not trigger the tool-calling context path and therefore will not get the persistence.

DeepSeek’s own recommendation for those frameworks is to use non-think models. If you are benchmarking Flash inside a harness that fakes tools as user turns, you are measuring the wrong path.

Quick Instruction, which is an inference-economics feature wearing a post-training costume

In a chat product, a handful of auxiliary decisions run before the real response: whether to trigger a web search, what the query should be, how authoritative a source needs to be, what domain the request belongs to, whether a pasted URL should be fetched.

The standard answer is a separate small model, which means a second prefill of the same prompt because it cannot share the big model’s KV cache.

DeepSeek trained special tokens for each of those tasks and appends them to the input sequence directly. The auxiliary task runs on the already-computed KV cache. There is no second prefill, several of the tasks run in parallel, the user-perceived time to first token drops, and there is no small model to maintain.

The published tokens are |action|, |title|, |query|, |authority|, |domain|, |extracted_url| and |read_url|. Read the list and it is obvious this is DeepSeek’s own chat product spilling into the model card, which is exactly what makes it interesting.

It is a vertical integration of the router into the weights, and it deletes a whole class of serving infrastructure that everyone else runs.

The sandbox, briefly

Agentic RL needs somewhere to execute, and DeepSeek built a platform they call DSec: three Rust components on top of their 3FS distributed filesystem, running hundreds of thousands of concurrent sandbox instances per cluster.

• One Python SDK abstracts four execution substrates behind one API, switchable by a parameter.

• Function Call dispatches stateless invocations to a pre-warmed pool with no cold start.

• Container is Docker-compatible with EROFS on-demand image loading. MicroVM is Firecracker for security-sensitive high-density work.

• FullVM is QEMU for arbitrary guest operating systems.

• Base images sit on 3FS as read-only layers shared across instances, writes go to a local copy-on-write layer, and snapshots chain, which gets them millisecond-scale resumption.

Each sandbox keeps a globally ordered trajectory log of every command and result, which serves three purposes: fast-forwarding after a preemption by replaying cached results rather than re-executing non-idempotent commands, provenance for every state change, and deterministic replay of any historical session.

None of this is in the model. All of it is why the model has agent scores.

What a node holds

Now the economics, and they start with capacity rather than speed.

Four B200 at 180 GB usable is 720 GB of HBM. The weights ship natively mixed: FP4 for routed experts, FP8 for attention, norms and router. That is 139 GB plus 6 GB, call it 145 GB resident.

At 85 percent HBM utilisation, leaving room for activations, the four-times-wider mHC residual stream and the compressor states, the KV pool is roughly 467 GB.

The same node, the same money, the same power draw. The difference is which axis was compressed.

129 concurrent one-million-token sessions on one four-GPU node. A V3.2-style stack on the same hardware holds five. At 128K, the working length of most real agent traffic, it is 1,031.

This is the actual product. Not the million-token window as a marketing number, but the ability to keep a thousand long-lived agent sessions warm on one node without evicting anyone.

Eviction is what makes long-context serving expensive, because every eviction is a re-prefill and a re-prefill of 128K tokens costs more than the entire conversation that followed it. Compressing the sequence axis turns that from a scheduling problem into a non-problem.

Subscribe now

The roofline, and what $0.28 buys

Decode on a large MoE is bandwidth-bound. At any batch deep enough to hit every expert, the node streams the full expert bank once per step: 278.1B parameters at FP4 is 139 GB, plus the dense stack replicated across four data-parallel ranks, giving 164 GB of weight traffic per step against 32 TB/s of aggregate bandwidth. A 5.12 ms floor, or 195 steps per second.

Artificial Analysis measures 122.7 output tokens per second per user on DeepSeek’s own API, which is 8.15 ms per step. The model achieves 63 percent of peak HBM bandwidth. That is a good number for something with this much elementwise work between matmuls, and it is a direct vindication of the fusion work in both vLLM and DeepSeek’s own kernels.

Holding that efficiency and adding KV read traffic gives node throughput, and node throughput times $0.28 per million gives revenue.

Break-even GPU rate from output revenue alone. The gold band is the rentable B200 market, roughly $3.35 reserved to $6.35 median as of August 2026.

At 32K of context you need a sustained decode batch of about 256 before $0.28 per million covers a mid-market B200 at $5.50 per GPU-hour. At 512 you clear 57 percent gross margin, at 1024 you clear 77.

At 1M of context it never clears: the KV pool caps the batch at 129 sequences and the break-even rate there is $2.92 per GPU-hour, below the cheapest reserved B200 anyone publishes.

That is the analysis I would’ve published if I had stopped there, and it is wrong in two ways that point in opposite directions. It prices only output tokens, which understates the revenue. And it prices GPUs at rental rates, which overstates the cost for the only company that matters here.

Nobody buys only output tokens

Real traffic has a shape. A coding agent sends tens of thousands of input tokens for every thousand it gets back, most of the input is a prefix it sent before, and DeepSeek charges for all three streams at three different prices.

A node has one time budget and has to split it between prefilling input and decoding output, so the sustainable output rate falls as the input ratio rises while total revenue climbs.

32K context, batch 1024, node time split between prefill and decode. Cache hits consume neither prefill nor decode, only a disk read.

Output-only pricing understates this card by roughly a factor of two. At a 20-to-1 input ratio with no caching, break-even is $55 per GPU-hour rather than $28.

At 200-to-1 with a 90 percent hit rate, which is what a long-running agent on a stable repository actually looks like, it is $64. Every one of those numbers is an order of magnitude above the rental market.

Cache hits are the mechanism. They consume no prefill and no decode, only a disk read of compressed entries plus a 5,504-token recompute, so raising the hit rate raises sustainable throughput without raising cost.

The $0.0028 price is low because the marginal cost is low, and it is worth having because it makes the traffic denser.

DeepSeek is not renting these GPUs

A rental rate contains a lessor’s margin, a scarcity premium that has been visibly volatile all year, and the lessor’s own financing cost. DeepSeek owns its fleet, and an owner’s cost is amortisation plus power.

Take a B200 at roughly $38,000 of capex, three years of life, 80 percent utilisation: $1.81 per GPU-hour. Add a kilowatt at PUE 1.3 and eight cents a kilowatt-hour: ten cents.

Call it $1.91 per GPU-hour all in, against a rental band of $3.35 to $6.35. An owner’s floor sits somewhere between 1.8 and 3.3 times below the price of renting the same silicon.

Note what power is and is not in that sum. A four-GPU node draws about 5.2 kW including overhead, which costs 42 cents an hour, which is under two percent of what the same node costs to rent. Power is not a meaningful line in the financial model.

It is a meaningful line in the physical one, and the report says so: the same paragraph that asks hardware vendors for a balance point also asks for more power headroom, because extreme kernel fusion drives compute, memory and network to high load simultaneously and power throttling becomes the limiter. Fusing everything is how you become thermally bound rather than bandwidth bound.

Which changes the verdict on the million-token window

The break-even line is what $0.28 per million pays at 1M context with the batch capped at 129 by the KV pool.

Serving a million-token context at $0.28 per million output tokens needs a cost basis under $2.92 per GPU-hour. Nobody renting Blackwell has one. DeepSeek does, with room.

The million-token window is not a loss leader. It is a moat, and a precisely shaped one. Owner economics against rental economics

So the million-token window is not a loss leader. It is a moat, and a precisely shaped one. The twenty providers currently reselling the preview weights on OpenRouter at 37 percent below DeepSeek’s own card can do that at short context, where deep batches make the arithmetic work on rented hardware.

They structurally cannot do it at a million tokens, at that price, on rented Blackwell, because the KV pool caps the batch and the capped batch does not clear the rent. Which is presumably why none of them advertise it.

An open-weights MIT model whose most expensive capability is only economic for an owner-operator is a strange and rather elegant object. DeepSeek gave away the architecture, the mega-kernel and the kernel library, and kept the one thing that does not fit in a repository: a fleet, a request volume large enough to make the on-disk cache pay, and a cost basis a third of what anyone else can rent.

What the whole card is betting on

Put the three prices next to the three cost structures and the strategy is legible. Prefill is compute-bound and enormously profitable: at 35 percent MFU a node prefills roughly 485,000 tokens per second, which at $0.14 per million is $244 an hour against a node that costs $22 to rent and $8 to own. Decode is bandwidth-bound and needs deep batches.

Cache hits cost almost nothing and are priced almost at nothing, which makes them a throughput multiplier rather than a revenue line.

The card rewards exactly one traffic shape: high input-to-output ratio, high prefix reuse, deep concurrency, moderate context.

That is coding-agent and tool-calling traffic, which is what the 0731 post-training pass targeted, what the native Responses API and Codex adaptation are for, and what Quick Instruction and interleaved thinking were built to make cheaper. The architecture, the post-training, the serving stack and the price list are one artefact pointed at one workload.

Point a different workload at it, long single-turn document analysis with no prefix reuse and shallow concurrency, and the margin thins toward nothing even for DeepSeek. The card does not distinguish, yet. I do not expect that to last.

Subscribe now

Reading the benchmarks in full

The report’s summary of its own results is accurate and selective, and the difference is worth spending a section on.

Start with the base models, which are the cleanest comparison in the document because all three ran in one internal harness with identical settings.

V4-Flash-Base carries 13B activated against V3.2-Base’s 37B, and 284B total against 671B. It should lose. It mostly does not.

Report Table 1. Same harness, same settings. Flash carries 35 percent of V3.2’s activated parameters and 42 percent of its total.

Plus 6.8 on FACTS Parametric, plus 6.7 on HumanEval, plus 4.5 on LongBench-V2, plus 3.5 on MultiLoKo, plus 2.8 on MMLU-Pro.

Getting more world knowledge out of fewer total parameters is the surprising one, because knowledge retention is supposed to scale with parameter count and Flash has 42 percent of them.

And then two regressions that nobody discussing this model has mentioned. BigCodeBench falls 7.1 points, from 63.9 to 56.8. MATH falls 3.1, from 60.5 to 57.4. Those are not noise, they are the two largest deltas in the table after FACTS, and one of them is a coding benchmark on a model being marketed for coding agents.

Post-training clearly recovers a great deal of it, since the instructed model’s code-agent scores are strong. But the base model is worse at BigCodeBench than its predecessor and the report does not discuss why.

My guess, and it is a guess: BigCodeBench is a three-shot benchmark on library-heavy code, which is a knowledge task about API surfaces rather than a reasoning task, and it is the sort of long-tail recall that a smaller expert bank should hurt.

The world-knowledge gains going the other way argue against that, which is why it stays a guess.

Then the long-context claim. The report’s own summary says V4-Pro-Max delivers strong results with a one-million-token window, “surpassing even Gemini-3.1-Pro on academic benchmarks.” That is totally true.

Report Table 6, DeepSeek’s own numbers, standardised configuration across models.

It is also the smaller half of the story. On LongMRCR at 1M, Claude Opus 4.6 scores 92.9, V4-Pro-Max 83.5, Gemini-3.1-Pro 76.3. On CorpusQA at 1M, Opus 71.7, V4-Pro-Max 62.0, Gemini 53.8. DeepSeek beats Gemini on both, comfortably, and loses to Opus by 9.4 and 9.7 points, in their own table, measured with their own harness.

Choosing Gemini as the comparison in the summary is a defensible framing decision and it is a framing decision.

The report is honest in other places where it did not have to be. It reports Terminal-Bench 2.0 at 67.9 on the original dataset while noting environment issues raised by another lab and disclosing that on the Verified subset V4-Pro scores about 72.0, which is higher.

It leaves cells blank for K2.6 and GLM-5.1 rather than filling them, saying those APIs were too busy to answer. It states that its reasoning performance trails the frontier by roughly three to six months.

That last sentence is in the report’s own summary of its own results and it is a more useful number than most third-party analysis of the same question.

Share

What this does to everyone else’s floor

List rates, August 2026. The gap to the cheapest Western flagship is more than two orders of magnitude.

Reuters, reporting Artificial Analysis figures, put V4-Flash at roughly 3 cents to complete the Intelligence Index battery, against 86 cents for Kimi K3, $1.86 for GPT-5.6 Sol and $3.15 for Claude Fable 5.

On the index itself Flash scores 50, tying Gemini 3.6 Flash, one point behind GLM-5.2 and Muse Spark 1.1, seven behind Kimi K3, and nine or more behind Opus 5, Fable 5 and GPT-5.6.

Artificial Analysis, August 2026. The vertical axis is nine points wide. The horizontal axis is two orders of magnitude.

The standard rebuttal to a cheap model is that it burns more tokens, so cost per task closes the gap that cost per token opens. It does not work here and the numbers say so precisely.

Flash generated 210M output tokens on the index against a 100M median, so it is 2.1 times more verbose than the typical model on identical work, for the documented reason that its Think Max system prompt instructs it to be. It is still 105 times cheaper per task than Fable 5, whose output token costs 179 times more.

Doubling token count against a 179x price advantage leaves 89x. The measured 105x is in that neighbourhood, and the verbosity tax is nowhere near large enough to matter.

What follows for the market is narrower than the headline suggests.

The nine-point index gap is not a rounding error. It is the difference between a model that finishes a hard task and one that plausibly fails it, and on hard agentic work a failed run costs more than the token price of a successful one.

At the top of the market price is not the binding constraint, and a 105x discount on a wrong answer is not a discount.

A 105x discount on a wrong answer is not a discount.On where the price pressure actually lands

The pressure is on the middle. Every workload running on a flagship because nobody bothered to route it, every classification and extraction and summarisation and first-draft call, is now paying somewhere between 60 and 105 times more than it needs to.

Routing is the mechanism that transfers that value and routing is engineering work most teams have not done. The models that should be nervous are not Opus 5 and Fable 5.

They are Haiku, Sonnet, the Flash and Terra and Luna tiers, everything between $1 and $5 per million output tokens where the capability gap to V4-Flash is small or negative and the price gap is 20x or more.

Two second-order effects are worth flagging. OpenRouter currently lists twenty providers serving the preview weights at $0.088 in and $0.176 out, 37 percent below DeepSeek’s own card.

That is an MIT-licensed model being resold below the price its author charges. Given the break-even analysis above, those hosts are running deeper batches, or cheaper capacity, or buying share. Only the first is durable.

And the weights are MIT, the mega-kernel is open inside DeepGEMM, the kernel library is open, and the fine-grained expert-parallel scheme was validated on Huawei Ascend as well as NVIDIA. The architecture is not the moat. The traffic shape is, and so is the on-disk cache that only pays off at DeepSeek’s request volume.

Subscribe now

Where we might be wrong

The CSA and HCA layer split for Flash is inferred. The report says the first two layers are pure sliding window and the rest interleave, which for 41 remaining layers gives either 21 CSA and 20 HCA or the reverse.

We assumed the interleave opens with CSA. If it opens with HCA the cache figure moves from 3.372 to 3.216 GiB, about 5 percent, the parameter reconstruction moves by 13M, and nothing in the argument changes.

The B200 configuration is nameplate: 180 GB usable per GPU, 8 TB/s, four GPUs, 85 percent HBM utilisation, and a 63 percent achieved bandwidth fraction calibrated against a single third-party throughput measurement.

A production deployment with prefill and decode disaggregated across separate node pools has different and probably better economics than the single-pool model we used, and DeepSeek describes that disaggregation themselves.

The owner-cost figure of $1.91 per GPU-hour is built from a $38,000 capex assumption, a three-year life and 80 percent utilisation, none of which DeepSeek publishes. Capex at $30,000 gives $1.53 and at $45,000 gives $2.24, so the conclusion that an owner clears the 1M break-even survives the range, but only just at the top of it. The utilisation assumption is the fragile one: at 50 percent utilisation the figure is $2.99 and the verdict flips.

The blended-revenue model splits node time between prefill and decode on a single pool. Real serving disaggregates them across separate node pools with different shapes, which changes the split and generally improves it.

It also assumes cache hits consume no node time beyond the disk read, which understates their cost at high hit rates where storage bandwidth starts to bind.

The FLOPs reconstruction lands at 12.2 percent of V3.2 at 1M against the report’s 10 percent. The report measures in equivalent FP8 FLOPs while V4’s experts run FP4, which has identical peak throughput on current silicon but which the report says could be a third cheaper on future hardware.

That accounting difference is the likely source. My V3.2 indexer model may also be too generous.

The prefill MFU of 35 percent is an assumption, not a measurement, and both prefill revenue and the blended break-even scale linearly with it. At 20 percent the prefill figure drops from $244 an hour to $140 and every blended break-even in the traffic-mix chart falls by roughly a third.

Our reading of BigCodeBench’s regression as an API-recall effect is speculation and I have flagged it as such in the text. I would drop it entirely if the world-knowledge results did not cut the other way.

And the largest one. Every agent benchmark in the 0731 release is vendor-reported, measured with a harness DeepSeek has announced but not shipped, at maximum reasoning effort, with sampling parameters DeepSeek chose. Two of the nine suites are DeepSeek’s own internal sets. I have reproduced none of them and neither has anyone else.

Seven predictions, dated

1. By June 2027, at least one major Western lab ships a production model that compresses the KV cache along the sequence axis rather than the head axis. The mechanism is cheap, the memory win is fifty times, and it has now been demonstrated at 32T tokens of pre-training rather than in an ablation.

2. By 31 December 2026, someone publishes a sub-linear top-k selector for compressed attention, most likely hierarchical or learned-hash, and demonstrates it on V4’s open weights. The indexer being four fifths of attention cost at long context is too visible a target, and the FlashMemory work has already aimed at it from the prefetch side.

3. By March 2027, DeepSeek introduces context-length-tiered pricing, a peak-hour surcharge that actually activates, or both. A flat rate across a thirty times span of serving cost is not stable, and a surcharge has already been announced without being switched on.

4. By September 2027, at least one of Anthropic, OpenAI or Google cuts a mid-tier model’s output price by more than 50 percent without a corresponding capability release. The squeeze is on the middle of the ladder, not the top.

5. By the end of December 2027, DeepSeek-V5 or its equivalent replaces SwiGLU with a gate-free elementwise activation. The report asks for this explicitly in its hardware proposals, gives the reason (removing the gate projection lets the intermediate dimension grow under a fixed budget, which raises the interconnect balance point), and labs that publish that kind of request are usually describing work already underway.

6. By late June 2027, an SMT or ILP solver appears in the compilation pipeline of at least one other major inference stack. Z3 inside TileLang’s algebraic system is the first production instance I know of, the payoff is a few seconds of compile time for stronger vectorisation and bound analysis, and the idea travels.

7. By 31 December 2027, no model with a one-million-token context window is profitable at that context length on rented NVIDIA hardware at published rates, while remaining profitable for owner-operators. The KV pool caps concurrency, concurrency is what pays for decode, and the gap between owning and renting is wider than the margin. This is the prediction I expect to age worst, and it fails if HBM capacity per package jumps faster than I think.

Confidence dossier

Tier A · Stated in a primary source · 13 claims

• 43 layers, d=4096, m=4, m’=128, top-k 512, 256+1 experts, 6 activated
  Report section 4.2.1, stated directly

• Partial RoPE on exactly the last 64 dimensions; inverse RoPE at position -i on the output
  Report section 2.3.3, stated directly

• HCA has one KV stream and no indexer; CSA has two overlapping streams
  Report equations 9 to 12 versus 20 to 23

• Attention sink logits per head, added to the softmax denominator
  Report equation 27

• Hybrid Newton-Schulz, 8 steps at (3.4445, -4.7750, 2.0315) then 2 at (2, -1.5, 0.5)
  Report section 2.4

• No QK-Clip, because RMSNorm on queries and KV entries already bounds the logits
  Report section 2.4, stated as a deliberate omission

• Dense attention for the first 1T tokens; sparsity introduced at 64K with an indexer warmup
  Report section 4.2.2

• Index scores quantised FP32 to BF16: 2x on top-k, 99.7 percent KV recall preserved
  Report section 3.4

• Interconnect condition C/B <= 2 d_ff; 6144 FLOPs per byte for Pro
  Report section 3.1, derived and evaluated there

• Three on-disk SWA strategies; zero-caching needs n_win x L tokens of recompute
  Report section 3.6.2

• Quick Instruction tokens reuse the existing KV cache to avoid a second prefill
  Report section 5.1.1, Table 5

• 0731 is the same structure and size as the preview; post-training only
  DeepSeek API changelog, 31 July 2026

• Rate card $0.14 / $0.0028 / $0.28 per million
  Artificial Analysis and DeepSeek changelog, mutually consistent

Tier B · Derived here and cross-checked · 8 claims

• Backbone reconstructs to 284.20B; MTP module excluded from the headline
  Derived here from published constants, 0.07 percent residual

• Attention is 1.74 percent of weights and 39.3 percent of activated
  Derived here; follows from the reconstruction

• mHC GEMM output dimension 24 = n_hc + n_hc squared + n_hc
  Derived here; matches the figure quoted in report section 3.3

• 3.372 GiB of KV per 1M sequence at production precision
  Derived here; four independent cross-checks, worst 10 percent

• Indexer is 79 percent of attention FLOPs at 1M, 50 percent at 32K
  Derived here from published dimensions

• Attention overtakes the expert bank at roughly 213K tokens
  Derived here; sensitive to the CSA/HCA split assumption

• Flash’s balance point is 4096 FLOPs per byte, 1.5x more demanding than Pro
  Derived here by applying the report’s own condition to Flash’s d_ff

• 5,504-token recompute restores the full SWA state, a 191x saving on prefill
  Derived here from n_win, L and the activated parameter count

Tier C · Model output, assumptions named · 7 claims

• 129 concurrent 1M sessions on one 4xB200 node
  Model; depends on 180 GB usable and 85 percent utilisation

• 63 percent of peak HBM bandwidth achieved at low batch
  Inferred from one third-party throughput measurement

• 1M context clears at owner cost and not at any rental rate
  Model output; hinges on the $38k / 3yr / 80 percent capex assumption

• Blended break-even is 32 to 64 dollars per GPU-hour on agentic traffic mixes
  Model output; single-pool prefill and decode, 35 percent prefill MFU

• Owner cost near $1.91 per GPU-hour against a $3.35 to $6.35 rental band
  Model output; capex and utilisation assumed, power from published TDP

• Break-even needs batch 256 at 32K against a $5.50 GPU-hour
  Model output; single-pool assumption, no disaggregation

• Prefill grosses $244 an hour against a $22 node
  Model output; 35 percent MFU is assumed, not measured

Tier D · Unreproduced or speculative · 3 claims

• BigCodeBench regression is a long-tail API recall effect
  Speculation, contradicted by the world-knowledge results

• Agent benchmark figures for the 0731 release
  Vendor reported, unshipped harness, two internal suites

• OpenRouter hosts undercutting DeepSeek by 37 percent are unprofitable
  Speculative; their batch depth and capacity costs are unknown

Share

Reproducing this

Everything numerical above comes from three short scripts with no dependencies beyond the standard library. None of it needs a GPU, because none of it is a measurement of a running model.

It is arithmetic over published constants, validated against figures published independently by vLLM and against ratios stated in the report itself.

The parameter reconstruction. Note the asymmetry between CSA and HCA in the projection count, which is the thing that is easy to get wrong:

L, d = 43, 4096
n_h, c, d_c = 64, 512, 1024          # query heads, head dim, query compression dim
n_hI, c_I   = 64, 128                # indexer heads, indexer head dim
g, d_g      = 8, 1024                # output projection groups
m, mp       = 4, 128                 # CSA and HCA compression rates
n_exp, n_sh, n_act, d_ff = 256, 1, 6, 2048
n_hc, V     = 4, 129280
n_swa, n_csa, n_hca = 2, 21, 20      # 2 SWA + 41 interleaved

per_expert = 3 * d * d_ff                        # SwiGLU: gate, up, down
moe_tot, moe_act = (n_exp+n_sh)*per_expert, (n_act+n_sh)*per_expert

def attn(kind):
    if   kind == ‘csa’: p = 4*d*c + 2*m*c        # two KV streams + positional biases
    elif kind == ‘hca’: p = 2*d*c + mp*c         # one KV stream  + positional bias
    else:               p = 1*d*c                # pure SWA, uncompressed
    p += d*d_c + d_c*(c*n_h)                     # query down then up
    if kind == ‘csa’:
        p += d_c*(c_I*n_hI) + d*n_hI             # indexer queries and per-head gate
    p += g*((n_h//g)*c)*d_g + (g*d_g)*d          # grouped output projection
    return p + n_h                               # attention sink logits

mhc = 2*(n_hc*d)*n_hc + (n_hc*d)*(n_hc**2)       # W_pre, W_post, W_res
att = n_swa*attn(’swa’) + n_csa*attn(’csa’) + n_hca*attn(’hca’)
backbone = L*moe_tot + att + 2*L*mhc + L*d*n_exp + 2*V*d
active   = L*moe_act + att + 2*L*mhc + L*d*n_exp

print(backbone/1e9, active/1e9)                  # 284.202  12.610
print(n_hc + n_hc**2 + n_hc)                     # 24, matching the mHC GEMM in section 3.3

The KV byte model, with all four calibration checks it has to pass before being used:

GiB, N = 1024**3, 1_048_576
c, c_I = 512, 128

def layer(kind, ent, idx, m=4, mp=128, n_win=128):
    if kind == ‘c4a’:   return (N//m)  * (ent + idx)
    if kind == ‘c128a’: return (N//mp) * ent
    if kind == ‘swa’:   return n_win * ent

# check 1: V3.2 bf16, MLA 512 latent + 64 rope, indexer 128, 61 layers
assert abs(61*(576*2 + 128*2)*N/GiB - 83.9) < 0.1              # vLLM publishes 83.9

# check 2: V4-Pro bf16, 30 c4a + 31 c128a, key and value shared
pro = 30*layer(’c4a’, c*2, c_I*2) + 31*layer(’c128a’, c*2, 0)
assert abs(pro/GiB - 9.62) < 0.01                              # vLLM publishes 9.62

# production precision: 64 rope dims bf16 + 448 fp8 = 576 B; fp4 indexer = 64 B
ent, idx = 64*2 + (c-64)*1, c_I//2
flash = 21*layer(’c4a’, ent, idx) + 20*layer(’c128a’, ent, 0) + 43*layer(’swa’, ent, 0)
v32   = 61*(64*2 + 512 + 128)*N

# check 3: the report’s own Figure 1 ratio
print(v32/flash)                       # 13.57  vs the report’s “13.7x smaller”

# check 4: the report’s claim that uncompressed SWA state is ~8x the compressed state
print(43*N*ent / (flash - 43*layer(’swa’, ent, 0)))            # 7.2  vs “approximately 8”

print(flash/GiB, flash/N)              # 3.372 GiB, 3453 bytes per context token
print(100*flash / (43*(2*8*128*2)*N))  # 1.96 percent of bf16 GQA-8, report says ~2

The decode roofline and break-even, for substituting your own hardware and rates:

NG, BW = 4, 8.0e12                                   # four B200, 8 TB/s each
w_step  = 278.1e9*0.5 + NG*6.2e9                     # FP4 experts + FP8 dense per rank
t_floor = w_step / (NG*BW)                           # 5.12 ms
eff     = t_floor / (1/122.7)                        # 0.63, from the measured 122.7 tok/s

def kv_bytes(ctx, ent=576, idx=64):
    return 21*((512+128)*ent + (ctx//4)*idx) + 20*(max(1, ctx//128)*ent + 128*ent)

def breakeven(batch, ctx, price=0.28):
    t = ((w_step + batch*kv_bytes(ctx)) / (NG*BW)) / eff
    return (batch/t) * 3600/1e6 * price / NG         # dollars per GPU-hour

print(breakeven(512, 32768))      # 14.76  clears the market comfortably
print(breakeven(256, 32768))      #  7.64  clears a mid-market B200
print(breakeven(129, 1048576))    #  2.93  clears nothing you can rent

# the report’s interconnect condition, applied to Flash
for name, d_ff in ((”Flash”, 2048), (”Pro”, 3072)):
    print(name, 2*d_ff, “FLOP/Byte ->”, 4.5e15/(2*d_ff)/1e12, “TB/s at 4.5 PFLOP/s FP8”)

Subscribe now

Sources

• DeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv:2606.19348, 26 April 2026. Sections 2.2 to 2.4 for architecture, 3.1 for expert parallelism and the interconnect balance point, 3.2 for TileLang, 3.3 for batch invariance and determinism, 3.4 for FP4 QAT, 3.5 for the training framework, 3.6 for KV cache management and on-disk storage, 4.2 for the constants and the training schedule, 5.1 for post-training, 5.2 for RL infrastructure and DSec, 5.3 for evaluation.

• vLLM Team. DeepSeek V4 in vLLM: Efficient Long-context Attention, 24 April 2026. Appendix contains the KV arithmetic used here for calibration, the derivation of why inverse RoPE is needed when key and value are shared, and the exact top-k values for c4a and c128a.

• LMSYS. DeepSeek-V4 on Day 0: From Fast Inference to Verified RL with SGLang and Miles, 25 April 2026, for the MXFP8 by MXFP4 fused MoE path and the split-K mHC pre-GEMM kernel.

• DeepSeek API changelog, 31 July 2026, for the 0731 release scope, the agent benchmark table and the harness settings.

• Artificial Analysis, model page for DeepSeek V4 Flash 0731, accessed 5 August 2026, for Intelligence Index v4.1, output speed, time to first token, token volume and the rate card.

• Reuters, via Quartz and Business Standard, 3 August 2026, for the cost-per-task comparison across V4-Flash, Kimi K3, GPT-5.6 Sol and Claude Fable 5.

• Hugging Face model cards for deepseek-ai/DeepSeek-V4-Flash and deepseek-ai/DeepSeek-V4-Pro, for the mixed FP4 and FP8 weight format and the reference inference implementation.

• DeepGEMM pull request 304, for the open-sourced MegaMoE fused expert-parallel kernel.

• FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention, arXiv:2606.09079, for the Neural Memory Indexer follow-up and its own account of its limitations.

• getdeploying.com B200 index (4 August 2026), gpuprice.fyi B200 index (31 July 2026) and published neocloud rate cards, for the $3.35 to $6.35 band.

• Anthropic, OpenAI, Moonshot and Z.ai published rate cards as of 4 August 2026, for the price ladder.

Thanks so much for reading The Software Frontier ! Your huge support keeps us motivated to do this work.

If you do really enjoy The Software Frontier, we would be really happy if you invited friends to subscribe and read with us. If you refer friends, you will receive a few benefits that give you special access to The Software Frontier.

Subscribe now

How to participate

1. Share The Software Frontier. When you use the referral link below, or the “Share” button on any post, you'll get credit for any new subscribers. Simply send the link in a text, email, or share it on social media with friends.

Refer a friend

2. Earn benefits. When more friends use your referral link to subscribe (free or paid), you’ll receive special benefits.

• Get a 1 month comp for 3 referrals

• Get a 3 month comp for 5 referrals

• Get a 6 month comp for 25 referrals

Visit the leaderboard

To learn more, check out Substack’s FAQ.

Thank you for helping get the word out about The Software Frontier! Without you, none of this could have been made possible.

Lorenzo Bradanini and Lorenzo Tettamanti.

For nine years the register file per SM didn’t grow of a byte, while tensor throughput per clock doubled almost four times. Blackwell’s answer was to take out the accumulator out of the register file, and to put it in an address space with its own allocator, its own barrier, and no coherence with anything. This is what that memory is, what the compiler actually emits for it, and the invariance that explains why it had to exist.

CUDA Mastery 2026

This article is a narrow part of one architecture. The actual guide is the wide one: 34 chapters and thousands words on CUDA 13.x, from memory model up through Hopper and Blackwell tensor cores, written the same way as everything here. Compile it, disassemble it, show the command, then explain what happened.

It also has a corrections page at the end. An early edition listed H100 sparse tensor rates as dense ones, and that single mislabel propagated into three roofline figures before a reader caught it.

That figures are fixed and the mistake is documented rather than quietly removed, which is the standard we would want from anyone selling us a technical book.

Get CUDA Mastery guide

Share The Software Frontier

Introduction

The very first thing I did with Blackwell was to compile something for it on a machine that has no GPU in it at all. That is not a counter sense.

ptxas, nvdisasm and cuobjdump are ordinary x86 programs, they ship as Python wheels on PyPI, and they will lower PTX for compute capability 10.0 on a laptop with a 40 mb download and no driver. You can’t run the result, but what you can do is to read every instruction, which for my own research, turned out to be the more useful half.

I wanted to know what a tcgen05.alloc costs, first was a small quest, but it got way bigger. It’s important because that’s the instruction that reserves Tensor Memory on Blackwell, it’s in every CUTLASS kernel for the architecture, and every explanation out there said the same three things: it takes a column count, it must be issued by one warp, and the result comes back through shared memory.

Nobody explained in full what the machine does. So, I wrote a small PTX kernel around it, assembled it for sm_100a, and disassembled the result.

What I saw was not an allocation in the sense a systems programmer means: it’s a uniform-datapath atomic against a per-SM pool, wrapped by the compiler in a spin loop with a NANOSLEEP backoff, guarded by three distinct trap handlers that ptxas injects on your behalf, with names like __cuda_sm10x_tcgen05_guardrail_trap_unallocated_columns_being_dealloced.

The tensor cores now have a memory allocator, with contention, with a retry path, and with a compiler-inserted runtime safety checker for use-after-free.

That’s the shape of the thing this article is about. Blackwell added an address space with its own instruction family, not just a cache with its own bus. On top of it we have its own allocator, a barrier, its own access-permission model, and zero coherence with anything else on the chip.

Almost everything written about Blackwell treats Tensor Memory as an implementation detail of the new MMA, but I found out it’s the opposite.

The MMA changed because the memory changed, which consequentially changed due to an arithmetic problem that had been building since 2017, and the consequences run outward through occupancy, epilogue design, quantization format choice, kernel portability, and eventually the cost per million tokens of anything you serve on hardware.

Neither of us owns a B200, unfortunately. Everything here that is measured was measured with a toolchain and a disassembler and is reproducible from the appendix. Derived things are in the open with the arithmetic on the page. All data taken from somebody else’s hardware is attributed and tiered in a dossier at the end.

Where the published literature and our disassembly disagree,(spoiler alert: in two places they do) both are shown.

Subscribe now

The number that did not move

We write this very often: every streaming multiprocessor NVIDIA has launched since Volta has exactly 65,536 registers of 32 bits. That is 262,144 bytes of register file per SM, and NVIDIA’s own tuning guides give the same 64 K figure for Ampere and, in the Blackwell tuning guide, for compute capability 10.0.

The same documents states the 255 register per thread ceiling, which we’ll see in use. what we have is: five architectures, three process nodes, four HBM generations, and the largest piece of storage in the SM has not changed capacity by one byte in nine years.

Over the same period tensor throughput per SM per clock went up eight times, and the doubling is exact rather than approximate; NVIDIA’s own numbers make this easily checkable without trusting anybody’s marketing.

A Volta tensor core does 64 fused multiply-adds per clock, eight of them per SM, so 1,024 FP16 FLOPs per clock per SM. Ampere doubled it to 2,048, then Hopper doubled it again to 4,096.

Multiply those out and the datasheets fall out to three digits: 108 A100 SMs times 2,048 times 1.410 GHz is 312 teraflops, which is exactly the published A100 figure. 132 H100 SXM SMs times 4,096 times 1.830 GHz is 989.7 teraflops against a published 989.4.

Run the same identity backwards on Blackwell. The B200 is two dies of 80 SMs with 74 enabled on each, so 148, and 2.25 petaflops dense FP16. That requires 8,192 FLOPs per clock per SM at 1.86 GHz.

Another doubling, and not only ours: the authors of the JAX scaling book run the same division and land on 2,048 FLOPs per tensor core per cycle across four tensor cores per SM, which is the same figure. So the ratio that actually matters, register file bytes per FP16 FLOP per clock, has fallen from 256 on Volta to 32 on Blackwell.

^{Figure 1. Register file capacity per SM against FP16 tensor throughput per SM per clock. The bars are flat by architectural fact. The line doubles every generation. Register file bytes per FLOP per clock: 256, 128, 64, 32.}

You can absorb a gap like that for one generation by being clever, and NVIDIA did, twice. Ampere added asynchronous copy so global to shared traffic stopped passing through registers.

Hopper added the Tensor Memory Accelerator so the address arithmetic for a tiled copy stopped consuming a warp’s registers, and added setmaxnreg so a producer warpgroup could donate its register budget to a consumer at runtime. Both are the same move: find something sitting in registers for no good reason and evict it.

By Hopper the evictable things were gone. What remained was the one thing that genuinely belongs to the math: the accumulator.

Subscribe now

The tile that cannot exist

Here is the constraint that decided Blackwell’s design, and it is a single line of arithmetic against a limit you can make the assembler confirm.

On Hopper, wgmma.mma_async accumulates into registers owned by the 128 threads of a warpgroup. The largest shape is m64n256k16. That accumulator is 16,384 FP32 values, which over 128 threads is 128 registers per thread.

As stated in the subtitle, the architectural ceiling is 255, and ptxas will tell you so directly:

$ ptxas -arch=sm_100a -maxrregcount=255 probe.ptx -o /dev/null
$ ptxas -arch=sm_100a -maxrregcount=256 probe.ptx -o /dev/null
ptxas warning : Too big maxrregcount value specified 256, will be ignored

So Hopper’s largest MMA already spends half of every thread’s addressable register space on the output tile, before any addressing, predication, loop state or epilogue arithmetic.

Now… let’s double it, which is what tcgen05.mma does. The largest single-CTA UMMA atom is m128n256k16, twice the area of the largest WGMMA atom. Its accumulator is 32,768 FP32 values. Over a warpgroup that is 256 registers per thread, against a ceiling of 255.

Blackwell’s headline matrix instruction produces a result that is, by one register, unrepresentable in the programming model of every NVIDIA GPU that came before it.

This is a representability problem and not a turning tradeoff. There is no register allocation, no spill policy, no compiler heroics that make a 128 by 256 FP32 tile live in the fragments of a warpgroup, because the fragment model tops out below the tile.

If you want that instruction to exist, its output must go somewhere that is not the register file. Everything else about Tensor Memory follows from that sentence.

^{Figure 2. The forcing function. Hopper’s largest MMA already spent half the addressable register space per thread on the output tile. Blackwell’s largest single-CTA MMA needs one register more than the architecture allows.}

There is a second argument in the same direction, less absolute but more expensive in practice. A register is thread-private, so an MMA that accumulates into registers is an operation the owning threads must be present for.

On Hopper this is a scheduling tax: a warpgroup issues wgmma, and although the instruction is asynchronous, that warpgroup cannot go and do something else with those registers, because the registers are the accumulator.

Warp specialization on Hopper is largely a set of arrangements to ensure the warps holding accumulators are not the warps doing anything interesting. Move the accumulator out and the whole class of tricks becomes unnecessary.

Share

What Tensor Memory actually is

Tensor Memory is 256 kilobytes per SM, organized as 128 lanes by 512 columns of 32 bit cells. A TMEM address is a 32 bit word whose high 16 bits are the lane index and low 16 bits are the column index. It is not a linear address space with a base and an offset, cause it’s a coordinate.

One consequence worth internalising before reading any CUTLASS TMEM layout: a stride of 65,536 in a TMEM tensor is not a large jump in memory, it is a step of exactly one lane.

Four properties matter more than the capacity, and none are properties of any other memory on the chip.

• It is allocated, not addressed. You call tcgen05.alloc with a column count, which must be a power of two and at least 32, and the hardware returns a base address which it writes into shared memory for you. Allocation is by column, and a column is all 128 lanes: there is no way to reserve part of one. You free it with tcgen05.dealloc, from the same warp that allocated it, or the columns stay claimed.

• Nothing computes on it. The only instructions that touch TMEM are the tcgen05 family. No ALU op, no ld.shared, no ldmatrix, no cp.async, no atomic, no texture path. Every pre-processing step happens before data enters and every post-processing step after it leaves. TMEM is the one memory on a Blackwell SM that a general purpose instruction cannot see.

• Access is partitioned by warp, in hardware. When threads read or write TMEM explicitly, warp 0 of a warpgroup reaches only lanes 0 to 31, warp 1 only lanes 32 to 63, and so on. This is the access model, not a guideline. One warp physically cannot read a full 128 lane accumulator tile, so draining one requires a whole warpgroup by construction, and CUTLASS’s make_tmem_copy is hardcoded to four warps for exactly that reason.

• It is a per-SM pool with a hard ceiling. 512 columns, shared by every CTA resident on that SM, and the largest UMMA accumulator occupies exactly 256 of them.

^{Figure 3. The ledger. Every Blackwell GEMM kernel is, underneath, a plan for carving up 512 columns. The largest accumulator takes half, block scale factors take a few more, and what is left is what you have for pipelining or for a second resident CTA.}

The instruction that uses all this is tcgen05.mma, which CUTLASS calls UMMA. Its operand rules invert everything before them. Operand A may be in shared memory or in Tensor Memory. Operand B has to be in shared memory. The accumulator must be in Tensor Memory. Registers appear nowhere in that sentence.

And it is issued by one thread. Not a warp, not a warpgroup: a single elected thread on behalf of the whole CTA, or of a pair of CTAs under cta_group::2, where two SMs sharing a texture processing cluster cooperate on one logical tile.

The consequence is visible in CUTLASS: the CuTe atom’s ThrID, which was Layout<_32> for warp-level MMA and Layout<_128> for Hopper’s warpgroup MMA, is now Layout<_1>, and the thread layouts have been repurposed as layouts of the CTAs collaborating on the instruction.

The abstraction the entire programming model is named after has been vacated at the top of the pipeline. There is still a thread. It does not do the math, does not own the inputs, and does not own the result.

Subscribe now

What the compiler actually emits

This is the part we did ourselves, so it is the part we trust most. The toolchain is three PyPI wheels and no GPU: ptxas 12.9.86 from nvidia-cuda-nvcc-cu12, and nvdisasm and cuobjdump 13.3.73 from their standalone packages. The full command sequence is in the appendix.

The probe does the minimum honest thing: reserve 128 columns of Tensor Memory, read back the base address, issue one UMMA into it, commit through an mbarrier, drain one fragment to registers, store it, free the columns.

// probe.ptx, assembled with ptxas 12.9.86 for sm_100a
.version 8.6
.target sm_100a
.address_size 64

.visible .entry umma_probe(.param .u64 p_out, .param .u64 p_adesc, .param .u64 p_bdesc)
{
    .reg .b32  %r<16>;   .reg .b64 %rd<8>;   .reg .pred %p<2>;
    .shared .align 16 .b32 tmem_slot[4];
    .shared .align 8  .b64 mbar[1];

    ld.param.u64 %rd1, [p_out];
    ld.param.u64 %rd2, [p_adesc];
    ld.param.u64 %rd3, [p_bdesc];

    mov.u32 %r1, 128;
    tcgen05.alloc.cta_group::1.sync.aligned.shared::cta.b32 [tmem_slot], %r1;
    tcgen05.relinquish_alloc_permit.cta_group::1.sync.aligned;
    ld.shared.b32 %r2, [tmem_slot];

    mov.u32 %r3, 0;
    setp.eq.u32 %p1, %r3, 0;
    tcgen05.mma.cta_group::1.kind::f16 [%r2], %rd2, %rd3, %r3, %p1;

    mbarrier.init.shared::cta.b64 [mbar], 1;
    tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.b64 [mbar];

    tcgen05.ld.sync.aligned.32x32b.x1.b32 {%r10}, [%r2];
    tcgen05.wait::ld.sync.aligned;
    st.global.u32 [%rd1], %r10;

    mov.u32 %r11, 128;
    tcgen05.dealloc.cta_group::1.sync.aligned.b32 %r2, %r11;
    ret;
}

Three facts fall out before the disassembler is even involved.

1. The minimum PTX ISA version is 8.6, which is CUDA 12.8. Below that the assembler names the requirement precisely: Feature 'tcgen05.alloc' requires PTX ISA .version 8.6 or later. We mention this because at least one widely linked public write-up puts it at 8.4, and 8.4 does not assemble.

2. tcgen05.commit requires .shared::cluster, and rejects .shared::cta with State space incorrect for instruction 'tcgen05.commit', even in a kernel with a trivial cluster and cta_group::1. The tensor core completion path is a cluster level mechanism whether or not you asked for a cluster.

3. ptxas does not validate the column count, even when it is a compile time constant. We assembled the probe with 16, 48, 96 and 1,024 columns, all of which violate the documented power-of-two and minimum-32 rules, and all of which assembled without a warning. The rule is enforced by hardware at runtime, not by the compiler. Given that the failure mode is a trap handler, this is a class of bug that only exists on a machine you may not own.

The allocator

Here is what tcgen05.alloc becomes. Address arithmetic trimmed, control flow still intact.

// nvdisasm -c probe_sm100a.cubin, excerpt
        ELECT P0, URZ, PT ;
   @!P0 BRA `(.L_x_2) ;
        DEPBAR.LE SB0, 0x36 ;
        UTCATOMSWS.FIND_AND_SET.ALIGN UP0, UR4, UR4 ;      // claim an aligned run of columns
        PLOP3.LUT P0, PT, PT, PT, UP0, 0x80, 0x8 ;
        SEL R0, RZ, 0xffffffff, !P0 ;
        ISETP.NE.AND P0, PT, R0, RZ, PT ;
    @P0 BRA `(.L_x_3) ;                                    // claimed, continue
.L_x_4:
        NANOSLEEP 0x64 ;                                    // back off, then retry
        UTCATOMSWS.FIND_AND_SET.ALIGN UP0, UR4, UR4 ;
        ...
   @!P0 BRA `(.L_x_4) ;                                    // spin
.L_x_3:
        ATOMS.OR RZ, [UR4], R2 ;                           // record the claimed mask in SMEM
        STS [UR6], R0 ;                                    // publish the base address
        ...
        UVIRTCOUNT.DEALLOC.SMPOOL 0x80 ;                   // SM pool virtual counter

The claim is a UTCATOMSWS.FIND_AND_SET.ALIGN, a uniform-datapath atomic that searches a per-SM pool for a free, aligned run of columns and sets them.

On failure the compiler emits a NANOSLEEP of 0x64 units and retries indefinitely. So a Blackwell kernel that allocates Tensor Memory has a spin loop on its critical path that no source line asked for, and the pool is genuinely contended, otherwise the retry path would not be there.

Then there are the guardrails. ptxas injects three named trap handlers and fifteen references to them, and it does so identically at every optimisation level from -O0 to -O3:

$__internal_0_$__cuda_sm10x_tcgen05_guardrail_trap_col_being_dealloced_not_returned_by_alloc
$__internal_1_$__cuda_sm10x_tcgen05_guardrail_trap_phase_invalid_during_alloc
$__internal_2_$__cuda_sm10x_tcgen05_guardrail_trap_unallocated_columns_being_dealloced

Read those names as a bug taxonomy. Freeing columns you did not allocate. Freeing a column that came from a different allocation. Allocating from an invalid phase.

That is a use-after-free checker, a double-free checker and a state machine assertion, compiled into every kernel, unconditionally. NVIDIA does not do this casually, and the fact that they did it tells you what the failure modes look like in practice.

Why this matters beyond trivia

Every mental model of a GPU kernel assumes resources are assigned at launch. Registers and shared memory are fixed by the compiler and the launch configuration, and occupancy is computed from them before a single instruction runs. Tensor Memory is the first first-class SM resource that is acquired at runtime, can block, can fail, and is arbitrated by an atomic. It moves part of the occupancy calculation out of the launch and into the kernel body, where no static tool can see it.

One number for scale. The probe above, whose actual work is one matrix multiply and one 32 bit drain, compiles to 152 SASS instructions at -O3 and 400 at -O0, using 14 registers and 24 bytes of shared memory.

Two of those 152 are the tensor core; the rest is allocation, election, barrier phase tracking, address reconstruction and guardrails.

The instruction

// dense f16
UTCHMMA      gdesc[UR14], gdesc[UR16], tmem[UR9], tmem[URZ], idesc[URZ], UPT ;
// same source with .sp: the sparsity metadata takes the fourth tmem slot
UTCHMMA      gdesc[UR14], gdesc[UR16], tmem[UR7], tmem[UR4], idesc[UR5], UPT ;
// same source with .cta_group::2
UTCHMMA.2CTA gdesc[UR12], gdesc[UR14], tmem[UR8], tmem[URZ], idesc[URZ], UPT ;
// NVFP4, block16 scaling: scale factors are a fifth tmem operand
UTCOMMA.4X   gdesc[UR14], gdesc[UR16], tmem[UR8], tmem[URZ], idesc[URZ], tmem[UR8], UPT ;

Every operand is a descriptor or a Tensor Memory coordinate, and every one lives in a uniform register, the UR file, not the per-thread register file. The U prefix is the same U as in UMOV, ULEA and UIADD3: the scalar, warp-uniform datapath NVIDIA added in Turing for address arithmetic. Blackwell’s matrix multiply runs entirely on it. There is no vector register in the instruction at all.

That is the cleanest evidence we have found that the CTA, not the thread, is now the unit of tensor computation. It is not an abstraction in CUTLASS or a convenience in PTX. It is visible in which register file the opcode reads.

One clarification the SASS supports and the secondary literature often gets wrong: under cta_group::2 the instruction is not issued by both CTAs in lock step. CUTLASS’s own two-SM tutorial states that only one of the two peer CTAs executes it, and names that one the leader. A single thread, in a single CTA, drives a matrix multiply spanning two SMs.

The dense form passes tmem[URZ], the zero register, in the fourth slot; the sparse form fills it with a real address.

So structured sparsity metadata also lives in Tensor Memory, alongside the accumulator and, for block scaled kinds, alongside the scale factors. Three different kinds of state, one pool of 512 columns.

The opcode family

We compiled the probe once per qualifier and disassembled each result. This mapping is measured.

This disagrees with the published literature in one place. The Delaware microbenchmarking paper, which is otherwise the most useful public measurement of this hardware and which we lean on later, reports in its Table IV that tcgen05.mma lowers to HMMA, QMMA, OMMA and IMMA, the same opcode names Volta through Hopper used.

On our disassembly it does not. The distinction is not cosmetic: HMMA reads and writes vector registers, UTCHMMA touches none. The most likely explanation is that the table was written from the family names rather than from a fresh disassembly.

Which chips can run any of this

Compiling against every target the assembler accepts produces a matrix sharper than the marketing, and the sharpest row is the one nobody mentions.

$ ptxas -arch=sm_100  probe.ptx  -o /dev/null
ptxas error : Instruction 'tcgen05.alloc' not supported on .target 'sm_100'

$ ptxas -arch=sm_100a legacy.ptx -o /dev/null
ptxas error : Instruction 'wgmma.fence' not supported on .target 'sm_100a'

$ ptxas -arch=sm_120a legacy.ptx -o /dev/null
ptxas error : Instruction 'wgmma.fence' not supported on .target 'sm_120a'

Three things to take from that table.

First, wgmma is not deprecated on Blackwell. It is removed, and it is removed from every Blackwell target including the consumer one. A Hopper kernel built on warpgroup MMA does not run slower on Blackwell, it fails at assembly time on all of sm_100a, sm_103a and sm_120a.

This is a harder break than any NVIDIA has shipped in the tensor core era, and it is worth stating precisely because the commonly repeated version of this claim, that consumer Blackwell falls back to mma.sync and wgmma, is wrong on the second half. There is no wgmma to fall back to.

Second, sm_100 without the trailing a, the forward compatible target whose PTX a future driver may recompile for a future chip, has neither tcgen05 nor wgmma.

The entire tensor path that defines Blackwell is available only on architecture specific targets, which by NVIDIA’s documented rules are not forward compatible with anything.

Third, the only instruction family that spans Hopper, datacenter Blackwell and consumer Blackwell is mma.sync, the warp-level, register-resident, m16n8k16 class of instruction that predates all of this. That is the portable subset now. It is also the one whose accumulator lives in the register file, which is the constraint we said the architecture outgrew.

The portable path and the fast path have fully separated. What runs everywhere is the instruction whose limits forced Tensor Memory into existence.

The practical consequence is a development loop, not a benchmark. You cannot write, debug or profile a tcgen05 kernel on a workstation. Not slowly, not at reduced fidelity, not at all. The instructions do not exist on hardware you can buy without a data center behind it.

In our compiler moat piece we argued the durable advantage sits at the ptxas and SASS layer and that the counter-technology is research rather than a new language. This adds a cruder second mechanism that has nothing to do with compilers: iteration on the instructions that matter now requires an allocation of scarce hardware.

We should immediately weaken that. The barrier is money and patience rather than access. B200 time is rentable by the hour, and the author of the best tcgen05 tutorial we have read, writing as gau-nernst, reports reaching 98 percent of cuBLAS on a 4096 cubed problem using rented capacity.

A single motivated person got there in a blog series. That is way a much weaker moat than a first reading suggests.

The cost of getting the answer back

tcgen05.ld takes a shape and a repetition count, and the count determines how many 32 bit values land in each thread’s registers.

We swept the count from 1 to 128, forced every loaded value to stay live by storing all of them, and read the allocation from ptxas -v.

^{Figure 4 The epilogue tax, measured. A single ldtm.x128 costs 134 registers per thread. Across a warpgroup that is 67 KiB of the 256 KiB register file, just over a quarter of it, checked out purely to hold data on its way from one on-chip memory to another.}

The curve is N plus six, and ptxas emits one LDTM.xN rather than N loads. So the register file did not stop being the bottleneck. It stopped being the bottleneck for the multiply and became the bottleneck for the epilogue.

On a Hopper kernel the accumulator is already in registers when you want to scale it, add a bias, apply an activation and cast down. On Blackwell you must first pay to bring it back, and the wider you pay the less room remains to do anything with the result.

Every fused epilogue on this architecture is a negotiation between drain width and register headroom, and no source-level construct expresses it.

Subscribe now

Why it had to be a separate memory

previous sections explained why the accumulator left the register file. It does not explain why the destination is a new addressable memory rather than a hidden internal buffer.

For that you need the traffic number, and the traffic number turns out to be more interesting than we expected when we first ran it.

Start with the shape rule, which is the load bearing fact. In CUTLASS’s Blackwell MMA traits the K extent of a UMMA atom is not a free parameter. It is computed:

// cute/atom/mma_traits_sm100.hpp
// Logical shape-K is always 256bits, transform to units of elements
static constexpr int K = 256 / cute::sizeof_bits<ValTypeA>::value;

K is always 256 bits, 32 bytes, of operand. FP16 gives K equal to 16. FP8 gives 32. FP4 gives 64. The K dimension scales as the inverse of element width, exactly.

Now compute the accumulator traffic. One UMMA over an M by N tile with FP32 accumulation reads the tile and writes it back, which is 8MN bytes, and performs 2MNK floating point operations. So:

accumulator bytes per FLOP  =  8MN / (2MNK)  =  4 / K  =  (bits per element) / 64

FP16   K=16   0.2500 B/FLOP   x  2.25 PFLOP/s  =  562.5 TB/s
FP8    K=32   0.1250 B/FLOP   x  4.50 PFLOP/s  =  562.5 TB/s
FP4    K=64   0.0625 B/FLOP   x  9.00 PFLOP/s  =  562.5 TB/s

per SM (148):                                       3.80 TB/s
B200 HBM3e:                                         8.00 TB/s
ratio, chip accumulator traffic to HBM:               70 x

The tile shape cancels, the clock cancels, and so does the precision. Accumulator traffic on a B200 is 562.5 terabytes per second at every precision the tensor cores support, because every time NVIDIA doubled the math rate by halving the element width, they simultaneously doubled K, which halved the accumulator traffic per FLOP. The two effects cancel to the digit.

That is not a coincidence and it is not a rounding artifact. It is the constraint the 256 bit K rule exists to satisfy.

Whatever structure holds the accumulator has to sustain a fixed bandwidth, and the format ladder was designed so that going from FP16 to FP4 buys four times the math without asking the accumulator path for a single additional byte per second.

^{Figure 5. The invariance. This is the strongest single argument for why Tensor Memory is a fixed size, fixed geometry, precision-agnostic array: the bandwidth it must sustain does not depend on what numbers you put in it.}

The other half, which explains where each operand lives

Run the same calculation on the inputs and the architecture stops looking like a set of choices and starts looking like a single constraint solved twice.

One UMMA reads an M by K slab of A and a K by N slab of B, so operand bytes are (M + N) times K times the element width, against the same 2MNK operations. The K cancels here too:

operand bytes per FLOP  =  (M+N)K b / (2MNK)  =  (M+N) b / (2MN)     // b = bytes per element

for the m128n256 tile:
FP16   b=2      0.01172 B/FLOP   x  2.25 PFLOP/s  =  26.4 TB/s
FP8    b=1      0.00586 B/FLOP   x  4.50 PFLOP/s  =  26.4 TB/s
FP4    b=0.5    0.00293 B/FLOP   x  9.00 PFLOP/s  =  26.4 TB/s

accumulator traffic / operand traffic  =  562.5 / 26.4  =  21.3 x

Invariant again, and for the same reason: halving the element width halves the operand bytes per FLOP at exactly the rate it doubles the FLOPs. So the precision ladder from FP16 down to FP4 is bandwidth neutral at both ends of the datapath.

Blackwell quadrupled its peak math without asking either the operand path or the accumulator path for one additional byte per second.

That is also the answer to a question the operand rules raise and never explain. Why must B sit in shared memory while D must sit somewhere else entirely? Because the accumulator moves twenty one times the traffic.

Operands are read once per instruction and are narrow by construction; the accumulator is read and written in full every single time, in FP32, no matter how few bits the inputs have. A general purpose memory can serve the first workload. Nothing general purpose serves the second.

It also explains a fact that otherwise looks like an oversight: Blackwell’s shared memory did not grow. 228 kilobytes per SM, identical to Hopper, in a generation that doubled math per SM per clock. It did not need to.

And where operand pressure did rise, NVIDIA answered with sharing rather than capacity: under cta_group::2 two SMs in a texture processing cluster consume the same operands for one logical tile, which halves the per-SM operand traffic instead of doubling the memory that carries it.

562.5 terabytes per second across the chip is seventy times the entire HBM bandwidth of a B200, and 3.8 terabytes per second per SM sustained. Neither a cache nor a register file survives that.

What survives is a small, banked array physically adjacent to the consumer, addressed in the coordinate system the datapath already uses, and free of every general purpose obligation: no coherence, no cache tags, no arbitrary indexing, no participation in the memory model, no ability to be read by an ALU.

Which is precisely the list of things TMEM cannot do. The restrictions are not a first generation compromise to be relaxed later. They are the reason the number is achievable.

It is worth putting our figure next to the one measurement that exists. The Delaware group reports roughly 16 terabytes per second of TMEM read bandwidth on a B200. That is about four times the sustained accumulator requirement we derive, which is the right shape of answer: a peak port figure with headroom for drains overlapping accumulation, not a number that contradicts ours.

We would rather show both than pretend they measure the same thing.

A caution about that paper’s peaksThe same paper reports achieved throughputs as percentages of theoretical peak: FP4 at 7,700 TFLOPS being 96.2 percent, FP16 at 1,929.6 being 96.5 percent. Those imply peaks of about 8,004 and 1,999 TFLOPS, where the B200 datasheet says 9,000 and 2,250 dense.

Both of their implied peaks are 11.1 percent below the datasheet, consistently, which is what you get from assuming a clock about 11 percent lower than the one the datasheet figures use. Their measurements are probably fine.

Their percentages are relative to a different baseline than NVIDIA’s, and should not be read as 96 percent of the number on the box.

Subscribe now

Occupancy stops meaning what it meant

Occupancy is the oldest performance heuristic in CUDA: resident warps per SM, limited by registers and shared memory, more of them meaning more latency to hide.

On Blackwell tensor kernels it is close to useless, and the reason is that a resource nobody’s occupancy calculator models now binds before the ones it does.

Start with what NVIDIA documents for compute capability 10.0, in the Blackwell tuning guide.

• Register file: 64 K 32-bit registers per SM, unchanged.

• Maximum concurrent warps per SM: 64, unchanged since Volta.

• Maximum thread blocks per SM: 32.

• Shared memory capacity per SM: 228 kilobytes, the same as Hopper, with 227 addressable by a single block after CUDA’s 1 kilobyte reservation.

• Combined L1, texture and shared memory: 256 kilobytes, also the same as Hopper.

Read that list again with the previous sections in mind. Across a generation that doubled tensor throughput per SM per clock, not one of the classical occupancy resources grew.

The only capacity Blackwell added to the SM is the 256 kilobytes of Tensor Memory, and Tensor Memory is the one resource with an allocation rule that quantizes hard.

Columns come in powers of two, minimum 32, from a pool of 512, and every column carries all 128 lanes. So the pool admits 16 concurrent allocations at the smallest legal size, 8 at 64 columns, 4 at 128, 2 at 256, and 1 if you take the whole thing. There is no middle. And 16, the best case, is already half the hardware ceiling of 32 blocks per SM.

The moment your tile needs more than the minimum allocation, which any tile worth issuing a UMMA for does, Tensor Memory is the binding constraint and nothing else is close.

^{Figure 6. What the pool admits. This is an upper bound from the allocation rule, not a measurement of scheduler behaviour. Whether the hardware will actually co-resident that many CTAs is a separate question, and one we cannot settle without a B200.}

That caveat is not decoration. Colfax’s post on the workstation part, contrasting it with the datacenter part, states flatly that on SM10x tcgen05.mma is locked to one CTA per SM. Colfax’s own tutorial kernel allocates all 512 columns for a single 128 by 256 tile and never revisits the question.

And the PTX manual describes tcgen05.relinquish_alloc_permit as a promise that the CTA will make no further allocations, which Colfax glosses as allowing future CTAs to queue up for the same SM. The word queue is doing a lot of work in that sentence.

It is consistent with a scheduler that admits a new CTA only when the pool can serve it, which would collapse figure 6 toward 1 for any realistic tile regardless of the arithmetic.

We cannot resolve that without hardware, and we would rather show the bound and name the uncertainty than assert a residency number we have not seen. What survives either reading is the shape of the problem: the resource that limits parallelism on a Blackwell SM is acquired at runtime, quantizes in powers of two, and does not appear in any static occupancy model.

Now the part that makes low occupancy survivable, which is the more interesting half.

What occupancy hid was memory latency: a warp stalls on a load, the scheduler runs another warp. In a Blackwell GEMM the loads are done by the Tensor Memory Accelerator, asynchronously, into shared memory, signalled by an mbarrier.

The math is done by a single elected thread issuing an asynchronous instruction that reads shared memory and writes Tensor Memory. Neither heavy operation is a thread stalling on anything. Latency is hidden by pipelining within one CTA, using barriers and multiple buffers, rather than by switching between CTAs.

The author writing as gau-nernst describes exactly this in a working kernel: multiple tcgen05.mma in flight, one mbarrier per stage, so different MMA stages can be waited on independently.

The Delaware measurements make the same point from the other side. Single instruction latency for tcgen05.mma is 11.0 to 11.4 clocks and nearly flat across tile shapes from m64n64k16 to m256n256k16. Hopper’s wgmma scales linearly with tile width, 32 clocks at m64n64k16 and 128 at m64n256k16.

A flat latency across a sixteen fold range of tile area is the signature of a spatial array, not a deeper pipeline.

^{Figure 7. Hopper’s MMA latency grows with the tile. Blackwell’s does not. Combined with single thread issue and a TMEM resident accumulator, this is what makes very low CTA counts survivable.}

So the tuning knobs invert. On Hopper you asked how many warpgroups fit and how to specialize them. On Blackwell you ask how many columns your accumulator needs, how many buffers fit in what remains, and whether the tile is worth a CTA pair.

The Nsight metric that used to matter, achieved occupancy, tells you almost nothing. The metric that matters has no counter.

Subscribe now

Quantization became an allocation problem

Here is the part we think is genuinely underappreciated, and it is the point where this architecture stops being a matter for kernel authors and starts being a matter for anyone who chooses a serving format.

Blackwell implements block scaling in hardware. A block scaled MMA computes D = C + (A x SFA) times (B x SFB), where SFA and SFB are vectors of scale factors, one per group of 16 or 32 elements along K.

The scale factors are not folded in beforehand and they are not applied afterwards. They are consumed by the tensor core as it runs. And, per the PTX ISA, they are consumed from Tensor Memory.

My disassembly shows this directly. The block scaled variants take a third and fourth tmem[] operand:

// kind::mxf4nvf4.block_scale.scale_vec::4X, NVFP4 with block16 scaling
UTCOMMA.4X gdesc[UR14], gdesc[UR16], tmem[UR8], tmem[URZ], idesc[URZ], tmem[UR8], UPT ;

// kind::mxf4.block_scale.scale_vec::2X, MXFP4 with block32 scaling
UTCOMMA    gdesc[UR14], gdesc[UR16], tmem[UR8], tmem[URZ], idesc[URZ], tmem[UR8], UPT ;

Read the consequence carefully. Your choice of numerical format now consumes the same scarce, power of two, 32 column granular, 512 column per SM resource that your accumulator consumes.

Finer scaling is not just more metadata bandwidth from HBM. It is columns you cannot use for the accumulator, for double buffering, or for a second CTA.

The two formats differ exactly where it hurts. MXFP4 follows the Open Compute microscaling specification: blocks of 32, scale in E8M0, a bare power of two exponent.

NVFP4 is NVIDIA’s own: blocks of 16, scale in E4M3, a real floating point number with a mantissa, plus a second level FP32 tensor-wide scale. Per the PTX rules, block32 must pair with E8M0, while block16 may use E8M0 or E4M3.

^{Figure 8 NVFP4 halves the block size and puts a mantissa in the scale, which is why it is more accurate than MXFP4. It also doubles the scale factor footprint, and that footprint lives in the same 512 column pool as the accumulator.}

This is a co-design decision hiding inside a numerics decision. NVIDIA defined a format whose accuracy advantage over the open standard comes from finer blocks and richer scales, then built the only silicon where those scales are consumed directly out of a dedicated on-chip memory rather than being unpacked into registers first.

On hardware without that memory, the same format is a software dequantization problem with a register cost. On Blackwell it is an operand.

The Delaware group reports the accuracy side: FP8 costs about 2 percent perplexity on Mistral 7B and Mixtral 8x7B, FP4 costs 8 to 9 percent, and their FP4 throughput is 2.5 times FP16 on the dense model and 2.7 times on the mixture of experts model.

Whether 8 percent perplexity is acceptable is a per-layer question and always was. My point is narrower: on this architecture the answer is also a per-SM capacity planning question, because the scale factors and the accumulator compete for the same 512 columns.

Numerics stopped being a property of the model and became a property of the memory allocator.

Subscribe now

What this does to serving economics

Everything above is architecture. Here is the part that shows up on an invoice.

Two rules from earlier sections collide here. The UMMA shape table offers exactly two values of M for a single CTA, 64 and 128; there is nothing smaller.

And allocation is by whole column, so the accumulator’s footprint is fixed by the tile you chose, not by the rows you filled.

During prefill neither rule bites. M is the token count of a chunk, thousands of rows, and the tile is full. During decode both bite at once. A decode step is a GEMM whose M is the number of sequences you are batching, and it is skinny by construction, so you pick the 64-row tile and leave most of it empty.

^{Figure 9 The decode penalty, stated against the most favourable tile the instruction set offers. At a batch of eight, an ordinary steady state for a latency sensitive endpoint, an eighth of the reserved accumulator carries anything. The rest is allocated, idle, and unavailable to any other block on that SM.}

Be precise about what is and is not new, because it is easy to overstate. The 64-row floor is not new: Hopper’s wgmma had exactly the same minimum M, and skinny GEMMs have underused tensor cores since Volta.

That is why prefill and decode get disaggregated onto separately provisioned pools in the first place, which we argued at length in a previous issue.

What is new is where the waste is recorded. Previously it was purely a throughput loss: you issued an MMA whose M dimension was mostly zeros and you got a fraction of peak flops, and the moment the instruction retired the machine was free again.

Now the same tile also holds a capacity reservation, in a 512 column pool, acquired through an atomic that other CTAs may be spinning on, held for the tile’s lifetime, and invisible to every static occupancy model. Underutilisation stopped being a transient and became an allocation.

Three practical consequences follow, and we hold them with decreasing confidence.

1. The first is that Blackwell widens the gap between prefill and decode economics rather than narrowing it, in a generation whose marketing is entirely about inference. The FP4 tensor cores are a prefill and large-batch story, and section 05 is the reason: the precision ladder is bandwidth neutral on chip, so what FP4 buys you is math you can only spend if you have rows to fill. Decode has neither. It remains bound by HBM bandwidth for weight movement, and now carries an on-chip capacity reservation on top. This is consistent with the Delaware measurements from a different angle: as precision drops from FP16 to FP4 their measured memory bandwidth utilization falls from 67 percent to 48 percent, which is what it looks like when a workload stops being bandwidth bound and starts being bound by something else.

2. The second is that this pushes harder toward every technique that manufactures M. Speculative decoding turns one sequence into k candidate tokens per step, and on Blackwell it is not only amortizing weight reads across more rows, it is filling rows of a tile that was reserved whether or not you filled them.

   Figure 9, read the other way, is a chart of how much accumulator a speculative draft gets for free. We modelled speculation as a throughput question in a previous issue. There is a second term in that model now, and it points the same way.

3. The third, and the one we are least sure of, is that grouped GEMM for mixture of experts becomes a harder allocation problem than it was. Each expert’s tile has its own M, determined by routing, varying per step. If your kernel allocates for the worst case it wastes columns on every expert that got fewer tokens; if it allocates per expert it pays the atomic repeatedly.

The one worklog we have found on optimizing NVFP4 grouped GEMM on Blackwell, by Mufeez Amjad, lands on cta_group::2 with a shared TMEM allocation across the CTA pair, and describes the shared allocation as one of the main benefits rather than the wider math tile. That is a hint about where the pressure actually is.

Subscribe now

What Rubin says about whether this was a one-off

A reasonable objection to everything above is that Tensor Memory might be a Blackwell specific hack, a way to ship a doubled MMA without redesigning the register file, and that the next architecture folds it back into something more general.

NVIDIA published enough about Rubin on 21 July 2026 to answer that. It is not going away, and every disclosed change points at the numbers in section 05.

Three details in NVIDIA’s own post are load bearing.

1. Tensor Memory gained a consumer that is not the MMA. In the long context attention path, the intermediate scores from the dense QK transpose are, in NVIDIA’s words, loaded from Tensor Memory into a structured 2 to 4 sparse compressed form, generating both the nonzero values and the metadata. TMEM is no longer only where accumulators land. It is a staging tier that a hardware compression path reads from. That is the same direction our disassembly already showed on Blackwell, where sparsity metadata and block scale factors are already TMEM operands.

2. Rubin doubles the K dimension per tensor core instruction. NVIDIA frames this as fewer K loop iterations and less loop overhead, which is true and is the reason a reader would care. Run it through section 05 and it is also something else. Accumulator bytes per FLOP is 4 over K. Doubling K halves it. If Rubin’s NVFP4 K goes from 64 to 128, accumulator traffic per unit of math drops by half at exactly the moment the math rate goes up. That is the same invariance trick applied across a generation instead of across a precision ladder, and it is the strongest evidence we have that accumulator bandwidth is a first order design constraint at NVIDIA rather than a consequence.

3. Softmax became the bottleneck, so they widened it. Rubin raises exponential throughput per clock per SM by 2 times for FP32 and 4 times for BF16 against Blackwell, with Blackwell Ultra at 2 times for both. You only build that if the matrix path has already pulled far enough ahead that the transcendental path is what is left.

Alongside those, Rubin adds inline descriptor updates for the Tensor Memory Accelerator, so a mixture of experts kernel keeps one descriptor and overrides the pointer and stride fields in the instruction instead of rewriting a descriptor in memory per expert, and counted writes for device initiated NVLink transfers so the receiver tracks completion without the barrier, acknowledgement and atomic flag sequence.

Every one of those is the same move: take a coordination cost that was paid in general purpose instructions and registers, and pay it in a dedicated mechanism instead. Tensor Memory was the first large instance. Rubin is the pattern applied to descriptors, to synchronization, to sparsity metadata and to the softmax path.

One sentence for what NVIDIA is doing to the SM across these two generations: they are disassembling the general purpose core into special purpose engines connected by dedicated memories and barriers, and leaving the threads to do bookkeeping.

The execution model is being hollowed out from the inside while its surface syntax stays the same.

Subscribe now

Where this could be wrong

• The accumulator traffic derivation assumes a full read and write per instruction. If the tensor core retains partial accumulator state internally across the K extent, or streams only the sub-tile currently in flight, then 562.5 terabytes per second is an upper bound. What makes us reasonably confident is not the absolute number but the invariance: the cancellation between element width and K is exact, holds at three precisions, and falls out of a rule visible in CUTLASS source. A different traffic model would change the constant and probably not the invariance. Note that this is a revision of an earlier draft of ours, which held K fixed at 16 across precisions and therefore produced a per-SM figure four times too large at FP4.

• We are reading the SASS of a probe, not of a real kernel. A kernel with one MMA gives ptxas no scheduling problem to solve. The allocation spin loop may be scheduled very differently, hoisted, or dominated by something else in a CUTLASS mainloop with a producer warpgroup and four pipeline stages. We are confident the instruction sequence exists and that the guardrails are unconditional. We are not confident about its cost in situ, and the 152 instruction figure is a property of our probe, not of production kernels.

• Figure 6 is an upper bound, not a residency measurement. The allocation arithmetic is exact, and the documented block and warp ceilings for compute capability 10.0 are exact. What we cannot verify is whether the scheduler will actually co-resident the CTAs the pool arithmetically admits. One published source states that SM10x is locked to one CTA per SM outright, and the PTX description of relinquish_alloc_permit hints at queueing rather than co-residency. If that reading is right, the correct version of figure 6 is a flat line at 1, the argument gets stronger rather than weaker, and our chart is still wrong.

• Figure 9 measures reservation, not throughput, and only for the tile we chose. The 64 row floor is documented and the arithmetic against it is exact, but a real decode kernel may prefer a different shape, may reuse one allocation across many tiles, or may batch several matrices into a single call in ways that change what fraction of the pool sits idle at any instant. The claim we will defend is narrow: below 64 sequences the accumulator is reserved for rows that do not exist. What that costs in dollars depends on a serving stack we have not profiled.

• The Blackwell per-SM per-clock figure of 8,192 is inferred, not published. Volta, Ampere and Hopper rates come from NVIDIA whitepapers and reproduce the published teraflops to three digits. For Blackwell we ran the identity backwards from 2.25 petaflops over 148 SMs, which requires 8,192 FLOPs per clock at 1.86 GHz. If the real SM count in the shipping part differs, or if the datasheet figure assumes a different clock, the doubling claim survives but the exact number moves.

• The portability finding is solid; the moat reading of it is not. Section 04 already discounts it and we would discount it further rather than less. The instruction family genuinely does not exist outside datacenter Blackwell. What follows from that commercially is much weaker than it first sounds.

Share The Software Frontier

Four predictions, each with its failure condition

Deliberately conservative, and each one checkable by a specific date against a specific artifact.

1. NVIDIA will not expose Tensor Memory to a general purpose instruction through the end of 2028. No ld, st, atomic or ALU operand outside the tcgen05 family, in Rubin or its successor. The invariance in section 05 is the reason: general purpose addressability is incompatible with the bandwidth. Wrong if a PTX ISA revision adds any TMEM access outside the dedicated opcode family.

2. By 31 December 2027, no open source compiler will generate a tcgen05 GEMM within 10 percent of CUTLASS on a mainstream shape without hand written PTX in its lowering path. Reaching the instruction is already happening. Reaching it through a general lowering that models column allocation, the warp-lane access partition and the drain width tradeoff is a different problem. Wrong if a mainline release of Triton, Mojo or tinygrad hits the bar with a pure compiler path.

3. Nsight Compute will ship a Tensor Memory occupancy or column pressure section before the end of 2028. The hardware is already tracking the pool, since the allocator does a find-and-set against it, and there is currently no static way to reason about a resource acquired at runtime. Wrong if no NVIDIA profiler release by then reports TMEM allocation state.

4. NVFP4 will remain the default four bit format in NVIDIA’s own inference libraries through 2027, and MXFP4 will not displace it there. Not an accuracy argument: the block16 path is the one with a dedicated opcode modifier and a hardware operand path on the silicon that matters. Wrong if TensorRT-LLM or NVIDIA’s quantization tooling defaults to block32 for four bit weights.

We have dropped two predictions that appeared in an earlier draft, on model architectures being shaped around column boundaries and on serving share ratios between formats, because neither had a failure condition we could actually check.

Confidence dossier

Subscribe now

Reproduce every tier A row on a machine with no GPU

Everything in tier A came out of the following, in a clean container, in under five minutes, with no CUDA installation and no driver.

# 1. the toolchain, as ordinary x86 binaries from PyPI
pip download nvidia-cuda-nvcc-cu12 --no-deps -d /tmp/w            # ptxas 12.9.86
pip download nvidia-cuda-nvdisasm nvidia-cuda-cuobjdump --no-deps -d /tmp/w   # 13.3.73
mkdir -p /tmp/tk && cd /tmp/tk && for f in /tmp/w/*.whl; do unzip -oq "$f"; done
PTXAS=/tmp/tk/nvidia/cuda_nvcc/bin/ptxas
DIS=/tmp/tk/nvidia/cu13/bin/nvdisasm

# A1, A2, A3, A4: assemble the probe and read the machine code
$PTXAS -arch=sm_100a probe.ptx -o probe.cubin && $DIS -c probe.cubin | less
$DIS -c probe.cubin | grep -oE '__cuda_sm10x_tcgen05_[a-z_]+' | sort -u
for o in 0 1 2 3; do $PTXAS -O$o -arch=sm_100a probe.ptx -o g.cubin
  echo "-O$o $($DIS -c g.cubin | grep -c guardrail) refs, \
$($DIS -c g.cubin | grep -cE '^\s+/\*[0-9a-f]{4}\*/') instructions"; done

# A1 continued: one build per .kind qualifier
for k in f16 tf32 f8f6f4 i8; do sed "s/kind::f16/kind::$k/" probe.ptx > k.ptx
  $PTXAS -arch=sm_100a k.ptx -o k.cubin && $DIS -c k.cubin | grep -oE 'UTC[A-Z0-9.]+'; done

# A5: three instruction families across five targets
for a in sm_90a sm_100a sm_103a sm_100 sm_120a; do
  for f in probe legacy_wgmma legacy_mmasync; do
    printf '%-9s %-16s ' $a $f; $PTXAS -arch=$a $f.ptx -o /dev/null 2>&1 | head -1; echo; done; done

# A6: minimum PTX ISA version
for v in 8.3 8.4 8.5 8.6 8.7 8.8; do sed "s/^.version 8.6/.version $v/" probe.ptx > v.ptx
  echo -n "$v "; $PTXAS -arch=sm_100a v.ptx -o /dev/null 2>&1 | head -1; echo; done

# A7: the epilogue register curve. widen tcgen05.ld to .xN and store every register
for n in 1 2 4 8 16 32 64 128; do $PTXAS -arch=sm_100a -v ld_$n.ptx -o /dev/null 2>&1 | grep Used; done

# A8, A9: the ceilings the compiler will and will not enforce
$PTXAS -arch=sm_100a -maxrregcount=256 probe.ptx -o /dev/null
for c in 16 48 96 1024; do sed "s/mov.u32 %r1, 128;/mov.u32 %r1, $c;/" probe.ptx > a.ptx
  $PTXAS -arch=sm_100a a.ptx -o /dev/null && echo "$c columns: accepted"; done

Two practical notes. ptxas from the 12.9 wheel caps at PTX ISA 8.8, which covers sm_103a and nothing above it, so pull a newer nvidia-cuda-nvcc for later targets. And nvdisasm from the CUDA 13 wheels reads CUDA 12 cubins, which is convenient because the CUDA 13 nvcc wheel did not build in our container while the standalone disassembler wheels did.

The value of this workflow is not that it replaces a GPU. It is that it separates two questions that get conflated constantly in GPU writing: what the machine does, which needs hardware, and what the compiler emits, which does not.

A surprising share of public claims about the CUDA moat are claims of the second kind, and the second kind is checkable by anyone with forty megabytes of disk.

Bibliography

1. NVIDIA, Parallel Thread Execution ISA: tensor memory addressing, tcgen05 MMA and its kind shapes, shared memory descriptors, instruction descriptors, data path layout organization, and the tcgen05 memory consistency model. The primary source for everything structural here.

2. NVIDIA, Blackwell Tuning Guide and Ampere Tuning Guide. Register file size, the register per thread ceiling, warp and thread block limits per SM, and shared memory capacities for compute capabilities 8.0, 10.0 and 12.0.

3. NVIDIA Volta, Ampere and Hopper architecture whitepapers. Source for 1,024, 2,048 and 4,096 FP16 tensor FLOPs per clock per SM, each of which reproduces the corresponding published teraflops figure to three digits.

4. NVIDIA, HGX B200 datasheet. Dense and sparse rates per precision, with the explicit note that dense is half of sparse, plus memory capacity and bandwidth.

5. Ryo, CUTLASS Tutorial: Writing GEMM Kernels Using Tensor Memory For NVIDIA Blackwell GPUs, Colfax Research, April 2025, updated November 2025. The clearest published account of TMEM allocation, UMMA operand rules and the CuTe abstractions over both.

6. Colfax Research, CUTLASS Tutorial: Hardware-supported Block-scaling with NVIDIA Blackwell GPUs. Scale vector rules and TMEM layouts of scale factors.

7. Colfax Research, NVFP4 Blockscaled GEMM on NVIDIA RTX Pro Blackwell GPUs (SM12x), June 2026. The explicit statement that SM12x has neither tcgen05 nor TMEM, and that SM10x is locked to one CTA per SM.

8. NVIDIA, CUTLASS source, include/cute/atom/mma_traits_sm100.hpp and copy_traits_sm100.hpp. The 256 bit K rule, the TMEM copy atoms, and the ThrID change.

9. Aaron Jarmusch and Sunita Chandrasekaran, University of Delaware, Microbenchmarking NVIDIA’s Blackwell Architecture: An in-depth Architectural Analysis, arXiv 2512.02189. The only systematic public measurement of B200 tensor core latency, TMEM behaviour and FP4 accuracy we are aware of.

10. NVIDIA, Inside NVIDIA Rubin GPU Architecture: Powering the Era of Agentic AI, 21 July 2026.

11. NVIDIA, CUTLASS documentation, Blackwell SM100 functionality. The seven tcgen05.mma instructions and the block scaled data types.

12. SemiAnalysis, Dissecting NVIDIA Blackwell: Tensor Cores, PTX Instructions, SASS, Floorsweep, Yield. The CuTe ThrID observation and the TPC scoped CTA pair framing.

13. gau-nernst, tcgen05 for dummies, December 2025. A working tutorial in plain CUDA and inline PTX.

14. Mufeez Amjad, Optimizing NVFP4 Grouped GEMM on Blackwell, worklog, March 2026. The CTA pair and shared TMEM allocation observation in section 08.

15. Rouhani et al., Microscaling Data Formats for Deep Learning, arXiv 2310.10537. The MX specification behind MXFP4 and MXFP8.

16. NVIDIA developer forums, thread on computing tensor core FP16 throughput per SM per clock from whitepaper figures. The identity used in section 01.

17. Chips and Cheese, Nvidia’s B200: Keeping the CUDA Juggernaut Rolling, December 2025. Die level SM counts: 74 enabled of 80 per die.

18. Austin et al., How To Scale Your Model, GPU chapter. Independent derivation of Blackwell’s per-SM per-clock tensor rate from the same datasheet figures.

19. Our own previous work: How the NVIDIA Compiler Moat Actually Works, How CUDA Binaries Actually Work, The Split and the Seam on prefill and decode disaggregation, and The Draft and the Ledger on speculative decoding economics.

    Share The Software Frontier

Intro

I spent a good part of last year writing an x86-64 assembler by hand, in C, for no more reason than wanting to know what an object file really is. The thing that surprised me was not the instruction encoding, but was the metadata.

A modern object file is essentially a negotiation between the compiler and the loader about few things the machine code itself cannot say properly: where the arguments are, how much stack to reserve, which symbols are entry points, what has to be patched before anything runs.

So when I started reading GPU binaries, the question I kept asking more and more was not “what does this instruction do.” It was “what is this file promising the driver.”

The answer turns out to be… well, a lot, and almost none of it is written down. NVIDIA just documents the container in one sentence and the disassembler in twelve pages. And that’s basically it.

The CUDA Binary Utilities manual says a cubin is “an ELF-formatted file which consists of CUDA executable code sections as well as other sections containing symbols, relocators, debug info, etc.” That is the specification, and the word “etc.” is where the whole platform lives.

I want you to imagine this piece as a medical dissection, if it’s possible to say so. Every single number, hex dump and section listing below, came out of a container on my own machine, and the exact commands are in the appendix so you can disagree with me precisely.

Two things made it possible: first thing is that you do not need a GPU to compile, link or disassemble CUDA binaries: ptxas, fatbinary, nvdisasm and cuobjdump are a bunch of ordinary x86 programs. The second is that the entire toolchain is on PyPI, so getting a byte-exact CUDA 13.3.73 install is actually just one pip install.

Methodology

Everything measured here is from CUDA 13.3.73 (built 9 June 2026) installed from the nvidia-cuda-nvcc, nvidia-cuda-cuobjdump and nvidia-cuda-nvdisasm wheels on x86-64 Linux, plus the cuBLAS 13.6.0.2 wheel for the library dissection.

I do not currently have a GPU, which means every claim here is about what the toolchain emits and not about what silicon does with it. There are a few parts where I’m reading structure out of bytes instead of documentation, dont’t worry: I’ll say so and I give it like a confidence tier at the end.

NVIDIA publishes no specification for the cubin section layout, the .nv.info attribute encoding, the fatbin entry header or the Mercury sections, and all the interpretations are mine, unless specified.

Subscribe now

Seven processes to compile one file

I want you to introduce the format and the pipeline now, because the shape of the output is derived from it. Keep in mind that nvcc is not a compiler, it’s just a driver that orchestrates other programs, and it will show you its plan.

$ nvcc -arch=sm_90 -c -o k.o saxpy.cu --dryrun
gcc      -D__CUDA_ARCH_LIST__=900 -E -x c++ -D__CUDACC__ ...        # host preprocess
cudafe++ --c++17 --static-host-stub --device-hidden-visibility ...  # split host/device
gcc      -D__CUDA_ARCH__=900 -E -x c++ -DCUDA_DOUBLE_MATH_FUNCTIONS # device preprocess
cicc     ... saxpy.cudafe1.gpu -o saxpy.ptx                        # C++ front end -> PTX
ptxas    -arch=sm_90 -m64 saxpy.ptx -o saxpy.sm_90.cubin           # PTX -> SASS
fatbinary -64 --cicc-cmdline=... --image3=...  -o saxpy.fatbin.c    # package
gcc      -c -x c++ saxpy.cudafe1.cpp -o k.o                        # host compile

We have two preprocessor passes over the same source with “different macros”, then a source-to-source splitter, a front end that emits PTX, an assembler that (you know) emits machine code, a packager that turns the machine code into a C array, and a host compiler that swallows the result.

The artifacts in the middle are real files you can save with --keep, and each one is a place where the format is decided.

The only thing that matters is where the close part actually is. We alredy know that cicc is the NVVM-based front end and it produces PTX, which is a published virtual instruction set, with all docs that you can read. But for ptxas,which takes PTX and produces the thing this article is about, we don’t know nothing.

Everything that works below PTX is the vendor’s private business, and everything upstream has alredy been made public: Clang, Triton, Julia, Mojo and tinygrad all emit PTX, and NVIDIA itself ships libNVVM, so they can do it.

The interesting feature in the CUDA stack is just one process wide.

Subscribe now

The container is an ELF file that lies about being one

Now, let’s compile a kernel to a standalone cubin and we will see that the file recognizes it immediately.

$ nvcc -arch=sm_90 -cubin -o saxpy.sm90.cubin saxpy.cu
$ file saxpy.sm90.cubin
saxpy.sm90.cubin: ELF 64-bit LSB executable, NVIDIA CUDA architecture, version 1,
                  statically linked, not stripped

It is a real ELF64, little endian, and readelf will walk its section table happily. Three fields in the 64-byte header carry the CUDA-specific part.

e_ident:  7f 45 4c 46 02 01 01 41 08 00 00 00 00 00 00 00
                              |  |
                              |  +-- EI_ABIVERSION = 8
                              +----- EI_OSABI      = 0x41
e_type    = 2        (ET_EXEC; ET_REL when compiled with -rdc=true)
e_machine = 190      (0xbe, EM_CUDA)
e_flags   = 0x06005a04

EM_CUDA = 190 is the one part of this that is available to everybody: it sits in LLVM’s BinaryFormat/ELF.h alongside every other machine type, because LLVM needs to read these files.

The OS ABI byte is where things get strange. That same header defines two CUDA values, ELFOSABI_CUDA = 51 and ELFOSABI_CUDA_V2 = 41, both written in decimal. Every cubin my 13.3 toolchain produces carries 65, which is 0x41. 51 is 0x33, so the first constant is the older value written in decimal, and the second…. looks like 0x41 transcribed as if it were decimal.

Obv. I don’t know where is the mistake, but if you are writing a tool that sniffs cubins, sniff for the byte 0x41 and don’t trust either constant. The ABI version byte is 8.

I know that e_flags is where the architecture lives, and the field names are recoverable from the disassembler: EF_CUDA_SM, EF_CUDA_PTX_SM, EF_CUDA_64BIT_ADDRESS and EF_CUDA_ACCELERATORS, alongside an EF_CUDA_SM10 through EF_CUDA_SM121 enumeration that covers fifteen years of hardware in one list.

The layout is easy peasy to recover by sweeping every target the compiler supports.

The same four-line kernel, one target per row. Bits 8 to 15 hold the compute capability as a plain integer. Byte 0 flips from 0x04 to 0x02 at the Blackwell boundary, and the file grows by 44% across that same boundary for identical source.

This is actually very good, but at least three things don’t match reality:

First, the virtual architecture is not in e_flags anymore, despite the name in the field. Compile the same kernel three ways, with the PTX architecture set to compute_75, compute_80 and compute_90 but the real target fixed at sm_90, and all three cubins carry e_flags = 0x06005a04.

The virtual architecture actually moved into a note section, where cuobjdump reports it as CUDA Virtual SM: sm_75 and so on. This used to be visible: CUDA 8-era disassembly printed a line reading .headerflags @"EF_CUDA_SM20 EF_CUDA_PTX_SM(EF_CUDA_SM20)", and current nvdisasm prints a plain .target sm_90 instead. If you have old tooling that reads the PTX architecture out of the flags word, you have seen zero for some years.

Second, the architecture-conditional suffixes do not show up here either. sm_90 and sm_90a produce identical e_flags, and so do all three of sm_100, sm_100a and sm_100f. For a kernel that uses no special features the sm_100 and sm_100f cubins are the same program byte for byte, and the 210 bytes that do differ between the two files are all downstream of one string: the note section records -arch sm_100f instead of -arch sm_100, which is one byte longer and shifts everything after it.

That is in line with what NVIDIA says about the feature, which is that code=sm_100 and code=sm_100f are aliases producing the same cubin when no family-specific features are in play. The suffix is recorded, but the recording is somewhere else and is “triggered” only when the kernel actually uses something.

Third, CUDA 13’s supported target list is very short, and that surprised even me. Turing is now the floor. ptxas in 13.3 accepts exactly sm_75, sm_80, sm_86, sm_87, sm_88, sm_89, sm_90, sm_90a, then the three-digit generation: sm_100, sm_103, sm_110, sm_120, sm_121 each with a and f variants, plus the matching lto_* targets.

Maxwell, Pascal and Volta were removed in CUDA 13.0, which NVIDIA’s release notes state directly: offline compilation and library support for those architectures are gone completely, and 12.x is the last line that can target them.

You have to know at least two numbers if you ever have to explain a build matrix. sm_110 is Jetson AGX Thor, renamed from sm_101 when CUDA 13.0 shipped, which means build scripts that hardcoded 101 broke on a rename rather than on a chip. NVIDIA states the rename in the nvCOMPDx release notes rather than anywhere prominent.

And sm_88, compute capability 8.8, is said by the community compatibility references to be the Nintendo Switch 2, which I cannot verify from the toolchain itself: ptxas will happily build for it and tells you nothing about what it is.

If that claim is right, the CUDA binary format has a target for a games console sitting in the same list as your H100.

Share The Software Frontier

Fifteen sections, but only one is code

Below I will show you the full section table of a cubin containing one trivial kernel, a SAXPY with four parameters, compiled for Hopper.

idx name                                   type        off   size  align
  1 .shstrtab                              STRTAB       64    295      1
  2 .strtab                                STRTAB      406    369      1
  3 .symtab                                SYMTAB      776    240      8
  4 .debug_frame                           PROGBITS   1016    104      1
  5 .note.nv.tkinfo                        NOTE       1120    164      4
  6 .note.nv.cuinfo                        NOTE       1284     32      4
  7 .nv.info                               CUDA_INFO  1316     36      4
  8 .nv.compat                             CUDA_COMPAT_INFO
                                                      1352     28      4
  9 .nv.info._Z5saxpyifPKfPf               CUDA_INFO  1380    132      4
 10 .nv.callgraph                          CUDA_CALLGRAPH
                                                      1512     32      4
 11 .rela.debug_frame                      RELA       1544     24      8
 12 .text._Z5saxpyifPKfPf                   PROGBITS   1664    512    128
 13 .nv.shared.reserved.0                   NOBITS     2176      0      1
 14 .nv.constant0._Z5saxpyifPKfPf           PROGBITS   2176    552      4

We have 512 bytes of machine code inside a 3,968 byte file. The italicised sections are the CUDA-specific ones, with section types in the 0x70000000 range that generic ELF tools print as “unknown”.

The code section is said to be “per kernel” and named after the mangled symbol, which is why a file with ten kernels has ten .text.* sections and ten of most other things too. Alignment is 128 bytes, one instruction cache line’s worth of paranoia.

Then, in a random order of how much they surprised me:

.note.nv.tkinfo records how the binary was built

This is a standard ELF note with owner string NVIDIA Corp, and cuobjdump will print it for you.

$ cuobjdump -elf saxpy.sm90.cubin | grep -A5 “CUDA Toolkit Information”
  NVIDIA Corp   140   NVIDIA CUDA Toolkit Information
    Note Version: 2
    Tool Name: ptxas
    Tool Version: Cuda compilation tools, release 13.3, V13.3.73
    Tool Branch: Build cuda_13.3.r13.3/compiler.38244171_0
    Tool Command Line Arguments: -arch sm_90 -m 64

Every cubin has the exact compiler build that produced it and the flags it was invoked with. That's a provenance record, and it’s sitting in every shipped binary on every machine learning platform you have ever installed and used.

If you want to know which toolkit built the kernels inside somebody’s wheel, no need to ask them. There is also a -verbose-tkinfo mode in the assembler that emits, in its own words, “object name and command line arguments which contains all arguments having file format,” meaning paths. I havent seen it on by default, and I’d look before shipping a binary built with it.

.nv.constant0 is the launch frame, and it keeps growing

Kernel parameters don’t come in registers, but in constant bank 0, and the cubin declares a per-kernel section for it. On Hopper we said this section is 552 bytes for a kernel whose parameters occupy 24. The user arguments start at offset 0x210; everything below that is owned by the driver and the ABI.

You can watch the exact boundary move across generations. It is at 0x160 for Turing through Ada, 0x210 for Hopper, and 0x380 for Blackwell and later, with the section growing from 380 to 556 to 924 bytes for the same kernel.

This is the sort of constant that reverse-engineering projects hardcode and then rediscover every two years.

You can see the effect in the disassembly, which is the easiest way I know to make constant bank 0 sort of…. real. Here is the same kernel on Hopper and on Thor.

sm_90                                    sm_110
LDC   R1, c[0x0][0x28]                   LDC   R1, c[0x0][0x37c]
S2R   R0, SR_TID.X                       S2R   R0, SR_TID.X
S2UR  UR4, SR_CTAID.X                    S2UR  UR4, SR_CTAID.X
LDC   R7, c[0x0][RZ]                     LDCU  UR5, c[0x0][0x380]
IMAD  R7, R7, UR4, R0                    LDC   R7, c[0x0][0x360]
ULDC  UR4, c[0x0][0x210]                 IMAD  R7, R7, UR4, R0
ISETP.GE.AND P0, PT, R7, UR4, PT         ISETP.GE.AND P0, PT, R7, UR5, PT
@P0   EXIT                               @P0   EXIT

Left: the first parameter is loaded from 0x210. Right: from 0x380, using LDCU, an instruction that loads a constant straight into a uniform register and that does not exist in the Hopper instruction set reference. Even the stack pointer moved, from 0x28 to 0x37c.

The symbol table has its own vocabulary

Kernels are ordinary global function symbols with two CUDA-specific decorations, but if you look carefully, ther's one symbol in a cubin that has nothing to do with your code.

index  value   size   info  other  shndx  name
  0x3      0      0    0x3      0    0xc  .text._Z5saxpyifPKfPf
  0x4      0    0x4   0x21      0      0  .nv.reservedSmem.offset0
  0x5      0      0   0x20   0xa0    0xd  __nv_reservedSMEM_offset_0_alias
  0x8      0  0x200   0x12   0x10    0xc  _Z5saxpyifPKfPf
  0x9      0      0    0x3      0    0xe  .nv.constant0._Z5saxpyifPKfPf

The kernel symbol carries st_other = 0x10, which nvdisasm prints as STO_CUDA_ENTRY STV_DEFAULT: a visibility byte reused to mark launchable entry points, which is how the driver tells a kernel from a device function without parsing names.

The reserved shared memory symbols are the interesting thing here. Even a kernel that declares “no shared memory” unexpectedly gets a reservation record, and on Blackwell that reservation has a non-zero size: somehw you see 64 bytes of shared memory that belong to the platform, not to you, aliased under a name beginning with __nv_. Anyone computing occupancy from source is computing it from the wrong number.

The same source is not the same program

I decided to write a whole paragraph to the architecture sweep, because it shows how much the cubin is influenced by the specific chip, rather than by your code.

The four-line SAXPY compiles to 24 instruction slots on sm_86 and 48 on sm_87, Orin, and the difference is not optimization.

sm_86 opcode histogram:   8 NOP   2 S2R   2 MOV   2 LDG.E   2 IMAD.WIDE  ...
sm_87 opcode histogram:  15 BMOV.32.CLEAR  12 NOP   2 S2R   2 LDG.E  ...

We see fifteen instructions clearing barrier state at kernel entry, on one embedded part, for a kernel that uses no barriers.

In other words, that piece of code inside the kernel isn’t there because your program desperately needs it. It’s here because NVIDIA is paying, through software, the price of a silicon limitation. And that proves that SASS is dependent not only on the PTX, but also on the hardware, the drivers and the workarounds embedded into the toolchain.

.nv.callgraph exists because device code can call things

Fixed 8-byte entries, and for a leaf kernel it is four rows of negative sentinels:

.nv.callgraph
<0,-1>   <0,-2>   <0,-3>   <0,-4>

The sentinels are the interesting part: a leaf kernel does not get an empty section, but four explicit “no callee of kind N” markers. This suggests that .nv.callgraph is not simply a list of callees, but a structured description of call relationships understood by the runtime.

That metadata matters because features such as indirect calls, dynamic parallelism, and separately compiled device functions require the driver to know the worst-case call depth before launch so it can size the per-thread stack correctly.

Even a kernel that calls nothing still participates in this contract, which is why absence is encoded explicitly rather than by omitting the section entirely.

Subscribe now

.nv.info is where the kernel describes itself

This is the section that let you see a loadable cubin, rather than merely readable. The format is a simple tag-length-value stream: one byte of format, one byte of attribute, then either nothing, one byte, two bytes, or a 16-bit length followed by that many bytes.

And….thirty-six bytes of it, at file level, look like this:

04 2f 08 00 | 08 00 00 00  0a 00 00 00      EIATTR_REGCOUNT
04 11 08 00 | 08 00 00 00  00 00 00 00      EIATTR_FRAME_SIZE
04 12 08 00 | 08 00 00 00  00 00 00 00      EIATTR_MIN_STACK_SIZE
 ^  ^  ^
 |  |  +-- value length (EIFMT_SVAL)
 |  +----- attribute code
 +-------- format: 1 = none, 2 = one byte, 3 = two bytes, 4 = length-prefixed

The first value that you read is a symbol table index, the second is the payload. The symbol 8 in each column is the kernel, and it needs 10 registers, no stack frame, no minimum stack. You can check that decode against the tool, because cuobjdump -elf knows the names.

$ cuobjdump -elf saxpy.sm90.cubin
.nv.info
  Attribute: EIATTR_REGCOUNT       Value: function: _Z5saxpyifPKfPf(0x8)  register count: 10
  Attribute: EIATTR_FRAME_SIZE     Value: function: _Z5saxpyifPKfPf(0x8)  frame size: 0x0
  Attribute: EIATTR_MIN_STACK_SIZE Value: function: _Z5saxpyifPKfPf(0x8)  min stack size: 0x0

The per-kernel section is way richer. Let’s see a four-parameter SAXPY:

.nv.info._Z5saxpyifPKfPf
  EIATTR_LANGUAGE            PTX
  EIATTR_CUDA_API_VERSION    0x85                      (133 = CUDA 13.3)
  EIATTR_KPARAM_INFO         Ordinal 0x3  Offset 0x10  Size 0x8  Space CBANK
  EIATTR_KPARAM_INFO         Ordinal 0x2  Offset 0x8   Size 0x8  Space CBANK
  EIATTR_KPARAM_INFO         Ordinal 0x1  Offset 0x4   Size 0x4  Space CBANK
  EIATTR_KPARAM_INFO         Ordinal 0x0  Offset 0x0   Size 0x4  Space CBANK
  EIATTR_SPARSE_MMA_MASK     0x0
  EIATTR_MAXREG_COUNT        0xff
  EIATTR_MERCURY_ISA_VERSION 1.1
  EIATTR_EXIT_INSTR_OFFSETS  0x70  0x120
  EIATTR_CBANK_PARAM_SIZE    0x18                      (24 bytes of parameters)
  EIATTR_PARAM_CBANK         0x9   0x180210            (symbol 9, size 0x18 at 0x210)
  EIATTR_SW_WAR              0x8
  EIATTR_NVSAL_SW_WAR        0x1

Kernel parameters are described entirely through size-and-offset metadata, with entries stored in something called “reverse ordinal order”.

This means the host runtime doesn’t have to understand the original programming language or the function signature. It can construct the kernel’s parameter buffer directly from the metadata, placing each argument at the correct location before launch. The compiled binary therefore exposes a language-independent ABI that the driver and runtime can consume.

EIATTR_EXIT_INSTR_OFFSETS contains the byte offset of every EXIT instruction generated for the kernel. At first glance this is considered to be like a minor detail, but it’s very useful for tooling.

A profiler, tracer, or instrumentation framework can find every kernel return point immediately, without performing a full disassembly or reconstructing the control-flow graph. If a tool have to inject timing code, coverage probes, or custom telemetry before kernel termination, these offsets provide the exact insertion points.

EIATTR_SW_WAR is a different kind of purpose here. It encodes software workarounds for known hardware errata as a bitmask. Modern GPUs sometimes require compiler- or driver-level mitigations to avoid wrong behavior in specific corner cases.

Rather than hard-coding these decisions somewhere else, the compiler do a record of the workarounds that are needed, in the binary metadata. The driver then inspects the flags at load time and enables the correct mitigation paths for the architecture in place.

All these attributes show you that CUDA metadata is not just a bunch of descriptions; it forms actively some parts of the contract between the compiler, runtime, driver, profiling tools, and the GPU itself, carrying the information needed to launch kernels, instrument execution, and safely navigate hardware-specific quirks.

Now a question arises naturally: how big is this vocabulary? The disassembler contains the enum names, so you can just ask it, and you’ll find out.

$ strings -a nvdisasm | grep -o “EIATTR_[A-Z0-9_]*” | sort -u | wc -l
113

Now you see it: 113 attribute kinds. A few of them are a decent map of what the hardware has learned to do since 2010.

• EIATTR_TCGEN05_1CTA_USED and EIATTR_TCGEN05_2CTA_USED flag use of Blackwell’s fifth-generation tensor cores, and the fact that there are separate flags for the one-CTA and two-CTA forms tells you the driver has to care which.

• EIATTR_CTA_PER_CLUSTER, EIATTR_MAX_CLUSTER_RANK, EIATTR_EXPLICIT_CLUSTER and EIATTR_BLOCKS_ARE_CLUSTERS are the Hopper cluster model.

• EIATTR_STACK_CANARY_TRAP_OFFSETS means device code has stack canaries now, and indeed nvcc has a flag for them.

• EIATTR_COROUTINE_RESUME_ID_OFFSETS is there for something that has not been announced.

• And a small family of attributes are named after bug numbers: EIATTR_SW1850030_WAR, EIATTR_SW2393858_WAR, EIATTR_SW2861232_WAR, EIATTR_WAR5829587_NEEDED.

There is also an internal ticket for each somewhere, and the workaround is load-bearing enough to have its own slot in the binary format.

The format is not a description of a program. It is a contract between a compiler and a driver that ship on different schedules and are written by the same company.

Subscribe now

Twenty-one bits per instruction that the hardware does not check

Starting with Volta, SASS instructions are 128 bits, and a piece of every instruction word is not the instruction at all, because it’s the scheduling metadata that the assembler computes and the hardware obeys without verifying.

You will find plenty of documentation about the structure of the field in the microbenchmarking literature, rather than by the vendor: a paper by Jia and colleagues describes stall counts, a yield flag, separate read and write dependency barriers, a wait mask that is a bitmask because an instruction can wait on several barriers at once, and reuse flags.

The exact bit positions below are the ones I decoded, and I trust them because the result is coherent with the docs: stall count in bits 105 to 108, yield flag at 109, write barrier index at 110 to 112, read barrier index at 113 to 115, a six-bit wait mask at 116 to 121, and four reuse flags at 122 to 125. Bits 126 and 127 come out zero on every instruction of every kernel I looked at, on five architectures.

You can decode it yourself with twenty lines of Python, and I think everyone who works on inference should do it at least once, because the mechanism explains more about the CUDA moat than any benchmark. Here is a real Hopper GEMM prologue with the fields pulled out.

off   stall  y  wr  rd    wait  reuse   instruction
0000      1  1   -   -  000000   0000   LDC R1, c[0x0][0x28]
0010      1  1   0   -  000000   0000   S2R R9, SR_CTAID.Y
0020      1  1   -   -  000000   0000   ULDC UR4, c[0x0][0x228]
0030      1  1   -   -  000000   0000   ULDC.64 UR6, c[0x0][0x208]
0040      1  1   -   -  000000   0000   MOV R0, UR4
0050      1  1   0   -  000000   0000   S2R R13, SR_TID.Y
0060      2  1   -   -  000000   0000   HFMA2.MMA R16, -RZ, RZ, 0, 0
0070      1  1   -   -  000000   0000   ISETP.GE.AND P0, PT, R0, 0x1, PT
0080      1  1   1   -  000000   0000   S2R R17, SR_CTAID.X
0090      1  1   1   -  000000   0000   S2R R7, SR_TID.X
00a0      2  0   -   -  000001   0000   LEA R9, R9, R13, 0x5
00b0      8  0   -   -  000010   0000   LEA R11, R17, R7, 0x5

Two special register reads allocate barrier 0, and the LEA that consumes them waits on barrier 0. Two more allocate barrier 1, and the next LEA waits on barrier 1. The dependency is not discovered at run time, but written into the binary.

That is the whole argument, in twelve instructions. For variable-latency instructions the assembler allocates one of six barriers and makes consumers wait on a bitmask.

For fixed-latency instructions it does not use barriers at all, it just writes a number of cycles into the stall field and the scheduler holds off that long. There is no interlock checking this: if the number is wrong, you do not get a stall, you get wrong answers.

Across a full tiled GEMM kernel the distribution looks like this.

The linker can rewrite the control bits

The relocation namespace makes the point better than I can. Pull the type names out of nvlink and there are 119 of them, and most encode a bit position in the name.

$ strings -a nvlink | grep -o “R_CUDA_[A-Z0-9_]*” | sort -u | wc -l
119

R_CUDA_ABS32_23        patch 32 bits at bit offset 23
R_CUDA_ABS32_HI_32     high half of a 64-bit address, at bit 32
R_CUDA_CONST_FIELD19_20  19-bit constant bank field at bit 20
R_CUDA_ABS55_16_34     55-bit value split across bits 16 and 34
R_CUDA_INSTRUCTION128  a whole 128-bit instruction word
R_CUDA_PCREL_IMM24_23  24-bit program counter relative branch at bit 23
R_CUDA_YIELD_CLEAR_PRED4_87   clear 4 bits at bit 87
R_CUDA_YIELD_OPCODE9_0        rewrite a 9-bit opcode field at bit 0

On a CPU, relocations patch whole bytes because operands are byte-aligned. In this case, the operands are bit fields inside a 128-bit word, so the relocation type has to name the field, and there is a separate type for the same width at each position it can occur.

The last two are the ones that really stopped me. A relocation that clears the yield predicate, and one that rewrites an opcode, mean the device linker’s contract includes editing scheduling and instruction selection after the assembler has finished.

The 21 control bits are not a compiler-internal detail, which is nested at the end of ptxas, because they are a key part of the object format, and a later stage is allowed to change them.

I have argued many times before that this field, not the language and not PTX, is the load-bearing part of the CUDA moat, and taking cubins apart has not changed my mind. It has sharpened one point though.

The reason the assembler is fully close is not that NVIDIA wants to hide optimizations and reverse-engineering, at least not the first point. What we also need to keep in mind is that the ptxas is correctness-critical infrastructure.

An external tool that emits SASS is not competing with a compiler on quality of code, it is taking over responsibility for a hazard-avoidance protocol with no runtime check and no error reporting, which is extremely dangerous.

NVIDIA’s own programming guide is blunt about the consequence: you read that binary compatibility is promised only for binaries created by NVIDIA tools: manual editing or generating binary code is not supported, and compatibility promises are invalidated if binaries are modified in any way.

Read that as a description of the support boundary rather than a threat, and the closure looks less like strategy and more like the only sentence a vendor can write once it has moved the interlock into software.

Share The Software Frontier

Blackwell cubins carry a second copy of every kernel

When I thought about writing this part, I didn’t even know what to expect. It was a complete surprise to me. Let’s do a concrete example: compile the same trivial kernel for Hopper and for Blackwell and count sections: 15 for sm_90, 22 for sm_100. The seven extra sections are not debug info.

idx name                                   type              flags     size
 12 .text._Z5saxpyifPKfPf                  PROGBITS              6      512
 13 .nv.shared.reserved.0                  NOBITS                3       64
 14 .nv.constant0._Z5saxpyifPKfPf          PROGBITS             42      920
 15 .nv.capmerc.text._Z5saxpyifPKfPf       0x70000016     10000000      258
 16 .nv.merc.debug_frame                   PROGBITS       10000000      112
 17 .nv.merc.nv.info                       0x70000083     10000000       36
 18 .nv.merc.nv.info._Z5saxpyifPKfPf       0x70000083     10000040      144
 19 .nv.merc.rela.debug_frame              0x70000082     10000040       24
 20 .nv.merc.nv.shared.reserved.0          0x70000015     10000003        0
 21 .nv.merc.symtab                        0x70000085     10000000      216

You are now staring at a complete parallel object: its own symbol table, its own info sections, its own relocations, its own shared memory reservation, and a “capmerc” text section. Section flag bit 0x10000000 marks the whole family.

The prefix is merc, and the name shows up in three other places. It is in the per-kernel info as EIATTR_MERCURY_ISA_VERSION 1.1. It is in a new section called .nv.compat, which exists on every target but says more on Blackwell, and it is all over the assembler binary.

$ cuobjdump -elf saxpy.sm100a.cubin | sed -n ‘/nv.compat/,/nv.info\._/p’
.nv.compat
  EICOMPAT_ATTR_CUDA_ACCELERATOR_TARGET                 0x1
  EICOMPAT_ATTR_ISA_CLASS                                0x1
  EICOMPAT_ATTR_INST_TCGEN05_MMA                        0x5
  EICOMPAT_ATTR_MERCURY_ISA_MAJOR_MINOR_VERSION_V2       1.1
  EICOMPAT_ATTR_MERCURY_ISA_MAJOR_MINOR_VERSION_V1       1.1
  EICOMPAT_ATTR_INST_TENSORMAP_V1                        0x0
  EICOMPAT_ATTR_CAN_FASTPATH_FINALIZE                   0x9 0x0

There is the missing suffix. EICOMPAT_ATTR_CUDA_ACCELERATOR_TARGET is 1 for sm_100a and 0 for both sm_100 and sm_100f, and the accelerated build declares EICOMPAT_ATTR_INST_TCGEN05_MMA, a capability it was permitted to use even though this kernel does not use it.

To really find out what the block really tracks, you need to build a kernel that needs a family-specific instruction. One line of inline PTX will do: tcgen05.fence::before_thread_sync, part of Blackwell’s fifth-generation tensor core interface.

Compiling it for five targets gives the feature tiers as a straight experiment rather than as a diagram.

$ for a in sm_100 sm_100f sm_100a sm_103f sm_120a; do nvcc -arch=$a -cubin -o t.cubin tc.cu; done

sm_100    Instruction ‘tcgen05.fence’ not supported on .target ‘sm_100’
sm_100f   OK  (5136 bytes)
sm_100a   OK  (5136 bytes)
sm_103f   OK  (5136 bytes)
sm_120a   Instruction ‘tcgen05.fence’ not supported on .target ‘sm_120a’

The baseline target refuses the instruction, but the family target accepts it, and so does the family target for the other member of the same family.

And the fully accelerated target for consumer Blackwell refuses it, because sm_120 is a different family and doesn’t have the hardware, which is the concrete answer to the frequently asked question of why a 5090 is not a small B200.

Now look at what the successful builds wrote into the object.

That is the piece I was missing when I first wrote this section, infact I had to rebuild it. The compat block is not a record of your compiler flags; it is a graded statement about what the machine code inside requires, computed from the code, and the flags only set the ceiling on what the assembler was allowed to reach for.

A resolver reading ISA_CLASS = 1 knows this object is safe anywhere in the generation; reading 2, it knows to check the family. This is the minimum viable metadata for the compatibility promise NVIDIA made in CUDA 12.9, and it is sitting in a 32-byte section nobody documents.

What reads it becomes obvious once you look inside ptxas. The strings are unambiguous.

$ strings -a ptxas | grep -iE “merc|finaliz” | sort -u
[Finalizer] fastpath optimization applied for off-target %u -> %u finalization
Generate Capsule Mercury
Specify the type of target ELF binary kind. Default on sm100+ is capmerc
Self check for capsule mercury (capmerc)
Specify the opportunistic finalization level. 0=default, 1=no opportunistic
  finalization, 2=intra family finalization only, or 3=intra and inter family
  finalization
Turns off the fast-path finalization optimization (allows normal refinaization)
Specify the ‘sm_’ name of the target architecture. If not specified, default
  behavior is on-target finalization
R_MERCURY_NONE  R_MERCURY_G64  R_MERCURY_ABS64  R_MERCURY_ABS32  R_MERCURY_ABS16 ...

Read together with the sections, this describes a two-stage back end. Mercury is an encoding of a kernel that is below PTX, but above final machine code.

A capsule wraps it with the metadata a finalizer needs: its own symbol table, register and barrier counts, shared memory usage, and its own relocation types. “Finalization” is the step that turns a capsule into executable SASS for a specific chip. On-target finalization is the ordinary case.

Off-target finalization is the same capsule being finalized for a different chip than the one it was compiled for, with a fast path when the assembler can prove the existing encoding is already valid. And the levels of “opportunistic finalization” go from none, to within a family, to across families.

That is what the f targets are made of. NVIDIA introduced family-conditional compilation in CUDA 12.9 and actually described it in terms of feature sets: an f binary may use the architecture-specific features that are common to a whole family, and will run on later members of that family.

The public framing is a compatibility promise. The mechanism, as far as I can see it in the binary, is that the cubin ships a re-finalizable representation next to the SASS, and the driver can produce correct machine code for a family member the compiler never saw.

Where I am reading, not knowing

Unfortunately, I havent observed finalization happen. At the moment, don’t own a Blackwell GPU, and none of this machinery has the ptxas command line: passing --binary-kind gets you Unknown option, so the option table I am quoting belongs to an internal or driver-side entry point.

What I am confident about is the presence and shape of the artifacts. An independent reverse-engineering effort on ptxas 13.0 reached the same reading and adds detail I could not verify, including a 328-byte capsule descriptor and a compilation-knob snapshot inside it. Treat the purpose as tier C, so a speculative claim.

There is more of it in the section type table than any single cubin shows. cuobjdump has to be able to print a name for every section type it might meet, so the names are in the binary, and the Mercury family is large.

$ strings -a cuobjdump | grep -oE “CUDA_[A-Z_]{3,}” | sort -u | grep -i merc
CUDA_CAPMERC
CUDA_MERCURY
CUDA_MERCURY_CONSTANT_DRIVER      CUDA_MERCURY_CONSTANT_PARAMS
CUDA_MERCURY_CONSTANT_IMGHDR      CUDA_MERCURY_CONSTANT_PIC
CUDA_MERCURY_CONSTANT_OPT         CUDA_MERCURY_CONSTANT_TOOLS
                                  CUDA_MERCURY_CONSTANT_USER
CUDA_MERCURY_RESOLVED_RELA
CUDA_MERCURY_SASS_MAP

Seven distinct constant bank section types, split by who owns the data: driver, image header, optimizer, parameters, position independent code, tools, user. A resolved-relocation type. And a section type called CUDA_MERCURY_SASS_MAP, which is hard to read as anything other than a correspondence between Mercury entities and SASS.

A format that needs a map to SASS is not an annotation on SASS, because it’s a representation of the program in its own right, and the map exists so that debuggers, profilers and the finalizer can move between the two.

All of which makes the size behaviour the really awkward part, because it argues against the simplest reading of what I just described.

If the capsule were a second copy of the instruction stream, in whatever denser encoding, its size would have to follow the SASS. Longer kernel, longer capsule. That is not what happens. Across seven kernels the SASS spans 384 to 2,048 bytes, a factor of 5.3, while the capsules span only 130 to 350, a factor of 2.7.

Correlations are carried almost entirely by a single long kernel: the coefficient across all seven is 0.76, and dropping that one point takes it to 0.33. Now, if you rank the kernels by capsule size and you get an ordering with no obvious relationship to how much code they contain.

The cleanest way to see it is a controlled pair. Two of these kernels compile to exactly 384 bytes of SASS. One stores a float, the other multiplies and adds in double precision. Same code size, same instruction count, and their capsules are 130 and 194 bytes, a 49% difference. Meanwhile a kernel using sqrtf compiles to 896 bytes of SASS, more than twice the double-precision kernel, and carries a smaller capsule at 184.

I want to be careful here, because when I first wrote this section I claimed the ordering tracked the variety of operations a kernel performs, and then I checked. It does not. Correlating capsule size against the number of distinct opcodes in each kernel gives 0.03, which is nothing at all. I could not find anything that predicts capsule size, and with seven kernels I would not trust a pattern even if I had found one.

What the double-precision pair does suggest is that whatever the capsule enumerates is closer to a set of requirements than to a program, and double precision is a plausible thing to find in such a set. Double-precision throughput is one of the sharpest differences within a Blackwell family: the datacenter parts and the consumer parts are not the same machine in that respect.

A representation whose job is to let a driver produce correct code for a family member the compiler never saw would need to record exactly this sort of thing, and would not need to grow much when you add another hundred instructions of the same kind.

So the sizes push against “a second copy of every instruction” and toward “a description of what this code needs.” They cannot separate the two readings that remain. A compact whole-kernel encoding in a much denser format, and a fixup table over the existing SASS naming only the sites a re-finalizer would have to revisit, would both be small and would both fail to scale with instruction count.

The section type called CUDA_MERCURY_SASS_MAP is the one piece of evidence that leans toward the first, since a table of fixups would not obviously need a map. I cannot settle it from sizes alone, and I doubt anyone can without a driver to watch.

What it costs is easier to state. On the ten-kernel workload, Mercury sections add 4.2 KB to a 57.8 KB cubin, about 7%, and Blackwell cubins are 2.0x the size of Turing cubins for identical source.

There is one more hint about direction of travel, in the disassembler rather than the assembler. nvdisasm in 13.3 has an option called --no-vliw, described as “conventional mode; disassemble paired instructions in normal syntax, instead of VLIW syntax.”

Nothing I compiled for any current target produced paired output. An option to turn off VLIW syntax implies a target where VLIW syntax is the default.

I would not build a thesis on a command-line flag, but if a future architecture issues statically paired instructions, the tooling has already been taught the word.

Subscribe now

The fatbin is a filesystem, and the driver is the resolver

A cubin runs on exactly one compute capability. Shipping software means shipping several, plus PTX for the ones that do not exist yet, and the container for that is the fatbin.

It starts with a 16-byte header whose magic is 0xBA55ED50, and it is followed by a run of entries, each with its own header and payload.

$ nvcc -gencode arch=compute_90,code=sm_90 \
       -gencode arch=compute_100,code=[sm_100,compute_100] -fatbin -o saxpy.fatbin saxpy.cu

kind     arch  hdrsz  payload   compressed  uncompressed  flags
CUBIN      90     64     3968            0             0  0x11
CUBIN     100    112     5712            0             0  0x1000011
PTX       100     80      400          399           922  0x8011

The entry header fields are recoverable by differential compilation: change one thing about the build, diff the bytes, name the field.

Reconstructed by construction, not from documentation. Every row was fixed by building the same kernel with one thing changed and diffing the header bytes. Fields I did not manage to move are omitted.

PTX is not the cheap option

The standard advice is to always ship PTX for the newest virtual architecture so future GPUs have something to JIT. The advice generally is right, and the reason people give for it is usually wrong.

PTX is not a compact representation. For the ten-kernel workload the PTX is 37.3 KB against a 35.9 KB Hopper cubin, and on Blackwell it is 48.0 KB against 57.8 KB. It is the same order of magnitude in both directions, and after compression the gap narrows further because PTX is text and compresses beautifully.

What PTX buys is coverage of chips that do not exist yet, at the cost of a JIT compile on first use, per process, unless the driver’s compute cache is warm.

What it does not buy is what most people think: forward compatibility does not extend to the instructions anyone actually cares about, because PTX built for an architecture-conditional target is not forward compatible at all.

If your kernel needs wgmma or tcgen05, it needs sm_90a or sm_100a, and the compatibility story ends there. The f targets exist precisely because that cliff was too steep, and the Mercury capsule is how they made it less steep.

That extended header is the good part. The sm_100 entry above has a 112-byte header instead of 64, and the extra 48 bytes are a length-prefixed copy of the .nv.compat capability block from inside the cubin.

header bytes at offset 0x40:
48 00 00 00  20 00 00 00     -- payload at 0x48, length 0x20
02 09 00 00  02 02 01 00  03 0d 01 01  03 07 01 01
02 03 00 00  04 0b 08 00  09 00 00 00  00 00 00 00
   ^^ the same TLV stream that .nv.compat holds inside the ELF

The capability claim is duplicated into the index so that whatever picks an entry never has to open the ELF. That is a loader optimization, and it is also a small confirmation of what the compat block is for: it is the thing a resolver matches against a device before committing to a payload.

Compression is off by default for the part that matters

CUDA has a --compress-mode flag with values none, speed, balance, size and default. Running all five over the same two-cubin-plus-PTX fatbin gives a result worth knowing.

The flag values behave like two different algorithms rather than one algorithm at three settings: speed sets flag bit 13 and gets 504 bytes out of 922, while balance and size set bit 15 and get 400.

On the larger ten-kernel workload across twelve targets, size mode takes 537 KB down to 75 KB, an 86% reduction, for no measurable build time cost at this scale.

If you ship CUDA binaries and have not tried it, that is the cheapest win in this article.

How it gets into your executable

The host side is one of the few places NVIDIA actually publishes the format, in fatbinary_section.h in the toolkit include directory.

#define FATBINC_MAGIC   0x466243B1
#define FATBINC_VERSION 1
typedef struct {
  int magic;
  int version;
  const unsigned long long* data;
  void *filename_or_fatbins;
} __fatBinC_Wrapper_t;

#define FATBIN_CONTROL_SECTION_NAME  “.nvFatBinSegment”
#define FATBIN_DATA_SECTION_NAME     “.nv_fatbin”

Compile a .cu file to an object and you get exactly that: a 24-byte wrapper in .nvFatBinSegment holding the magic and a relocation pointing at the fatbin bytes in .nv_fatbin, plus a section called __nv_module_id, plus a constructor.

$ readelf -x .nvFatBinSegment saxpy.o
  0x00000000 b1436246 01000000 00000000 00000000 .CbF............
$ readelf -rW saxpy.o | grep nvFatBinSegment
  R_X86_64_64  .nv_fatbin + 0

$ cat saxpy.cudafe1.stub.c        # generated by nvcc --keep
static void __nv_cudaEntityRegisterCallback(void **__T0) {
  __cudaRegisterEntry(__T0, (void(*)(int,float,const float*,float*))saxpy,
                      _Z5saxpyifPKfPf, (-1)); }
static void __sti____cudaRegisterAll(void) __attribute__((__constructor__));
static void __sti____cudaRegisterAll(void) {
  __cudaRegisterBinary(__nv_cudaEntityRegisterCallback); }

void __device_stub__Z5saxpyifPKfPf(int p0, float p1, const float *p2, float *p3){
  __cudaLaunchPrologue(4);
  __cudaSetupArgSimple(p0, 0UL);  __cudaSetupArgSimple(p1, 4UL);
  __cudaSetupArgSimple(p2, 8UL);  __cudaSetupArgSimple(p3, 16UL);
  __cudaLaunch((char *)saxpy); }

So a CUDA program registers its device code from an ELF constructor, before main. The argument offsets in the stub, 0, 4, 8 and 16, are the same offsets the cubin declared in EIATTR_KPARAM_INFO.

Host and device agree on the layout because both were generated from the same front end, and the binary carries the agreement in two places.

Two variants are worth noting because they trip people up. With -rdc=true, the device code is relocatable, the cubin becomes ET_REL with .rela.text.* sections and an EIATTR_EXTERNS attribute, and it lands in a differently named section, __nv_relfatbin, until the device link step turns it into a normal .nv_fatbin. The cost shows up in the machine code, and it is not subtle:

$ nvcc -arch=sm_90 -rdc=true -c a.cu b.cu && nvcc -arch=sm_90 -dlink -o dl.o a.o b.o
$ cuobjdump -sass dl.o
    Function : _Z5applyPfPKffi
        /*0100*/  CALL.ABS.NOINC 0x0 ;
    Function : _Z5scaleff
        /*0000*/  FFMA R4, R4, R5, 1 ;
        /*0010*/  RET.ABS.NODEC R20 0x0 ;

A one-instruction device function became a real call, with a stack frame, a return address register and a separate .text section, where whole-program compilation would have inlined it into a single FFMA.

That is what -dlto exists to undo: with LTO the fatbin entry is kind 8 and the instruction selection is deferred to link time so the inlining can happen after all translation units are visible.

In my sandbox the LTO device link fails for want of a driver-side link library, so I can report the container shape (1,944 compressed bytes expanding to 2,548) but not the linked output.

What the driver does with all this

The loading path is the reason the format looks the way it does, and it is worth stating plainly because the layering is easy to lose. The constructor calls __cudaRegisterBinary with a pointer to the wrapper.

The runtime hands the fatbin to the driver, which reads the index and selects one payload for the device it is about to use: an exact-architecture cubin if there is one, otherwise a compatible one, otherwise PTX to be compiled on the spot. The selected payload becomes a module.

The module’s .nv.info tells the driver how many registers each kernel needs, how much shared memory to reserve, how big a stack to allocate given the call graph, and where in constant bank 0 to write the arguments. Only then can a launch happen, and a launch is essentially a memcpy of the argument block into the frame the cubin described, followed by a grid dispatch.

Everything in the binary that is not machine code exists to let that sequence happen without the driver consulting anything else. That is why the parameter layout is in the object rather than in a header, why the exit offsets are precomputed, why the capability claim is duplicated into the fatbin index, and why the reserved shared memory has a symbol. A loader that had to infer any of it would be slower and more fragile.

Two changes to this path are worth knowing. Since CUDA 12.0 there is a second module abstraction, the library and kernel API (cuLibraryLoadData, cuKernel), which decouples a loaded artifact from a specific context, and it exists largely because the old model made lazy loading awkward.

And since 12.2 on Linux and 12.3 everywhere, lazy loading is the default: modules load on first use of a symbol from them, and kernels within a module load on first cuModuleGetFunction.

The requirement is a runtime of 11.7 or later, statically linked into whatever built the library, which means an old dependency in your stack can quietly opt you out. NVIDIA’s own documentation puts the requirement plainly: libraries compiled against older runtimes load all modules eagerly.

Subscribe now

What the format costs, in bytes and seconds

All of the above has a price, and it is paid by everyone who ships a wheel.

This is why PyTorch’s release engineering discussions read the way they do. When the CUDA 12.8 build matrix had to add Blackwell, the team explicitly rejected keeping the older architectures because of binary size. The trade is not subtle: adding a target adds a copy of every kernel in the library, and machine learning libraries have a great many kernels.

You can see the endpoint of that logic by taking apart what NVIDIA itself ships. I pulled the cuBLAS 13.6.0.2 wheel from PyPI and parsed the .nv_fatbin sections directly.

Two libraries, 6,860 fatbin entries, 178 MB on disk expanding to 1.8 GB of device code. And every single entry is compressed, which tells you that NVIDIA does not ship its own libraries with the default compression mode.

The right-hand panel is worth sitting with. In cuBLASLt, 101 MB is host code, and 78 MB is a section called .cask_resource.

One detail worth noting before anyone adds those numbers up: the library also declares 99 MB of .bss and .lbss, and those are NOBITS sections, which means they cost 99 MB of memory at run time and zero bytes on disk.

Counting only the sections that actually occupy file space gets you 491.4 MB of the 492.7 MB file, and the rest is headers and padding. I cannot tell you what CASK stands for, because NVIDIA has never said, and the name only surfaces publicly in cuDNN and TensorRT diagnostics.

What I can tell you is that the strings in that section include 2,429 mangled C++ names beginning N11cask, so it is a namespace, and that the rest of the section is a catalog. Pull strings out of it and you get entries like this:

cutlass3x_sm103_bstensorop_s256x256x96gemm_block_scaled_ue4m3xe2m1_ue4m3xe2m1_
  f32_bf16_ue4m3xe2m1_256x256x768_0_tnn_align...

A CUTLASS 3.x kernel for sm_103 doing a block-scaled GEMM with FP8 and FP4 operand types at a 256x256x96 tile.

Counting distinct kernel-shaped names across the resource and read-only sections gives 25,401, with the generation prefixes still visible in the histogram: 5,329 mentioning sm90, 5,015 cutlass3x, 1,956 ampere, 808 volta, 274 turing.

That number is the moat, expressed as an artifact count rather than an argument. Not one clever kernel. Twenty-five thousand parameterised ones, plus 78 MB of metadata describing which to pick, plus 101 MB of host code doing the picking. A competitor can match the code generator.

Matching the catalog and the selection heuristics is a different kind of work, and it is the kind that does not benefit much from being smart.

NVIDIA is visibly feeling the weight of it. cuDNN 9.5 shipped a build configuration called GRAPH_JIT_ONLY that keeps the runtime kernel generation engines and drops the precompiled ones specifically to cut binary size, which is the vendor arriving at the same trade its users have been making by hand: generate at run time or ship the archive.

It also explains why lazy loading became the default. If a process eagerly loaded all 5,309 cubins in cuBLASLt it would spend its startup budget on kernels it will never call.

Lazy loading arrived as opt-in in CUDA 11.7, became the Linux default in 12.2 and the universal default in 12.3, and defers module and kernel loading until first use.

In a world where a single library holds thousands of independently loadable objects, that stops being an optimization and becomes a requirement.

Debug information is not free either

For the same ten kernels on sm_90: 35.9 KB release, 75.6 KB with -lineinfo, 205.9 KB with -G. The last one is not just extra sections.

It changes code generation, taking .text from 16.0 KB to 57.5 KB, because full device debug turns off most optimization. -lineinfo is the option you want in production builds: it doubles the cubin and leaves the code alone.

Share

Why any of this matters if you are not writing a compiler

Three reasons, in increasing order of how much I care about them.

The first is operational. Binary size and load time are real costs in inference deployments, container images and cold starts, and they are almost entirely determined by choices in the format: how many targets, PTX or not, which compression mode, lazy or eager. Those are four flags. Most teams have never touched them.

The second is diagnostic. A cubin tells you its register count, its shared memory usage, its stack frame, its parameter layout, the toolkit that built it and the flags it was built with, and you can read all of that out of a shipped wheel without running anything.

If you are trying to work out why somebody else’s kernel spills, or which architectures a vendor actually optimized for versus which they merely support, the binary is a better source than the changelog. The -res-usage flag on cuobjdump is the fastest occupancy audit I know.

Auditing somebody else’s wheel

cuobjdump -lelf libfoo.so          # which architectures, how many cubins
cuobjdump -lptx libfoo.so          # is there PTX for anything newer
cuobjdump -res-usage libfoo.so     # registers, shared memory, stack, per kernel
cuobjdump -elf libfoo.so | grep -A5 “Toolkit Information”   # which ptxas built it
readelf -SW libfoo.so | grep nv_fatbin                      # how much of the file is device code

Five commands, no GPU, no source. The last one is the one that surprises people: if .nv_fatbin is 85% of a library, you are shipping a kernel archive with a thin API on top, and any conversation about image size has to start there.

The third is strategic, and it is why I went looking in the first place.

There is a long line of work reverse-engineering these formats: decuda for the earliest chips, asfermi for Fermi, KeplerAs, Scott Gray’s maxas for Maxwell (the toolchain behind the fast convolution kernels of the deep learning era), turingas, CuAssembler for everything through Ampere, NVBit for dynamic instrumentation, and now work lifting SASS back into typed compiler IR.

Every one of these projects had to rediscover the container before it could touch the code, and every one of them targets an architecture at least two generations behind current silicon. That lag is the moat’s actual shape.

It is not that the format is unknowable, instead it’s that knowing it is a full-time job that resets on a yearly cadence, and the number of people doing it is small enough to name.

Mercury is the part of this that I think changes the picture, and not in the direction I expected. Publishing an interface and keeping the lowering is NVIDIA’s oldest move: PTX is public, ptxas is not.

What the capsule does is insert a second, private layer at exactly the point where a competitor would want to attach, and it does it for a reason customers asked for: one binary that keeps working on the next chip in the family. The compatibility benefit is real.

The side effect is that the interesting representation of a Blackwell kernel is no longer the SASS you can disassemble, and the format that carries it has no public name, no documented encoding, and its own relocation types.

Subscribe now

Four things I would change, and one I would not

Having read a few hundred of these files, I have opinions about the format that have nothing to do with the moat.

Publish the metadata, keep the encoding. There is no competitive information in EIATTR_KPARAM_INFO. It is a calling convention. Documenting the .nv.info attribute codes and the fatbin entry header would cost NVIDIA nothing and would stop every profiler, virtualization layer, checkpointer and build tool from rediscovering the same twenty structs. The instruction encoding is a different argument and I understand the refusal. The container is not.

Make size the default compression mode. The current default compresses PTX and leaves the SASS raw, which is the wrong way round: SASS is the bulk and the part that never gets read by a human. NVIDIA already ships its own libraries with everything compressed. Every framework wheel on PyPI is paying for a default its vendor does not use.

Give the toolkit a way to strip PTX. There is nvprune for removing architectures, but the common case in a container build is “I know exactly which GPUs this image runs on, remove the JIT fallback.” Doing that today means unpacking and repacking fatbins with nvFatbin, which is a strange amount of work for what should be a link flag.

Version the format visibly. Between the ABI version byte, the OS ABI byte that changed value, the toolkit note, the API version attribute and the Mercury ISA version, there are five different notions of “which format is this” in one file, and no single field that a tool can check to know whether it will understand what follows. A cubin that a future tool cannot parse should say so in its first sixteen bytes.

What I would not change personally is the decision to make the binary self-describing rather than pushing the information into headers or a side-channel.

It is the reason NVBit can instrument a kernel it has never seen, the reason cuobjdump -res-usage works on a stranger’s wheel, and the reason this article was possible on a machine with no GPU in it. Verbose, redundant, self-describing binaries are a gift to everyone downstream, including the people trying to work out what you shipped.

Most of what is wrong with the CUDA binary format is that it is undocumented. Very little of it is that it is badly designed.

Where I might be wrong

• The capsule might be much less than I think. If it turns out to be a small fixup table rather than a re-encodable program, then “second copy of every kernel” is the wrong phrase and the right one is “compatibility annotations.” The sublinear size scaling in the capsule plot above is evidence for the smaller reading; the existence of a section type called CUDA_MERCURY_SASS_MAP is evidence for the larger one. I could not settle it.

• I have no Blackwell hardware. Nothing here was executed. A claim about what a driver does with a capsule is, from my position, a claim about what an artifact appears designed for. Everything about run time behaviour, including JIT cost and the actual effect of lazy loading, is cited rather than measured.

• The OS ABI byte is a mess and I may be reading the mess wrong. LLVM lists 51 and 41; cubins carry 65. My reading is that the second constant is 0x41 written as decimal, but it is equally possible NVIDIA changed the value again and LLVM has not caught up. I only checked 13.3 output.

• The fatbin flag bits are inferred from a handful of data points. I set them by construction using the compression modes and the LTO path; other bits in that word certainly mean things I did not exercise. Byte 0 of e_flags, which flips from 4 to 2 at Blackwell, I could not explain at all.

• My library numbers are one version of one vendor library. cuBLAS is the extreme case for kernel count. Generalizing “27% device code” to other libraries would be wrong.

• The feature-tier experiment used one instruction. tcgen05.fence behaves as described, and I have no reason to think the mechanism differs for other family-specific instructions, but one probe is one probe.

Predictions

1. By the end of 2027, at least one open-source project will publish a working parser for the Mercury capsule format, and it will be motivated by GPU virtualization or checkpointing rather than by performance.

2. NVIDIA will not document the capsule encoding, on the same terms it has never documented SASS encodings, through at least CUDA 15.

3. By the end of 2027, -compress-mode=size or its equivalent will be the default in at least one major framework’s release build, driven by wheel size limits rather than by anyone reading the flag documentation.

4. The next architecture after Blackwell will ship a SASS encoding with statically paired instructions, and nvdisasm‘s VLIW syntax will be why we find out.

5. Family-conditional targets will absorb the architecture-conditional ones in practice: by the end of 2028, the fraction of shipped cubins carrying a suffixes will fall as libraries move to f, because one binary per family is a better deal than one per chip.

Subscribe now

Dossier

Tier A, measured directly and reproducible from the appendix.

• ELF header field values and the e_flags architecture encoding, including the absence of the virtual architecture from that word.

• Section inventories and sizes for all twelve targets.

• The .nv.info and .nv.compat attribute contents as decoded by cuobjdump, including the ISA_CLASS transition when a family-specific instruction is used.

• The rejection of tcgen05.fence on sm_100 and sm_120a and its acceptance on sm_100f, sm_100a and sm_103f.

• The 113 attribute names and the 119 relocation names. Parameter base offsets and constant bank sizes.

• Control field bit positions and their per-kernel distribution.

• Fatbin entry field offsets, compression flag bits and all size and timing measurements.

• The CALL.ABS.NOINC cost of relocatable device code.

• The cuBLAS section and entry statistics, the 25,401 kernel names and the 2,429 cask mangled names.

• The presence, sizes and scaling of the Mercury sections.

Tier B, documented by NVIDIA, in shipped headers, or in third-party source.

• The fatbin wrapper struct and section names (fatbinary_section.h).

• The family and architecture-conditional target semantics, including the statement that sm_100 and sm_100f are aliases absent family features (Programming Guide, PTX ISA, the CUDA 12.9 blog post).

• Lazy loading history and defaults.

• Dropped architecture support in CUDA 13.0 and the Thor renumbering.

• EM_CUDA, ELFOSABI_CUDA and ELFOSABI_CUDA_V2 in LLVM.

• cuDNN’s GRAPH_JIT_ONLY configuration.

• The structure of the control field, which is documented in the literature rather than by the vendor.

Tier C, inferred from artifacts.

• Mercury is a re-finalizable intermediate representation and that the capsule exists to serve family-conditional compatibility.

• That the duplicated compat block in the fatbin entry header is a loader fast path.

• ptxas‘s hidden finalizer options correspond to a driver-side code path.

• .cask_resource is a kernel selection catalog.

• That the ELFOSABI_CUDA_V2 constant is a hexadecimal value written as decimal.

Tier D, speculation, flagged as such in the text.

• --no-vliw implies a future paired-issue ISA.

• EIATTR_COROUTINE_RESUME_ID_OFFSETS points at an unannounced feature.

• sm_88 is the Nintendo Switch 2, which comes from community references and not from anything I could check.

Subscribe now

Reproduce this

No GPU required. On x86-64 Linux:

pip install nvidia-cuda-nvcc==13.3.73 nvidia-cuda-cuobjdump==13.3.73 \
            nvidia-cuda-nvdisasm==13.3.73 nvidia-cuda-crt==13.3.73 \
            nvidia-cuda-runtime nvidia-cuda-cccl
export PATH=$(python3 -c “import sysconfig,os;print(os.path.join(sysconfig.get_paths()[’purelib’],’nvidia/cu13/bin’))”):$PATH

cat > saxpy.cu <<’EOF’
#include <cuda_runtime.h>
__global__ void saxpy(int n, float a, const float* x, float* y) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) y[i] = a * x[i] + y[i];
}
EOF

# the container and its metadata
nvcc -arch=sm_90  -cubin -o k90.cubin  saxpy.cu
nvcc -arch=sm_100 -cubin -o k100.cubin saxpy.cu
cuobjdump -elf k90.cubin | less
diff <(cuobjdump -elf k90.cubin | sed -n ‘/^Sections:/,/^$/p’) \
     <(cuobjdump -elf k100.cubin | sed -n ‘/^Sections:/,/^$/p’)

# the capability claim, and the accelerator bit
for a in sm_100 sm_100a sm_100f; do nvcc -arch=$a -cubin -o c.cubin saxpy.cu
  echo “== $a”; cuobjdump -elf c.cubin | grep -A1 EICOMPAT | grep -E “Attr|Value”; done

# the attribute namespace and the finalizer options
strings -a $(command -v nvdisasm) | grep -o “EIATTR_[A-Z0-9_]*” | sort -u
strings -a $(command -v ptxas)    | grep -iE “capmerc|finaliz|R_MERCURY”

# the feature tiers, as an experiment rather than a diagram
cat > tc.cu <<’EOF’
#include <cuda_runtime.h>
__global__ void k(unsigned* o){
  asm volatile(”tcgen05.fence::before_thread_sync;” ::: “memory”);
  o[0]=1;
}
EOF
for a in sm_100 sm_100f sm_100a sm_103f sm_120a; do printf “%-9s “ $a
  nvcc -arch=$a -cubin -o t.cubin tc.cu 2>&1 | head -1; done
for a in sm_100f sm_100a; do nvcc -arch=$a -cubin -o t_$a.cubin tc.cu
  echo “== $a”; cuobjdump -elf t_$a.cubin | grep -A2 EICOMPAT | grep -E “Attribute|Value”; done

# is the virtual architecture in e_flags? (no)
for v in 75 80 90; do nvcc -gencode arch=compute_$v,code=sm_90 -cubin -o v.cubin saxpy.cu
  python3 -c “import struct;print(’compute_$v -> 0x%08x’%struct.unpack_from(’<I’,open(’v.cubin’,’rb’).read(),48)[0])”
  cuobjdump -elf v.cubin | grep “Virtual SM”; done

# the field names nvdisasm knows
strings -a $(command -v nvdisasm) | grep -o “EF_CUDA_[A-Z0-9_]*” | sort -u

# scheduling control bits, 21 bits starting at 105 of each 128-bit word
nvdisasm -c -hex k90.cubin | head -40
python3 - <<’PY’
import struct
d=open(’k90.cubin’,’rb’).read()
o,=struct.unpack_from(’<Q’,d,0x28); es,n,si=struct.unpack_from(’<HHH’,d,0x3a)
sh=[struct.unpack_from(’<IIQQQQIIQQ’,d,o+i*es) for i in range(n)]
st=sh[si][4]; nm=lambda x: d[st+x:d.index(b’\0’,st+x)].decode()
for s in sh:
    if not nm(s[0]).startswith(’.text.’): continue
    b=d[s[4]:s[4]+s[5]]
    for i in range(0,len(b),16):
        lo,hi=struct.unpack_from(’<QQ’,b,i); w=(hi<<64)|lo
        print(f”{i:04x} stall={(w>>105)&0xf} yield={(w>>109)&1} “
              f”wr={(w>>110)&7} rd={(w>>113)&7} wait={(w>>116)&0x3f:06b} “
              f”reuse={(w>>122)&0xf:04b}”)
PY

# the fatbin container and the cost of coverage
nvcc -gencode arch=compute_90,code=sm_90 \
     -gencode arch=compute_100,code=[sm_100,compute_100] -fatbin -o f.fatbin saxpy.cu
for m in none speed balance size default; do
  nvcc -compress-mode=$m -gencode arch=compute_90,code=sm_90 \
       -gencode arch=compute_100,code=[sm_100,compute_100] \
       -fatbin -o f_$m.fatbin saxpy.cu; ls -l f_$m.fatbin; done

# the host side
nvcc -arch=sm_90 -c -o k.o saxpy.cu
readelf -SW k.o | grep -E “nv_fatbin|nvFatBinSegment|module_id”
readelf -x .nvFatBinSegment k.o
nvcc -arch=sm_90 --keep -c -o /dev/null saxpy.cu && cat saxpy.cudafe1.stub.c

The fatbin entry parser used for the library dissection is thirty lines: walk from the 0xBA55ED50 header, then for each entry read the kind at offset 0, header size at 4, padded payload size at 8, compressed size at 0x10, architecture at 0x1c, flags at 0x28, and skip header_size + payload_size to the next one.

Subscribe now

Read more

Intro

I spent, more or less, about a year writing an x86-64 assembler in C. Did Lexer, expression evaluator, instruction encoder, ELF writer, relocations, symbol resolution, eventually a macro preprocessor. The thing I keep coming back to from that year is how little information the assembler needed to know.

x86-64 encoding is somwthing baroque. ModRM and SIB are a whole mess, REX prefixes leak into everything, and the exact same mnemonic can have six encodings depending on operand width and register numbering.

But none of that is semantic. I emit the bytes and the machine figures out the rest in some ways. If I put two dependent instructions back to back, the processor detects the hazard, stalls, and my program suddently becomes slow. It is almost never wrong. The whole apparatus of scoreboarding, register renaming and out-of-order issue exists so that the assembler can behave like a dumb table lookup with a symbol table stapled on.

Then I started reading SASS, and that assumption stopped holding.

On modern NVIDIA hardware, if the compiler gets the scheduling metadata wrong, you do not get a slow kernel, you get directly a wrong answer. The dependency interlock for fixed-latency instructions is not in the hardware.

It is basically in a 21-bit field that ptxas writes into every instruction, and that field is undocumented, unstable across various architectures, and produced by exactly one program on earth.

Just that, and not the C++ dialect, and not cudaMalloc, is where I believe the compiler moat actually lives. What follows is a personal attempt to be precise about which layer is load-bearing and which layers people keep mistaking for it, because the public version of this argument is usually pitched at the wrong altitude.

Subscribe now

Five moats wearing one name

“CUDA” is, in various conversations, a giant bag holding at least five separate things, with wildly different levels of defensibility. We have:

1. The source language. __global__, threadIdx, <<<grid, block>>>, a C++ dialect with a few extensions. This is the" “easiest” thing everyone points at and, by design, it is the least defensible layer. Clang has compiled CUDA for more than a decade. The NVPTX backend is just upstream LLVM. libNVVM is a documented public API and NVVM IR is a specified subset of LLVM IR. NVIDIA gave the frontend away years ago, precisely because giving away the frontend costs them nothing.

2. The virtual ISA and the JIT distribution channel. PTX plus the compiler that ships inside libcuda.so. This is a genuine moat, one that I’ve personally studied in the past few months, but it is a distribution moat, not a performance one. It is the reason a fatbinary built in 2019 still runs on Blackwell.

3. ptxas and SASS. This is interesting. We’re talking about a closed optimizing compiler that does register allocation, scheduling, and the encoding of hardware control state, pointing to an instruction set with no public specification and no official assembler.

4. The libraries and the layout algebra. cuBLAS, cuDNN, CUTLASS, CuTe, NCCL, TensorRT-LLM. Thousands of pre-tuned kernel variants plus a template metaprogramming layer that encodes the tensor core’s operand layout requirements. It’s partly open, but the real thing is that it is entirely co-designed with silicon that is not public.

5. The numerics contract and the install base. Now we’re serious. Reduction orders, TF32 defaults, FP8 and NVFP4 scaling recipes, atomics behaviour, the accumulated set of hyperparameter recipes that were tuned on top of all of it. I don’t know why yobody writes about this one, because it’s worth more than you can think.

Layer 1 is defintely gone. Layer 2 is a legal and logistical lock, not a technical one. Layers 3, 4 and 5 are “the moat”, and they defend against completely different attacks.

The vast majority “CUDA alternative” projects pick a layer, do good work, and then basically die on a different layer they did not budget for.

Share

The frontend left the building a long time ago

nvcc is not a compiler. It is a driver script, and you can watch it work:

nvcc --dryrun -arch=sm_90 kernel.cu 2>&1 | grep -v '^#\$ *[A-Z_]*=' 

What you get back is a pipeline: cudafe++ splits host from device, cicc (the NVVM-based device compiler) turns device code into PTX, the ptxas compiler turns PTX into a cubin, fatbinary staples the cubins and the PTX together, nvlink handles device linking, and your host compiler of choice gently handles all the rest. Six things doing 6 separate tasks.

One of those, cicc, is an LLVM. NVIDIA has been explicit about this for the last few years: NVVM IR is just LLVM IR with a documented set of intrinsics and address space conventions, and libNVVM is a shipped, documented library you can use.

If you don’t want to use it, Clang’s own CUDA support will be happy to emit PTX for you, and it has been extreme quality, at least since Google put it there to build TensorFlow.

So the frontend nowadays is a commodity. That is why Triton, Mojo, tinygrad, Julia etc… can emit PTX, all without hurting NVIDIA’s position. Every one of those projects walks up to the same closed door at the end of the hallway.

I just want to be careful here, because “the frontend is commoditized” is often stated as if it were an indictment of NVIDIA’s openness, but it’s not. Ceding the frontend was correct and (probably) a deliberate choice.

Various frontends just increase the number of paths into PTX, and every path into PTX eventually terminates in a program NVIDIA controls.

Subscribe now

The 21-bit field

Here is the part that changed how I think about this.

Kepler version removed hardware dependency checking for fixed-latency instructions and moved the responsibility into the compiler. Then we had Maxwell refining it.

Volta made it permanent by widening the instruction word from 64 bits to 128 bits and embedding the control payload directly in every instruction rather than in separate scheduling words interleaved every few instructions.

The reconstructed layout of that payload, for Volta and later, looks more or less like this:

First. four bits of operand reuse cache hints. Then, a six-bit mask of scoreboard barriers this instruction waits on. Now on, two three-bit indices naming the barriers it sets for variable-latency results. A yield bit that tells the warp scheduler whether to prefer this warp or switch away.

And finally we have four bits of static stall count: the number of cycles the scheduler must wait before issuing the next instruction from this warp.

That last field is the one that matters for real. For fixed-latency instructions, the hardware does not check read-after-write hazards at all. The microarchitecture research on this is mostly unambiguous: if you set the stall counter wrong and the program produces incorrect results, because there is no scoreboard behind you to catch it.

The register status table and the wires from the issue logic to it were purposefully deleted from the design, and the area and energy they would have cost were spent on something else. The compiler is the scoreboard.

On x86, the ISA is a contract about meaning and the microarchitecture is free to reorder underneath it as long as the contract holds. On NVIDIA hardware since Kepler, a meaningful piece of the microarchitecture’s correctness lives in bits the compiler writes.

ptxas is not a tool that targets the machine, because it is a component of the machine that happens to run on your workstation. Those are the obvious consequences.

You cannot open source that boundary without publishing the timing characteristics of every functional unit in every SM variant of every SKU, including the ones you have not announced. And if you publish it, you have frozen it, because now third-party code depends on it and any change basically breaks correctness, not just performance.

The closedness is not primarily a business decision. It is downstream of an architectural decision that was made for area and power reasons around 2012, and the strategic value came along for the ride.

I find that way interesting than the general story, even thoigh it’s more depressing, because it means the moat is not a policy anyone can be lobbied to reverse.

Occupancy is a compiler policy, not a hardware property

The other thing ptxas decides on your behalf, and the one you can actually measure without a disassembler, is how many registers each thread gets.

We alredy know that an H100 SM has a 256 KB register file: 65,536 registers of 32 bits, allocated to warps at a granularity of eight registers per thread. The SM can hold at most 64 resident warps. Those three numbers set the whole occupancy calculation, and the only free variable is the register count ptxas chose.

Take a kernel where ptxas -v reports 168 registers per thread. Each warp consumes 168 x 32 = 5,376 registers, so the SM can hold floor(65536 / 5376) = 12 warps, which is 12 of 64, or 18.75 percent occupancy. Now suppose you pass -maxrregcount=128.

The same warp now costs 4,096 registers, 16 warps fit, occupancy goes to roughly 25 percent, and ptxas pays for the difference by pushing the excess to local memory, which is to say to L1 and then to L2 and then, if you are unlucky and you have a giant working set, to HBM.

Neither number is right on its own. It depends on what the kernel spends its time waiting for. A kernel that waits on memory wants more warps. When one warp stalls on a load, the scheduler switches to another one, and if enough warps are resident, the waiting never shows up in the runtime. Occupancy is what keeps that kernel fed.

On the opposite side, a kernel that lives in the tensor cores usually wants the registers. It already hides its own waiting, by fetching the next tile while the current one is being multiplied. It has a pipeline, so it has nothing to gain from having other warps to switch to, and it will run happily at eighteen percent occupancy.

That tradeoffs are the single most consequential decision in GPU performance work, and they are made by a heuristic inside a binary that I believe is impossible to read, tune, or replace. -maxrregcount and __launch_bounds__ are the two knobs you get, and they are blunt.

I want to highlight this because it is the everyday version of the argument. You don’t need to care about control bits to be affected by the closed backend. If you have ever watched a two-line source change move register count from 168 to 176 and cost you a resident warp, you have already been on the wrong side of this wall.

There is no official SASS assembler. Instead, there’s a disassembler, nvdisasm, and cuobjdump --dump-sass, and the documentation for the instruction set amounts to a table of mnemonics with one-line descriptions. The opcode encodings, the control field layout, and the exact latency table are reverse engineered by a small group of people, generation by generation.

The lineage of that work is short enough to list: asfermi for Fermi, Scott Gray’s maxas for Maxwell, KeplerAs, TuringAs, CuAssembler for Pascal all the way through Ampere, openptxas under the gpuocelot umbrella for Maxwell and Pascal. Every one of these is a heroic, partial, architecture-locked effort that goes stale the moment a new chip make it to the public.

The payoff of that is real, which is the frustrating part in my opinion. Gray’s maxas SGEMM reached figures close to theoretical peak on GM204, ahead of the cuBLAS of that era, purely by scheduling better than ptxas did.

More recently there is paper work applying reinforcement learning to SASS schedules directly (CuAsmRL), reordering instructions inside cubins that ptxas already produced and finding wins. If ptxas were optimal, that entire research direction would return zeros.

I have personally enjoyed one single detail, that I’m gonna show you below:

/*0080*/ LDG.E.128 R8,  desc[UR6][R2.64]    ;  /* [----:B----:R-:W2:-:S01] */
/*0090*/ LDG.E.128 R12, desc[UR6][R4.64]    ;  /* [----:B----:R-:W3:-:S01] */
/*00a0*/ IMAD.WIDE R2,  R0, R7, c[0x0][0x168] ;/* [----:B----:R-:-:-:S02] */
/*00b0*/ HMMA.16816.F32 R20, R8, R12, R20   ;  /* [--23:B----:R-:-:-:S04] */
                          ^          ^
                          |          waits on write barriers 2 and 3
                          |          set by the two loads above
                          stall 4 cycles before the next issue

This is a reconstructed fragment in the shape that cuobjdump --dump-sass prints, not literal output from a specific compilation, but the bracketed control payload follows the documented conventions.

Please take a deep look at the two loads first. Each one sets a write barrier, 2 and 3 respectively, because a global load has variable latency and the compiler cannot know statically when the data lands.

Then the IMAD in between carries no barrier at all (only a stall count) because integer multiply-add is fixed latency and the compiler knows exactly how long it takes. The tensor core instruction waits on both barriers before it issues.

Now notice what is doing the work. The S02 on the IMAD is not an optimization hint. It is the compiler asserting the pipeline depth of the integer unit on this specific SM. Change the SM, keep the number, and the machine reads a register before the previous instruction has written it.

One last thing: NVIDIA Research’s own SASS instrumentation framework, SASSI, was distributed as a closed-source fork of ptxas binaries. Their own researchers, inside the company, shipped a patched binary rather than source. That suggests you something about how the artifact is treated internally.

Share The Software Frontier

PTX is a treaty, not an ISA

I often read posts about people describing PTX as “NVIDIA’s assembly language,” which is so wrong in a way that hides the actual mechanism. PTX is a virtual ISA with unbounded virtual registers, no notion of the physical register file, no scheduling, and no control bits.

It is much closer to LLVM IR than to x86 assembly. Writing PTX by hand does not get you anywhere near the metal; it gets you a slightly lower-level input to the same closed optimizer.

DeepSeek’s V3 work is the canonical example, and it is routinely misreported. They partitioned twenty of the H800’s 132 SMs to handle inter-node communication, used warp specialization, and wrote custom PTX to reduce L2 pressure and interference with the compute kernels.

That is obv. excellent engineering under export-control constraints, but is not an escape from CUDA. It is a deeper commitment to CUDA: PTX pins you to NVIDIA harder than CUDA C++ does, simply because CUDA C++ at least has clean-room reimplementations and PTX has one consumer.

What PTX actually buys NVIDIA is the forward compatibility story, and it is a genuinely excellent piece of platform engineering. Ship PTX inside your fatbinary, and when your binary lands on hardware it has never seen, the driver compiles it at load time. libcuda.so contains a compiler. Every NVIDIA GPU in the world is shipped with a copy of the last-mile toolchain baked into the driver.

The strategic consequences of that are larger than the technical ones:

• Any stack that emits PTX, no matter how open, has a closed NVIDIA compiler in its runtime path. Triton is open source and ships PTX to ptxas. tinygrad in PTX=1 mode does the same. Open at the top, closed at the bottom.

• Compatibility is a one-way ratchet. Old PTX runs on new hardware. New PTX never runs on old hardware. Every generation, the set of instructions you need to hit peak throughput moves into a .target sm_XXa gate that the previous generation’s driver cannot parse.

• The world’s binaries contain PTX. That corpus is a compatibility obligation NVIDIA has taken on, and it is also an asset nobody else can serve.

The compatibility promise does not cover the instructions that matter

This is the part of the PTX story I have not seen written down anywhere, but I do believe it changes the conclusion.

PTX forward compatibility applies to base architecture targets. sm_90, sm_100, and so on. But the instructions that reach peak throughput do not live there. wgmma requires sm_90a. The tcgen05 family requires sm_100a or sm_103a.

That “a suffix” means architecture-specific, and code compiled for an a target is explicitly not forward compatible: it runs on that architecture and nowhere else, ever. NVIDIA later added f family targets to soften this within a family, which is an admission that the problem is real.

Put the two facts next to each other. The forward compatibility that everyone cites as CUDA’s great platform virtue covers the instruction set you use when you do not care about performance. The moment you write a kernel that actually saturates the tensor cores, you have opted out of it, and you are shipping architecture-locked binaries exactly like everyone else.

So the JIT compatibility story is not a technical moat at all; it is a convenience for the long tail, and the long tail is not where the money is. What it does buy NVIDIA is that the world’s software ships PTX, and PTX has one consumer.

And then there is the legal layer, which people mention and then move past, maybe too quickly. The CUDA EULA has, since 2021 online and since the 11.6 installed files, carried a restriction on reverse engineering, decompiling or disassembling the output generated using SDK elements for the purpose of translating those artifacts to target a non-NVIDIA platform.

Read it as written: it is a restriction on what you may do with the output of the compiler, not just the compiler. Whatever its enforceability, and I am not a lawyer, its practical effect is a chilling one on exactly the class of project that would otherwise attack layer 2.

Subscribe now

The libraries are an admission, not a flex

The standard framing is that cuBLAS and cuDNN are evidence of NVIDIA’s software superiority. I want you to read them the opposite way, like I do.

If nvcc and ptxas could reliably generate peak code from readable source, cuBLAS would be a thin wrapper over a few templated kernels. Instead it is a multi-hundred-megabyte binary containing thousands of pre-compiled kernel variants, selected at runtime by heuristics, many of them tuned by hand at the SASS level by people who work at NVIDIA.

cuDNN is even worse. The size of those .so files is a direct measurement of the gap between what the compiler can do and what the hardware can do.

That gap is the real product. NVIDIA is not selling you a compiler that makes your code fast. It is selling you the output of a large, expensive, permanently-employed team of kernel engineers, wrapped in an API, in a form you cannot fork.

And the gap is widening, on purpose

The tensor core programming model has changed shape three times in six years, and each change moved way further away from anything a general compiler can target from ordinary source.

Ampere gave us mma, warp-level, operands in registers, with layout requirements you could hold in your head if you tried. Hopper introduced wgmma, warpgroup-scoped and asynchronous, plus TMA for bulk asynchronous copies driven by descriptors, plus mbarrier for the synchronization that async now requires.

Blackwell deprecated wgmma entirely and introduced the tcgen05 family: a dedicated on-chip Tensor Memory for accumulators, a single thread issuing the MMA on behalf of everyone, and cta_group::2 letting two CTAs on a TPC cooperate on one MMA with shared operands.

Read that again as a compiler person. The CuTe MMA atoms for Blackwell use a thread ID layout of one, where Hopper used 128. At peak performance, the SIMT model is a fiction.

What is actually happening is that one elected lane fires a descriptor at an accelerator, and the rest of the code is a software pipeline built out of barriers, async copies and a memory space whose allocation you manage explicitly.

This is why the ceiling moved from the compiler to the layout algebra. You will see some numbers, because the abstraction hides how physical this is.

Tensor Memory on Blackwell is 256 KB per SM, organised as 128 lanes of 512 columns of 32 bits. You allocate it explicitly, in columns, in power-of-two units with a minimum of 32.

A single warp can only reach the 32 lanes matching its position in the warpgroup, which is why the copy instruction has multicast modes to duplicate a tile across all four lane quadrants.

Block-scaled narrow formats add their own geometry on top: MXFP8 carries one E8M0 scale per block of 32 elements, NVFP4 carries an E4M3 scale per block of 16 plus a second-level FP32 tensor scale, and the scale factors themselves have to be staged into tensor memory in a specific layout before the MMA will consume them.

None of that is expressible as “the compiler will figure it out”. It is a data layout problem with hardware-imposed alignment constraints that propagate back into your choice of tile shape, which propagates back into your choice of pipeline depth, which propagates back into your register budget.

This is why the ceiling moved from the compiler to the layout algebra. You cannot express a competitive Blackwell GEMM by writing loops and hoping.

You express it as a set of layouts, and the algebra of those layouts is CuTe, and CuTe is co-designed with the silicon by the people who designed the silicon, eighteen months before you can buy it.

The single most clarifying data point I have seen on this: FlashAttention 4, the first version written in CuTe DSL rather than C++ templates, reportedly hits about 1613 TFLOP/s on B200 for bf16 head-dim 128 causal attention at 8K sequence length, roughly 71 percent of peak, about 1.3 times faster than cuDNN 9.13, and around 2.7 times faster than Triton on the same hardware.

Three conclusions fall out of one benchmark. NVIDIA’s own closed, hand-tuned attention library was not the ceiling.

The tile-level DSL, not the C++ template library, is now the productive authoring layer. And Triton, the great portable hope, is nowhere near peak on the newest tensor cores, because its abstractions do not yet expose what tcgen05 and TMA require.

Subscribe now

What is actually in a cubin

I want to discuss and be clear about one thing worth opening up, because it is the part that felt most familiar to me coming from ELF.

A cubin is an ELF64 object. S ame header layout you already know, with a CUDA-specific machine type. Inside, you get a .text section per kernel holding the SASS, a .nv.constant0 section per kernel holding the kernel parameters and grid constants (this is why parameters arrive via c[0x0][...] in the disassembly rather than in registers), .nv.info sections carrying metadata like register counts and parameter layout, plus the usual relocation sections.

The fatbinary that nvcc embeds in your host object is itself a container of these, in a .nv_fatbin section, with a small descriptor segment telling the runtime what is inside.

You can walk all of it here:

cuobjdump -elf a.out            # fatbin contents, section by section
nvdisasm -elf kernel.cubin      # ELF structure plus disassembly
readelf -S kernel.cubin         # it really is just ELF

The greatest absent, once you have looked, is the same one as before. Every part of this container is inspectable: the relocations and the metadata are readable, the instruction stream is disassemblable. What you can’t do is produce a valid one from modified SASS, because the only program that writes the .text section is the one you do not have.

Coming from x86, that asymmetry is the strangest part of the whole stack. I can hand-assemble an ELF object, link it, and run it. Here I can read everything and write nothing.

Subscribe now

What is not being told

Most of the commentary stops at “CUDA is sticky.” Here are the parts I think are underweighted, roughly in order of how much they change the conclusion.

The compiler is correctness-critical, not just performance-critical

Again, sorry for that, but I have not seen this stated plainly anywhere outside the microarchitecture literature. Every discussion of open SASS toolchains treats the problem as “we would generate slower code”.

The actual problem is that a third-party SASS assembler that mis-models the latency of one functional unit on one SKU generates silently incorrect results, in a data-dependent way, on a subset of hardware.

That is a categorically harder engineering problem than performance parity, and it is the reason no serious commercial effort has attempted it. The risk profile is closer to writing a memory controller than to writing a backend.

The moat is a labor market

Strip away the branding and ask what actually cannot be replicated and you’ll find out that is not the compiler source.

It is the few hundred people worldwide who can look at a cuobjdump listing, reason about the control bits, and know from experience that a particular shared memory access pattern will hit a bank conflict on this generation but not the last.

A large fraction of them work at NVIDIA. Most of the rest work at four or five labs and two or three GPU startups. That reframing matters because it tells you what the actual counter-technology is. Many have attemped to address with it, but no, it’s not a new language.

It is just search: autotuners, superoptimizers, and increasingly LLM-driven kernel generation, all of which convert scarce expert labour into abundant compute. If the moat is a hiring problem, then the thing that erodes it is the thing that makes kernel expertise reproducible.

There is a telemetry loop nobody else has

Nsight, DCGM, the driver, customer escalations from every large training run on the planet. NVIDIA sees the performance pathologies of the entire industry’s kernels, continuously, at a scale no competitor approaches.

Then it fixes the top ones in the next library release and, where the fix has to be in hardware, in the next architecture. Every “CUDA maturity” argument is really an argument about the number of laps this loop has run.

The collectives are compiled kernels, and they eat your SMs

I listed the interconnect under layer 5 and then almost walked past it, which would have been a mistake, because NCCL is where the compiler moat and the network moat turn out to be exactly the same thing.

A collective is not a library call that hands work to a NIC, it’s a set of CUDA kernels. NCCL builds rings and trees over the discovered topology, splits each collective across some number of channels, and every channel is a resident thread block.

Which means bandwidth on the wire is bought with SMs that are then not available for compute. It also means the protocol matters: NCCL picks between a low latency mode that trades payload efficiency for skipping memory fences, a 128 byte variant that recovers most of the bandwidth but wants NVLink underneath, and a plain mode that gets full bandwidth at higher latency.

The choice is made by heuristics against measured topology, and the tuning knobs are environment variables. We see probably at least two consequences that are worth holding onto.

First, this is exactly what DeepSeek was doing when they carved twenty SMs out of 132 for communication. Their work was quite revolutionary because they were taking manual control of a tradeoff that NCCL normally makes for you, because on export-limited H800s with halved inter-GPU bandwidth, NCCL’s defaults were wrong for their shape. All of that, as stated before, working with CUDA, not around it or without it.

Second, in-network reduction changes the arithmetic entirely. When the switch itself can perform the reduction, the collective stops being a kernel that moves the same bytes several times and becomes a kernel that ships bytes once.

That capability lives in the NVSwitch and InfiniBand silicon, it is exposed through the same library, and there is no version of “port your kernels” that gets you it. AMD’s answer, RCCL, is a real port of the same design, and the vLLM tuning guidance for it involves pushing the channel count up sharply on MI300X, which tells you how much of the performance is in the topology-specific constants rather than the algorithm.

If you are keeping a scorecard of which moats are compiler-shaped, this one is a hybrid, and it is the least portable thing in the stack.

Port a model from cuBLAS to hipBLASLt and the arithmetic is not bit-identical for a few reasons. Split-k strategies differ, even reduction orders do not behave the same.

TF32 is on by default in one place and not another.

FP8 scaling granularity, per-tensor versus per-block versus the MX and NVFP4 formats, is a semantic choice baked into the kernels.

For inference this usually shows up as a small quality wobble that people can get along with. For training, it shows up as a recipe that diverges at step 40,000 for some random reasons nobody can bisect, on a piece of hardware where you have less tooling.

The cost of that risk almost never appears in the total-cost-of-ownership spreadsheets that compare dollars per FLOP, and it is a real reason large labs pay the NVIDIA premium.

Watch which artifact they do not publish

There is a pattern in NVIDIA’s open sourcing, and once you see it you can predict the next move. They publish the specification and the dialect, while keeping the lowering.

NVVM IR: specified, with a public library. cicc: closed. PTX: exhaustively documented, hundreds of pages, updated every release. ptxas: closed. CUTLASS: open, permissively licensed, genuinely excellent.

The SASS it lowers to: closed. And now CUDA Tile IR: the MLIR dialect, the Python bindings, the bytecode format and the conformance suite are on GitHub. The compiler that turns tile bytecode into cubin is not.

This is a coherent, repeated strategy. Publishing the interface grows the number of producers. Keeping the lowering means every producer terminates in your binary.

It is the same move IBM made with channel architecture and the same move ARM makes with the ISA versus the implementations, and NVIDIA runs it more cleanly than either.

AMD is the counterexample that breaks the simple story

Here is the fact that should discipline every “closed compiler is the moat” argument: AMD publishes its ISA documents. The AMDGPU backend is upstream LLVM. ROCm is open source, top to bottom, including the compiler and the runtime.

You can read the machine encodings for CDNA3 and CDNA4 in a PDF on AMD’s website But (unexpectedly) even this has not collapsed the moat.

So openness is not the mechanism. If it were, AMD would have won already a few years ago. The mechanism is the co-design loop, the library labour, the numerics contract, the install base and the interconnect, and NVIDIA’s closed compiler is a symptom of the same architectural strategy that produces the rest of it.

Anyone whose plan is “make an open CUDA” has already been run as an experiment, by a company with $25 billion of revenue, for at least a decade.

The moat is thinning at the top and thickening at the bottom

Two things are true at once and they point in opposite directions, which is again interesting because it sounds counterintuitive and understandable at the same time.

At the top, the default path from PyTorch to the GPU is now a portable tile language. torch.compile emits Triton, not CUDA C++. Liger, FlexAttention and a large fraction of production fused kernels are…. yes, again, Triton. That layer is vendor-neutral by construction, and it is where most new kernel code is being written.

At the bottom, the instructions you need to reach peak on the newest silicon are less reachable from a generic compiler than at any point since 2007. TMA descriptors, tensor memory allocation, CTA-pair MMA, block-scaled narrow formats with alignment constraints that constrain your tile shapes.

The gap between “compiles and runs” and “hits peak” is the widest it has ever been in 19 years.

Now the interesting question is not whether the moat holds. It is more which layer the industry ends up standardizing on, and NVIDIA has just made a very large move to answer that question in its own favour.

Subscribe now

Tile IR: the moat gets rebuilt one level up

The most important compiler event of the last twelve months got almost no coverage outside compiler circles and a restricted group of GPU programmers.

CUDA 13.1 introduced CUDA Tile, a tile-based programming model with its own intermediate representation. Tile IR is an MLIR dialect. The dialect, the Python bindings, the bytecode serializer and a conformance test suite are open source at NVIDIA/cuda-tile.

The bytecode format is explicitly specified as stable, versioned, and forward and backward compatible: bytecode emitted by an older compiler is readable by a newer compiler or driver, and a driver accepts bytecode up to its supported version.

That paragraph should sound familiar, because it is the PTX contract, restated one abstraction level higher, in MLIR, with a published binary encoding. Tile IR is the new PTX.

The user-facing language is cuTile, a Python DSL, with a Julia binding already shipping at near parity on simple kernels. And critically, NVIDIA wrote a Triton backend that targets Tile IR, so Triton programs can be compiled through the new path instead of through PTX.

Meta has filed an RFC to add a cuTile backend to PyTorch Inductor, with the stated motivation that Tile IR gives bytecode portability, performance portability across GPU generations, and access to a tile-specific optimizing compiler, and that it is a prerequisite for pointing Helion at Tile IR as well.

Pro tip: read the sequence of moves rather than the individual announcements.

Triton became the default codegen target of PyTorch. That made the tile abstraction, the layer where new kernels get written, and that layer is portable across vendors. It was the most serious structural threat to the moat in fifteen years, because it made the frontend irrelevant and gave AMD and Intel a credible path to the same source.

NVIDIA’s response was not to fight Triton: it was to build a better tile IR, open source the dialect and the spec, write the Triton backend themselves, and offer everyone a faster path through their own infrastructure. If it works, the portable tile layer that was supposed to route around NVIDIA becomes a frontend for NVIDIA’s IR, and the optimizing compiler underneath, the one that turns tiles into cubin, is closed, just like ptxas.

It is the cleanest execution of the “publish the interface, keep the lowering” strategy I have seen so far, and it happened while everyone was arguing about whether ROCm had caught up.

I want to flag my uncertainty honestly: it is early. Tile IR shipped in December 2025, the Inductor RFC is in progress, and I have not seen independent third-party benchmarks of cuTile against hand-written CuTe on Blackwell at production shapes.

NVIDIA’s own claim is that cuTile is competitive with cuBLAS for large GEMMs and cuDNN for large attention. Competitive with cuDNN is a lower bar than it used to be, given FA4.

Subscribe now

The alternatives, honestly

Every project below is real engineering by capable people. What I want to do is say which layer each one attacks and where it runs out of road, because most of them are described by their advocates as if they attack all five.

HIP and ROCm

Attacks: layer 1 and layer 4. A CUDA-shaped API, a source translation tool, and a library set with matching names.

ROCm in 2026 is a genuinely different product from ROCm in 2023, and people who formed their opinion during the Frontier era are working from stale data.

ROCm 7.2 ships tuned hipBLASLt GEMM kernels for FP8, BF16 and FP16 on MI350 and MI355X. AITER provides fused kernels that, inside vLLM, deliver large multiples over the legacy ROCm attention path. vLLM now carries seven attention backends on ROCm.

MI355X ships 288GB of HBM3E, and capacity per package is a real architectural advantage that removes tensor parallelism from model sizes where NVIDIA needs it.

Where it runs out of road: the API-clone strategy structurally guarantees you arrive second. hipify ignores inline PTX, which means exactly the kernels that were worth hand-optimizing are the ones that do not port. Matrix core layouts (MFMA) differ enough from mma, wgmma and tcgen05 that a “ported” kernel is a rewrite at the layout level. And the performance story has historically been per-model tuning: a specific configuration of a specific model at a specific batch size is fast because someone at AMD made it fast, which is a different asset from a compiler that makes your unseen kernel fast.

Honest read: on standard transformer inference, the gap is small enough that the decision is now economic rather than technical. On training, on novel architectures, and on anything requiring bespoke kernels, it is not close.

SCALE (Spectral Compute)

Attacks: layer 1 and layer 3 simultaneously, which makes it the most technically interesting project in this space.

SCALE is a clean-room, Clang and LLVM based drop-in replacement for nvcc that compiles nvcc-dialect CUDA source, including inline PTX, directly to AMD machine code. This is not translation, not even middleware: an actual second implementation of the CUDA language.

The company is small, London based, founded 2018, and funded its own compiler work off consulting. Published benchmarks claim roughly 5.94 times over HIP on AMD hardware, and as of May 2026 they are claiming speedups over nvcc itself on B300 with CUDA 13.

The detail I keep turning over is that they joined NVIDIA Inception in June 2026. A company whose product is CUDA portability, partnering with NVIDIA. Which makes sense if you believe NVIDIA’s interest is in CUDA remaining the standard more than in CUDA remaining exclusive, but it is a strange shape in my view.

Where it runs out of road: the CUDA-X library surface. There are hundreds of libraries above the language, and a compiler that perfectly compiles the language still leaves you needing cuDNN, cuTENSOR, cuDF, NCCL and the rest. Spectral is working on delegating those to ROCm equivalents, which means SCALE inherits ROCm’s library ceiling at exactly the point where it matters most.

ZLUDA

Attacks: layer 2. Binary and PTX level translation, no recompilation.

ZLUDA is the project people cite most and understand least. It gained ROCm 7 support in December 2025 and shipped v6 in June 2026.

That release also announced that commercial funding had ended again and the project is back to being one person’s weekend work, which is why v6’s headline features are 32-bit PhysX and Blender textures rather than anything relevant to inference.

Where it runs out of road: everywhere, and the reasons are structural rather than technical. Binary translation is a treadmill against a vendor who ships a new virtual ISA extension every generation and a whole new IR every few years. The EULA restriction points directly at this approach. It has now lost funding twice, from two different backers. I would treat the binary translation path as closed, not because the engineering is bad, but because the economics have been tested twice and failed twice.

Triton

Attacks: layer 1 and, at least partially, layer 4.

Triton is the most consequential thing on this list because of where it sits: it is what torch.compile emits, which means it is the default kernel authoring layer for most of the industry whether they know it or not. It runs on NVIDIA, AMD and Intel from the same source.

Where it runs out of road: two places. On NVIDIA it emits PTX, so it never escapes ptxas; it relocates the frontend, it does not cross the wall. And on the newest hardware it is a long way from peak, because the abstractions do not expose TMEM, CTA-pair MMA and the descriptor-driven async machinery that Blackwell needs. FlashAttention 4 leaving Triton for CuTe DSL is the load-bearing evidence there.

And now NVIDIA is offering to compile it through Tile IR, which resolves the second problem in exchange for reintroducing the first at a higher altitude.

Helion, TileLang, Pallas, and the search-based approach

Attacks: the labour market, which as argued above is the real moat.

Helion is a Python DSL from Meta’s PyTorch compiler team that sits above Triton: you write PyTorch-shaped code with tile loops and it autotunes over hundreds of generated Triton implementations, spending roughly ten minutes of compute per kernel to do it.

The reported numbers are the interesting part. Around 1.05 times over torch.compile and 1.44 times over hand-written Triton on AMD on average, with individual kernels much higher, and on H100 roughly 1.2 to 1.85 times over Triton.

Beating hand-written Triton on average, by searching. That is the anti-moat technology, and it is not a language feature. It is a compute substituting for expertise, and it gets better every time GPUs get cheaper, which is a nice recursion.

The same logic explains AMD’s agentic kernel work (GEAK), LLM kernel generators, and the growing body of RL-over-schedules research including the SASS-level work mentioned earlier. If your competitor’s advantage is several hundred irreplaceable engineers, the correct response is not to hire several hundred engineers. It is to make the engineers less necessary.

Where it runs out of road: search needs a correct, expressive, fast-to-evaluate space to search over, and on Blackwell the space that contains peak performance is only reachable through the vendor’s layout algebra. You can search brilliantly within Triton’s expressible set and still be 2.7 times off.

tinygrad

Attacks: layers 1, 2 and 3, by refusing the premise.

tinygrad generates its own kernels, can render directly to PTX, and has experimental backends that talk to AMD and NVIDIA hardware without the vendor userspace runtime at all.

Whatever you think of the framework’s ambitions, the AMD path is an existence proof that the vendor compiler and userspace stack are not physically required.

Where it runs out of road: peak tensor core utilization on the newest silicon, and the entire ecosystem surface. It is a demonstration of what is possible for a small team, not a production alternative for a lab training frontier models. Its real contribution is epistemic: it proves the stack is thinner than the marketing implies.

Mojo and MAX

Attacks: layers 1 and 4, with an MLIR-native language designed for kernel authoring across vendors.

Modular’s own writing on this is refreshingly unromantic: they note that write-once-run-everywhere in GPU land has usually meant write-once-run-slowly-everywhere, that CUTLASS does not attempt portability beyond NVIDIA and is often locked within a generation, and that Triton’s performance degrades off NVIDIA.

Their bet is that a properly designed compile-time metaprogramming language can express hardware-specific structure without forking the source.

Where it runs out of road: it is a proprietary language asking developers to adopt it on faith, competing against a Python DSL layer that PyTorch already emits by default. The technology is pretty good. The distribution problem is brutal.

The vertical integrators: TPU, Trainium, and friends

Attacks: the entire column, by not participating in the compatibility game at all.

Google’s TPU plus XLA is the only stack that has demonstrably trained frontier models at scale outside NVIDIA, sustained, for years. It did not do that by cloning CUDA.

It did it by owning the chip, the interconnect, the compiler, one framework, and one enormous first-party customer whose workloads shape the hardware roadmap. Trainium plus NKI is the same play at an earlier stage.

The lesson is uncomfortable for everyone selling portability: the proven counter to vertical integration is vertical integration. Compatibility layers have a zero-for-many record. Owning the column has a one-for-one record.

Where it runs out of road: you cannot buy this. It requires a decade, a captive workload, and a willingness to eat several generations of worse hardware.

The ones I am not covering properly, and why

Three more categories exist and deserve at least an honest pointer rather than silence.

Huawei’s CANN with the Ascend NPUs is the most complete non-Western vertical stack I’ve ever seen, with its own kernel language, its own graph compiler, and a captive domestic customer base that gives it the one thing compatibility layers never have, which is a workload that shapes the roadmap.

I do not have enough independent benchmark data to say anything useful about where it sits on peak utilization, and I would rather say that than guess randomly. The same caveat applies to Moore Threads and Biren, both of which ship CUDA-adjacent programming models.

On the standards side, chipStar compiles HIP and CUDA to SPIR-V for OpenCL and Level Zero targets, and Intel’s SYCLomatic migrates CUDA source to SYCL with a reported majority of code converting automatically and a manual remainder. Both are layer 1 plays with the layer 4 problem intact.

And there is a fourth category I have deliberately excluded from the comparison table because it is not a competitor at all: NVIDIA’s own Python surface. cuda.core, Numba, Warp, nvmath, and now cuTile are a coordinated effort to make sure that when kernel authoring moves fully into Python, which it is doing, it moves into NVIDIA’s Python rather than someone else’s.

The cautionary tale: OpenCL

Worth remembering, because the failure mode repeats. OpenCL standardized the API and the language and left the libraries, the tuning, and the last-mile compiler quality to each vendor.

Portable source, unportable performance, no cuBLAS equivalent, no tensor core story, committee cadence against NVIDIA’s annual co-design cadence.

SYCL is a better designed successor with the same structural problem: it is a standard for the layer that was never the moat.

Subscribe now

What replacing the kernel layer actually costs

Let me put a rough number on the library moat, since that is the part people assume is unbuyable.

Take LLM inference specifically, not the whole CUDA-X surface. The kernels that matter for a modern decoder-only model at serving time are: dense GEMM in several shapes, grouped or batched GEMM for MoE, prefill attention, decode attention with paged KV, an MLA variant if you serve DeepSeek-shaped models, quantize and dequantize paths for FP8 and FP4, RMSNorm, RoPE, fused SwiGLU, sampling, KV cache gather and copy, speculative verification, and the collectives.

Call it 30 to 50 kernels that appear in profiles, of which eight or so carry most of the time.

Assume a fully loaded peak-quality kernel engineer at somewhere between $350,000 and $600,000, call it $450,000 to be some kind of “average”. Assume a top-tier attention or GEMM kernel for a new architecture takes three to nine engineer-months including autotuning, numerics validation and the long tail of shape coverage, and the secondary kernels take one to two.

• Eight critical kernels at six months each: roughly $1.8M

• Forty secondary kernels at 1.5 months each: roughly $2.2M

• Test infrastructure, numerics validation, CI across shapes and dtypes: call it 50 percent overhead

That lands somewhere around $6M to $15M per hardware generation to build a competitive inference kernel layer from scratch. For a company selling accelerators, that is a rounding error. It is roughly one percent of a single generation’s mask set and tape-out cost.

Which explains something that otherwise looks strange: AMD is actually close on inference and not close on training. The inference kernel surface is small enough to buy.

The training surface, plus the full CUDA-X library set, plus the numerics validation across a decade of recipes, is somewhere between ten and fifty times that, and the co-design loop that makes the kernels good on day one instead of month nine is not purchasable at any price.

Treat these figures as a stated model with stated assumptions, not as a measurement. The point is the ratio, not the absolute value: the part everyone talks about is the cheap part.

Where I might be wrong

Four ways this argument could be badly calibrated, in descending order of how much they would cost me.

The backend gap may be small. My case for layer 3 rests partly on the maxas era, which was Maxwell, and partly on recent reinforcement learning work over SASS schedules that finds wins by reordering ptxas output. If the residual there is three to five percent on typical kernels rather than the double digits the Maxwell results implied, then ptxas is close enough to optimal that the backend is not the moat at all, the libraries are, and I have spent a lot of words on the wrong floor.

I have not seen a rigorous modern measurement of that residual across a representative kernel set, and I would very much like to.

The labour market framing may be a category error. I argue the scarce resource is people who can read control bits, and therefore that search and autotuning erode it.

But the advantage might be organisational rather than headcount: eighteen months of pre-silicon access, a hardware team down the hall, and the ability to change the chip when the kernel is awkward. If that is the real mechanism, then no amount of autotuning compute closes it, because the loop being run is design, not optimization.

Tile IR may be commoditization rather than capture. I read the open dialect, the published bytecode format and the conformance suite as a way of owning the layer Triton was about to own. The more generous reading is that an MLIR dialect with a stable binary encoding and a test suite is precisely the artifact another vendor needs in order to target the same source, and NVIDIA has just handed it over. Both readings are consistent with the facts as of today. Which one is right will be visible in whether anyone outside NVIDIA ships a Tile IR backend.

And the boring one. I do not have a B200. Everything in this piece at the Blackwell layer comes from documentation, from other people’s benchmarks, and from reading disassembly conventions rather than fresh disassembly. The control field layout is reverse engineered by researchers, not published.

Where I have marked things Tier B or C, that is what the marking means, and I would treat the Hopper and Blackwell specifics as directionally right rather than precisely right until someone with the hardware checks them.

Subscribe now

Five dated calls

Predictions with dates, so this piece can be scored rather than admired.

1. No third party ships a production-quality SASS assembler for Hopper or Blackwell before 2030. Production quality meaning it passes a public correctness suite across a non-trivial kernel set on more than one SKU. Falsified by any project that does.

2. NVIDIA never open sources the Tile IR to cubin lowering. The dialect, the bytecode, the spec and the conformance suite stay open. The optimizing backend does not. Falsified trivially and publicly if wrong.

3. Before the end of 2027, an automated kernel generator beats a vendor library on a headline GEMM or attention shape on non-NVIDIA hardware, with reproducible numbers. The search-versus-labour thesis stands or falls on this one.

4. AMD reaches within fifteen percent of NVIDIA on tokens per second per dollar for standard transformer inference on a like-for-like generation by end of 2027, and does not close the equivalent gap on training in the same window. The asymmetry is the prediction, not the number.

5. At least one more CUDA compatibility project loses its funding before the end of 2027. ZLUDA has now done it twice. The economics of chasing a virtual ISA that moves every eighteen months have been tested and they do not work, and I expect the next attempt to discover this independently.

Share

What would actually work (for real)

I am sorry to say it, but I do not think the moat gets breached. I think it gets routed around, in specific places, for specific workloads, and here is where I would put effort if that were my job.

Own the tile layer, and make it a spec with teeth. Not a language, not a framework: a specification with a versioned binary encoding and a conformance test suite that vendors must pass. NVIDIA just told you this layer is the strategic one by shipping exactly that artifact. The mistake would be to let Tile IR become the only one.

Make numerics a specification too. This is the gap nobody is filling. A conformance suite that pins reduction order tolerances, accumulation types, scaling granularity for narrow formats, and determinism guarantees, with reference outputs. Right now “matches cuBLAS” is folklore transmitted through issue threads. Turning it into a testable contract would remove a real and unpriced switching cost.

Attack at inference, not at parity. The kernel surface is small, the workloads are known, the customers are cost-sensitive, and memory capacity per package is a lever NVIDIA does not always win. Trying to be CUDA-complete is the losing strategy; being excellent at forty kernels is the winning one.

Spend compute instead of headcount. Every dollar into autotuning, superoptimization and learned scheduling attacks the actual scarce resource. Helion beating hand-written Triton on average is the proof of concept. This is the only lever in the list that gets cheaper over time.

And keep the compiler argument in proportion. The binding constraint on who gets to serve AI workloads in 2026 is HBM supply, advanced packaging capacity, rack-scale networking and power. A perfect compiler on a chip you cannot buy, in a rack you cannot cool, connected by a fabric that stalls at 72 GPUs, wins nothing. The compiler moat is real and it is roughly the fourth most important moat NVIDIA has.

Subscribe now

Check it yourself

Nothing above requires trusting me. The relevant artifacts are all inspectable with the toolkit you already have installed.

See the driver script decompose:

nvcc --dryrun -arch=sm_90 kernel.cu

Every stage, in order, with the real command lines. cicc, ptxas, fatbinary, nvlink.

Watch the virtual ISA:

nvcc -arch=sm_90 -ptx kernel.cu -o kernel.ptx

Note the unbounded %r virtual registers and the total absence of scheduling information.

Watch the last mile do the work:

ptxas -arch=sm_90 -v kernel.ptx -o kernel.cubin

The -v output tells you registers used, spill stores, spill loads, shared memory. That register count is a policy decision ptxas made on your behalf, and it sets your occupancy.

Look at the control bits:

nvdisasm -c -g kernel.cubin
cuobjdump --dump-sass kernel.cubin

The bracketed fields next to each instruction are the scheduling payload. Compare the same kernel compiled with and without -maxrregcount and watch the stall counts and barrier usage change.

Confirm the frontend is a commodity:

clang++ -x cuda --cuda-gpu-arch=sm_90 --cuda-device-only -S kernel.cu -o clang.ptx

Diff that against nvcc‘s PTX. They differ. Then push both through ptxas and compare the SASS. That comparison is the whole argument of this piece in two commands: two independent frontends, one backend, and the backend is where the differences stop mattering.

Find the boundary: try to turn a modified SASS listing back into a cubin using only NVIDIA tooling. There is no such command. That absence is the moat.

Subscribe now

Confidence dossier

Tier A, directly verifiable from primary sources or first-party documentation

• nvcc is a driver invoking cicc, ptxas, fatbinary and nvlink; observable via --dryrun

• NVVM IR is a documented LLVM IR subset with a public libNVVM API; Clang’s NVPTX backend is upstream

• PTX is a virtual ISA; the driver JIT-compiles it, providing forward compatibility

• There is no NVIDIA-provided SASS assembler; nvdisasm and cuobjdump disassemble only

• Blackwell deprecated wgmma and introduced tcgen05, Tensor Memory, single-thread MMA issue and cta_group::2

• CUDA Tile IR shipped in CUDA 13.1; the MLIR dialect, bytecode format and conformance suite are open source; the bytecode is specified as stable and versioned

• A Triton to Tile IR backend exists, authored by NVIDIA

• The CUDA EULA restricts translating SDK-generated output artifacts to non-NVIDIA platforms

• wgmma requires the sm_90a architecture-specific target and tcgen05 requires sm_100a or sm_103a; code for a targets is not forward compatible

• A cubin is an ELF64 object with per-kernel .text, .nv.constant0 and .nv.info sections

• Blackwell Tensor Memory is 256 KB per SM, 128 lanes by 512 columns of 32 bits, allocated in power-of-two column counts

• MXFP8 uses one E8M0 scale per 32 elements; NVFP4 uses an E4M3 scale per 16 elements plus a tensor-level FP32 scale

Tier B, well-supported by multiple independent secondary sources or peer-reviewed measurement

• The Volta-and-later control field layout: reuse, wait mask, read and write barrier indices, yield, stall count, packed into the instruction word

• Fixed-latency instructions lack hardware RAW interlocks; incorrect stall counts produce incorrect results

• Instruction encoding widened from 64 to 128 bits at Volta

• H100 register file of 65,536 32-bit registers per SM, 64 maximum resident warps, eight-register allocation granularity, and the occupancy arithmetic that follows from them

• FlashAttention 4 in CuTe DSL outperforming cuDNN 9.13 and Triton by the margins cited

• ROCm 7.2 library and AITER performance improvements on MI300 through MI355X

• Helion’s reported speedups over Triton and torch.compile

• NCCL’s channel-per-thread-block structure, its protocol selection, and the fact that collectives consume SMs that are then unavailable for compute

• In-network reduction in NVSwitch and InfiniBand switch silicon

• SCALE’s benchmark claims against HIP and nvcc

Tier C, single-source, historical, or reported rather than independently verified

• maxas SGEMM efficiency figures on GM204 versus contemporaneous cuBLAS

• The annotated SASS fragment, which is reconstructed in the documented format rather than captured from a specific compilation

• DeepSeek’s specific SM partitioning and custom PTX details, as described in the V3 paper and subsequent reporting

• Spectral Compute’s NVIDIA Inception membership and its strategic reading

• cuTile’s competitiveness with cuBLAS and cuDNN at large shapes, which is a first-party claim awaiting third-party benchmarks

Tier D, my model or my opinion, flagged as such

• The $6M to $15M per-generation estimate for an inference kernel layer, and the 10x to 50x multiplier for the full surface

• The claim that layer 3 is the load-bearing wall rather than layer 1

• The reading of Tile IR as a strategic response to Triton’s position in PyTorch

• The assertion that the moat is best modelled as a labour market

• The ranking of the compiler moat as roughly fourth behind supply, packaging and networking

• All five dated calls, which are forecasts and should be read as such

• The four self-criticisms, which are my own estimate of where this argument is weakest

Subscribe now

Bibliography

Primary documentation

• NVIDIA, Parallel Thread Execution ISA. https://docs.nvidia.com/cuda/parallel-thread-execution/

• NVIDIA, CUDA Binary Utilities (cuobjdump, nvdisasm, SASS instruction set tables). https://docs.nvidia.com/cuda/cuda-binary-utilities/

• NVIDIA, NVVM IR Specification. https://docs.nvidia.com/cuda/nvvm-ir-spec/

• NVIDIA, CUDA Compiler Driver NVCC. https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/

• NVIDIA, License Agreement for NVIDIA Software Development Kits. https://docs.nvidia.com/cuda/eula/index.html

• NVIDIA, CUDA Tile. https://developer.nvidia.com/cuda/tile

• NVIDIA, Tile IR: Binary Format. https://docs.nvidia.com/cuda/tile-ir/latest/sections/bytecode.html

• NVIDIA, cuTile Python documentation. https://docs.nvidia.com/cuda/cutile-python/

• NVIDIA, CUTLASS documentation: tcgen05 MMA Programming Guide. https://docs.nvidia.com/cutlass/latest/media/docs/pythonDSL/mma_docs/tcgen05_programming.html

• NVIDIA, CUTLASS: Blackwell SM100 functionality. https://docs.nvidia.com/cutlass/4.2.1/media/docs/cpp/blackwell_functionality.html

• AMD, ROCm documentation: vLLM inference optimization. https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/vllm-optimization.html

Microarchitecture and reverse engineering

• Jia, Maggioni, Staiger, Scarpazza, Dissecting the NVIDIA Volta GPU Architecture via Microbenchmarking, arXiv:1804.06826. https://arxiv.org/pdf/1804.06826

• Huerta, Abaie et al., Analyzing Modern NVIDIA GPU cores, arXiv:2503.20481. https://arxiv.org/pdf/2503.20481

• Dissecting and Modeling the Architecture of Modern GPU Cores, MICRO 58, 2025. https://dl.acm.org/doi/10.1145/3725843.3756041

• Luo et al., Benchmarking and Dissecting the Nvidia Hopper GPU Architecture, arXiv:2402.13499

• CuAsmRL: Optimizing GPU SASS Schedules via Deep Reinforcement Learning, arXiv:2501.08071. https://arxiv.org/html/2501.08071v1

• GPA: A GPU Performance Advisor Based on Instruction Sampling, arXiv:2009.04061

• Finn, What happens when you run a CUDA kernel. https://fergusfinn.com/blog/what-happens-when-you-run-a-gpu-kernel/

Assembler and toolchain projects

• Gray, maxas, Maxwell assembler and control code notes. https://github.com/NervanaSystems/maxas/wiki/Control-Codes

• CuAssembler. https://github.com/cloudcores/CuAssembler

• openptxas, gpuocelot. https://github.com/gpuocelot/openptxas

• NVlabs, SASSI: Flexible GPGPU instrumentation, distributed as a closed-source ptxas fork. https://github.com/NVlabs/SASSI

• NVIDIA, cuda-tile. https://github.com/NVIDIA/cuda-tile

Kernel authoring layers

• Colfax Research, CUTLASS Tutorial: Writing GEMM Kernels Using Tensor Memory for NVIDIA Blackwell GPUs. https://research.colfax-intl.com/cutlass-tutorial-writing-gemm-kernels-using-tensor-memory-for-nvidia-blackwell-gpus/

• Colfax Research, CUTLASS Tutorial: Hardware-supported Block-scaling with NVIDIA Blackwell GPUs. https://research.colfax-intl.com/cutlass-tutorial-hardware-supported-block-scaling-with-nvidia-blackwell-gpus/

• SemiAnalysis, Dissecting Nvidia Blackwell: Tensor Cores, PTX Instructions, SASS, Floorsweep, Yield. https://newsletter.semianalysis.com/p/dissecting-nvidia-blackwell-tensor

• PyTorch, Helion: A High-Level DSL for Performant and Portable ML Kernels. https://pytorch.org/blog/helion/

• pytorch/helion repository. https://github.com/pytorch/helion

• PyTorch RFC, Inductor cuTile Backend, issue 175311. https://github.com/pytorch/pytorch/issues/175311

• NVIDIA Developer Blog, Advancing GPU Programming with the CUDA Tile IR Backend for OpenAI Triton. https://developer.nvidia.com/blog/?p=112268

• Red Hat Emerging Technologies, From hand-tuned to generated: a reproducible Triton GPU kernel benchmark across different vendors. https://next.redhat.com/2026/02/12/from-hand-tuned-to-generated-a-reproducible-triton-gpu-kernel-benchmark-across-different-vendors/

• Modular, Structured Mojo Kernels Part 4: Portability and the Road Ahead. https://www.modular.com/blog/structured-mojo-kernels-part-4-portability-and-the-road-ahead

• Mojo: MLIR-Based Performance-Portable HPC Science Kernels on GPUs for the Python Ecosystem, arXiv:2509.21039

Alternatives and portability

• Spectral Compute, SCALE. https://scale-lang.com/ and https://github.com/spectral-compute/scale-docs

• HPCwire, Spectral Compute Aims to Set CUDA Free. Will It Succeed?, July 2026. https://www.hpcwire.com/2026/07/09/spectral-compute-aims-to-set-cuda-free-will-it-succeed/

• SCALE: Ahead-Of-Time Compilation of CUDA for AMD GPUs, Middleware 2024 demo track. https://dl.acm.org/doi/10.1145/3704440.3704782

• ZLUDA, Update Q1 and Q2 2026: back to the roots. https://vosen.github.io/ZLUDA/blog/zluda-update-q1q2-2026/

• Phoronix, ZLUDA For CUDA On Non-NVIDIA GPUs Enables AMD ROCm 7 Support. https://www.phoronix.com/news/ZLUDA-ROCm-7

• Tom’s Hardware, Nvidia bans using translation layers for CUDA software, March 2024. https://www.tomshardware.com/pc-components/gpus/nvidia-bans-using-translation-layers-for-cuda-software-to-run-on-other-chips-new-restriction-apparently-targets-zluda-and-some-chinese-gpu-makers

• vLLM Blog, Beyond Porting: How vLLM Orchestrates High-Performance Inference on AMD ROCm. https://vllm.ai/blog/2026-02-27-rocm-attention-backend

• AMD, Speed is the Moat: Inference Performance on AMD GPUs. https://www.amd.com/en/developer/resources/technical-articles/2026/inference-performance-on-amd-gpus.html

• tinygrad repository and release notes. https://github.com/tinygrad/tinygrad

• Pall et al., GROMACS on AMD GPU-Based HPC Platforms: Using SYCL for Performance and Portability, arXiv:2405.01420

Model and systems work referenced

• DeepSeek-AI, DeepSeek-V3 Technical Report, arXiv:2412.19437

• Tom’s Hardware, DeepSeek’s AI breakthrough bypasses industry-standard CUDA for some functions. https://www.tomshardware.com/tech-industry/artificial-intelligence/deepseeks-ai-breakthrough-bypasses-industry-standard-cuda-uses-assembly-like-ptx-programming-instead

• Dao et al., FlashAttention series, most recently FlashAttention 4 on Blackwell in CuTe DSL

• Bauer et al., Singe: Leveraging Warp Specialization for High Performance on GPUs, PPoPP 2014

Corrections and disagreements are welcome and get published. If you have measured ptxas output against hand-scheduled SASS on Hopper or Blackwell, I would like to see the numbers.

Share The Software Frontier

Intro

Moonshot AI released Kimi K3 precisely on July 16. The headline number is 2.8 trillion parameters, which makes it the largest open-weight model ever announced.

We spent the launch week reading almost everything published about it, and the more we read, the less the parameter count felt like the point. The point is that a 2.8T model is being served today, at Claude Sonnet prices, with a flat rate across a one million token context window.

The reasons that is possible sit exactly at the layer this newsletter cares about: attention design, expert routing, quantization format, and cache economics.

A caveat before the numbers. The technical report is not out yet. Moonshot’s launch blog states that architecture, training, and evaluation details will arrive alongside the report, and several figures below come from community analysis of the launch documentation rather than from a paper.

Where we compute something ourselves, the assumptions are stated inline and marked as ours, and there is a confidence appendix at the end.

This is a launch-week analysis, not a postmortem.

Subscribe now

What shipped

The facts first, because launch weeks always tend to generate a lot of useless fog. The model went live on July 16 through kimi.com, the Kimi apps, Kimi Work, Kimi Code, and the Moonshot API at api.moonshot.ai under the model ID kimi-k3.

It is also listed on OpenRouter at the same rates as the direct API. What did not ship is the checkpoint: Moonshot says the full weights land by July 27, and until the files appear on Hugging Face, K3 is a hosted model you can call but not download.

Verdent’s launch guide reports the weights are coming under a modified MIT license, which would basically match the K2 line, but the license text is unconfirmed until the release itself.

The specification, per Moonshot: 2.8 trillion total parameters, a mixture of experts activating 16 of 896 experts per token, native vision input rather than a bolted-on adapter, and a context window of 1,048,576 tokens. The model reasons before every answer. At launch, reasoning effort is locked to max, with lower effort modes promised in later updates. That detail matters more than it sounds, and we will get to it in the pricing section.

The timing is not subtle either. VentureBeat notes the release landed just a few days before the World Artificial Intelligence Conference in Shanghai, and frames it as a comeback move for a company whose position had eroded badly during DeepSeek’s rise.

Moonshot is backed by Alibaba, which also builds the Qwen line, so the Chinese open-model field is now crowded with players who share an investor and compete anyway. Moonshot’s own blog claims that for nine of the past twelve months, Kimi models have held the frontier of open-source scale.

Whatever you think of that framing, the release cadence behind it is real: K2 in July 2025 at one trillion total parameters, K2 Thinking in November with quantization-aware INT4, and now K3 at nearly three times the K2 total.

Share

The shape of the model

Three important architectural pieces define K3: the sparsity of the expert layer, the hybrid attention stack, and a modified residual path. Each one is a serving decision as much as a modeling decision.

My advice is to start with sparsity, because the trend line is the clearest signal of where frontier MoE design is going. K2 activated 8 of 384 experts, roughly 32B active parameters out of 1T total, an activation ratio around 3.1 percent. K3 activates 16 of 896.

Moonshot has not published the active parameter count; the community model card analysis on Hugging Face estimates roughly 50B active equivalent, which against 2.8T total is circa 1.8 percent. Total capacity nearly tripled while active compute grew by maybe half.

That is the whole game: parameters are cheap to store and expensive to move, so you scale what sits in memory and hold nearly flat what has to cross the datapath every token.

DeepSeek formalized this direction at 671B total and 5.5 percent active; Moonshot is now running it harder than anyone with an open checkpoint.

The lineage is worth one sentence: the sparse expert layer goes back to Shazeer’s 2017 outrageously large networks paper, was made trainable at datacenter scale by GShard and Switch, and was pushed toward fine-grained experts with a shared trunk by the DeepSeek line; K3 extends that same axis to 896, and the references at the end walk the chain to the primary sources.

The expert layer runs on what Moonshot calls Stable LatentMoE. According to the launch documentation, routing happens in a latent space rather than directly on token representations, load is managed by a mechanism called Quantile Balancing, and overflow tokens are soft-dropped rather than hard-bounced.

All three claims obviously await the technical report, but the intent is legible: at 896 experts, classical routing instability and load skew are the failure modes that kill both training and serving, and everything named here is aimed at them.

If you have ever watched a MoE deployment where two hot experts saturate their devices while the other few hundred idle, you know why a lab would lead with the word Stable.

The attention stack is a hybrid, and it has a paper trail. K3 is built on Kimi Delta Attention, KDA, which Moonshot introduced with the Kimi Linear report in late 2025.

KDA is essentially a gated refinement of DeltaNet: a delta-rule state update with fine-grained per-channel decay instead of a scalar forget gate, implemented with chunked parallel kernels so the recurrence trains at practical speed.

In the Kimi Linear configuration these layers were interleaved with full attention at a three to one ratio, the global layers used multi-head latent attention, and the paper reported on the order of a 75 percent KV cache reduction and up to a reported 6x decode throughput gain at million-token lengths against a full-attention baseline.

K3’s global layers use Gated MLA, a gated variant of the same latent attention. Moonshot has not confirmed K3’s layer ratio, but the lineage tells you the design logic: KDA layers carry a fixed-size recurrent state that costs the same at position one million as at position one hundred, and only the global layers accumulate position-indexed KV, compressed into latents at that.

The million-token window is not priced flat out of generosity. It is priced flat because most of the stack does not pay for length.

For readers who want the update rule, KDA in the Kimi Linear formulation maintains a matrix-valued state S, rewritten each step as

S_t = (I - beta_t k_t k_t^T) Diag(alpha_t) S_{t-1} + beta_t k_t v_t^T

read right to left: decay the previous state channel by channel through Diag(alpha_t), erase the stale association along the incoming key direction, then write the new key-value pair at strength beta_t.

The per-channel diagonal gate is the refinement over Gated DeltaNet’s scalar decay, restoring some of the selectivity full attention gets for free, and the chunked kernel that makes the recurrence trainable at scale, a specialized diagonal-plus-low-rank formulation, is the same computational shape the prefill discussion below returns to.

The third piece is Attention Residuals, AttnRes, described as a drop-in replacement for standard residual connections. Instead of every layer adding onto one uniformly accumulated stream, each layer can selectively retrieve representations from arbitrary earlier layers.

The claimed motivation is depth: in very deep MoE stacks, different experts fire at different depths, and a selective residual path keeps early information reachable without forcing it through every intermediate transformation.

It is the kind of change that sounds small and touches everything, and it is the single item on this list we most want to see ablated in the report.

Around the edges: a custom activation Moonshot calls SiTU, a sigmoid tanh unit replacing the usual SwiGLU family, and a per-head variant of the Muon optimizer, scheduling learning rates at the level of individual attention heads.

Muon is a K2-era inheritance; per-head scheduling is new. Moonshot’s aggregate claim is that the architectural and data changes together give K3 roughly 2.5 times the scaling efficiency of K2, meaning more capability per unit of training compute. That number is vendor-measured, marked as unverified, and should be treated accordingly.

The vision path deserves more than a scope note, because it is half the modality story and it touches everything above. What is known: vision is native rather than adapter-based per the community model card reading, continuing the multimodal line Moonshot opened with K2.5, which means image tokens enter the same transformer stack and the same 896-expert router as text, and the multimodal scores ship on day one rather than in a point release.

The API mechanics are documented even where the economics are not: images go in as base64 payloads or platform file references, public image URLs are not accepted, and OpenRouter lists image input at the standard rates.

The agent framing leans on vision explicitly, a model pitched for iterating against screenshots, logs, and runtime feedback rather than captioning, and Moonshot’s own multimodal numbers, 81.6 on MMMU-Pro and 94.3 on MathVision, sit in the launch tables.

What is not known at all is the exchange rate. The rate card is a single line, and whether an image bills at a multiplier over the $3 text rate or simply lands as however many tokens the tokenizer emits is unconfirmed, a gap the pricing trackers have flagged since day one.

The serving consequences do not wait for the answer. Whatever the patch geometry turns out to be, images are prefill load: a screenshot-heavy agent loop is a long-prompt workload wearing a different costume, and it collides with prefix caching in a way text does not, because a refreshed screenshot mid-transcript invalidates every cached token downstream of it.

The append-only discipline from the harness section applies doubly to media: attach new frames at the tail, never edit them in place.

And there is a broad research question waiting in the checkpoint. Early fusion means text and vision tokens share the router and the expert pool.

Whether the 896 experts partition by modality, some effectively becoming vision specialists, or blend, is exactly the kind of question the July 27 weights let anyone answer with an afternoon of routing statistics, and the answer feeds straight back into pruning, because a modality-partitioned pool compresses very differently from a blended one.

Subscribe now

Four bits from the start

The decision that makes 2.8T servable is not in the attention stack. It’s in the number format. K3 was trained with quantization-aware training from the supervised fine-tuning stage onward: MXFP4 for weights, MXFP8 for activations.

This is not a post-training quantization pass applied to a bf16 checkpoint. The model learned to live inside four-bit weights, compensating for quantization error during training instead of absorbing it afterward.

Terms, briefly, because the acronym is doing a lot of work. MXFP4 is the four-bit member of the OCP Microscaling family: weights are grouped into blocks of 32, each element is an E2M1 float, one sign bit, two exponent bits, one mantissa bit, and every block shares a single 8-bit power-of-two scale.

The element grid has eight magnitudes, 0, 0.5, 1, 1.5, 2, 3, 4, 6, spaced logarithmically rather than uniformly. Activations ride one tier up as MXFP8, eight-bit elements under the same block-scale scheme, where the precision matters for numerical stability.

The migration from K2 Thinking, which shipped INT4 quantization-aware training in November 2025, to floating-point four bits here is a change of grid, and grids have consequences you can measure. So we measured, on a synthetic but realistic weight distribution, a Gaussian with and without a sprinkle of 30-sigma outliers, block size 32, both formats implemented per spec:

# MXFP4 (E2M1 elements, shared E8M0 block scale) vs INT4 groupwise, block size 32.
# Question: what does each grid do to a realistic weight distribution?
import numpy as np
rng = np.random.default_rng(7)

E2M1 = np.array([0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0])   # magnitudes

def quant_mxfp4(w, block=32):
    out = np.empty_like(w)
    for i in range(0, w.size, block):
        b = w[i:i+block]
        amax = np.abs(b).max()
        if amax == 0:
            out[i:i+block] = 0; continue
        shared_exp = np.floor(np.log2(amax)) - 2          # emax of E2M1 is 2
        scale = 2.0 ** shared_exp                          # E8M0: power of two only
        x = np.clip(np.abs(b) / scale, 0, 6.0)
        idx = np.abs(x[:, None] - E2M1[None, :]).argmin(1) # round to nearest level
        out[i:i+block] = np.sign(b) * E2M1[idx] * scale
    return out

def quant_int4(w, block=32):
    out = np.empty_like(w)
    for i in range(0, w.size, block):
        b = w[i:i+block]
        amax = np.abs(b).max()
        if amax == 0:
            out[i:i+block] = 0; continue
        scale = amax / 7.0                                 # fp16 scale, uniform grid
        out[i:i+block] = np.clip(np.round(b / scale), -7, 7) * scale
    return out

def report(name, w):
    for label, q in (("mxfp4", quant_mxfp4(w)), ("int4-g32", quant_int4(w))):
        rel = np.linalg.norm(w - q) / np.linalg.norm(w)
        small = np.abs(w) < np.quantile(np.abs(w), 0.99)   # error on the 99% mass
        rel_small = np.linalg.norm(w[small] - q[small]) / np.linalg.norm(w[small])
        dead = (q[small] == 0).mean() * 100                # small weights crushed to zero
        print(f"{name:18s} {label:9s} relRMSE {rel:.4f}  relRMSE(99% mass) {rel_small:.4f}  zeroed {dead:5.2f}%")

w_gauss = rng.normal(0, 0.02, 1 << 20)
report("gaussian", w_gauss)

w_out = w_gauss.copy()                                     # 0.2% outliers at 30x sigma
hit = rng.choice(w_out.size, w_out.size // 500, replace=False)
w_out[hit] = rng.choice([-1, 1], hit.size) * 0.6
report("gaussian+outliers", w_out)

Executed output:

gaussian           mxfp4     relRMSE 0.1141  relRMSE(99% mass) 0.1102  zeroed  8.19%
gaussian           int4-g32  relRMSE 0.0970  relRMSE(99% mass) 0.1012  zeroed 13.37%
gaussian+outliers  mxfp4     relRMSE 0.1919  relRMSE(99% mass) 0.2353  zeroed 13.01%
gaussian+outliers  int4-g32  relRMSE 0.1499  relRMSE(99% mass) 0.2585  zeroed 18.40%

Read the table honestly and neither format dominates.

INT4 with a floating-point scale per block really wins global reconstruction error in both regimes, because its fifteen uniform levels and exact scale beat eight logarithmic levels under a power-of-two scale on Gaussian mass.

What the FP4 grid buys is essentially at the bottom of the distribution: it zeroes meaningfully fewer small weights, 8.2 versus 13.4 percent on clean data, 13.0 versus 18.4 with outliers, and it carries the bulk of the mass with lower error when an outlier inflates a block’s scale, because the log-spaced levels keep resolution near zero exactly where a uniform grid goes coarse.

Small weights are where accumulated damage shows up at scale, so this is not a cosmetic difference, but it is also not the headline reason for the format.

The headline reasons are that the exponent-only scale is free to apply in hardware, that Blackwell tensor cores execute block-scaled MXFP4 natively at full rate, with the Hugging Face community analysis reporting the same for AMD’s MI400 class, and that quantization-aware training exists precisely to claw back whatever the grid loses.

A lab that wanted a research artifact would release bf16 and let the community fight over quants. A lab that wants its model actually served releases the four-bit weights it trained, in the format the current accelerator generation runs at full rate.

The arithmetic consequence: 2.8 trillion parameters at four bits is 1.4 TB of element payload, and closer to 1.5 TB once the per-block scales ride along, since a 32-element block carries 136 bits, 4.25 effective bits per weight.

The same model in FP16 would be about 5.6 TB. That factor of nearly four compounds through the memory system: fewer devices to hold the model, and a quarter of the bandwidth per token spent reading weights. Be precise about what the four does and does not buy on the wire, though.

Expert dispatch and combine move activations, and activations are MXFP8, so all-to-all traffic halves relative to a bf16 model rather than quartering. The full 4x applies to weight movement: initial loading, host offload, expert migration and rebalancing.

Weights and activations shrink by different factors, and conflating them is how serving estimates go wrong by 2x.

There is a subtler consequence for the open release. When the weights drop on July 27, they will be born four-bit.

There will be no ambiguity about which community quant is faithful, because the FP4 checkpoint is the reference, not a lossy derivative. The flip side is that the fine-tuning story gets strange: LoRA and QLoRA on top of an already-quantized MoE base is thinly explored territory, a point the Hugging Face overview raises as an open research question.

The first serious K3 fine-tunes will be experiments in method, not just in data. One loose thread already dangles here. One launch tracker describes OpenRouter’s route as Moonshot’s hosted INT4 endpoint.

That may be nothing more than four-bit shorthand for the MXFP4 checkpoint, or it may mean the serving fleet runs a different quantization than the training format, and the difference matters to anyone who plans to benchmark API behavior against the weights they download. File it with the questions the report owes.

Subscribe now

What it takes to serve it

Napkin arithmetic first, ours and labeled as such, all of it downstream of the ~50B active estimate.

A single decode token’s forward pass touches 16 experts per MoE layer plus the shared trunk, on the order of 25 GB of weight reads at four bits. Against the roughly 8 TB/s of HBM bandwidth on a current Blackwell-class part, that is about three milliseconds of pure weight traffic, an upper bound near 320 tokens per second even if one device could hold the whole model, which it cannot.

The ratio is the point of the exercise: at FP16 the same pass would read 100 GB per token and the model would be bandwidth-strangled on any realistic hardware. Four-bit weights are not an optimization here. They are the enabling condition.

That 25 GB figure is also a batch-size-one number, and batch size one is the worst case this architecture has. Here is the mechanism, because it is the actual thesis of extreme sparsity. Dense-layer weights are read once per forward pass and amortize cleanly across every token in the batch.

Expert weights amortize only when tokens share experts, and with 16 of 896 routing, sharing takes scale.

Rather than assert the curve, we simulated it, sixteen trials per point, with two popularity models: uniform routing, which is what Quantile Balancing is trying to buy, and a moderately skewed distribution with a coefficient of variation around 0.7, which is what real routers produce when you let them:

# How expert weight traffic per token falls with batch size, and what routing
# skew does to it. E=896, k=16, ~1.5 GB per expert at MXFP4 (assumption, tier E).
import numpy as np
rng = np.random.default_rng(3)

E, K, GB_PER_EXPERT, TRIALS = 896, 16, 1.5, 16
batches = [1, 8, 64, 256, 1024, 4096]

def run(pop):                       # pop: expert popularity distribution, sums to 1
    uniq_gb, imb = [], []
    for B in batches:
        u, m = [], []
        for _ in range(TRIALS):
            draws = np.concatenate([rng.choice(E, K, replace=False, p=pop)
                                    for _ in range(B)])
            counts = np.bincount(draws, minlength=E)
            u.append((counts > 0).sum())
            m.append(counts.max() / (B * K / E))          # max load over mean load
        uniq_gb.append(np.mean(u) * GB_PER_EXPERT / B)
        imb.append(np.mean(m))
    return uniq_gb, imb

uniform = np.full(E, 1 / E)
skewed = rng.dirichlet(np.full(E, 2.0))                    # cv ~0.7, mild hotness
g_u, i_u = run(uniform)
g_s, i_s = run(skewed)

print(f"{'batch':>6} {'uniform GB/tok':>15} {'imbalance':>10} {'skewed GB/tok':>14} {'imbalance':>10}")
for j, B in enumerate(batches):
    print(f"{B:>6} {g_u[j]:>15.2f} {i_u[j]:>10.1f} {g_s[j]:>14.2f} {i_s[j]:>10.1f}")
print(f"floor at large batch: {E * GB_PER_EXPERT:.0f} GB / B")
print("EP sizing at MXFP4: EP64 ->", E // 64, "experts/GPU,", E // 64 * GB_PER_EXPERT,
      "GB | EP128 ->", E // 128, "experts/GPU,", E // 128 * GB_PER_EXPERT, "GB")

Executed output:

 batch  uniform GB/tok  imbalance  skewed GB/tok  imbalance
     1           24.00       56.0          24.00       56.0
     8           22.72       14.9          21.82       18.8
    64           14.42        4.9          12.65        7.4
   256            5.20        2.7           4.80        5.5
  1024            1.31        1.8           1.30        4.6
  4096            0.33        1.4           0.33        4.6
floor at large batch: 1344 GB / B
EP sizing at MXFP4: EP64 -> 14 experts/GPU, 21.0 GB | EP128 -> 7 experts/GPU, 10.5 GB

Three things in that table are worth internalizing. First, the amortization cliff: per-token expert traffic barely moves from batch 1 to batch 8, because eight tokens drawing 128 expert slots land on about 120 distinct experts, which is to say almost no sharing at all.

It falls to 5 GB per token at batch 256, where the batch has touched nearly the whole 1,344 GB expert pool once, and approaches the 1,344 divided by B floor beyond a thousand concurrent tokens.

Second, the imbalance column, max expert load over mean, is the one your step time actually obeys, because the slowest device sets the pace of every synchronous layer.

Under uniform routing it decays toward 1.4 as the law of large numbers kicks in. Under skew it plateaus at about 4.6 and never recovers: the hot expert stays hot, no matter how big the batch gets. Third, notice the trade hiding between the columns: skew slightly reduces traffic, popular experts get reused, while wrecking balance.

Quantile Balancing, read through this table, is Moonshot choosing the uniform column, paying the worst-case traffic to buy the converging imbalance, because a 4.6x hot spot at expert parallelism width is a 4.6x tax on every token.

The structural conclusion stands either way: a 1.8 percent activation ratio is only cheap at large batch under wide expert parallelism, where every device holds a slice of the pool, reads its residents once per step, and serves whichever tokens the all-to-all delivers.

At EP64 that slice is 14 experts and 21 GB per GPU; at EP128, 7 experts and 10.5 GB, numbers that fit comfortably beside a trunk shard and a KV allocation.

Extreme sparsity does not just permit big-batch serving. It demands it, which is why models shaped like this favor operators with deep pools of concurrent traffic and punish anyone trying to run them hot for three users.

The all-to-all itself can be sized on the back of the same napkin. Per MoE layer, each token’s hidden state ships to its 16 expert devices and 16 partial outputs ship back: roughly 2 x k x d_model bytes at FP8.

K2’s hidden width was 7,168 and K3’s is unpublished, so treat this as parameterized: 2 x 16 x 7,168 is about 229 KB per token per MoE layer, double K2’s 8-way routing at the same width.

Across an assumed 60 MoE layers that is roughly 14 MB of fabric traffic per generated token, and at 15,000 tokens per second of aggregate throughput the expert-parallel group is moving on the order of 200 GB/s. Inside an NVLink domain at 1.8 TB/s per GPU that is background noise.

Stretched across racks on 400G links at 50 GB/s each, it is the budget. We have written before about the all-to-all tax and the rack-scale networking built to pay it; K3 is precisely the class of model that hardware exists for.

If the 2 x k x d term feels abstract, here is the entire MoE data path, route, dispatch, grouped GEMM, combine, in one page of numpy, checked against a dense reference:

# The whole MoE data path in one page: route, dispatch, grouped GEMM, combine.
# Toy sizes so it prints; the shapes are the ones a serving engine juggles.
import numpy as np
rng = np.random.default_rng(0)

T, D, E, K = 8, 16, 32, 4                  # tokens, hidden, experts, top-k
x = rng.normal(size=(T, D)).astype(np.float32)
W = rng.normal(size=(E, D, D)).astype(np.float32) * 0.1    # one matrix per expert
router = rng.normal(size=(D, E)).astype(np.float32)

logits = x @ router                                        # [T, E]
topk = np.argsort(-logits, axis=1)[:, :K]                  # [T, K] expert ids
gates = np.exp(logits[np.arange(T)[:, None], topk])
gates /= gates.sum(1, keepdims=True)                       # softmax over the k winners

# dispatch: replicate each token k times, then sort rows by destination expert
flat_expert = topk.ravel()                                 # [T*K]
flat_token  = np.repeat(np.arange(T), K)                   # [T*K]
order = np.argsort(flat_expert, kind="stable")             # the permutation
xp = x[flat_token[order]]                                  # [T*K, D] permuted buffer

# grouped GEMM: one segment per expert, ragged sizes
counts = np.bincount(flat_expert, minlength=E)
offsets = np.concatenate([[0], np.cumsum(counts)])
yp = np.empty_like(xp)
for e in range(E):                                         # in CUDA: one grouped kernel
    s, t = offsets[e], offsets[e + 1]
    if t > s:
        yp[s:t] = xp[s:t] @ W[e]

# combine: unpermute and gate-weighted sum back to [T, D]
y = np.zeros_like(x)
np.add.at(y, flat_token[order], yp * gates.ravel()[order][:, None])

# reference: dense loop over tokens and their experts
y_ref = np.zeros_like(x)
for t in range(T):
    for j in range(K):
        y_ref[t] += gates[t, j] * (x[t] @ W[topk[t, j]])
print("matches dense loop:", np.allclose(y, y_ref, atol=1e-5))
print("permuted buffer rows:", xp.shape[0], f"({K}x the token count)")
bytes_per_tok = 2 * K * D * xp.itemsize
print(f"wire bytes per token, this layer: 2 x k x d x {xp.itemsize} = {bytes_per_tok}")
print("tokens per expert:", counts.tolist())

Executed output:

matches dense loop: True
permuted buffer rows: 32 (4x the token count)
wire bytes per token, this layer: 2 x k x d x 4 = 512
tokens per expert: [2, 0, 1, 1, 0, 2, 0, 0, 2, 0, 1, 1, 1, 2, 0, 1, 2, 1, 1, 1, 0, 2, 0, 0, 2, 0, 0, 2, 2, 3, 1, 1]

Every serving-engine complication is already visible in the toy. The permuted buffer holds k times the token count, so top-16 doubles the activation memory and permutation traffic of a top-8 model before a single expert FLOP is spent.

The per-expert segments are ragged, 0 to 3 tokens here, thousands in production, which is why the expert matmul is a grouped GEMM with runtime-sized groups rather than a clean batched one. And the scatter-add in the combine is the reduction the fabric has to carry back.

Scale the token count by six orders of magnitude and the expert count by 28, and the sort, the offsets, and the segment loop become the router kernel, the dispatch layout, and the grouped GEMM that the serving stacks will spend the next quarter optimizing.

Memory next. Weights at ~1.49 TB fit inside a single 8x B200 node, 1,536 GB, with about 50 GB to spare, and that sentence is a trap: the spare must hold every KV block, activation buffer, routing table, and graph on the box, so a one-node K3 is a demo, not a deployment.

Moonshot’s own launch guidance recommends a supernode of at least 64 accelerators, which community sizing reads as eight nodes of eight 80 GB GPUs, around 5 TB aggregate, a floor with headroom for cache and parallelism.

On a GB200 NVL72 the weights occupy roughly a tenth of the rack’s 13.4 TB of HBM. The DeepSeek 671B generation already made rack-scale expert parallelism a normal conversation; K3 raises the resident-byte requirement by roughly 4x…. and makes the full NVL72 look like the natural unit for a single replica rather than an extravagance.

The long-context economics deserve their own bytes, because the flat 1M pricing rests on them. In the DeepSeek V3 configuration that MLA descends from, each token stores a compressed latent per global layer: a 512-wide KV latent plus a 64-wide decoupled positional key, 576 elements, 576 bytes at FP8.

If K3 carries something like the Kimi Linear ratio, call it 15 to 20 global layers out of the stack, a full million-token sequence holds roughly 9 to 12 GB of KV, ours again and resting on the assumed ratio. A comparable dense model, 60 layers of grouped-query attention with 8 KV heads of dimension 128 at FP16, holds about 246 KB per token, 246 GB at the same window, per sequence.

That is a factor of about 25, and it is the difference between a million-token session being a scheduling problem and being a physical impossibility. The KDA layers contribute a fixed-size state regardless of length, which is a rounding error next to either number.

This is the architecture showing through the rate card: Moonshot can price the window flat because most of the model does not pay for it.

Prefill is the other half, and it is why the serving system is disaggregated. Filling the full window on a cache miss costs $3.15 of input at the sticker rate, $0.31 on a hit, and the compute behind the miss is brutal.

Linear-layer FLOPs alone run about 2 x 50B x 1M tokens, on the order of 10^17, several seconds on a full H100 node at perfect efficiency and the better part of half a minute at realistic utilization, before the attention term, which grows quadratically on the global layers and dominates at this length even confined to a quarter of the stack.

The KDA layers prefill differently again: a chunked scan, parallel within each chunk with recurrent state carried across chunk boundaries, a kernel shape with nothing in common with FlashAttention.

Prefill at 1M is a compute event; decode is a bandwidth event; the two want different hardware shapes and different batching. Moonshot serves K3 on Mooncake, its KVCache-centric disaggregated architecture, which splits prefill and decode across separate node pools and treats the cache as a first-class pooled resource between them.

Mooncake is not opaque vendor infrastructure: the design won the best paper award at FAST 25, the code is open-sourced, and its transfer engine has been integrated into the mainstream serving stacks, including vLLM and SGLang.

The company reports cache hit rates above 90 percent on coding workloads. That figure is vendor-reported, but it is plausible for agent traffic, where the same system prompt, repository context, and conversation prefix recur on every loop iteration, and the published architecture is engineered to produce exactly that number.

The hit rate is what makes the pricing model work, which brings us to the bill by way of the software that has to exist first.

Extreme sparsity does not just permit big-batch serving. It demands it.

Subscribe now

What has to change in the serving stacks

Self-hosting K3 is not a config file away, and it is worth being specific about where the engineering actually lands, because the July 27 story will be written in pull requests.

Start at the router. Every MoE layer produces logits of shape tokens by 896, takes a top-16, and must land tokens in per-expert segments fast enough not to shadow the GEMMs.

The grouped GEMM behind it now has up to 896 runtime-sized groups per layer per rank, and the tokens-per-expert variance the toy above showed becomes tile quantization waste at CUDA scale: an expert with 37 tokens still occupies whole tensor-core tiles.

The mature answers are persistent grouped kernels that pull ragged work from a queue instead of launching per group, and the CUTLASS grouped GEMM machinery most stacks already wrap.

What changes for K3 is scale, 896 groups against the 256 the current kernels were tuned around, and precision, because the groups now multiply block-scaled FP4 weights against FP8 activations.

Precision is where the hardware generations split. Blackwell’s fifth-generation tensor cores execute block-scaled MXFP4 natively, scales applied in the MMA itself, so B200-class serving runs the checkpoint as stored.

Hopper has no FP4 MMA path at all: an H100 deployment dequantizes weights to FP8 or BF16 in registers on the way into the matmul, the Marlin and Machete family of mixed-input kernels, which exist and are fast for dense weight-only GEMMs but are immature for 896-group grouped MoE at launch.

That dequant tax is not cosmetic. It is compute spent on format conversion inside the innermost loop, and it lands directly on the self-hosting ledger below: the 37 tokens per second per GPU that beats the sticker price assumes the grouped W4A8 path works well on Hopper, and today that is an assumption.

The communication layer has a shelf to raid but not a finished product. DeepSeek open-sourced DeepEP, NVSHMEM-based dispatch and combine kernels with FP8 payloads, IBGDA for the internode path, and a low-latency mode for decode, and it is the obvious starting point.

It was also built and tuned around the 256-expert, top-8 shape of the V3 generation. K3 doubles the per-token payload with top-16 and multiplies the destination fan-out with 896 experts, so the kernel launch geometry, the SM budget reserved for communication, and the overlap schedule that hides all-to-all latency behind expert compute all need retuning.

The dual-microbatch overlap trick, one microbatch’s communication hidden under another’s GEMMs, matters more here, not less, because the fabric term we sized above grows with k while the compute term holds near flat.

The subtlest problem is the cache manager, and it is the one we would watch first. Serving frameworks assume a homogeneous per-layer paged KV cache; the hybrid allocators built for the Mamba-generation models already broke that assumption, and K3 stresses it further: paged 576-byte latents for the global layers next to fixed-size recurrent state for the KDA layers.

Decode is straightforward. Prefix caching is not, because the entire cache-hit economy assumes you can resume from a stored prefix, and for a recurrent layer that means storing the state at the reuse boundary, not just the tokens.

A KV pool like Mooncake’s holds paged latents naturally; checkpointed KDA states at chunk boundaries are a second object type with different lifetime and size semantics, and how Moonshot handles state-restore for the linear layers is, to us, the single most interesting undisclosed detail in the serving stack.

Whoever implements it in the open stacks will be making a real design decision, not just a port.

The good news is how much is already on the shelf. The chunked delta-rule kernels for KDA shipped in the open with the Kimi Linear release through the flash-linear-attention line.

FlashMLA covers the global layers on Hopper. DeepEP covers the fabric, pending retuning. Mooncake’s transfer engine is merged where it needs to be.

What does not exist yet, anywhere public, is the integration: one engine that schedules block-scaled grouped GEMMs, hybrid attention with two kernel families, a two-type cache, and disaggregated state transfer, for a 1.5 TB checkpoint.

That is the artifact to watch for in the weeks after July 27, and its arrival date, more than the weights themselves, decides when open K3 is real.

Subscribe now

The bill

The rate card, from Moonshot’s platform pages:

All figures per million tokens. Web search tool calls bill separately at $0.004 each, though the tool ships flagged as being updated and one guide advises against production use for now. Vision input pricing is unconfirmed, per the vision section above.

The first thing the sticker tells you is positioning. K2 built its reputation as the cheap frontier model. K3 abandons that identity: at $3 and $15 it matches Claude Sonnet 5 to the cent, undercuts Opus 4.8 at $5 and $25 and GPT-5.6 Sol at $5 and $30, and costs several times its own K2.6 sibling.

Within the Chinese open cohort it is the expensive option, with DeepSeek V4 Pro at roughly a sixth of its input price and GLM 5.2 around half. Moonshot is telling you it believes K3 competes on capability, and it has priced away the discount narrative to prove it.

The second thing is that the sticker is not the effective price. The cache tier is. At a 90 percent hit rate, effective input cost is 0.9 times $0.30 plus 0.1 times $3.00, about $0.57 per million tokens.

Agent workloads with stable prefixes live near that floor. Caching is automatic, with no cache ID or TTL management, which removes the usual engineering excuse for missing it, and OpenRouter’s traffic view says the same thing from the demand side: average realized K3 prices run 60 to 80 percent below list once caching is counted.

Two more levers are worth knowing. K3 ships with no batch tier, the 60 percent batch discount on Moonshot’s menu covers only the K2 line, and the API defaults max_completion_tokens to 131,072, configurable up to the full window, a cap worth leaving in place until you have watched what max-effort thinking does to a bill.

The third thing cuts the other way. K3 always reasons, at max effort, and reasoning tokens bill as output at $15. Artificial Analysis measured 130 million output tokens to run its Intelligence Index evaluation, against a median near 63 million for comparable reasoning models.

The same measurements put decode speed at 62 tokens per second, under the 72.7 median for the price tier, and split latency into the two numbers an always-thinking model needs: about 2 seconds to the first streamed token, 34.24 seconds to the first answer token, which at the measured decode rate is roughly 2,000 thinking tokens spent before the answer begins on a standard 10,000-token workload.

Running the full index cost $2,709.75, nearly all of it output. Independent testing has recorded over thirteen thousand thinking tokens on a single short task, roughly twenty cents of deliberation before the first answer token.

Artificial Analysis puts the blended cost at $2.31 per million tokens on a 7:2:1 cache, input, output mix, but your mix will vary, and a maximally verbose reasoner shifts the mix toward the expensive column. “Price it on your own traces” is easy to say obv, so here is the whole model in twenty lines, run against three workload archetypes at launch rates:

# Request-level cost model for kimi-k3 at launch pricing.
# $3/MTok input on miss, $0.30 on cache hit, $15/MTok output.
# Thinking is always on and bills as output.
IN_MISS, IN_HIT, OUT = 3.00, 0.30, 15.00

def request_cost(in_tokens, cached_frac, answer_tokens, thinking_tokens):
    inp = in_tokens * ((1 - cached_frac) * IN_MISS + cached_frac * IN_HIT)
    out = (answer_tokens + thinking_tokens) * OUT
    return inp / 1e6, out / 1e6

archetypes = [
    # name, input, cached share, answer, thinking, requests
    ("short chat turn",      2_000, 0.20,   300, 13_000, 1),
    ("agent loop, 40 iters", 80_000, 0.95,   500,  6_000, 40),
    ("1M-doc analysis",     900_000, 0.00, 2_000, 10_000, 1),
]
print(f"{'workload':<22}{'$ input':>9}{'$ output':>9}{'$ total':>9}{'output share':>14}")
for name, i, c, a, th, n in archetypes:
    ci, co = request_cost(i, c, a, th)
    ci, co = ci * n, co * n
    print(f"{name:<22}{ci:>9.3f}{co:>9.3f}{ci+co:>9.2f}{co/(ci+co):>13.0%}")

Executed output:

workload                $ input $ output  $ total  output share
short chat turn           0.005    0.200     0.20          98%
agent loop, 40 iters      1.392    3.900     5.29          74%
1M-doc analysis           2.700    0.180     2.88           6%

The distribution of pain is the finding. On a small chat turn, deliberation is 98 percent of the bill: the answer costs half a cent and the thinking costs twenty.

A forty-iteration agent loop, the workload K3 is aimed at, still spends three of every four dollars on output even with a 95 percent cache hit rate doing everything right on the input side. Only the cold million-token document flips the ratio, and that shape pays $2.70 of prefill precisely once, after which it becomes an agent loop too.

The lever ordering falls out directly: for two of the three archetypes that matter, the promised low and high effort modes are worth more than any cache optimization you can do, and until they ship, the K2.6 rate card at $0.95 and $4 remains the right tool for every request that does not need a frontier model deliberating at full depth.

Subscribe now

The self-hosting ledger

The weight release invites an obvious question:

at what point does running K3 yourself beat paying Moonshot?

The napkin again, ours. Take the community floor configuration, 64 H100-class GPUs, and a rental price of $2 per GPU-hour: $128 per hour for the cluster.

To generate tokens cheaper than the $15 output rate, the cluster must sustain about 2,370 tokens per second aggregate, or 37 per GPU. That is an achievable number for a well-tuned MoE deployment with real batch depth, with the caveat the kernel section earned: on Hopper it assumes the grouped W4A8 dequant path matures, because every register spent converting FP4 is throughput the break-even does not get.

To beat the $2.31 blended rate that cached API traffic actually pays, the same cluster must sustain about 15,400 tokens per second, 240 per GPU, which is a very hard number for a model this size on Hopper-generation hardware.

Independent color corroborates the memory wall from an unexpected direction: an early community stress test found a 1.5 TB unified-memory Mac Studio cluster at the edge of what a K3-class million-token stack can carry, which is the same wall the 1.49 TB storage figure predicts.

The comparison is loose by construction: the API’s $15 covers output alone while your cluster serves whole requests, the blended target moves with your cache profile, and $2 per H100-hour is a spot figure that swings with region and term.

The asymmetry survives all of that. Self-hosting K3 competes with Moonshot’s sticker prices and loses badly to Moonshot’s cache economics, because the cache discount is not a margin decision you can copy, it is a serving architecture, Mooncake’s pooled KV plus a workload with 90 percent prefix reuse, that a single-tenant cluster cannot replicate without the traffic to feed it.

The teams for whom self-hosting pencils out are the ones who need it for reasons the rate card does not price: data residency (the hosted API processes in Singapore, per an EU procurement review), fine-tuned variants, latency floors, or sovereignty.

For everyone else, the July 27 download is leverage in a pricing negotiation, not a cost reduction.

Subscribe now

Benchmarks, with the usual caution

Moonshot’s own evaluation places K3 at frontier level but behind the two strongest closed models, Claude Fable 5 and GPT-5.6 Sol, an admission worth respecting because labs rarely volunteer it.

On GDPval-AA v2, the aggregate knowledge-work evaluation, the reported ordering is Fable 5 Max at 1815, GPT-5.6 Sol Max at 1747.8, K3 at 1687, and Opus 4.8 at 1600. On the science and multimodal side, Moonshot reports 93.5 on GPQA-Diamond, 81.6 on MMMU-Pro, and 94.3 on MathVision.

The coding numbers are where it gets interesting. K3 posts 88.3 on Terminal-Bench 2.1, half a point behind GPT-5.6 Sol, and takes the top score outright on Program Bench at 77.8 and on SWE Marathon at 42.0. SWE Marathon is the long-horizon one, and a model leading there while merely placing elsewhere is consistent with the design: a million-token window that holds an entire repository plus the full history of a long session is worth more on sustained work than on sprints.

FrontierSWE lands at 81.2, against a reported 86.6 for Fable 5 on the same benchmark, and DeepSWE at 67.5.

On agentic evaluations, VentureBeat’s read of the launch documentation has K3 first in four of eight real-world automation benchmarks, including Automation Bench, SpreadsheetBench 2, and BrowseComp, and second to Fable 5 in most of the rest.

The reported scores include 91.2 on BrowseComp, a 95.0 F1 on DeepSearchQA, and 30.8 on Automation Bench, which tells you as much about Automation Bench as about the model.

The claim we find most interesting is methodological: Moonshot says these results came from a single-agent setup running on the raw million-token window, with no context compression or management scaffolding (BrowseComp reads 90.4 in that configuration against the 91.2 headline figure, both vendor numbers).

Taken at face value, that is a real data point in the context-versus-orchestration argument, raw window plus strong retrieval beating elaborate multi-agent plumbing, and it needs independent replication before it graduates from press release to argument.

Two launch-week numbers deserve a somewhat explicit skepticism. K3 jumped from rank 18 to rank 1 on LMArena’s Frontend Code Arena within hours, at a score of 1679.

The placement itself has real sourcing, the Associated Press reported K3 at the top of the frontend ranking, but arena scores during a hype cycle measure attention as much as ability; check back in a month.

BenchLM’s weighted leaderboard still lacked a K3 row at press time, an absence worth noting, but the first independent aggregate has landed: Artificial Analysis scores K3 at 57 on its Intelligence Index, fourth of 189 models tracked, against an average of 31 for comparable models, and the index rolls up nine evaluations including GDPval-AA v2, Terminal-Bench 2.1, GPQA Diamond, and Humanity’s Last Exam.

Frontier company, by an outside ruler. The 2.5x scaling efficiency figure, the 90 percent cache hit rate, and the single-agent protocol are all vendor-reported. None of this means the numbers are wrong. It means the confirmations are scheduled for after July 27.

One demo is worth relaying because it is verifiable in kind if not yet in fact. Moonshot reports that K3, in a single 48-hour autonomous run, designed a serving chip for a nano model on its own architecture using open-source EDA tools against the Nangate 45nm library: four square millimeters, timing closed at 100 MHz, 1.46 million standard cells, 0.277 MB of SRAM, an INT4 multiply-accumulate array with fused dequantization, and a simulated decode throughput above 8,700 tokens per second. A staged demo, obviously, and simulation is not silicon.

But as a choice of demo it is telling: the lab wanted to show long-horizon agency on exactly the kind of task this audience does for a living.

A second staged demo aims at researchers: to reproduce the I-Love-Q universal relations in computational astrophysics, K3 reportedly cross-validated more than twenty papers, evaluated over three hundred equations of state, caught inconsistencies in published formulas, and shipped three thousand lines of Python plus an interactive dashboard in about two hours, against Moonshot’s estimate of one to two weeks for an experienced researcher.

Vendor-staged, like the chip, and chosen with the same message: long horizons are the product.

Subscribe now

Where it strains

Moonshot names three weaknesses, and all three have operational consequences.

The first is thinking history sensitivity: agent harnesses that truncate or rewrite the model’s reasoning traces degrade quality significantly. Launch coverage adds the reason: K3 was trained in a preserved thinking history mode, so a harness that drops the thinking, or a session handed mid-flight from another model, is operating outside the training distribution.

For agent builders this is a real constraint, and it compounds with the cache economics, because the two failure modes share a root cause: mutating the transcript.

Strip the thinking and quality drops; touch anything upstream in the serialized prefix and you lose the $0.30 tier and re-prefill from the perturbation point.

The defense against both is the same discipline, an append-only history with a verifiable prefix, cheap enough to enforce in a dozen lines:

# K3 penalizes two things at once: perturbing the prompt prefix (you lose the
# $0.30 cache tier) and truncating its reasoning traces (quality degrades).
# Same defense for both: an append-only history with a verifiable prefix.
import hashlib, json

def fingerprint(messages):
    blob = json.dumps(messages, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(blob.encode()).hexdigest()[:16]

def prefix_intact(old, new):
    return len(new) >= len(old) and fingerprint(new[:len(old)]) == fingerprint(old)

history = [
    {"role": "system", "content": "You are a code review agent."},
    {"role": "user", "content": "Review PR 4217."},
    {"role": "assistant", "content": "<thinking>...</thinking> Two issues found."},
]
appended  = history + [{"role": "user", "content": "Fix the first one."}]
truncated = [history[0], history[1],
             {"role": "assistant", "content": "Two issues found."}]  # thinking stripped
reordered = [history[1], history[0], history[2]]                     # system moved

for name, h in (("append-only", appended), ("thinking stripped", truncated),
                ("system reordered", reordered)):
    print(f"{name:<20} prefix intact: {prefix_intact(history, h)}")

# Client shape (not executed here; the endpoint is OpenAI-compatible):
#   client = OpenAI(base_url="https://api.moonshot.ai/v1", api_key=KEY)
#   r = client.chat.completions.create(model="kimi-k3", messages=history)
#   history.append({"role": "assistant", "content": r.choices[0].message.content})

Executed output:

append-only          prefix intact: True
thinking stripped    prefix intact: False
system reordered     prefix intact: False

The two False rows are the two expensive mistakes. A harness that “helpfully” compacts old reasoning fails the quality constraint and the cache constraint in the same commit; one that rebuilds the message list per call, reordering tools or regenerating timestamps, silently pays full prefill price on every iteration.

Fingerprint the prefix in development, assert it in production, and design summarization as a new branch rather than an edit to history. On long sessions where the window genuinely fills, the append-only rule forces the honest version of the tradeoff: fork the conversation with an explicit summary and eat one cold prefill, rather than quietly degrading the model mid-branch.

The second weakness is overeagerness. In ambiguous situations K3 tends to act rather than ask. On benchmarks that rewards decisiveness. In production it is how an agent cheerfully migrates the wrong database. The mitigation is boring and necessary: explicit confirmation gates in the harness, because the model will not supply them.

The third is polish. Moonshot concedes the subjective experience trails Fable 5 and GPT-5.6 Sol even where benchmark numbers are close. Benchmark parity and product parity remain different finish lines.

Share The Software Frontier

What July 27 decides

The weight release is the actual event; July 16 was the trailer.

The license text comes first. Modified MIT is reported across the launch coverage and would match the K2 family precedent, and the precedent has teeth worth knowing: the modified MIT that ships with K2.7 Code adds an attribution requirement that triggers at 100 million monthly active users or 20 million dollars of monthly revenue.

If K3 inherits the clause it is a display obligation for giants rather than a commercial restriction, but the binding text is the LICENSE file that lands with the weights, and nobody should sign a deployment plan before reading it.

Day-one inference support is the second gate, and the stacks section above is the checklist: block-scaled grouped GEMM at 896 groups, retuned dispatch and combine at top-16, the two-type cache with KDA state restore, and a Hopper W4A8 path that does not bleed the break-even dry.

Moonshot says it is working with inference partners ahead of the drop, and the Mooncake integrations already sitting in those codebases suggest the relationship is real; the commits will tell.

Third, reproduction. Someone outside Moonshot needs to run the benchmark suite, and the single-agent long-context protocol specifically.

Fourth, the research surface the checkpoint opens. With 896 experts in the open, expert specialization can finally be probed at frontier scale. Pruning experiments will ask whether the pool compresses to something that fits a smaller cluster, and what the quality curve looks like on the way down.

AttnRes can be ablated at small scale to see whether it generalizes or only pays at depth. And someone will attempt the first LoRA on a natively FP4 base and discover what breaks.

Then, the field is not holding still. The DeepSeek V4 line is already live at a fraction of K3’s price, and community reporting has a V4 refresh in staged testing; GLM 5.5 and MiniMax Pro are both expected around the trillion-parameter mark, and Qwen 4 is on the horizon. Whatever lead K3 holds is measured in weeks, and DeepSeek’s answer is the one to watch.

Our read, calmly: K3 does not win the frontier. By Moonshot’s own tables it sits a step behind Fable 5 and Sol. What it breaks is the assumption that open weights trail the frontier by six months and a capability class.

The gap is now single-digit percentage points and a dated download link, at Sonnet prices, on an architecture whose every major choice, the sparsity ratio, the hybrid attention, the four-bit training, the disaggregated serving, was made with the inference bill in mind. The link goes live on July 27, and that is where the work starts.

Sources and confidence

A, primary: Moonshot’s K3 launch blog and platform documentation: release dates, architecture component names, context window, pricing, stated limitations, chip design demo, benchmark tables. Published background used for parameterized math and the serving-stack analysis: the OCP Microscaling format specification, the Kimi Linear technical report (KDA design, the three to one hybrid ratio, KV reduction and decode speedup figures, the open kernel release), the Mooncake paper and open-source release (FAST 25 best paper; transfer engine integrated in mainstream serving stacks), the DeepSeek V3 MLA configuration (512 plus 64 latent dimensions), and the DeepEP communication library.

B, independent measurement: Artificial Analysis: Intelligence Index score of 57 (fourth of 189), output token consumption on the index run (130M vs a 63M median), decode speed of 62 tokens per second against a 72.7 tier median, latency of about 2 seconds to first token and 34.24 seconds to first answer token, evaluation cost of $2,709.75, and blended cost ($2.31/M at 7:2:1). OpenRouter: listing at matching rates, and platform-reported realized prices 60 to 80 percent below list under caching.

C, informed secondary: VentureBeat on benchmark placements, release timing, and corporate context. The Hugging Face community model overview for the ~50B active parameter estimate, MXFP4 and MXFP8 details, and deployment sizing. Verdent and multiple pricing guides for the comparative rate card and the modified MIT report.

D, vendor-claimed, unverified: the 2.5x scaling efficiency over K2, the 90 percent cache hit rate, the single-agent no-compression benchmark protocol, launch-week arena rankings, and the chip demo results.

E, our arithmetic and code: the batch amortization simulation, the quantization grid experiment, the dispatch reference implementation, the cost model, the prefix fingerprint demo, all-to-all sizing, KV cache bytes, prefill FLOPs, and the self-host break-even. Every code block in this issue was executed as printed, CPython 3.12.3 with numpy 2.4.4, with fixed seeds; outputs are shown verbatim. Assumptions are stated inline; every figure downstream of the ~50B active estimate, the assumed 7,168 hidden width, the ~60 MoE layer count, the ~1.5 GB per expert, and the assumed global-layer ratio inherits their uncertainty and will be recomputed when the technical report publishes the real dimensions.

The Kimi K3 technical report had not been published at the time of writing. Numbers in this issue may be revised when it appears, and we will follow up after the July 27 weight release.

Share

The “fact check” ledger

Every load-bearing claim, its source tier, and how it was checked. Rows marked recomputed were re-derived at build time by the scripts shipped with this issue.

Subscribe now

References

Primary and background sources. arXiv identifiers for the Kimi Linear and Gated DeltaNet entries were verified against arXiv during fact-checking; the remaining identifiers are standard citations for their papers.

1. Moonshot AI. Kimi K3 Tech Blog: Open Frontier Intelligence. kimi.com/blog/kimi-k3, July 2026.

2. Moonshot AI. Kimi K3 quickstart and platform pricing. platform.kimi.ai, July 2026.

3. Kimi Team. Kimi Linear: An Expressive, Efficient Attention Architecture. arXiv:2510.26692, 2025.

4. Yang, S. et al. Gated Delta Networks: Improving Mamba2 with Delta Rule. arXiv:2412.06464, 2024.

5. Yang, S. et al. Parallelizing Linear Transformers with the Delta Rule over Sequence Length. arXiv:2406.06484, 2024.

6. DeepSeek-AI. DeepSeek-V3 Technical Report. arXiv:2412.19437, 2024.

7. DeepSeek-AI. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv:2405.04434, 2024.

8. Qin, R. et al. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. arXiv:2407.00079; USENIX FAST 2025, best paper.

9. Rouhani, B. et al. Microscaling Data Formats for Deep Learning. arXiv:2310.10537, 2023. See also the OCP Microscaling Formats Specification v1.0.

10. Micikevicius, P. et al. FP8 Formats for Deep Learning. arXiv:2209.05433, 2022.

11. Shazeer, N. et al. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. arXiv:1701.06538, 2017.

12. Lepikhin, D. et al. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. arXiv:2006.16668, 2020.

13. Fedus, W. et al. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. arXiv:2101.03961, 2021.

14. Wang, L. et al. Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts. arXiv:2408.15664, 2024.

15. Kimi Team. Kimi K2: Open Agentic Intelligence. arXiv:2507.20534, 2025.

16. Liu, J. et al. Muon is Scalable for LLM Training. arXiv:2502.16982, 2025.

17. Ainslie, J. et al. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. arXiv:2305.13245, 2023.

18. Kwon, W. et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. arXiv:2309.06180, 2023.

19. Zheng, L. et al. SGLang: Efficient Execution of Structured Language Model Programs. arXiv:2312.07104, 2023.

20. Open-source repositories referenced: MoonshotAI/Kimi-Linear, fla-org/flash-linear-attention, deepseek-ai/DeepEP, deepseek-ai/FlashMLA, kvcache-ai/Mooncake, on GitHub.

21. Launch-week coverage and trackers cited in text: VentureBeat, the Hugging Face community overview, Artificial Analysis, OpenRouter, BenchLM, and the pricing and readiness guides from Verdent, eesel, aireiter, kie.ai, avenchat, wan27.org, NxCode, digitalapplied, and TECHi, all July 2026.

Qualcomm just agreed to pay roughly $3.9 billion for a 150 person compiler company. Tesla says it rebuilt the FSD compiler and runtime on the same infrastructure. NVIDIA’s newest Python kernel DSL compiles through it. That infrastructure is MLIR.

This is the full tour: the object model, the dialect system, one MLP block lowered step by step into genuine sm_90 PTX, a custom dialect built from scratch, and the economics of who now controls the narrowest point in the AI software stack.

Subscribe now

Three headlines, one substrate

On June 24, 2026, Qualcomm announced it would acquire Modular, the AI infrastructure company founded in 2022 by Chris Lattner and Tim Davis.

Qualcomm’s own press release gives no price; Reuters and the Wall Street Journal reported an all-stock deal worth approximately $3.9 billion, expected to close in the second half of 2026. Modular employs roughly 150 people, per the industry analysts at NAND Research.

Do the arithmetic and Qualcomm is paying something like $26 million per person for a company whose flagship language, Mojo, reached its first 1.0 beta seven weeks before the announcement, and whose compiler is not even open source yet. Qualcomm CEO Cristiano Amon justified it in one line: developers, he said, “demand a more open and modern software foundation.”

Two months earlier, in April 2026, Tesla shipped FSD v14.3. Buried in the release notes, as documented by the release trackers at Tesla Oracle, was the claim that the AI compiler and runtime had been rewritten “from the ground up with MLIR”, with Tesla attributing a 20 percent improvement in vehicle reaction time to the new stack.

And a year before that, in May 2025, NVIDIA shipped CUTLASS 4 with something unprecedented for the company: CuTe DSL, a Python interface for authoring peak-performance GPU kernels which, per NVIDIA’s own documentation, are JIT compiled through MLIR and then handed to ptxas.

A $3.9 billion acquisition by a mobile silicon giant. A safety-critical automotive inference stack. The kernel library that NVIDIA itself uses to demonstrate what Blackwell can do.

Three announcements, three companies with three entirely different agendas, one common substrate underneath. That substrate is MLIR, and if you work anywhere near GPUs, inference serving, or the economics of accelerated compute, you are running on top of it whether you know it or not.

In the previous installment of this series, How LLVM Works: The IR That Took Over Modern Computing, we told the story of the intermediate representation that unified the CPU world: one typed SSA language in the middle, every source language on one side, every instruction set on the other.

This piece is about what happened when that model met machine learning and broke, and about the infrastructure Lattner’s team built in response. MLIR is routinely described as “LLVM for AI.” It is not.

It is something stranger and more consequential: not a compiler, but a machine for building compilers, and it has quietly become the coordination point for the entire AI hardware buildout.

How to read the code in this piece

Every block of IR, TableGen, and C++ below was run through real tools before publication: mlir-opt, mlir-runner, mlir-tblgen, and GCC against the MLIR 20 headers, all from the stock LLVM 20.1.2 packages on Ubuntu 24.04.

A verified badge means the snippet parses and passes the verifier; executed means we compiled it to native code and ran it, and the output you see is what the machine printed.

Share

The problem LLVM could not see

LLVM IR’s contract, the one that conquered the CPU world, is deliberately narrow: scalar values in SSA form, a flat control flow graph of basic blocks, loads and stores against a flat memory model, one abstraction level for everything.

That narrowness is why thirty-odd frontends and every serious instruction set could meet in the middle. It is also precisely what makes LLVM IR the wrong meeting point for machine learning.

Consider what a tensor compiler needs to know about a matrix multiplication.

• That the two input buffers are dense 2-D arrays with known strides.

• That the triple loop nest around it is two parallel dimensions and one reduction.

• That the accumulator is transient and could live in registers or shared memory.

• That the whole operation is one node in a graph whose neighbors it might profitably fuse with.

By the time a matmul has been lowered to LLVM IR, every one of those facts has been destroyed. The arrays are opaque pointers, the loop structure is a thicket of compare-and-branch, the parallelism is gone, and the fusion opportunity evaporated three layers up.

Recovering that information from LLVM IR is the auto-vectorization problem, a research area with forty years of partial credit. The lesson the ML compiler generation drew was blunt: do not recover structure. Never lose it.

The second problem was combinatorial rather than semantic. Lattner has told the origin story himself, most recently in January 2026 in “Democratizing AI Compute, Part 8,” on Modular’s blog: MLIR began in 2018 inside Google, in the middle of the TPU buildout, when the TensorFlow ecosystem had accumulated a zoo of graph representations, converters, and codegen paths that shared almost nothing.

Every framework wanted to reach every accelerator. Every accelerator vendor was rebuilding the same 80 percent of a compiler: parsers, printers, pass managers, verifiers, canonicalizers, testing harnesses.

With N frontends and M targets, the industry was signing up to build and staff something like N times M compilers.

The insight that became MLIR was to go one level more abstract than LLVM did. LLVM’s answer to fragmentation was a single fixed IR in the middle.

But ML compilation does not have one natural middle: it has a graph level, a tensor algebra level, a loop level, a vector level, a hardware intrinsic level, and programs need to descend through all of them.

So instead of shipping another fixed IR, ship the infrastructure that makes intermediate representations cheap to build, and let them all coexist in one framework, one type system, one pass manager, one textual format.

Google open sourced the project in April 2019 and donated it to the LLVM Foundation, with the code landing in the LLVM monorepo at the end of that year.

The design paper, “MLIR: Scaling Compiler Infrastructure for Domain Specific Computation” by Lattner, Mehdi Amini, Uday Bondhugula, Albert Cohen, and colleagues, appeared at CGO 2021 and is still the best single description of the architecture.

The name expands to Multi-Level Intermediate Representation, and the plural is the entire point. One artifact can hold, simultaneously and legally, a tensor-algebra view of one function, an explicit loop nest view of another, and raw hardware intrinsics for a third, with every level checked by the same verifier and transformed by the same pass infrastructure.

The rest of this article is a tour of how that actually works, at the level of real IR, because the abstractions only make sense once you have watched them move.

Share The Software Frontier

One idea, applied without exception

Strip away everything else and MLIR is one data structure applied with unusual discipline. The universe contains operations.

An operation has a name, takes SSA values as operands, produces values as results, carries compile-time constants called attributes, and, this is the structural leap, may contain regions: nested lists of blocks which themselves contain operations. Values have types.

That is the whole object model. There are no statements, no expressions, no special forms. Here is the smallest interesting function, in the friendly syntax you will see in every MLIR dump:

01_axpy.mlir verified: mlir-opt 20.1.2

func.func @axpy(%a: f32, %x: f32, %y: f32) -> f32 {
  %0 = arith.mulf %a, %x : f32
  %1 = arith.addf %0, %y : f32
  return %1 : f32
}

Friendly syntax is sugar. Ask mlir-opt to print the same module in generic form and the uniformity becomes visible:

mlir-opt --mlir-print-op-generic 01_axpy.mlir verified: actual tool output

"builtin.module"() ({
  "func.func"() <{function_type = (f32, f32, f32) -> f32, sym_name = "axpy"}> ({
  ^bb0(%arg0: f32, %arg1: f32, %arg2: f32):
    %0 = "arith.mulf"(%arg0, %arg1) <{fastmath = #arith.fastmath<none>}> : (f32, f32) -> f32
    %1 = "arith.addf"(%0, %arg2) <{fastmath = #arith.fastmath<none>}> : (f32, f32) -> f32
    "func.return"(%1) : (f32) -> ()
  }) : () -> ()
}) : () -> ()

Read that carefully, because it is the most important dump in this article. The module is an operation named builtin.module with one region. The function is an operation named func.func whose “body” is just a region and whose name and signature are ordinary attributes.

Even return is an operation. There are no built-in language constructs at all; the things that look built in live in a dialect that happens to be called builtin. Every structure you will meet from here on, loop nests, GPU kernels, entire schedules, is an operation containing regions containing operations, all the way down.

Regions are what LLVM IR never had. LLVM gives you one flat control flow graph per function, so a loop exists only as a cycle among basic blocks, something analyses must rediscover with dominator trees.

In MLIR, an operation like a loop contains its body as a region, so the nesting of the source program survives as nesting in the IR. A GPU kernel is an operation containing the kernel body. A module is an operation containing functions.

SSA and dominance still hold inside each region, so classic optimization theory transfers intact, but structure is now first class rather than archaeological.

Discipline this strict pays off in the machinery. Because everything is an operation, one parser, one printer, one pass manager, one multithreading strategy, and one verification framework serve every abstraction level ever built on MLIR, including ones that do not exist yet.

Extensibility is kept honest by three mechanisms.

• Verifiers let each operation enforce its own structural invariants, so malformed IR dies at the boundary instead of corrupting a pass 40 steps later.

• Traits declare properties like “this op has no side effects” or “operands and results share a type.”

• And interfaces are the load-bearing wall: a pass written against LoopLikeOpInterface or DestinationStyleOpInterface works on any operation implementing the interface, including operations invented years after the pass was written by people who never met the pass author.

That is the property that makes what follows possible: generic infrastructure over an open universe of abstractions.

Subscribe now

Dialects: a periodic table you can extend

Operations are grouped into dialects: namespaces that bundle related ops, types, and attributes together with their verifiers, canonicalization patterns, and documentation.

The prefix before the dot tells you the dialect: arith.mulf, func.func, linalg.matmul. A dialect is not a language; it is closer to a library of IR vocabulary, and dialects mix freely in one function.

The stock distribution is already a small civilization:

the in-tree roster verified: actual tool output

$ mlir-opt --show-dialects
Available Dialects (48):
  acc, affine, amdgpu, amx, arith, arm_neon, arm_sme, arm_sve, async,
  bufferization, builtin, cf, complex, dlti, emitc, func, gpu, index,
  irdl, linalg, llvm, math, memref, mesh, ml_program, mpi, nvgpu,
  nvvm, omp, pdl, pdl_interp, polynomial, ptr, quant, rocdl, scf,
  shape, sparse_tensor, spirv, tensor, test, test_dyn, tosa,
  transform, ub, vector, x86vector, xegpu

Forty-eight dialects ship in LLVM 20.1.2, and the count only grows: CPU vector extensions (amx, arm_sve, x86vector), GPU vendor exits (nvvm, rocdl, xegpu), parallel runtimes (omp, async), quantization, sparse tensors, hardware description.

Out of tree, the population explodes; we will meet the important settlers in a few sections. What matters is the shape of the traffic: programs enter at high abstraction and descend. A few landmarks, each verified:

scf, structured control flow. Loops and conditionals as region-holding operations rather than branch spaghetti. Loop-carried state is explicit SSA, threaded through iter_args:

dot product in scf verified: mlir-opt 20.1.2

// scf: structured control flow with loop-carried values
func.func @dot(%a: memref<1024xf32>, %b: memref<1024xf32>) -> f32 {
  %c0 = arith.constant 0 : index
  %c1 = arith.constant 1 : index
  %n  = arith.constant 1024 : index
  %zero = arith.constant 0.0 : f32
  %sum = scf.for %i = %c0 to %n step %c1
         iter_args(%acc = %zero) -> (f32) {
    %x = memref.load %a[%i] : memref<1024xf32>
    %y = memref.load %b[%i] : memref<1024xf32>
    %m = arith.mulf %x, %y : f32
    %s = arith.addf %acc, %m : f32
    scf.yield %s : f32
  }
  return %sum : f32
}

affine, the polyhedral inheritance. Same loops, harsher rules: bounds and subscripts must be affine expressions of the induction variables. In exchange, dependence analysis becomes decidable.

The compiler can prove that iterations are independent, that tiling is legal, that this access pattern never aliases that one, instead of guessing:

1-D stencil in affine verified: mlir-opt 20.1.2

// affine: loops + accesses restricted to affine expressions, so the
// compiler can *prove* dependence facts instead of guessing
func.func @blur1d(%in: memref<1026xf32>, %out: memref<1024xf32>) {
  affine.for %i = 0 to 1024 {
    %l = affine.load %in[%i]     : memref<1026xf32>
    %c = affine.load %in[%i + 1] : memref<1026xf32>
    %r = affine.load %in[%i + 2] : memref<1026xf32>
    %s0 = arith.addf %l, %c : f32
    %s1 = arith.addf %s0, %r : f32
    affine.store %s1, %out[%i] : memref<1024xf32>
  }
  return
}

tensor and memref, the two memories. This pair encodes the single most useful distinction in the whole stack.

A tensor is an immutable SSA value: no address, no aliasing, safe to reorder and fuse aggressively, the natural currency of ML graphs.

A memref is a view of actual memory: base pointer, sizes, strides, and, critically for this audience, an address space. The types are expressive enough to carry real kernel engineering:

types that will look familiar verified: mlir-opt 20.1.2

func.func private @smem_tile(%t: memref<64x64xf16, strided<[68, 1]>, 3>)
func.func private @kv_page(%p: tensor<16x8x128xbf16>)

The first is a 64x64 half-precision tile with a padded row stride of 68 elements, the classic plus-four skew that kills shared memory bank conflicts, living in address space 3: GPU shared memory.

The second is a bf16 paged-KV block of the kind every serving engine slings around. MLIR’s type system says these things natively; nothing here is a comment or a convention.

vector, the portable SIMD layer. Fixed-shape virtual registers and the operations that move them: vector.fma, transfers, contractions, shuffles. Target independent until the last moment, then peeled into NEON, AVX-512, or GPU instructions:

an 8-wide fused multiply-add verified: mlir-opt 20.1.2

// vector: SIMD/SIMT-width-explicit compute on virtual registers
func.func @vec_fma(%a: vector<8xf32>, %b: vector<8xf32>, %c: vector<8xf32>) -> vector<8xf32> {
  %0 = vector.fma %a, %b, %c : vector<8xf32>
  return %0 : vector<8xf32>
}

And beneath everything, the exits: the gpu dialect for launch grids and kernels, nvvm and rocdl wrapping the vendors’ intrinsics, and the llvm dialect, which models LLVM IR itself as just another MLIR dialect so that the handoff to the old empire is an ordinary rewrite rather than a file format negotiation.

One table to orient the descent:

Read the right-hand column top to bottom and you have the core operating principle of the whole system, the one we will watch in motion next: information is only destroyed downward, so every optimization should run at the highest level where its enabling facts still exist.

Fusion happens in tensor algebra, where aliasing cannot exist. Tiling happens where loops are structured objects. Bank conflict avoidance happens where address spaces are types.

By the time you reach LLVM, you are not optimizing anymore; you are translating.

Share

linalg: the algebra at the waist

If MLIR has a center of gravity, it is the linalg dialect: the tensor-algebra layer where most serious codegen pipelines, IREE’s, Modular’s, the upstream CPU flows, do their thinking.

Its design principle is the anti-autovectorizer stance taken to its conclusion. Instead of encoding a matmul as loops and hoping to rediscover its nature, linalg encodes the nature and treats loops as one of several possible spellings, generated on demand.

The workhorse is linalg.generic, which describes a perfectly nested computation with three pieces of data. First, indexing maps: one affine map per operand saying which element each point in the iteration space touches.

Second, iterator types: each loop dimension is declared parallel or reduction, which is the fact autovectorizers spend their lives failing to prove. Third, a region holding the scalar payload, the body computed at every point.

Everything the dialect knows, it knows structurally. A matmul’s maps are (m, n, k) -> (m, k), (m, n, k) -> (k, n), (m, n, k) -> (m, n) over iterators [parallel, parallel, reduction]; named ops like linalg.matmul are simply memorable spellings of specific generics.

Here is the payload we will spend the rest of the article compiling, a real inference fragment: a linear layer with fused bias and ReLU.

02_mlp.mlir: matmul + fused bias/ReLU on tensors verified: mlir-opt 20.1.2

#id2d   = affine_map<(m, n) -> (m, n)>
#bcast  = affine_map<(m, n) -> (n)>

func.func @mlp_block(%x: tensor<64x512xf32>,
                     %w: tensor<512x256xf32>,
                     %b: tensor<256xf32>) -> tensor<64x256xf32> {
  %c0 = arith.constant 0.0 : f32

  // Materialize the destination and zero-init the accumulator.
  %empty = tensor.empty() : tensor<64x256xf32>
  %acc   = linalg.fill ins(%c0 : f32)
                       outs(%empty : tensor<64x256xf32>) -> tensor<64x256xf32>

  // y = x @ w
  %mm = linalg.matmul ins(%x, %w : tensor<64x512xf32>, tensor<512x256xf32>)
                      outs(%acc : tensor<64x256xf32>) -> tensor<64x256xf32>

  // out = max(y + b, 0), one fused elementwise op
  %out = linalg.generic
           {indexing_maps = [#id2d, #bcast, #id2d],
            iterator_types = ["parallel", "parallel"]}
           ins(%mm, %b : tensor<64x256xf32>, tensor<256xf32>)
           outs(%empty : tensor<64x256xf32>) {
  ^bb0(%y: f32, %bias: f32, %o: f32):
    %sum  = arith.addf %y, %bias : f32
    %relu = arith.maximumf %sum, %c0 : f32
    linalg.yield %relu : f32
  } -> tensor<64x256xf32>

  return %out : tensor<64x256xf32>
}

Three details deserve attention. The bias’s indexing map, (m, n) -> (n), expresses broadcasting as pure index algebra: no replicated buffer, no runtime rule, just a map that ignores m.

The elementwise op fuses the add and the clamp into one region, because a region can hold any scalar computation, which is how epilogue fusion is spelled at this level.

And every linalg op writes into an explicit outs operand it also returns: destination-passing style. On immutable tensors the destination looks redundant, and semantically it is.

It exists as a standing declaration of where results could live, which is exactly the hint that lets bufferization later plan in-place updates instead of solving a global aliasing puzzle. Keep an eye on that %empty; it is about to earn its keep.

Share The Software Frontier

The walkthrough: one MLP block to sm_90 PTX

Now the demonstration the whole article is built around. We will take that block and walk it down the ladder one verified step at a time, watching what each level adds and what it forgets.

This is the same journey your PyTorch graphs take through Triton, that JAX programs take through XLA, that Mojo kernels take through MAX; we are just taking it on foot, with the intermediate files open.

Step 1. Tiling, written as IR

The first performance decision on any matrix multiply is blocking it for the memory hierarchy. In most compilers, tile sizes hide inside a C++ heuristic.

In MLIR, the transformation itself can be scripted in IR, using the transform dialect: a schedule language whose operations manipulate other operations. We append this to the file:

03_tile.mlir, the schedule part verified: mlir-opt 20.1.2

// ---- Schedule: also IR, in the transform dialect ----
module attributes {transform.with_named_sequence} {
  transform.named_sequence @__transform_main(%root: !transform.any_op {transform.readonly}) {
    %mm = transform.structured.match ops{["linalg.matmul"]} in %root
      : (!transform.any_op) -> !transform.any_op
    %tiled, %loops:3 = transform.structured.tile_using_for %mm tile_sizes [16, 32, 64]
      : (!transform.any_op) -> (!transform.any_op, !transform.any_op, !transform.any_op, !transform.any_op)
    transform.yield
  }
}

The handles typed !transform.any_op are SSA values whose runtime contents are payload operations: the schedule matches every linalg.matmul, then tiles each one 16 by 32 by 64.

Run mlir-opt --transform-interpreter and the payload function comes back rewritten:

output: the tiled matmul verified: actual tool output, constants elided as marked

  func.func @matmul(%arg0: tensor<64x512xf32>, %arg1: tensor<512x256xf32>, %arg2: tensor<64x256xf32>) -> tensor<64x256xf32> {
    // ... index constants (0, 16, 32, 64, 256, 512) elided ...
    %0 = scf.for %arg3 = %c0 to %c64 step %c16 iter_args(%arg4 = %arg2) -> (tensor<64x256xf32>) {
      %1 = scf.for %arg5 = %c0_0 to %c256 step %c32 iter_args(%arg6 = %arg4) -> (tensor<64x256xf32>) {
        %2 = scf.for %arg7 = %c0_1 to %c512 step %c64_2 iter_args(%arg8 = %arg6) -> (tensor<64x256xf32>) {
          %extracted_slice = tensor.extract_slice %arg0[%arg3, %arg7] [16, 64] [1, 1] : tensor<64x512xf32> to tensor<16x64xf32>
          %extracted_slice_3 = tensor.extract_slice %arg1[%arg7, %arg5] [64, 32] [1, 1] : tensor<512x256xf32> to tensor<64x32xf32>
          %extracted_slice_4 = tensor.extract_slice %arg8[%arg3, %arg5] [16, 32] [1, 1] : tensor<64x256xf32> to tensor<16x32xf32>
          %3 = linalg.matmul ins(%extracted_slice, %extracted_slice_3 : tensor<16x64xf32>, tensor<64x32xf32>) outs(%extracted_slice_4 : tensor<16x32xf32>) -> tensor<16x32xf32>
          %inserted_slice = tensor.insert_slice %3 into %arg8[%arg3, %arg5] [16, 32] [1, 1] : tensor<16x32xf32> into tensor<64x256xf32>
          scf.yield %inserted_slice : tensor<64x256xf32>
        }
        scf.yield %2 : tensor<64x256xf32>
      }
      scf.yield %1 : tensor<64x256xf32>
    }
    return %0 : tensor<64x256xf32>
  }

One schedule op became a three-deep scf.for nest. tensor.extract_slice carves 16x64 and 64x32 tiles from the operands, an inner linalg.matmul, still the full structured op, now shaped 16x64 times 64x32, computes each 16x32 block, and tensor.insert_slice threads the result through the loop-carried tensor.

Notice what did not happen: we have loops, but we did not lose the algebra. The inner op is still a matmul with its maps and iterator types intact, still available for vectorization, for lowering to tensor core intrinsics, or for another round of tiling.

Structure survives the transformation because the transformation is structure-aware.

Step 2. Fusion: FlashAttention thinking, six lines long

Tiling one op is table stakes. The move that defines modern kernel engineering, compute the producer inside the consumer’s tile so intermediates never round-trip through HBM, is the same idea that makes FlashAttention FlashAttention.

Here it is as a schedule: tile the bias/ReLU consumer into a parallel grid, then pull the matmul into each tile.

04_fuse.mlir, the schedule part verified: mlir-opt 20.1.2

module attributes {transform.with_named_sequence} {
  transform.named_sequence @__transform_main(%root: !transform.any_op {transform.readonly}) {
    // 1. Tile the *consumer* (bias+relu) into a parallel grid of 16x32 tiles.
    %gen = transform.structured.match ops{["linalg.generic"]} in %root
      : (!transform.any_op) -> !transform.any_op
    %tiled, %grid = transform.structured.tile_using_forall %gen tile_sizes [16, 32]
      : (!transform.any_op) -> (!transform.any_op, !transform.any_op)

    // 2. Pull the matmul producer inside each tile of that grid.
    %mm = transform.structured.match ops{["linalg.matmul"]} in %root
      : (!transform.any_op) -> !transform.any_op
    %fused, %grid2 = transform.structured.fuse_into_containing_op %mm into %grid
      : (!transform.any_op, !transform.any_op) -> (!transform.any_op, !transform.any_op)
    transform.yield
  }
}

output after --transform-interpreter -canonicalize -cse verified: actual tool output

#map = affine_map<(d0) -> (d0 * 16)>
#map1 = affine_map<(d0) -> (d0 * 32)>
#map2 = affine_map<(d0, d1) -> (d0, d1)>
#map3 = affine_map<(d0, d1) -> (d1)>
  func.func @mlp_block(%arg0: tensor<64x512xf32>, %arg1: tensor<512x256xf32>, %arg2: tensor<256xf32>) -> tensor<64x256xf32> {
    %cst = arith.constant 0.000000e+00 : f32
    %0 = tensor.empty() : tensor<64x256xf32>
    %1 = linalg.fill ins(%cst : f32) outs(%0 : tensor<64x256xf32>) -> tensor<64x256xf32>
    %2 = scf.forall (%arg3, %arg4) in (4, 8) shared_outs(%arg5 = %0) -> (tensor<64x256xf32>) {
      %3 = affine.apply #map(%arg3)
      %4 = affine.apply #map1(%arg4)
      %extracted_slice = tensor.extract_slice %arg0[%3, 0] [16, 512] [1, 1] : tensor<64x512xf32> to tensor<16x512xf32>
      %extracted_slice_0 = tensor.extract_slice %arg1[0, %4] [512, 32] [1, 1] : tensor<512x256xf32> to tensor<512x32xf32>
      %extracted_slice_1 = tensor.extract_slice %1[%3, %4] [16, 32] [1, 1] : tensor<64x256xf32> to tensor<16x32xf32>
      %5 = linalg.matmul ins(%extracted_slice, %extracted_slice_0 : tensor<16x512xf32>, tensor<512x32xf32>) outs(%extracted_slice_1 : tensor<16x32xf32>) -> tensor<16x32xf32>
      %extracted_slice_2 = tensor.extract_slice %arg2[%4] [32] [1] : tensor<256xf32> to tensor<32xf32>
      %extracted_slice_3 = tensor.extract_slice %arg5[%3, %4] [16, 32] [1, 1] : tensor<64x256xf32> to tensor<16x32xf32>
      %6 = linalg.generic {indexing_maps = [#map2, #map3, #map2], iterator_types = ["parallel", "parallel"]} ins(%5, %extracted_slice_2 : tensor<16x32xf32>, tensor<32xf32>) outs(%extracted_slice_3 : tensor<16x32xf32>) {
      ^bb0(%in: f32, %in_4: f32, %out: f32):
        %7 = arith.addf %in, %in_4 : f32
        %8 = arith.maximumf %7, %cst : f32
        linalg.yield %8 : f32
      } -> tensor<16x32xf32>
      scf.forall.in_parallel {
        tensor.parallel_insert_slice %6 into %arg5[%3, %4] [16, 32] [1, 1] : tensor<16x32xf32> into tensor<64x256xf32>
      }
    }
    return %2 : tensor<64x256xf32>
  }

The consumer became an scf.forall, a parallel-by-construction grid of 4 by 8 tiles, and inside each tile sits a private 16x512 by 512x32 matmul feeding the fused bias-and-clamp directly.

The 64x256 intermediate matrix that used to exist between the two ops is gone from the program: each tile’s slice is produced, consumed, and discarded in place. canonicalize and cse then deleted the original full-size matmul as dead code.

This is a working, verified skeleton of producer fusion, the transformation entire serving stacks are built around, expressed in six lines of schedule that a performance engineer can read, diff, and review like any other code.

Step 3. Bufferization: the value world ends here

Everything so far lived on immutable tensors, where fusion and reordering are trivially safe because aliasing cannot be expressed. Hardware, unfortunately, sells mutable bytes.

The crossing is one-shot bufferization, a whole-module analysis that assigns every tensor a buffer while proving where it can reuse memory instead of copying. Run it on the unfused MLP and look at what comes back:

mlir-opt -one-shot-bufferize=”bufferize-function-boundaries” verified: actual tool output

#map = affine_map<(d0, d1) -> (d0, d1)>
#map1 = affine_map<(d0, d1) -> (d1)>
module {
  func.func @mlp_block(%arg0: memref<64x512xf32, strided<[?, ?], offset: ?>>, %arg1: memref<512x256xf32, strided<[?, ?], offset: ?>>, %arg2: memref<256xf32, strided<[?], offset: ?>>) -> memref<64x256xf32> {
    %cst = arith.constant 0.000000e+00 : f32
    %alloc = memref.alloc() {alignment = 64 : i64} : memref<64x256xf32>
    linalg.fill ins(%cst : f32) outs(%alloc : memref<64x256xf32>)
    linalg.matmul ins(%arg0, %arg1 : memref<64x512xf32, strided<[?, ?], offset: ?>>, memref<512x256xf32, strided<[?, ?], offset: ?>>) outs(%alloc : memref<64x256xf32>)
    linalg.generic {indexing_maps = [#map, #map1, #map], iterator_types = ["parallel", "parallel"]} ins(%alloc, %arg2 : memref<64x256xf32>, memref<256xf32, strided<[?], offset: ?>>) outs(%alloc : memref<64x256xf32>) {
    ^bb0(%in: f32, %in_0: f32, %out: f32):
      %0 = arith.addf %in, %in_0 : f32
      %1 = arith.maximumf %0, %cst : f32
      linalg.yield %1 : f32
    }
    %cast = memref.cast %alloc : memref<64x256xf32> to memref<64x256xf32, strided<[?, ?], offset: ?>>
    return %alloc : memref<64x256xf32>
    // ...

Tensors became memrefs; function boundaries acquired conservative strided layouts so callers can pass any compatible view.

But the line to study is the fused elementwise: it now reads from %alloc and writes into %alloc. The analysis proved the bias/ReLU can execute in place on the matmul’s buffer, so the second 64KB allocation that a naive translation would emit simply does not exist.

That is destination-passing style paying out: because every linalg op declared its destination back in value land, the planner had the aliasing story handed to it.

This pass, more than any other, is where MLIR’s two-world design earns its complexity: algebra above the line, memory below it, and one well-tested analysis holding the border.

Step 4. Proof of life: the IR actually runs

Before descending to GPU intrinsics, a checkpoint that separates this article from a whiteboard exercise.

The remaining distance to a CPU binary is mechanical: linalg to loops, loops to branches, memrefs and arithmetic to the llvm dialect, then JIT.

We wrapped the MLP in a tiny main with known inputs, chose a bias of [-10, 1] so the ReLU visibly clamps, and ran the pipeline:

the full descent, one command executed: mlir-runner, LLVM 20.1.2

$ mlir-opt mlp_run.mlir \
    -one-shot-bufferize="bufferize-function-boundaries" \
    -convert-linalg-to-loops \
    -expand-strided-metadata -lower-affine \
    -convert-scf-to-cf -finalize-memref-to-llvm \
    -convert-arith-to-llvm -convert-cf-to-llvm -convert-func-to-llvm \
    -reconcile-unrealized-casts \
  | mlir-runner -e main -entry-point-result=void \
      -shared-libs=$LLVM/lib/libmlir_runner_utils.so.20.1,$LLVM/lib/libmlir_c_runner_utils.so.20.1

what the machine printed

Unranked Memref base@ = 0x55b27c52f500 rank = 2 offset = 0 sizes = [2, 2] strides = [2, 1] data = 
[[0,   6], 
 [0,   12]]

By hand: x@w is [[4, 5], [10, 11]]; add the bias to get [[-6, 6], [0, 12]]; clamp at zero for [[0, 6], [0, 12]]. The machine agrees.

Every abstraction in this article compiles to electrons, and the fused ReLU is provably doing its job on the negative entries.

Step 5. The GPU descent, ending in genuine PTX

Now the branch this audience came for. Same matmul, memref form, aimed at a GPU.

First, expose the parallelism that linalg has known about all along: -convert-linalg-to-parallel-loops turns the two parallel iterators into an scf.parallel over (m, n) with the reduction as an ordinary inner loop:

parallel loops verified: actual tool output

  func.func @matmul(%arg0: memref<64x512xf32>, %arg1: memref<512x256xf32>, %arg2: memref<64x256xf32>) {
    %c0 = arith.constant 0 : index
    %c64 = arith.constant 64 : index
    %c1 = arith.constant 1 : index
    %c256 = arith.constant 256 : index
    %c512 = arith.constant 512 : index
    scf.parallel (%arg3, %arg4) = (%c0, %c0) to (%c64, %c256) step (%c1, %c1) {
      scf.for %arg5 = %c0 to %c512 step %c1 {
        %0 = memref.load %arg0[%arg3, %arg5] : memref<64x512xf32>
        %1 = memref.load %arg1[%arg5, %arg4] : memref<512x256xf32>
        %2 = memref.load %arg2[%arg3, %arg4] : memref<64x256xf32>
        %3 = arith.mulf %0, %1 : f32
        %4 = arith.addf %2, %3 : f32
        memref.store %4, %arg2[%arg3, %arg4] : memref<64x256xf32>
      }
      scf.reduce 
    }
    return
  }
}

Next, the mapping decision: -gpu-map-parallel-loops -convert-parallel-loops-to-gpu assigns parallel dimensions to the grid and materializes a launch:

a kernel is born verified: actual tool output

    gpu.launch blocks(%arg3, %arg4, %arg5) in (%arg9 = %0, %arg10 = %1, %arg11 = %c1_0) threads(%arg6, %arg7, %arg8) in (%arg12 = %c1_0, %arg13 = %c1_0, %arg14 = %c1_0) {
      %2 = affine.apply #map1(%arg3)[%c1, %c0]
      %3 = affine.apply #map1(%arg4)[%c1, %c0]
      scf.for %arg15 = %c0 to %c512 step %c1 {
        %4 = memref.load %arg0[%2, %arg15] : memref<64x512xf32>
        %5 = memref.load %arg1[%arg15, %3] : memref<512x256xf32>
        %6 = memref.load %arg2[%2, %3] : memref<64x256xf32>
        %7 = arith.mulf %4, %5 : f32
        %8 = arith.addf %6, %7 : f32
        memref.store %8, %arg2[%2, %3] : memref<64x256xf32>
      }
      // ... gpu.terminator

Be honest about what we are looking at: a 64 by 256 grid of blocks with a single thread each, the most naive mapping a GPU has ever been insulted with.

Production pipelines tile before mapping, so blocks get tiles and threads get elements, then layer on shared memory staging and tensor core ops from the nvgpu dialect (mma.sync and friends).

We are deliberately taking the unoptimized path because the plumbing, not the schedule, is today’s subject; later sections covers who builds the good schedules. -gpu-kernel-outlining then splits host from device, the step every CUDA programmer performs mentally when writing __global__:

outlined device module verified: actual tool output, trimmed as marked

  gpu.module @matmul_kernel {
    gpu.func @matmul_kernel(%arg0: index, %arg1: index, %arg2: memref<64x512xf32>, %arg3: memref<512x256xf32>, %arg4: memref<64x256xf32>, %arg5: index) kernel attributes {known_block_size = array<i32: 1, 1, 1>} {
      %block_id_x = gpu.block_id  x
      %block_id_y = gpu.block_id  y
      %thread_id_x = gpu.thread_id  x
      %thread_id_y = gpu.thread_id  y
      %0 = affine.apply #map1(%block_id_x)[%arg0, %arg1]
      %1 = affine.apply #map1(%block_id_y)[%arg0, %arg1]
      // ... same loop body, now reading gpu.block_id instead of loop IVs

The loop body is unchanged, but the block indices now arrive from gpu.block_id instead of loop induction variables, and the kernel records its launch invariants (known_block_size) as attributes for later passes to exploit.

From here, -convert-gpu-to-nvvm rewrites device code into the llvm dialect sprinkled with NVIDIA’s intrinsics:

nvvm: the vendor boundary verified: excerpt of actual tool output

gpu.module @matmul_kernel {
  // signature abbreviated: 24 pointer and index arguments
  llvm.func @matmul_kernel(%arg0: i64, ..., %arg23: i64)
      attributes {gpu.kernel, gpu.known_block_size = array<i32: 1, 1, 1>,
                  nvvm.kernel, nvvm.maxntid = array<i32: 1, 1, 1>} {
    // ...
    %24 = nvvm.read.ptx.sreg.ctaid.x : i32
    %25 = llvm.sext %24 : i32 to i64
    %26 = nvvm.read.ptx.sreg.ctaid.y : i32
    // ... address arithmetic in llvm dialect: getelementptr, mul, add ...
  }
}

gpu.block_id x became nvvm.read.ptx.sreg.ctaid.x, a name any CUDA disassembly veteran will greet like an old acquaintance: the special register file, now visible in the IR.

Finally, the packaged pipeline --gpu-lower-to-nvvm-pipeline="cubin-chip=sm_90 cubin-format=isa" runs the whole descent and invokes LLVM’s NVPTX backend, embedding the result in a gpu.binary op. Extract it and you are holding 90 lines of the real thing:

out/matmul.ptx generated: LLVM 20.1.2 NVPTX backend, sm_90

//
// Generated by LLVM NVPTX Back-End
//

.version 7.8
.target sm_90
.address_size 64

	// .globl	matmul_kernel
	// ... prologue: bounds check, address setup ...
$L__BB0_2:
	ld.global.f32 	%f4, [%rd35];
	ld.global.f32 	%f5, [%rd34];
	mul.rn.f32 	%f6, %f4, %f5;
	add.rn.f32 	%f7, %f7, %f6;
	st.global.f32 	[%rd6], %f7;
	add.s64 	%rd36, %rd36, %rd17;
	add.s64 	%rd35, %rd35, %rd8;
	add.s64 	%rd34, %rd34, %rd10;
	setp.lt.s64 	%p2, %rd36, %rd19;
	@%p2 bra 	$L__BB0_2;
	// ... epilogue ... (90 lines total)

There is a final teaching moment hiding in that inner loop, and it is the thesis of this article restated by the machine. Look at st.global.f32: the accumulator is written back to global memory every iteration of k.

The value lives happily in register %f7, yet the store stays, because at this level nothing can prove the output buffer does not alias the inputs, so the backend must assume a reader might observe every intermediate sum.

Four levels up, in linalg on tensors, that fact was free: tensors cannot alias. We destroyed the information on the way down, exactly as the ladder predicted, and the machine code is paying for it, one redundant HBM transaction per multiply.

Every dollar of kernel engineering ever spent is, in some form, the cost of putting facts back that a lowering threw away. MLIR’s entire bet is that it is cheaper to never throw them away.

Be precise about the distance between this demonstration and a kernel you would actually deploy, because the gap is the whole subject of the next section.

A production pipeline tiles before it maps to hardware, so a block owns a tile and a thread owns a register-sized sliver rather than the one-thread-per-block insult we emitted.

It stages the operand tiles through the padded shared-memory memref from section 4, turning the plus-four skew into live bank-conflict avoidance.

And instead of scalar mul.rn.f32 plus that redundant store, it lowers the inner tile op through the nvgpu dialect into mma.sync or, on Hopper, wgmma, so the loop body becomes a warp-level matrix instruction feeding tensor cores, with the accumulator held in registers across the entire reduction and written back exactly once.

Here is the point that matters for understanding the whole system: every one of those improvements is a different schedule over the identical set of passes we just ran. The naive descent and the peak one share their entire lowering machinery; what separates 2 percent of peak from 90 percent is the sequence of tiling, fusion, and mapping decisions applied on the way down.

Which is why the ability to write that sequence as a first-class, reviewable, searchable artifact, rather than bury it in compiler C++, is not a curiosity. It is the ballgame.

The schedule is also IR

Steps 1 and 2 of the walkthrough smuggled in the most radical idea in modern compiler engineering, so let us stop and look at it directly.

Classically, a compiler’s transformations are opaque C++ controlled by flags and heuristics; if the heuristic tiles your attention kernel wrong, your options are a rebuild or a prayer.

Halide’s great insight, back in 2012, was to split the algorithm from the schedule and make the schedule a first-class program. MLIR’s transform dialect imports that split into the compiler infrastructure itself: the schedule is IR, in the same file format, checked by the same verifier, with SSA handles pointing at the payload operations it manipulates.

The consequences compound. A schedule that is data can be diffed and code reviewed: the six lines in step 2 are a reviewable artifact in a way a C++ pass never is.

It can be searched: an autotuner can emit thousands of candidate schedules, run the interpreter, and measure, without recompiling the compiler; the transform interpreter is exactly the substrate you would build a kernel search system on.

It can be shipped per workload: IREE has used transform-dialect scripts to pin codegen strategies for specific dispatches, which is how you get reproducible kernels out of a general compiler.

And it fails loudly: apply a tiling to an op that cannot be tiled and the interpreter reports a definite error against a definite handle, rather than silently generating slow code.

For a CUDA-native audience the honest framing is this: the transform dialect is the compiler conceding that you, the performance engineer, sometimes know better, and giving you a typed, verifiable console instead of a fork of the codebase.

The upstream vocabulary already covers the daily verbs, tile_using_for, tile_using_forall, fuse_into_containing_op, vectorize, pad, plus pattern application and PDL-based matching for everything else.

It is not the default path for most users and may never be; it is the expert path, and its existence changes what the expert path costs.

Build a dialect before lunch

The claim that MLIR makes IRs cheap deserves a demonstration with a bill attached. Suppose we run inference infrastructure and want the compiler to reason about our world: quantized weights, dequantization groups, fused epilogues.

In a classical compiler, adding a first-class operation means touching the parser, the printer, the verifier, the serializer, and a dozen switch statements.

In MLIR it means describing the op once, declaratively, in TableGen’s Operation Definition Specification, and generating the rest. Here is a real op for a real concern, weight-only quantized matmul, in 28 lines:

serve_ops.td verified: mlir-tblgen 20.1.2

include "mlir/IR/OpBase.td"
include "mlir/Interfaces/SideEffectInterfaces.td"

def Serve_Dialect : Dialect {
  let name = "serve";
  let summary = "Ops for LLM serving kernels";
  let cppNamespace = "::mlir::serve";
}

class Serve_Op<string mnemonic, list<Trait> traits = []>
    : Op<Serve_Dialect, mnemonic, traits>;

def Serve_DequantMatmulOp : Serve_Op<"dequant_matmul", [Pure]> {
  let summary = "matmul with fused groupwise weight dequantization";
  let description = [{
    Computes out = act @ dequant(weights, scales) without ever
    materializing the dequantized weight tensor in memory.
  }];
  let arguments = (ins
    TensorOf<[F16]>:$act,
    TensorOf<[I8]>:$weights,
    TensorOf<[F16]>:$scales,
    I64Attr:$group_size
  );
  let results = (outs TensorOf<[F16]>:$out);
  let assemblyFormat =
    "$act `,` $weights `,` $scales attr-dict `:` functional-type(operands, results)";
}

Feed it to mlir-tblgen and the machinery materializes:

the leverage, measured verified: actual tool output

$ mlir-tblgen -gen-op-decls -I /usr/lib/llvm-20/include serve_ops.td > ServeOps.h.inc
$ mlir-tblgen -gen-op-defs  -I /usr/lib/llvm-20/include serve_ops.td > ServeOps.cpp.inc

# 28 lines of ODS in, 259 lines of generated C++ declarations out:
class DequantMatmulOp;
    ...

Twenty-eight declarative lines became 259 lines of generated C++ declarations, plus definitions: typed accessors (getAct(), getGroupSize()), builders, parser and printer honoring our assemblyFormat, and verifier scaffolding enforcing the type constraints, an f16 activation tensor, i8 weights, f16 scales, or the IR does not construct.

Nine-to-one leverage on the boilerplate, and every generated line is the same battle-tested code path the 48 upstream dialects use.

This is what “infrastructure” means concretely: our bespoke serving op gets round-trippable syntax, verification, and pass-manager citizenship for the price of a lunch break, which is why every hardware startup in section 9 could afford a real compiler at all.

Ops are half a dialect; the other half is transformations, and those are written in C++ against the pattern infrastructure. Here is a complete, compiling pass with a pattern this audience will feel in their DRAM bills: detect a matmul whose accumulator is a fresh zero fill, and tag it so a later lowering can emit a beta-equals-zero GEMM instead of a memset followed by a GEMM, saving one full write-and-read pass over the output through HBM:

TagZeroInit.cpp verified: g++ -fsyntax-only against MLIR 20 headers, exit 0

#include "mlir/Dialect/Arith/IR/Arith.h"
#include "mlir/Dialect/Linalg/IR/Linalg.h"
#include "mlir/IR/BuiltinOps.h"
#include "mlir/IR/PatternMatch.h"
#include "mlir/Pass/Pass.h"
#include "mlir/Transforms/GreedyPatternRewriteDriver.h"

using namespace mlir;

namespace {
// Match  linalg.fill(+0.0) feeding a linalg.matmul accumulator and tag
// the matmul, so a later lowering can emit a beta = 0 GEMM instead of
// a memset followed by a GEMM (one less trip through HBM).
struct TagZeroInitMatmul : OpRewritePattern<linalg::MatmulOp> {
  using OpRewritePattern::OpRewritePattern;

  LogicalResult matchAndRewrite(linalg::MatmulOp op,
                                PatternRewriter &rewriter) const override {
    if (op->hasAttr("serve.zero_init"))
      return failure();          // already rewritten: stop, or loop forever
    auto fill =
        op.getDpsInitOperand(0)->get().getDefiningOp<linalg::FillOp>();
    if (!fill)
      return failure();
    auto cst =
        fill.getInputs()[0].getDefiningOp<arith::ConstantFloatOp>();
    if (!cst || !cst.value().isPosZero())
      return failure();
    rewriter.modifyOpInPlace(op, [&] {
      op->setAttr("serve.zero_init", rewriter.getUnitAttr());
    });
    return success();
  }
};

struct TagZeroInitPass
    : PassWrapper<TagZeroInitPass, OperationPass<ModuleOp>> {
  MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(TagZeroInitPass)
  StringRef getArgument() const final { return "serve-tag-zero-init"; }
  StringRef getDescription() const final {
    return "Tag matmuls whose accumulator is a fresh zero fill";
  }
  void runOnOperation() override {
    RewritePatternSet patterns(&getContext());
    patterns.add<TagZeroInitMatmul>(&getContext());
    if (failed(applyPatternsGreedily(getOperation(), std::move(patterns))))
      signalPassFailure();
  }
};
} // namespace

The anatomy generalizes to every pattern you will ever write. Match structurally by walking use-def edges (getDefiningOp through the destination operand, courtesy of destination-passing style).

Guard exhaustively, including against your own previous rewrite, because the greedy driver reapplies patterns to fixpoint and an unguarded pattern is an infinite loop.

Mutate only through the rewriter, never behind its back, so the driver’s worklist stays coherent. Fifty-ish lines, and it composes with every canonicalization and lowering upstream ships.

The last pillar of the workflow is cultural: everything above gets a FileCheck test, a .mlir file whose RUN line invokes the pass and whose CHECK lines assert on the output IR.

Because IR is text with stable syntax, transformation tests are diffs, reviewable by humans and replayable by CI. Teams that build on MLIR inherit this discipline for free, and it shows: it is the reason a project like

Triton can refactor its entire layout system without regressing a thousand kernels silently.

The honest total cost of a production dialect is of course higher than lunch, the semantics and the verifier discipline are the real work, but the floor has moved by an order of magnitude, and floors are what determine who gets to play.

Where MLIR is hiding in your stack

Infrastructure succeeds when it disappears. Seven years after open sourcing, MLIR has disappeared into more of the AI stack than almost anyone tracks, so let us make the map explicit.

Triton, which means PyTorch. When OpenAI rebuilt Triton on MLIR in 2022, the language stopped being a research artifact and became a compiler stack.

Python is traced into the triton dialect (TTIR), a target-independent tile program; conversion to the triton_gpu dialect (TTGIR) then makes the performance-critical decision explicit by attaching a layout encoding to every tensor, describing exactly which thread of which warp owns which element, with the LinearLayout framework giving those mappings a uniform algebra; vendor sub-dialects handle Hopper and CDNA specifics before the drop into the llvm dialect and PTX or AMDGCN.

Since PyTorch 2 made Inductor the default compile path and Inductor emits Triton for GPU graphs, the practical consequence is that most compiled PyTorch GPU workloads on the planet route through MLIR today, whether their owners have heard of it or not.

And in 2025 the Triton team surfaced the lower rungs of its own ladder as a product: Gluon, documented in the Triton repository, hands expert users the TTGIR-level controls, layouts, shared memory, warp specialization, while reusing the same dialect stack underneath.

Which means your serving engine, too. The inference stacks this publication’s readers actually operate, vLLM and SGLang chief among them, are not usually thought of as MLIR consumers, but trace the dependency and the connection is direct.

Both lean heavily on Triton for their fused and custom kernels: fused RMSNorm and RoPE, fused MoE routing, quantized GEMM epilogues, and increasingly the attention paths through backends in the FlashInfer lineage that mix hand-written CUDA with Triton-generated variants.

Every one of those Triton kernels is a program descending through TTIR and TTGIR, which is to say through MLIR, before it reaches ptxas. The abstraction that a serving engine calls “a Triton kernel” is, one layer down, exactly the layout-annotated dialect IR.

And the data structure those engines sling most obsessively, the paged KV cache, is precisely the strided, address-spaced memref from section 4 made concrete: a block table indexing fixed-size buffers, the same view-of-memory abstraction the type system was built to name.

When you profile a vLLM deployment on a B300 and find time in a fused decode kernel, you are, whether the dashboard says so or not, looking at the output of the compiler stack this article has been dissecting. The waist is not upstream of your infrastructure; it is inside it.

NVIDIA, voluntarily. CUTLASS 4’s CuTe DSL lets engineers author kernels in Python that, per NVIDIA’s documentation, JIT through MLIR into optimized code finished by ptxas, with the CuTe layout algebra as the programming model.

Read that strategically: the company with the most to gain from CUDA C++ lock-in decided its flagship kernel library’s productivity layer should be built on the industry’s shared compiler substrate.

NVIDIA is not ceding the moat, ptxas and SASS remain closed, but it has conceded where the moat is not: the language layer above the driver is now contested ground, and NVIDIA chose to contest it with MLIR rather than against it.

Google, the origin. XLA remains the TPU’s compiler, and since 2022 its portable front door is StableHLO, an MLIR dialect with versioned serialization that lets JAX, TensorFlow, and PyTorch programs travel between compilers without marrying one.

IREE, under the OpenXLA umbrella, is the fully MLIR-native end-to-end: StableHLO or Torch in, linalg in the middle, compiled artifacts for CPU, CUDA, ROCm, Vulkan, and mobile out. It is the closest thing to the textbook hourglass shipped as a product, and its codegen is where much of the transform-dialect and structured-codegen machinery in this article was hardened.

The silicon insurgents, which is the actual story. Look at who builds their entire software identity on this infrastructure. Tenstorrent’s tt-mlir stack, public on GitHub since 2024, compiles graphs through TTIR into its TTNN library ops for Wormhole and Blackhole silicon.

AMD ships MLIR-AIE, its 1.2 release landing in January 2026 per Phoronix, as the low-level programming layer for the NPUs in every Ryzen AI laptop. Qualcomm’s compiler group published Hexagon-MLIR on arXiv this spring: a stack that ingests, notably, both PyTorch graphs and Triton kernels and lowers them onto Hexagon NPUs.

Tesla, per the FSD v14.3 release notes, rebuilt its in-car AI compiler and runtime on MLIR and claims a 20 percent reaction-time improvement from the new stack, which, whatever discount you apply to vendor claims, is a statement about MLIR’s fitness for safety-critical, latency-bounded deployment that no benchmark suite could make.

Every one of these organizations faced the same choice: a decade of compiler archaeology from scratch, or a running start on shared infrastructure. None of them chose archaeology.

And beyond AI entirely, the quieter colonization: Flang, LLVM’s Fortran frontend, is built on MLIR dialects; CIRCT applies the same infrastructure to hardware design and verification; ClangIR is the upstream effort to give C and C++ themselves a structured MLIR layer above LLVM IR.

The pattern across all of it is the one Lattner himself named in January’s “Democratizing AI Compute, Part 8”: MLIR the infrastructure won more or less totally, while the original dream of one shared AI compiler on top of it fractured into vigorous, incompatible ecosystems.

The waist is shared; the programs flowing through it are not. Which is precisely what makes the waist worth money.

The economics of the waist

Regular readers know this publication’s core lens: in accelerated computing, margins pool wherever software friction is highest.

CUDA’s twenty-year lesson is that the compiler and kernel layer is not a cost center; it is the tollbooth. MLIR changes the tollbooth’s economics in three distinct ways, and the June acquisition finally put a market price on one of them.

First, it collapsed the entry ticket. Before this infrastructure existed, a credible accelerator software stack meant hundreds of compiler engineers for the better part of a decade; that is what CUDA cost, and it is why so many silicon startups died with excellent chips and unusable toolchains.

Previous sections showed the mechanism by which that changed: parsing, printing, verification, pass management, testing culture, all amortized across the industry, leaving a new entrant to spend only on what is genuinely theirs, the semantics of their silicon.

The evidence is the roster in Figure 4. Tenstorrent, a company of startup scale, fields a full graph-to-silicon compiler. AMD stands up an NPU stack as a side effort. This does not guarantee anyone beats NVIDIA; it guarantees the attempt no longer costs a billion dollars, and lowering the cost of attempts is how oligopolies erode.

Second, it standardized the engineers, not just the code. A compiler developer who learns dialects, patterns, and the linalg descent is productive at Google, AMD, Qualcomm, Tenstorrent, Modular, or any of a hundred teams, because they all speak the same infrastructure.

Before MLIR, compiler talent was siloed by proprietary IR; now there is a labor market for the waist. Every technology that has standardized its practitioners, from TCP/IP to Kubernetes, has seen the layer above it commoditize and the layer below it consolidate. Keep that in mind when reading the next paragraph.

Third, it created a control point you can buy. Qualcomm did not pay approximately $3.9 billion, per the Reuters-reported terms, for Modular’s revenue. It paid for roughly 150 people, per NAND Research’s headcount estimate, who include the founding architects of both MLIR and LLVM, plus MAX and Mojo, the most complete independent attempt to unify the layer above everyone’s silicon.

That is on the order of $26 million per employee, ten times acquihire norms, for a company whose language hit 1.0 beta in May, per its own PyPI listing, and whose compiler remains closed source with an open sourcing commitment for later this year.

The premium is not for what Modular sells; it is for what Qualcomm was structurally unable to build alone, demonstrated by the fact that its own Hexagon-MLIR effort was already consuming the ecosystem Modular’s founders created.

In the framework of our kernel-moat analysis: when the moat migrates from a proprietary language to a shared substrate, the scarce asset becomes the people who define the substrate’s direction, and June 24 was the day that asset got marked to market.

NVIDIA’s response is the most sophisticated play on the board: embrace the waist from above.

Contributing to Triton’s NVIDIA backends and shipping CuTe DSL on MLIR costs NVIDIA the exclusivity of its language layer, which was already eroding, while making its hardware the best-supported target of the shared stack everyone else depends on, and keeping the truly proprietary layers, ptxas, SASS scheduling, NVLink-domain libraries, exactly as closed as before.

Commoditize your complement, in textbook form. For the neoclouds and inference operators this publication serves, this is not abstract; it changes the arithmetic that turns capex into margin.

A fleet’s single largest source of pricing power against its accelerator vendor is credible multi-homing: the ability to qualify a second silicon supplier and actually move workloads.

Historically that ability died at the kernel layer, because a serving stack tuned to CUDA was a serving stack married to NVIDIA, and the cost of re-qualifying every fused kernel on a new target was prohibitive enough that “we could switch” was rarely true.

The shared waist is what makes the threat credible, incrementally. Every serving stack that targets the waist, vLLM and SGLang through Triton, IREE, MAX, is quietly reducing the switching cost between a B300 fleet and an MI400 fleet at the kernel layer.

Portability never arrives as a binary event, and CUDA-only paths (FlashAttention’s fastest branches, NCCL, the long tail of fused ops) remain very real. But price convergence per token happens at the margin, and the margin is now programmable.

Watch whether accelerator price-performance spreads narrow over the next four quarters as these stacks mature; that spread is the purest financial expression of what this article has been describing.

Subscribe now

The counter-narrative

The strongest version of the case against, because a thesis untested against its critics is marketing.

“The library ate the compiler.” The performance frontier on NVIDIA hardware is still held by artisanal kernels: FlashAttention lineages, cuDNN, CUTLASS templates, hand-scheduled decode paths.

Much of what shipping compilers do is orchestrate calls into those libraries, and a fair reading of the TPP-MLIR work (arXiv 2404.15204), which reached roughly 90 percent of expert-written kernel performance on CPU benchmarks using upstream MLIR, is that the last ten percent is precisely where the money lives.

The rebuttal is not that compilers beat ninjas; it is that MLIR is the substrate the ninjas themselves are migrating to, CuTe DSL and Gluon being exhibits A and B. But the point stands: the waist wins by hosting excellence, not by automating it away.

“LLMs will write the kernels.” If models generate CUDA or PTX directly on demand, does a shared IR layer matter less? Plausibly it matters more: generated code needs verification, search needs a space with structure, and an IR with machine-checkable semantics and a transform language is a far better substrate for automated kernel search than free-form C++.

But intellectual honesty requires flagging this as the genuine wildcard: a world of cheap, correct, model-generated kernels reshuffles every layer of this analysis, ours included.

“Dialect soup.” Forty-eight upstream dialects, hundreds downstream, and no blessed end-to-end path is real fragmentation with real costs: two teams can both “use MLIR” and share nothing but a parser.

Lattner’s own Part 8 critique makes essentially this charge, and the upstream community’s response, charters, governance reform, the StableHLO-style versioning discipline spreading to more dialects, is work in progress, not a solved problem.

Relatedly, core MLIR offers no stable cross-release IR contract: downstream projects pay a real rebase tax every LLVM release, a tax Triton and IREE budget actual headcount for.

“The C++ tax.” The infrastructure’s expressiveness rides on heavy C++ templates and TableGen metaprogramming; build times are punishing, the learning curve is a cliff, and debugging a misapplied pattern through the greedy driver is an acquired skill.

Python bindings and tooling soften the edges for users, but dialect authors live in C++, and that gates the contributor pool. Fair, true, and priced in: the roster in Figure 4 is the market’s judgment that the tax is worth the leverage.

Every one of these criticisms describes a middle-aged infrastructure project with genuine adoption. None of them describes an alternative, and in infrastructure, the alternative is the only criticism that kills.

What to do with this

If this article did its job, MLIR stopped being a logo on other people’s architecture slides and became legible machinery.

Here is the shortest path from legible to useful, calibrated for readers who already know what a memory coalescing problem feels like.

Learn to read dumps before writing anything. The single highest-leverage habit is inspecting the IR your existing tools already produce. Triton will show you its TTIR and TTGIR for any kernel you own; start with a matmul you understand and find the layout encodings.

On your own experiments, mlir-opt --mlir-print-ir-after-all is the x-ray: every pass, before and after. When output confuses you, --mlir-print-op-generic strips the sugar and shows you the honest structure, exactly as in section 3.

Replay this article. Everything here ran on stock Ubuntu 24.04 packages: apt install mlir-20-tools llvm-20 libmlir-20-dev and you have mlir-opt, mlir-runner, mlir-translate, mlir-tblgen, and mlir-reduce (the last being delta-debugging for IR: feed it a crashing module and a script, get back a minimal reproducer).

The dossier below lists every command. An afternoon of replaying the walkthrough will teach you more than a month of architecture diagrams.

Then climb the same ladder the ecosystem climbed. The canonical study sequence: the Toy tutorial on mlir.llvm.org for the object model; the linalg dialect rationale for the structured-ops philosophy of section 5; the transform dialect tutorial for section 7’s machinery; then the CGO 2021 paper, which reads completely differently once you have touched the IR.

For a working CUDA engineer, a realistic 30-day arc is: week one, read IR and replay the walkthrough; week two, lower your own toy op through the same pipeline and break things on purpose; week three, build the previous section dialect and make the pattern fire on real IR; week four, open the dumps of one production Triton kernel you own and annotate every layout decision.

At the end of that month you will be conversant in the layer where, as section 10 argued, an increasing share of this industry’s margin gets decided.

Seven years ago this was a whiteboard sketch about taming TensorFlow’s compiler zoo. Today it compiles the kernels NVIDIA brags with, the NPU stacks of three chip giants, the vision system in a million cars, and it just priced a 150 person team at nearly four billion dollars.

The previous installment of this series ended by observing that LLVM won by being the infrastructure nobody had an incentive to replace. MLIR is running the same play one level up, on the layer where the AI hardware war is actually fought.

The empire had one IR. The dominion has as many as you can define, verify, and lower, and now you know how it is done.

Verification dossier

House methodology: every load-bearing claim is tiered, every number is sourced or explicitly derived, and for this piece, every code artifact was executed against real tools before publication.

Environment: Ubuntu 24.04, stock packages mlir-20-tools, llvm-20, libmlir-20-dev, all LLVM/MLIR 20.1.2.

Syntax note: MLIR evolves between LLVM releases; snippets are guaranteed against 20.1.2 and may need small spelling changes on older or newer toolchains.

Code verification manifest

Claims and confidence

Tiers: A primary source or machine-verified. B reputable secondary reporting or vendor primary for vendor facts. C reasoned inference or derived arithmetic, flagged as such in text. D interpretation and forecast: our analysis, argued not asserted.

Subscribe now

Sources

1. Qualcomm, “Qualcomm to Acquire Modular,” investor press release, June 24, 2026. investor.qualcomm.com/news-events/press-releases/news-details/2026/Qualcomm-to-Acquire-Modular/default.aspx

2. Modular, “Qualcomm to acquire Modular,” company blog, June 2026. modular.com/blog/qualcomm-to-acquire-modular

3. Quartz, deal report citing Reuters and WSJ terms, June 24, 2026. qz.com/qualcomm-acquires-modular-ai-software-stock-deal-062426

4. NAND Research, “Qualcomm acquires Modular for its hardware-agnostic AI software layer,” June 2026. nand-research.com

5. Modular repository newsline (funding history, releases). github.com/modular/modular

6. Chris Lattner, “Democratizing AI Compute, Part 8: What about the MLIR compiler infrastructure?”, Modular blog, January 2026. modular.com/blog/democratizing-ai-compute-part-8-what-about-the-mlir-compiler-infrastructure

7. Modular, “The Path to Mojo 1.0”; PyPI, mojo-compiler release history (1.0.0b1, May 7, 2026). modular.com/blog/the-path-to-mojo-1-0; pypi.org/project/mojo-compiler

8. NVIDIA, CUTLASS documentation: CuTe DSL introduction and overview. docs.nvidia.com/cutlass/latest/media/docs/pythonDSL/cute_dsl_general/dsl_introduction.html

9. Triton project, Gluon documentation and dialect sources. triton-lang.org/main/gluon

10. Qualcomm et al., “Hexagon-MLIR,” arXiv 2602.19762, 2026. arxiv.org/pdf/2602.19762

11. Phoronix, “AMD MLIR-AIE 1.2,” January 2026. phoronix.com/news/AMD-MLIR-AIE-1.2

12. Tesla Oracle, “Tesla rolls out FSD v14.3 (2026.2.9.6): better reaction time, rewritten AI compiler with MLIR,” April 8, 2026. teslaoracle.com

13. Lattner, Amini, Bondhugula, Cohen, Davis, Pienaar, Riddle, Shpeisman, Vasilache, Zinenko, “MLIR: Scaling Compiler Infrastructure for Domain Specific Computation,” CGO 2021.

14. “TPP-MLIR” upstream performance study, arXiv 2404.15204. arxiv.org/pdf/2404.15204

15. MLIR project documentation: Toy tutorial, linalg rationale, transform dialect tutorial. mlir.llvm.org

16. StableHLO specification and repository. github.com/openxla/stablehlo

17. Tenstorrent, tt-mlir repository. github.com/tenstorrent/tt-mlir

Intro

This is the story of how LLVM’s intermediate representation ate modern computing, told at the level of the instructions themselves, with the source numbers checked.

LLVM won because of two decisions made in one semester in the year 2000: a single, strongly typed, target-independent intermediate representation in SSA form that can be shipped and executed on its own, and a modular library architecture that let anyone reuse a compiler pass without dragging the whole compiler along.

Everything else, the trillion-dollar reach across mobile, cloud, consoles and AI, is a consequence of those two decisions.

The current release, LLVM 22.1, shipped in February 2026 and still carries the exact architecture Chris Lattner sketched over a winter break twenty-five years ago.

Subscribe now

The hole that GCC left

Why the dominant open compiler of 2000 could not become infrastructure

Compilers are the one technology every other technology has to pass through. They translate every line a human writes into something a processor can run, which means a compiler that is easy to build on is a lever under the entire software economy.

By the year 2000 the open lever was the GNU Compiler Collection, and it was showing its age.

GCC, launched in the 1980s, had become foundational: it built every Linux system, every Apple computer of the era, and a large share of the open source world.

But as Vikram Adve and Chris Lattner recount in their June 2026 retrospective in Communications of the ACM, GCC lagged on the techniques that were becoming central to modern compilation.

It was written in C rather than a modern object-oriented language, it lacked cross-module interprocedural optimization, and it did not gain Static Single Assignment form until GCC 4.0 in 2005.

Static Single Assignment, or SSA, is the representation on which most serious dataflow optimization is built, and GCC arrived late to it.

The deeper problem was structural. GCC was monolithic. You could not lift a piece of it out, an optimizer, a code generator, an analysis, and reuse that piece without pulling most of the compiler with it.

That single fact foreclosed most of the interesting futures: load-time and just-in-time compilation of mobile code, embedded scripting languages, sandboxed browser extensions, graphics shader compilation.

The managed-language runtimes of the day, the Jikes RVM for Java, Mono and later Roslyn for the CLR, were genuinely excellent inside their domain and advanced garbage collection and JIT compilation considerably.

But each imposed a language object model and a set of runtime semantics that ruled it out for the vast majority of other languages, and for anything that needed to ship mobile code without a managed runtime underneath it.

So the opening was specific and it was large: no open system combined ahead-of-time optimizing compilation of the kind static languages need with the self-contained, shippable, just-in-time-capable code representation that managed languages have, and offered both to every language rather than one family. Filling that hole is the whole of what follows.

The federal money that made it possible

Adve had come to Illinois with a research program aimed at flexible dynamic compilation for arbitrary languages.

His CAREER proposal to the National Science Foundation‘s Next Generation Software program, submitted in 2000, funded the work; the acknowledgments in the CACM piece name grant NSF EIA-0093426, and Adve’s own record lists the CAREER award at roughly $499,211.

That grant, Adve and Lattner write, “would prove pivotal to the group’s early work on LLVM,” and it supported both authors through most of the project’s early life.

The point they press, and it is worth pressing, is that this was blue-sky research with an unknowable outcome, funded on the condition that the artifacts be released as open source. The trillion-dollar downstream was not forecast. It was seeded.

A naming footnote

LLVM originally stood for Low Level Virtual Machine. The name outgrew the acronym years ago; the project’s own position, stated on llvm.org, is that “LLVM” is now simply the name of the umbrella project and has “little to do with traditional virtual machines.” We use it as a proper noun throughout.

Subscribe now

The IR is the entire argument

Everything LLVM can do is downstream of a single design object

If you understand one thing about LLVM, understand the intermediate representation, because the reach of the project is a property of the IR and not of any particular optimizer or backend.

LLVM IR is a strongly typed, RISC-like virtual instruction set. It abstracts away the machine, there are no physical registers and no addressing modes, but it stays low enough to represent operating systems, kernels and application code without imposing any language’s object model on top.

Three things about it do most of the work.

It is in SSA form. Every value is assigned exactly once, and each use points back to a single, unambiguous definition. Where control flow merges, an explicit phi instruction selects the incoming value based on which predecessor block you arrived from.

This is the representation that Ron Cytron and colleagues formalized in 1991, and it is what makes dataflow optimization tractable: because there is one definition per value, the flow from definitions to uses is explicit, and reordering transformations are freed from spurious dependencies.

Consider, for example, a trivial loop.

int sum_to(int n){
  int s = 0;
  for (int i = 0; i < n; i++)
    s += i;
  return s;
}

Compiled at -O0, Clang first emits the naive version: local variables become stack slots via alloca, and every read and write is an explicit load or store.

Note the pointer type is simply ptr, a point we return to below.

define i32 @sum_to(i32 %n) {
entry:
  %i   = alloca i32
  %s   = alloca i32
  store i32 0, ptr %s
  store i32 0, ptr %i
  br label %cond
  ; ... loads and stores on every iteration ...
}

Then the mem2reg pass (memory-to-register promotion, one of the first transformations any pipeline runs) proves those stack slots never escape and rewrites them into pure SSA values.

The stack traffic disappears, and phi nodes appear at the loop header and at the exit merge:

define i32 @sum_to(i32 %n) {
entry:
  %pos = icmp sgt i32 %n, 0
  br i1 %pos, label %loop, label %exit

loop:                                ; preds = %entry, %loop
  %i = phi i32 [ 0, %entry ], [ %i.next, %loop ]
  %s = phi i32 [ 0, %entry ], [ %s.next, %loop ]
  %s.next = add nsw i32 %s, %i
  %i.next = add nsw i32 %i, 1
  %done = icmp eq i32 %i.next, %n
  br i1 %done, label %exit, label %loop

exit:                                ; preds = %entry, %loop
  %r = phi i32 [ 0, %entry ], [ %s.next, %loop ]
  ret i32 %r
}

The nsw flags mark the additions as having no signed wraparound, a promise the front end makes on the compiler’s behalf so later passes can optimize more aggressively.

This is the texture of LLVM IR: low-level operations, explicit control flow, and a scattering of flags and metadata that carry higher-level facts down to where they can be exploited.

It carries a real type system. Integers of arbitrary bit width (i1, i8, i32, i64), floating point types, vectors as first-class values, arrays and structures.

Vectors matter more than they look: because a vector is a first-class value, the same IR expresses Intel SSE, AVX and AVX-512, Arm Neon and SVE, and the tensor operations that machine-learning front ends emit, and the auto-vectorizer can create them from scalar loops using dependence analysis.

Aggregate access goes through getelementptr, the address-arithmetic instruction that computes a field or element address without loading it:

; struct Point { i32 x; i32 y; };  compute &p->y
%y.ptr = getelementptr inbounds { i32, i32 }, ptr %p, i32 0, i32 1
%y     = load i32, ptr %y.ptr

It is extensible through intrinsics. Intrinsic functions are built-ins with defined semantics that behave like ordinary calls, so a pass that does not care about them can ignore them safely, while selected passes and the backend can recognize llvm.memcpy, llvm.masked.load, the matrix and vector-predication intrinsics, atomics and the rest, and lower them specially.

This is the mechanism that let the IR absorb decades of new hardware features without a language change every time.

The complexity was not free, and the authors are candid about it. LLVM IR launched in 2003 with fewer than thirty-five polymorphic operations. It now carries more than one hundred and seventy-five, including in excess of one hundred and ten vector operations defined as intrinsics, plus hundreds of further semantic intrinsics for garbage collection, memory ordering, synchronization and stack manipulation.

Most of that can be ignored when writing a front end or an optimization pass, but it makes writing a new backend, or a formal semantics for the IR, genuinely hard.

Three forms of one thing, and the opaque-pointer turn

The IR exists in three isomorphic forms: a human-readable textual assembly (.ll), a dense binary serialization called bitcode (.bc), and an in-memory form the compiler manipulates directly.

They are interconvertible without loss, which is exactly what lets the IR be shipped and compiled later rather than only used in-process.

clang -S -emit-llvm foo.c -o foo.ll   # textual IR
clang -c -emit-llvm foo.c -o foo.bc   # bitcode
llvm-as foo.ll -o foo.bc              # text  -> bitcode
llvm-dis foo.bc -o foo.ll             # bitcode -> text

The IR is not frozen. Two recent changes show the project still reshaping its own foundation. The larger one is opaque pointers. For most of LLVM’s life a pointer carried its pointee type, so you wrote i32*.

That information was redundant with the types on the loads and stores that actually used the pointer, and it forced a swarm of no-op bitcast instructions.

Per the official documentation, opaque pointers, a single untyped ptr, became the default in LLVM 15 (2022), and typed pointers were removed entirely in LLVM 17 (2023). Every IR sample in this article uses ptr for that reason.

The second, newer change landed in LLVM 22: a dedicated ptrtoaddr instruction that extracts a pointer’s integer address while separating it from provenance tracking, which the older ptrtoint conflated.

It is a small instruction with a real purpose, sharper semantics for the memory model, and it is the kind of change that keeps the IR honest as verification tools grow more demanding.

Subscribe now

Passes, and the separation of concerns

The modular library architecture is the part that actually won

The most cited technical novelty of LLVM is the IR. The most consequential design decision, the authors argue and we agree, is not technical at all in the usual sense: it is the modular, library-based architecture, built on a strict adherence to separation of concerns.

A compiler built on LLVM is assembled from libraries with clean interfaces rather than carved out of a monolith. That is why pieces of LLVM turn up in circuit-design tools, symbolic-execution engines, browser sandboxes and quantum-computing compilers, uses the authors freely admit they never imagined.

Concretely, transformation is organized as a pipeline of passes. Analysis passes compute facts (dominator trees, alias analysis, loop information, scalar evolution) and cache them; transformation passes consume those facts and rewrite the IR, declaring which analyses they preserve so the manager knows what to recompute.

Since LLVM 13 (2021) the default engine is the new pass manager, a rewrite that made analysis caching explicit and pass composition cheaper.

A canonical -O2 pipeline threads dozens of passes together. A representative slice:

• SROA and mem2reg to lift memory into SSA values;

• EarlyCSE and GVN to eliminate redundant computation;

• InstCombine to canonicalize and simplify instruction patterns;

• SimplifyCFG to clean up control flow; the inliner to pull small callees into their callers;

• LICM to hoist loop-invariant code;

• SCCP for sparse conditional constant propagation;

• IndVars and LoopUnroll for loops;

then the two vectorizers, the loop vectorizer and the SLP (superword-level parallelism) vectorizer, near the end. You can watch any of it run.

# run one pass and print the result
opt -passes=“mem2reg” -S sum.ll -o -

# see the full default -O2 pipeline the pass builder constructs
opt -passes=“default<O2>” -print-pipeline-passes -S sum.ll -o /dev/null

Writing a pass is deliberately cheap, which is the whole point. The 2003 release announcement claimed new users could “write their first LLVM pass in hours,” and the modern out-of-tree plugin form holds to that. Here is a complete, loadable function pass in the new pass manager:

#include “llvm/IR/PassManager.h”
#include “llvm/Passes/PassBuilder.h”
#include “llvm/Passes/PassPlugin.h”
using namespace llvm;

struct CountAllocas : PassInfoMixin<CountAllocas> {
  PreservedAnalyses run(Function &F, FunctionAnalysisManager &) {
    unsigned n = 0;
    for (BasicBlock &BB : F)
      for (Instruction &I : BB)
        if (isa<AllocaInst>(I)) ++n;
    errs() << F.getName() << “: “ << n << “ allocas\n”;
    return PreservedAnalyses::all();   // we changed nothing
  }
};

extern “C” LLVM_ATTRIBUTE_WEAK ::llvm::PassPluginLibraryInfo
llvmGetPassPluginInfo() {
  return {LLVM_PLUGIN_API_VERSION, “CountAllocas”, LLVM_VERSION_STRING,
    [](PassBuilder &PB) {
      PB.registerPipelineParsingCallback(
        [](StringRef Name, FunctionPassManager &FPM,
           ArrayRef<PassBuilder::PipelineElement>) {
          if (Name == “count-allocas”) {
            FPM.addPass(CountAllocas()); return true;
          }
          return false;
        });
    }};
}

clang++ -shared -fPIC CountAllocas.cpp -o CountAllocas.so \
  $(llvm-config --cxxflags)
opt -load-pass-plugin=./CountAllocas.so \
    -passes=“count-allocas” -disable-output sum.bc

The IR was the innovation everyone cites. The library architecture was the innovation that let ten thousand projects reuse a compiler without inheriting one.

Subscribe now

Five capabilities no one else has together

A twenty-year-old assertion from the CGO 2004 paper that still holds

In their 2004 paper at the International Symposium on Code Generation and Optimization, Lattner and Adve made a specific claim: LLVM provides a combination of five capabilities that no other system offers at once.

The paper won the conference’s Most Influential Paper award ten years later, and the retrospective restates the claim as still true more than twenty years on. The five:

CapabilityWhat it meansPersistent program informationThe IR can be preserved across an application’s whole lifetime, so optimization can happen at compile time, link time, load time, run time and even idle time between runs on the user’s machine.

Offline code generationRegardless of persistence, code can be lowered to efficient native machine code offline, using techniques too expensive to run at runtime.User-specific profilingProfile data can be gathered from deployed software as the end user actually runs it, then fed back into profile-guided optimization.

Transparent runtime modelThe IR imposes no object model, exception semantics or runtime, so it serves any language or mix of languages.Uniform whole-program compilationBecause it is language-independent, an entire application, including its language runtime and system libraries, can be optimized together after linking.

The argument is not that each property is unique; it is that the conjunction is. Managed virtual machines like the JVM or the .NET CLI deliver user-specific profiling and partially deliver persistence and whole-program compilation, but they do not deliver a transparent runtime model, which is what confines them to a language family.

And when they do offer offline code generation, the authors note, they do it at the expense of persistence and profiling. GPU instruction sets such as CUDA’s PTX and OpenCL’s SPIR-V, and the JITs behind JavaScript, Python and WebAssembly, cover some combination of persistence and profiling and none of the rest.

GCC, superb at offline code generation, was never designed to ship a language-independent representation for lifelong compilation at all.

Subscribe now

From IR to silicon

Instruction selection, register allocation, and the TableGen machine

Optimized IR is still target-independent. Turning it into machine code is the backend’s job, and LLVM has, notably, two backends that do it.

The mature path is SelectionDAG. Each basic block is turned into a directed acyclic graph of operations; the DAG is legalized so every node is something the target can actually do; instruction selection matches subgraphs against target patterns to pick real machine instructions; the result is scheduled into a linear order; and then register allocation and emission follow.

The newer path is GlobalISel, designed to eventually replace SelectionDAG. It works on the whole function rather than one block at a time and skips the DAG entirely: an IRTranslator lowers IR to generic machine instructions, a Legalizer makes them legal, RegBankSelect assigns register banks, and an InstructionSelect pass chooses final opcodes.

As of the current release GlobalISel is the default at -O0 on AArch64 and is used for AMD GPUs, and, as an ongoing LLVM forum discussion from February 2026 shows, work to enable it more broadly on AArch64 is still live. Two instruction selectors, maintained in parallel for years, is itself a statement about how much the project values getting this layer right.

Register allocation is where virtual registers, of which the IR has infinitely many, meet the finite physical register file. The default at -O1 and above is the greedy allocator, which orders live ranges by priority and spills to the stack when it must; -O0 uses a fast allocator that trades quality for speed.

Both are among the passes now being handed to machine learning, a thread we pick up in section 10.

Almost none of this is hand-written per target. Targets are described declaratively in TableGen, a domain-specific language whose .td files specify registers, instructions, calling conventions, scheduling models and selection patterns; a generator turns them into the C++ tables the backend uses.

A single instruction definition ties an assembly syntax to a SelectionDAG pattern:

def ADDrr : Instruction {
  let Namespace       = “MyTarget”;
  let OutOperandList = (outs GPR:$dst);
  let InOperandList  = (ins GPR:$src1, GPR:$src2);
  let AsmString      = “add $dst, $src1, $src2”;
  // match (add a, b) in the DAG and emit this instruction
  let Pattern = [(set i32:$dst, (add i32:$src1, i32:$src2))];
}

This is why standing up a new architecture in LLVM, RISC-V being the most visible recent example, is a matter of writing target descriptions rather than a compiler.

The LLVM 22 notes alone record new scheduling models for NVIDIA’s Olympus CPU, additional Arm C1 cores, Intel Wildcat Lake and Nova Lake, and fresh RISC-V vector extensions, all arriving as target-description work on top of the shared infrastructure.

Share The Software Frontier

The link-time frontier, and why ThinLTO exists

Whole-program optimization that actually scales to a datacenter build

Persistent IR unlocks link-time optimization: instead of discarding each source file’s IR after compiling it, you keep the bitcode, hand all of it to the linker, and optimize across module boundaries, inlining a function defined in one file into a caller in another, propagating constants program-wide.

Classic full LTO merges every module into one giant IR module and optimizes it as a unit. The result is excellent and the cost is brutal: the whole program has to fit in one process, and the work is largely serial, which is a non-starter when the program is a datacenter binary with tens of thousands of translation units.

ThinLTO, first presented at EuroLLVM 2015 by Teresa Johnson, Mehdi Amini and David Li and merged into LLVM over the following year, is the answer that made link-time optimization practical at industrial scale. Each module is compiled to bitcode alongside a compact summary: a thin index of its call graph and symbols.

A cheap global step reads only the summaries, decides cross-module actions such as which functions to import for inlining, and then the heavy per-module optimization runs in parallel, importing only the specific functions each module needs. It recovers most of full LTO’s benefit at a fraction of the memory, and it parallelizes. The numbers are quite stark.

Full LTO can inflate compile time two to three fold and demands that the whole program fit in a single process; ThinLTO’s per-module summaries add under one percent to bitcode size, roughly 0.8 percent for Clang built without debug information, and keep build time and peak memory close to a non-LTO build.

On some benchmarks ThinLTO even edges out full LTO, because its scalable design lets each module run a more aggressive backend pipeline than a single monolithic module could afford.

LLVM 22 is now upstreaming Distributed ThinLTO (DTLTO), which pushes those parallel backend jobs across a build cluster, with caching for incremental rebuilds. Turning it on is a flag.

clang -flto=thin -O2 a.c b.c c.c -o app   # scalable cross-module LTO
clang -flto      -O2 a.c b.c c.c -o app   # full (monolithic) LTO

This is the mechanism underneath one of the headline claims: that all of Google’s and Meta’s datacenter C and C++ is built with LLVM. It is not merely that Clang compiles the files; it is that link-time optimization across an entire service is tractable because ThinLTO made it so.

Share

What LLVM actually runs

The reach, with the adoption dates checked against primary sources

The scale claims in the retrospective are large, and they hold up. LLVM long ago replaced GCC as the foundation of Apple’s software across every device line. Google uses it for Android and the Android NDK. Applications on Qualcomm’s Snapdragon processors are built in substantial part with LLVM.

The two dominant CPU vendors for PCs and servers, Arm and Intel, retired their proprietary compilers, ARMCC and ICC, in favor of LLVM-based toolchains. Google’s and Meta’s datacenter C and C++ is compiled with it. Sony’s PlayStation 4 and 5 and the Nintendo Switch use Clang as their primary application toolchain.

And the AI stack leans on it heavily, through NVIDIA’s CUDA compiler, through the Triton compiler used by PyTorch and OpenAI, and most broadly through MLIR.

The adoption did not happen at once, and the sequence matters, because it shows a single seed (Apple, via Lattner’s hiring) catalyzing an industry.

The bitcode chapter is the sharpest illustration of the “lifelong compilation” idea reaching production. For years, developers shipped iOS, watchOS and tvOS apps to Apple’s App Store not as machine code but as LLVM bitcode, and Apple compiled and specialized each app for the target device on its side.

That let one submission serve 32-bit and 64-bit Arm iPhones, low-power Apple Watch variants and Apple TV, shrank downloads, and let Apple roll out new compiler optimizations without developers rebuilding.

Apple deprecated bitcode in Xcode 14 in 2022, once its hardware had converged on Arm64 and the flexibility was no longer worth the friction. The feature retired, but it stands as the largest deployment of shippable IR the industry has seen.

Sizing the surface area

The authors put a figure on all this: software compiled with LLVM, they write, represents hundreds of billions of dollars in annual revenue across mobile, cloud, desktop, server, gaming, supercomputing and AI.

We think that undercounts it, and it is worth being precise about what can actually be measured, because “revenue compiled by LLVM” is not a single clean quantity.

The honest move is to measure the top-line revenue of the platforms whose flagship products are demonstrably built with LLVM, treat it as a floor rather than an attribution, and be explicit that this is revenue riding on LLVM-compiled software, not value the compiler captures.

Do that and the picture is not subtle. In their most recent full fiscal years three companies alone, Apple, Alphabet and NVIDIA, booked about 1.04 trillion dollars in combined revenue, and the flagship products behind every one of those dollars are built with LLVM: Apple’s entire device line through Clang and Swift, Android and Google’s hyperscale C and C++ through Clang and ThinLTO, and the AI-accelerator stack through the CUDA, Triton and MLIR compilers that lower into LLVM.

We do not sum the rows of that table: the smartphone market overlaps the Apple and Alphabet figures, and the point is the surface area, not a grand total.

But the three company revenues in Figure 4 are independently additive and clear a trillion dollars between them, which is why we are comfortable calling LLVM’s reach a trillion-dollar one. The precise figure is unauditable and we mark it accordingly in the dossier; the order of magnitude is not in doubt.

A large majority of end users across computing are, at some layer, running code that LLVM compiled.

The academic and language dividend

Two quieter forms of impact compound over time. LLVM underlies a generation of programming languages that an easy-to-target, modular back end made feasible: Swift, Rust, Julia, Halide and Mojo among them.

Rust’s memory-safety guarantees and Julia’s JIT-driven scientific performance both rest on LLVM code generation.

And in the academy, LLVM democratized compiler research and teaching: a graduate student can write a novel pass in a day and test it on large real programs immediately, and the authors report meeting undergraduates who arrived at college having already built compiler projects on LLVM in high school, “for fun.”

Leave a comment

MLIR, and the compiler layer under modern AI

When one IR was not enough, the answer was a framework for making IRs

This is the part that matters most for anyone working on inference infrastructure, so we will spend a moment on it. LLVM IR is deliberately low-level, which is a strength for representing machines and a weakness for representing a matrix multiply.

By the time a tensor operation has been shredded into scalar loads, adds and branches, the structure a compiler most wants to exploit, the fact that this is a dense linear-algebra kernel, is gone. That loss is fine for C. It is expensive for machine learning, where the highest-value optimizations (tiling, fusion, layout, targeting a systolic array) live at exactly the level LLVM IR throws away.

MLIR, the Multi-Level Intermediate Representation, introduced in 2019 and presented at CGO 2021, is the response, and it is characteristically an infrastructure answer rather than a point solution. Instead of one IR, MLIR provides the machinery to define many coexisting IRs, called dialects, that all share the same structural substrate of operations, regions and blocks.

A compiler keeps a computation at a high level of abstraction as long as that is useful, then progressively lowers it, dialect by dialect, until it reaches the llvm dialect and finally ordinary LLVM IR, where the mature backends take over. A tensor program can begin like this:

// a matmul on tensors, before any lowering
func.func @matmul(%A: tensor<128x256xf32>,
                  %B: tensor<256x512xf32>,
                  %C: tensor<128x512xf32>) -> tensor<128x512xf32> {
  %0 = linalg.matmul
         ins(%A, %B : tensor<128x256xf32>, tensor<256x512xf32>)
         outs(%C   : tensor<128x512xf32>) -> tensor<128x512xf32>
  return %0 : tensor<128x512xf32>
}

From there the same operation descends through structured and loop dialects (linalg, affine, scf, tensor), gets its buffers assigned (memref, bufferization, vector), is retargeted to a device dialect (gpu, nvvm, rocdl, spirv), and finally lands in the llvm dialect and LLVM IR.

Each step is a rewrite in the same framework, and each step is where a tiling or fusion decision can be made while the structure is still visible.

For an inference audience the payoff of this layering has a name: fusion. Because progressive lowering keeps tensor-level structure visible until late, the compiler can fuse a chain of operations, a matmul followed by a bias add followed by an activation, into a single kernel, so the intermediate tensors are never written out to high-bandwidth memory and read back.

That matters because decode, the token-by-token phase of LLM serving, is bound by memory bandwidth rather than arithmetic: the accelerator spends its time moving weights and activations, not multiplying them.

Collapsing those HBM round trips is therefore one of the largest levers on tokens per second per GPU, and it is a compiler transformation, made at exactly the abstraction level that raw LLVM IR throws away.

This is the concrete reason every serious inference stack now ships a compiler rather than a fixed library of kernels, and why the layer we are describing sits directly upstream of inference cost.

This is not a research curiosity. MLIR is the backbone of TensorFlow’s XLA compiler and the broader OpenXLA ecosystem, of parts of PyTorch through Torch-MLIR, of IREE, and of OpenAI’s Triton, the kernel language a great deal of contemporary GPU work is written in. Even components of NVIDIA’s own CUDA toolkit use it.

And the underlying LLVM backend still does the final, unglamorous work of register allocation and scheduling for all of them. When people say the AI stack runs on LLVM, this is the mechanism they are pointing at: not that models are written in C, but that the compilers turning tensor graphs into GPU code are, almost universally, MLIR lowering into LLVM.

The hardware-design world is following the same path through CIRCT, which extends MLIR into digital circuit design.

The one thing LLVM is bad at

Compile speed, and the backends being built to route around it

A retrospective written by the project’s founders is not the place to look for LLVM’s weaknesses, so we will supply the missing side.

LLVM is engineered for the quality of the code it emits, and it pays for that in the time it takes to emit it. Even with optimizations disabled the pipeline is heavy, and for a developer sitting in an edit, compile, run loop the wait is a real tax.

As bjorn3, the author of Rust’s Cranelift backend, puts it, LLVM is optimized for output quality at the cost of compilation speed even when optimizations are turned off. This is the most common and most legitimate complaint about the project, and two major language communities are now building their own code generators specifically to escape it.

Rust took the incremental path. Its alternative rustc_codegen_cranelift backend uses Cranelift, a code generator built for speed rather than peak output and aimed squarely at debug builds.

The Rust team’s 2025 measurements on large real projects, among them Zed, Tauri and hickory-dns, show roughly a twenty percent cut in code-generation time and about a five percent speedup in total clean-build time, and getting the backend production-ready for local development is an official 2025 project goal.

The bargain is stated plainly: Cranelift produces code almost as fast as LLVM with optimizations off, in exchange for compiling far more quickly.

Zig went further and wrote its own backends outright. In mid-2025 it made its self-hosted x86-64 backend the default for debug builds on Linux and macOS, bypassing LLVM entirely on that path, and reported wall-clock compile improvements ranging from five to fifty percent across real projects.

The rationale is a direct indictment: because compilation time was dominated by LLVM, and because LLVM’s heavy use of shared state kept Zig’s LLVM path effectively single-threaded while its own backend parallelized code generation across cores, the only way to get materially faster was to leave.

Notably, Zig still emits LLVM bitcode when it wants LLVM’s optimizer for release builds. That is the pattern worth seeing.

Keep LLVM for the optimized release build, where its code quality is still unmatched. Route around it for the fast debug build, where its latency hurts.

The shape of both the Rust and the Zig response to LLVM’s one real weakness

That shared pattern is not a threat to the empire so much as a map of its single soft border, and it is worth being honest that the border exists. There is a second, quieter tension living inside LLVM’s own success.

MLIR’s openness invites a proliferation of dialects, and a representation that can be anything risks fragmenting into many incompatible somethings; the whole force of the original LLVM argument was that the IR won because it was one thing.

“The IR won because it was one representation” and “the future is many coexisting representations” are in genuine tension, and how MLIR holds its dialects together as their number grows is one of the more interesting open questions in the field. So far the shared substrate has held.

The bet is that it keeps holding.

Subscribe now

Where LLVM goes next

Verified compilation, auto-generated backends, machine-learned heuristics, hardened code

The retrospective closes on active research, and several threads are worth stating precisely because they point at the next decade of the project.

Proving the optimizer correct

A compiler bug silently miscompiles correct source into wrong machine code, and optimizers are where these bugs breed.

Alive2 (Nuno Lopes and colleagues, PLDI 2021) attacks this with bounded translation validation: it takes an IR function before and after a transformation and uses an SMT solver to check whether the rewrite is sound for all inputs, encoding LLVM’s tricky semantics, including undefined behavior and poison, into the query.

Alive2 runs continuously against LLVM and has found real miscompilations in core passes such as InstCombine. The ptrtoaddr instruction we met in section 02 is part of the same trend: the project is steadily sharpening its own semantics so that this kind of verification can go further.

Generating the compiler from the chip

Two projects challenge the assumption that backends and peephole optimizations must be written by hand. Hydride (ASPLOS 2024) and MISAAL (published in the ACM’s Proceedings on Programming Languages, 2025) use program synthesis to generate compiler components, target-independent IR operations, front-end translators and backend code generators, directly from a vendor’s formal specification of an instruction set.

The reported result is striking: auto-generated compilers that match or beat a heavily hand-engineered production compiler for Halide while requiring roughly an order of magnitude less manual effort.

As instruction sets multiply, matrix and vector extensions arriving on nearly every new chip, generating the compiler from the specification rather than writing it by hand is a plausible future for the backend.

Machine learning inside the compiler, and about the compiler

There are two distinct AI-and-compiler stories, and they are easy to confuse. The first is machine learning inside LLVM: Google’s MLGO framework replaces hand-tuned heuristics for inlining-for-size and register-allocation eviction with policies trained by reinforcement learning, and these have shipped in production builds.

The second is LLVM as training data for models about code. Meta’s LLM Compiler (2024), built on Code Llama and published at Compiler Construction 2025, was trained on a corpus of 546 billion tokens of LLVM IR and assembly, then instruction-tuned to predict the effect of optimization passes; the flag-tuning and disassembly variants add a further 164 billion tokens for 710 billion in total.

Meta reports the model reaching 77 percent of the optimization benefit of an autotuning search, and a 45 percent round-trip on disassembling machine code back to IR.

The reason such a model can exist at all is the reason LLVM matters everywhere else: the sheer volume of LLVM-compiled open source makes a compiler-scale training corpus possible.

One number, corrected

The CACM retrospective states the LLM Compiler was fine-tuned on “537 billion tokens.” The figure in Meta’s paper and in the peer-reviewed Compiler Construction 2025 version is 546 billion tokens of LLVM IR and assembly. We use 546 billion, which is what the primary source reports.

Hardened code and heterogeneous parallelism

Two more directions round out the picture.

On security, LLVM and Clang are the delivery vehicle for a generation of hardening: forward-edge control-flow integrity, shadow call stacks, and the Arm hardware defenses of pointer authentication (PAC), branch target identification (BTI) and memory tagging (MTE), alongside the sanitizer family (AddressSanitizer, ThreadSanitizer, MemorySanitizer, UndefinedBehaviorSanitizer) that made whole classes of C and C++ bugs findable.

LLVM 22 even added the AArch64 build-attribute machinery that lets linkers reason about PAC and BTI compatibility across object files.

On parallelism, the HPVM project (IEEE Micro 2022) layers a hierarchical dataflow graph over LLVM IR to target heterogeneous systems, CPUs, GPUs and accelerators, from a single representation, and national laboratories are extending LLVM for exascale machines through efforts such as Los Alamos’s Kitsune and the NNSA-backed Flang Fortran front end.

What the story is actually about

Cadence, and the transferable lesson under the technology

Strip away the specifics and LLVM is a case study in how foundational infrastructure gets built, and the ingredients are unglamorous. Federal money funded early-stage research whose payoff no one could forecast, on the condition that the results be released openly.

An academic effort took known techniques, SSA, separation of concerns, retargetable code generation, interprocedural analysis, and applied all of them together more thoroughly than any production compiler had.

A commercially neutral open license, eventually Apache 2.0 with LLVM exceptions, let the result be reused, again and again, in products that competed with each other. Strong technical and community leadership, first inside Apple and then across the industry, drove adoption.

And a family of successful derivatives, Clang, Swift and MLIR chief among them, widened the reach far beyond the original core.

The result compounds on a schedule. Twenty-three years after the first release, LLVM ships a full toolchain, Clang, LLD, LLDB, libc++, compiler-rt, MLIR, Flang, Polly, BOLT, on a strict six-month train.

Adve and Lattner end their retrospective on the claim that a small, federally funded academic project produced “world-changing, foundational infrastructure,” and on the evidence it is not an overstatement.

The instruction set that Lattner sketched over a winter break in 2000, three isomorphic forms of one strongly typed, target-independent, shippable IR, is visibly the same architecture running underneath the compiler in every phone, console, datacenter and AI accelerator we have discussed.

The empire is real. It was always, underneath, an argument about a representation.

Verification dossier

Every load-bearing claim above, checked against a primary or authoritative source. Confidence tiers: A primary source, direct confirmation · B authoritative secondary · C reasonable inference or the authors’ own estimate · D flagged discrepancy or figure we corrected.

A. LLVM 1.0 was first released in October 2003

Chris Lattner’s announcement, “The LLVM 1.0 Release is finally available!”, is dated October 24, 2003; Wikidata records the same inception date. Discrepancy noted: the CACM retrospective says “October 2003” in its introduction but “December 2003” in two later places. October 24, 2003 is correct; the December date appears to be the release-notes page’s last-modified timestamp (Dec 8, 2003), not the release.

A. Current stable release is LLVM 22.1, from February 2026

LLVM/Clang 22.1 was released February 24, 2026 per the project and Phoronix; the latest point release, 22.1.8, is dated June 16, 2026 on GitHub. LLVM 23 is in development as 23.0.0git. Consistent with the project’s post-2017 six-month cadence.

A. The CGO 2004 paper won Most Influential Paper; the team won the 2012 ACM Software System Award

“LLVM: A Compilation Framework for Lifelong Program Analysis and Transformation” (CGO 2004, pp. 75 to 86, DOI 10.1109/CGO.2004.1281665) was named Most Influential Paper of CGO 2004, awarded 2014. Nuance: the 2012 ACM Software System Award went to Adve, Lattner and Evan Cheng; the CACM retrospective names only the two authors. The “more than 8,000 citations” figure is the authors’ own and is conservative against public citation indices.

D. The Meta LLM Compiler was trained on 546 billion tokens, not 537 billion

The CACM piece states 537 billion tokens. Meta’s paper (arXiv 2407.02524) and the peer-reviewed Compiler Construction 2025 version (DOI 10.1145/3708493.3712691) both report 546 billion tokens of LLVM IR and assembly, with a further 164 billion for flag-tuning and disassembly (710 billion total). We use 546 billion.

A. Opaque pointers became default in LLVM 15; typed pointers were removed in LLVM 17

Confirmed by the official Opaque Pointers documentation: default in LLVM 15 (2022), typed-pointer support removed in LLVM 17 (2023). The ptrtoaddr instruction is a genuine LLVM 22 addition, per the 22.1 release coverage.

A. Apple deprecated bitcode in Xcode 14 (2022)

Xcode 14 release notes: the App Store no longer accepts bitcode submissions and Xcode no longer builds bitcode by default. This bounds the “mid-2010s through 2022” window the retrospective gives for App Store bitcode.

B. Industry adoption: Apple, Google, Arm, Intel, Sony, Nintendo, Qualcomm

Apple’s OpenGL/iOS SDK work began around 2005 and the macOS OpenGL JIT shipped in 2006 (Clang and LLVM histories). Intel switched its C/C++ stack to LLVM (ICX / oneAPI DPC++); Arm shipped LLVM-based Arm Compiler 6 and drove the Apache 2.0 relicensing over patent concerns. Sony uses Clang for PS4/PS5. These are well documented across primary announcements and vendor pages.

C. ”Hundreds of billions of dollars in annual revenue” compiled by LLVM

This is the authors’ order-of-magnitude estimate, not an audited figure, and we present it as such. The qualitative claim, that a large majority of end users run LLVM-compiled code at some layer, is well supported by the adoption evidence above.

A. MLIR underpins XLA, PyTorch (Torch-MLIR), IREE and Triton; GlobalISel is default at -O0 on AArch64

MLIR was presented at CGO 2021 (DOI 10.1109/CGO51591.2021.9370308) and is used across the named AI compilers. GlobalISel’s default-at-O0 status on AArch64 and ongoing work to broaden it are documented in current LLVM forum threads (February 2026).

B. ThinLTO: first presented at EuroLLVM 2015; summaries add under one percent to bitcode; full LTO costs two to three times the compile time

ThinLTO was introduced by Teresa Johnson, Mehdi Amini and David Li (LLVM Project blog, 2016; presented EuroLLVM 2015). The summary overhead of about 0.8 percent for Clang without debug info, and the design goal of build time and memory close to a non-LTO build, are from that write-up and the Clang ThinLTO documentation. The two-to-three-fold compile-time cost of full LTO is a widely reported figure (for example, the Gentoo LTO documentation). “Sometimes beats full LTO” reflects the authors’ own SPEC results, where ThinLTO’s more aggressive per-module pipeline occasionally wins.

C. LLVM’s reach can be sized as a trillion-dollar surface area

Apple FY2025 revenue 416.16 billion dollars (Form 8-K, quarter and year ended Sept 27, 2025); Alphabet 2025 revenue 402.84 billion dollars (Form 10-K / Annual Report 2025); NVIDIA fiscal 2026 revenue 215.9 billion dollars, data center 193.7 billion, for the year ended Jan 25, 2026 (Form 8-K). Combined, about 1.04 trillion dollars. This is top-line company revenue for firms whose flagship products are built with LLVM, presented as a floor, not value attributed to or captured by the compiler. Smartphone units of about 1.25 billion for 2025 are from IDC and Omdia. We do not treat the constructed total as audited.

A. Rust (Cranelift) and Zig are building non-LLVM backends to escape LLVM compile times

Rust’s rustc_codegen_cranelift reports roughly a twenty percent reduction in code-generation time and about five percent in clean-build time on large projects, and production-readiness is a 2025 Rust project goal (rust-project-goals, 2025h2). Zig made its self-hosted x86-64 backend the default for Debug builds on Linux and macOS in mid-2025, reporting five to fifty percent wall-clock improvements and citing LLVM as the compile-time bottleneck (Zig devlog, 2025). Both retain an LLVM path for optimized release builds.

Sources

1. V. Adve and C. Lattner, “The LLVM Compiler Infrastructure,” Communications of the ACM, June 2026. cacm.acm.org

2. C. Lattner and V. Adve, “LLVM: A Compilation Framework for Lifelong Program Analysis and Transformation,” CGO 2004, pp. 75 to 86. DOI 10.1109/CGO.2004.1281665.

3. C. Lattner, “The LLVM 1.0 Release is finally available!”, llvm-announce, October 24, 2003.

4. LLVM Project, “Opaque Pointers.” llvm.org/docs/OpaquePointers.html

5. Arm, “What is new in LLVM 22,” 2026; M. Larabel, “LLVM/Clang 22.1 Released,” Phoronix, Feb 24, 2026.

6. LLVM Project, release tags through 22.1.8. github.com/llvm/llvm-project

7. Apple, “Xcode 14 Release Notes,” 2022. developer.apple.com

8. N. P. Lopes et al., “Alive2: Bounded Translation Validation for LLVM,” PLDI 2021. DOI 10.1145/3453483.3454030.

9. C. Lattner et al., “MLIR: Scaling Compiler Infrastructure for Domain Specific Computation,” CGO 2021. DOI 10.1109/CGO51591.2021.9370308.

10. C. Cummins et al., “Meta Large Language Model Compiler,” arXiv 2407.02524, 2024; CC 2025, DOI 10.1145/3708493.3712691.

11. A. Kothari et al., “Hydride,” ASPLOS 2024, DOI 10.1145/3620665.3640385; A. R. Noor et al., “MISAAL,” PACMPL 2025.

12. A. Ejjeh et al., “HPVM: Hardware-agnostic programming for heterogeneous parallel systems,” IEEE Micro 42(5), 2022.

13. V. Adve, faculty biography and CV, University of Illinois. vikram.cs.illinois.edu

14. T. Johnson, M. Amini, D. Li, “ThinLTO: Scalable and Incremental LTO,” LLVM Project Blog, 2016 (presented EuroLLVM 2015); Clang “ThinLTO” documentation. blog.llvm.org

15. Rust Project Goals 2025h2, “Production-ready Cranelift backend.” rust-lang.github.io

16. Zig, Devlog 2025 (self-hosted x86-64 backend default for Debug builds). ziglang.org/devlog/2025

17. Apple Inc., Form 8-K, fiscal 2025 fourth-quarter and full-year results (year ended Sept 27, 2025), revenue 416 billion dollars. SEC EDGAR

18. Alphabet Inc., Annual Report / Form 10-K for 2025, total revenue 402.84 billion dollars. SEC EDGAR

19. NVIDIA Corp., Form 8-K, fourth quarter and fiscal 2026 (year ended Jan 25, 2026), revenue 215.9 billion dollars, data center 193.7 billion. SEC EDGAR

20. IDC and Omdia, worldwide smartphone shipments for 2025, about 1.25 billion units.

Inference.Engineering · Independent analysis of LLM serving infrastructure, GPU economics, and the compiler layer beneath them. This piece is built on the June 2026 CACM retrospective by Vikram Adve and Chris Lattner and cross-checked against LLVM 22.1 documentation, release notes and the primary research literature. Vendor-independent; every figure sourced. Corrections and disputes welcome.

Leave a comment

Intro

For three years, the cost of making a token fell every quarter while the budget to buy tokens grew without a meter. In H1 2026 both curves reversed at once: memory pass-through is raising the cost floor of every token served, while enterprises institutionalize token budgets at the ceiling.

This is the anatomy of inference’s first margin squeeze: the wafer math, the proof of pass-through, modeled cost floors marked to real market prices, and the engineering playbook that defends the spread.

Two curves reversed in the same half-year

Every industry eventually meets the quarter when its two defining curves cross. For LLM inference, that quarter just happened.

Since late 2022, the economics of this business rested on two assumptions so reliable nobody wrote them down: the cost of producing a token falls every quarter (process nodes, kernel engineering, quantization, and a relentless price war), and the budget for buying tokens grows without a meter, the era of “tokenmaxxing,” when Meta and Salesforce were publicly urging employees to consume as many tokens as possible. In the first half of 2026, both assumptions died within months of each other.

Jaw one: the cost floor turned upward. That is the core of this issue: the deepest memory supercycle in the forty-year history of DRAM, transmitted measurably into GPU rental rates with a 4-6 week lag ( Fig. 7). For the first time since the ChatGPT moment, the marginal cost of serving a token is inflecting up, not down: driven by the single most supply-constrained commodity in technology.

Jaw two: the price ceiling got institutionalized. SemiAnalysis published field data on July 2 from conversations with 50+ enterprises: Uber burned through its annual Claude Code and Codex budget in four months and responded with a $1,500/month per-employee cap; an aerospace manufacturer’s $250/month caps were exhausted by power users in four days; companies are switching off premium model tiers and downgrading defaults; enterprise budgets now range from $250 to tens of thousands per employee per month, but they are budgets, reviewed by finance.

Read the finding precisely, because it is not a demand-collapse story: SemiAnalysis concludes there is no material risk to 2H26 AI budgets and API spend keeps compounding. The jaw is not falling demand, it is the arrival of price sensitivity.

Unbounded willingness-to-pay is over; every serving provider now negotiates against a CFO’s cap instead of an enthusiast’s curiosity.

The quarter the memory market broke

Start with the raw numbers, because they have no precedent in the forty-year history of the DRAM industry. TrendForce’s Q1 2026 contract-price revisions came in at +90-95% quarter-over-quarter for conventional DRAM: revised upward from an initial forecast of +55-60% that analysts had already called unprecedented.

PC DRAM was expected to more than double in a single quarter; server DDR5 rose roughly 90%; NAND contract prices climbed 33-38% in the same window. Counterpoint tracked spot DRAM up 80-90% inside the quarter.

For the full year, TrendForce projects DRAM up more than 70% on top of the Q1 step-change; Bank of America has DRAM industry revenue +51% YoY and NAND +45%.

The supply side tells you why this is not a blip. SK Hynix announced in October 2025 that its entire 2026 production capacity was already sold out. Kioxia has said the same of its 2026 NAND output.

Micron shut down its Crucial consumer business outright (a two-decade brand, liquidated) to route every wafer toward hyperscaler and GPU-grade contracts, and its executives describe being able to cover at most two-thirds of medium-term demand for some customers.

Micron’s Boise fabs ramp in 2027-2028; the New York megafab in 2030. Jensen Huang, asked at CES 2026 whether gamers should resent AI for GPU prices, answered in effect that the world needs more memory factories.

As stated even before, in the previous issues: when the largest memory buyer on Earth says the fix is on the supply side, the message downstream is: you will not negotiate your way to allocation relief.

Three structural breaks from every previous cycle

1. The demand driver is durable, not episodic. Prior shortages came from fab fires, earthquakes, supply discipline, or one-off demand pulses (crypto, pandemic PCs). This one comes from inference workloads compounding in production. IDC’s assessment calls it a “permanent reallocation” of the world’s wafer capacity, not a cyclical shortage, and IDC does not deploy the word “permanent” casually.

2. The margin gradient is a one-way valve, and the wafer math is brutal. HBM carries gross margins 3-5× commodity DRAM, so every rational fab reallocates cleanroom toward it.

But because of TSV formation, die thinning, stacking, and compounded assembly yield, one gigabyte of HBM consumes ≈4× the wafer capacity of one gigabyte of standard DRAM; GDDR7 consumes ≈1.7×. AI’s effective wafer claim reaches ~20% of world DRAM capacity in 2026 against total bit-supply growth of only 10-16% per year.

The arithmetic cannot close without starving someone, and “someone” is every non-AI buyer plus every AI buyer without a long-term agreement.

3. The price convergence matters more than the price level. HBM3e historically sold at 4-5× server DDR5 per bit. TrendForce expects that gap to compress to 1-2× by end-2026, not because HBM got cheaper, but because DDR5 inflated toward it. Sit with that.

The entire tiered-memory architecture of modern serving (KV offload to host DRAM, paged hierarchies, DDR5-backed prefix caches, NVMe cold tiers) was designed in a world where the tier below HBM was 4-5× cheaper per bit.

When the discount for descending a tier collapses from ~80% to ~30-50%, a large fraction of the “obvious” tiering optimizations of 2024-2025 stop being obvious. Section 5 does that math properly.

One supply-side note that matters for H2: HBM4 becomes the mainstream HBM in the second half of 2026, with a doubled 2,048-bit interface, ~2 TB/s per stack, and a manufacturing-complexity premium above 30% over HBM3e. NVIDIA’s Rubin ships with up to 288 GB of HBM4 per GPU, and every one of those stacks is wafer capacity that used to be laptops.

The KV cache ate the memory supply

The lazy narrative is “AI ate the memory market.” The precise narrative is that the KV cache ate the memory market, and the industry’s shift from training-dominated to inference-dominated compute is what lit the fuse.

Training demand is large but concentrated and schedulable: a fixed pool of HBM for weights, activations, and optimizer states, planned quarters ahead. Inference demand is elastic and per-user: every concurrent conversation, agent trajectory, and 200K-token codebase instantiates its own slab of state that must live somewhere in the hierarchy for the life of the request, and increasingly beyond it, as prefix caches persist context between turns.

Industry estimates via TrendForce put cloud high-speed memory consumption near 3 exabytes in 2026, with core inference platforms (Gemini-, Bedrock-, ChatGPT-class serving) accounting for ~750 PB of live memory demand before redundancy roughly doubles it.

TrendForce separately projects that by 2029 inference (not training) becomes the primary driver of AI server demand outright, and North American CSPs are already bulk-buying high-capacity DDR5 RDIMMs specifically for inference fleets, the demand class that used to be the safety valve absorbing DRAM oversupply.

The per-token physics, derived from first principles

Everything in this issue hangs on one number: bytes of KV state per token. Take the workhorse case, a Llama-3-class 70B dense model with grouped-query attention: 80 layers, 8 KV heads, head dimension 128.

A single 128K sequence on the GQA-8 model pins ~41 GB at FP16: half an H100’s entire HBM for one user’s context, before a single weight is stored. This is why effective batch size at long context is a memory-capacity calculation, not a compute calculation, and why every marginal concurrent user at long context is, economically, a memory purchase.

Multiply by the fleet: 10,000 concurrent long-context sequences at FP8 KV pins ~200 TB of state across HBM, DDR5, and NVMe, state that produces no tokens itself; it exists purely so decode can stream it past the compute units at terabytes per second.

The 2026 memory market is what happens when the industry’s aggregate KV state, growing super-linearly with agent adoption and context inflation, collides with wafer supply growing 10-16% a year.

Two second-order effects complete the picture

NAND got dragged in. NVMe became the cold tier for prefix caches and the staging layer for weights (including GPU-direct storage paths). Kioxia now says nearly half its future NAND demand could come from AI. Meanwhile Samsung and SK Hynix cut NAND wafer output in 2024-2025 while chasing HBM margins. Omdia has Samsung’s NAND wafers falling 4.9M→4.68M and SK Hynix’s 1.9M→1.7M. There is no cheap tier left to hide in.

The host-DRAM tax on every GPU node inflated. A serious 2026 inference node pairs its GPUs with 1-2 TB of DDR5 for KV offload, CPU-side batching, and cache tiers. At Q1 2026 server-DRAM pricing, the host memory on a single node can cost what a mid-range GPU did two years ago. No 2024-vintage TCO model has this line item at anywhere near its current size.

Roofline: why prefill and decode were never the same workload

Before the silicon story makes sense, you need the roofline argument stated with numbers, not vibes. A kernel’s attainable throughput is bounded by min(peak_FLOPs, intensity × bandwidth), where arithmetic intensity is FLOPs performed per byte moved from memory.

The machine has a ridge point, the intensity at which it stops being bandwidth-bound and becomes compute-bound.

The corollary that most cost models miss: decode throughput is not a FLOPs number, it is a bandwidth budget divided by bytes-per-step. Per decode step the device must stream the weights once plus every live sequence’s KV:

Read those three lines again. Same GPU, same model, same bandwidth: 16× throughput difference between short-context and long-context traffic, and a clean 2× recovered at 128K purely by halving KV bytes. Context length is not a feature flag; it is a cost multiplier that acts through memory.

Every KV byte you remove converts directly into either more concurrent sequences or faster steps: both of which are revenue.

Rubin CPX is a memory trade wearing a GPU costume

Now read NVIDIA’s 2026 inference roadmap through the memory market and it snaps into focus in a way most launch coverage missed. If prefill is compute-bound and decode is bandwidth-bound (§03), then serving both phases on identical HBM-stuffed GPUs means that during prefill you are paying for the most supply-constrained, margin-rich commodity on the planet (HBM bandwidth) and not using it.