Introduction
In August, while working through NVIDIA’s Groq 3 LPX, we derived a number I haven’t been able to stop thinking about. If you put the KV cache back into the roofline of a decode step, the critical batch (the batch at which a GPU stops waiting on memory and starts doing math) stops being a number and turns into a pole.
Past a certain context length no batch size makes the GPU compute bound, because every request you add brings bytes faster than it brings FLOPs. For a dense 31-billion-parameter model that context came out between 84 and 162 tokens on every recent GPU we checked.
I filed it as a curiosity, and I was wrong to. Attention-FFN disaggregation, which is how NVIDIA now proposes to serve long-context decode at high speed, is why the Vera Rubin platform has gained a chip that does not run CUDA, and probably part of why a GPU announced in September 2025 to speed up prefill disappeared from the roadmap six months later. It’s also the subject of at least four papers this year and an experimental vLLM plugin, and they do not all agree.
This piece is our attempt to work out the arithmetic underneath all of it. I don’t run a fleet: everything below is roofline math on published model configurations and hardware specifications, checked against numbers other people measured, plus a careful read of this year’s literature.
Where the arithmetic and the papers disagree, we say so. If you operate AFD in production and something here is wrong, I would like to hear about it.
Two kinds of bytes
Take one decode step and follow the bytes through a layer. Some of them are shared. The weights of the QKV and output projections, and above all those of the feed-forward network or the experts, are the same for every request in the batch.
They are read once and used for every token in the step, so doubling the batch doubles the FLOPs and leaves those bytes where they were. This is why batching works at all, and why the critical batch from our earlier pieces, B* = P·b / (2·Bmem), with P the chip’s FLOPs, b the bytes per weight and Bmem its memory bandwidth, depends only on the chip and the number format.
The rest are private. The keys and values a request has accumulated belong to that request alone. To produce its next token, attention reads all of them, or, in the sparse variants, a selected subset plus an index. Double the batch and these bytes double too, so the number of FLOPs attention performs per byte it reads is fixed by the architecture, whatever the batch.
For grouped-query attention that number takes one line. Every cached token carries a key and a value per KV head. Each of the g query heads that share a KV head does a dot product with the key and a multiply-accumulate with the value: 4·d FLOPs per query head against 2·d·b bytes per KV head, where d is the head dimension and b the bytes per cached element. The intensity is 2g/b FLOPs per byte.
Qwen3-235B has 64 query heads on 4 KV heads, so g is 16, and with an FP8 cache its attention does 32 FLOPs per byte read. gpt-oss-120b and Qwen3-Next have g equal to 8, which gives 16. A BF16 cache halves both figures.
Now put the machines next to them. The ridge point of a chip is its dense FLOPs divided by its memory bandwidth, the intensity at which memory and math finish at the same time. In FP8 it is 412 FLOPs per byte for an H200, 563 for a B200 and 625 for one GPU of a GB200.
Grouped-query attention runs 13 to 39 times below the ridge of the machine executing it, and while a Blackwell GPU is doing it, its tensor cores have useful work for 3 to 6 percent of the time.
The one design near the ridge is DeepSeek’s multi-head latent attention. Once the up-projections are absorbed into the query and output sides, all 128 query heads share a single 576-wide latent row, so each cached row feeds 278,528 FLOPs.
FlashMLA stores that row in 656 bytes (512 FP8 values, four FP32 scales and 64 RoPE values kept in BF16) and does its math in BF16. That is 425 FLOPs per byte against a GB200 BF16 ridge of 312, so the kernel is compute bound, and DeepSeek reports 410 TFLOPS for it in a compute-bound configuration on the H800.
With a BF16 cache MLA does 242 FLOPs per byte, close to the H800’s BF16 ridge of 295, and it is probably not a coincidence that it came from a lab serving on H800s. StepFun went the other way on purpose: Step-3’s multi-matrix factorization attention, designed together with the disaggregated system that serves it, lands at 128 with an 8-bit cache.
The LP30, whose line sits near the left edge of the chart, is the chip inside NVIDIA’s Groq 3 LPX: 500 MB of SRAM at 150 TB/s, and 1.2 PFLOPS of FP8 if NVIDIA’s tray figure is split evenly across its 16 chips. Its ridge is 8, which is where grouped-query attention lives.
It’s the only machine on the chart whose balance matches attention, and the one with the least memory: a rack of 256 chips holds 128 GB.
The ratio
Disaggregation follows directly. If one kind of byte amortizes with batch and the other does not, put them on different machines and size each pool for its own bottleneck. The attention pool holds the caches and runs attention for many requests.
The FFN pool holds the expert weights and sees the tokens of all those requests at once, so its batch, and with it its arithmetic intensity, can be made as large as you need.
ByteDance called this disaggregated expert parallelism in MegaScale-Infer; StepFun called it attention-FFN disaggregation in the Step-3 paper, and that is the name most work uses now. The question every deployment then has to answer is how many attention devices to run per FFN device. In the throughput regime, where the FFN pool runs at its compute peak, the answer is a single equality.
Attention time per request per step is the private bytes it reads divided by the attention device’s bandwidth. FFN time per token is twice the active FFN parameters divided by the FFN device’s FLOPs. Make the two pools finish together and you get:
Here s is the private bytes read per token of context, L the context length, Pffn the FFN parameters a token touches, and R = FF / BA, which on a homogeneous cluster is just the ridge point. κ = s / (2·Pffn) belongs to the model alone: bytes of private state per token of context, per FLOP of FFN work. R belongs to the machine alone, and L to the workload.
The ratio grows linearly with context. For Qwen3-235B on GB200-class GPUs it is about 18 attention GPUs per FFN GPU at 8K tokens, 70 at 32K and 278 at 128K.
It does not depend on precision, as long as the cache and the math move together. Halve the bytes per cached element and double the FLOPs of the format, and κ halves while R doubles. This is the same invariance that pinned the critical batch in our piece on the GPU software gap, for the same reason.
The 1:1 point, where the two pools are the same size, is where κ·L·R equals one, so L = 2·Pffn·BA / (FF·s). That is the saturation context from August, written for a mixture-of-experts model with only the FFN parameters on the batching side.
We did not expect the two to be the same quantity, but they are: the context past which a colocated GPU cannot be compute bound at any batch is the context at which disaggregation needs as many attention devices as FFN devices. For Qwen3-235B on a GB200 GPU it is 472 tokens.
Counting the attention projections on the attention side, which do not grow with context, the pools are already equal at 228. Below that point attention is a minority of the work and AFD has little to rebalance; above it, the attention pool grows in proportion to context.
A formula this simple has to be checked against someone else’s measurement before it means anything. AFD-Ledger, a paper from Tsinghua University and Meituan published in August, re-examined FastAFD’s Qwen3-235B deployment on 72 GB200 GPUs at 8K context and re-provisioned it under a 40 ms per-token target.
Its best AFD configuration used 68 attention GPUs and 4 FFN GPUs, 17 to 1, at 25.0 ms per output token. The one-line formula, with the projections and the LM head counted on the attention side, gives 17.9. With their batch of 122 requests per attention GPU, our attention stage comes out at 24.5 ms and our FFN stage at 23.5 ms.
FastAFD’s own published configuration used 28 attention GPUs per 4 FFN GPUs, and AFD-Ledger’s reconstruction shows it was attention bound, which is what a ratio below the balanced one should produce.
Tables 2 and 3 run four widely deployed open models through the same accounting.
DeepSeek-V3.2’s sparse attention moves κ by a factor of 20 relative to Qwen3-235B. After the lightning indexer picks 2,048 rows per query, the only read that still grows with context is a 132-byte index key per token per layer, so at 128K context its ratio is 15.6, not 278.
Sparse attention also pulls capacity and bandwidth apart: the cache still stores 48.1 KB per token of context, but each step reads only 8.1 KB per token plus a fixed 82 MB for the selected rows.
A design-space study from Georgia Tech, Intel and Google found the empirical version of this. On its agentic workload on 128 B200s, it gave DeepSeek-V3.2 an attention slice of 2 GPUs and Qwen3-235B a slice of 8.
Qwen3-Next is the opposite case. Its hybrid design keeps a growing cache in only 12 of its 48 layers, which cuts its KV per token by a factor of 7.8 relative to Qwen3-235B.
Its FFN work per token is 8.5 times smaller too, so κ ends up about 9 percent higher than Qwen3-235B’s. At 8K its ratio in Table 3 is still about twice Qwen3-235B’s, because of a recurrent state it reads in full at every step, which we come back to below.
Two operating points
κ·L·R is the ratio that keeps the FFN pool at its compute peak, and at long context nobody can pay for it. At 278 attention GPUs per FFN GPU, the FFN GPU is a rounding error on the bill and almost all of it is HBM. In practice the FFN pool is sized by something else, and what that is depends on the speed being sold.
At the throughput end, where most of the 2025 AFD work lived, the per-token deadline is generous. MegaScale-Infer, ByteDance’s system, was evaluated against a 150 ms time per output token, about 7 tokens per second per user.
With that budget the micro-batches can be large, the ping-pong pipeline hides the network behind computation, and the FFN pool can be pushed toward its peak. At this end AFD is a utilization tool: each expert GPU sees the tokens of many attention GPUs, so its batch can grow to the size the GPU needs.
At the interactivity end the deadline is the product. NVIDIA makes its claim of up to 35 times the throughput per megawatt for Vera Rubin with LPX over GB200 NVL72 at 400 tokens per second per user, for a 2-trillion-parameter MoE with 400K tokens of input context, and its launch post describes generation speeds approaching 1,000.
At those speeds the batch per step is far too small to make the FFN compute bound, so FFN time is set by how fast the expert weights can be streamed.
DeepSeek-V3.2’s FFN weights take 658 GB in FP8, and with uniform routing a hundred tokens per step already touch 96 percent of the 256 routed experts in each layer, so every step reads nearly all of them. Reading all of them 400 times a second takes 263 TB/s: the HBM bandwidth of 33 GB200 GPUs, or the SRAM bandwidth of 1.8 LP30 chips.
At 1,000 tokens per second it is 82 GB200s or 4.4 LP30s. On SRAM the bandwidth problem nearly disappears. The capacity problem does not: holding those weights takes 1,315 LP30s, a little over five LPX racks, at 8 bits per weight.
So NVIDIA puts the FFN on the LPU and attention on the GPU, which is the opposite of what the first section suggests. At the interactivity end both halves of the layer are bandwidth bound, and what differs is what the bandwidth reads.
The FFN streams a fixed set of shared weights, a few hundred gigabytes that fit in a handful of SRAM racks if you accept the price per gigabyte. Attention streams private state that grows with every user and every token, and only HBM can hold that.
What decides the placement is that the FFN’s bytes are finite, not that the FFN is heavy on compute. Our LPX piece reached the same place from the hardware side, where SRAM came out 914 times cheaper than HBM per TB/s and 4.3 times more expensive per GB.
The wire
Splitting a layer in two puts a network inside it. Every layer that crosses sends each token’s hidden state from the attention side to the FFN side and brings the result back.
For DeepSeek-V3.2 that is 7,168 bytes out in FP8 and 14,336 bytes back in BF16 per token per layer, or eight times the outbound figure if the attention side sends each token straight to the GPUs that hold its eight experts. These are small messages, but there are two per layer, 122 per token for a 61-layer model, and every one of them is on that token’s critical path.
At a given speed target, each layer gets one second divided by the product of tokens per second per user and layers, and that time has to cover attention, the FFN and both crossings. For a 61-layer model at 400 tokens per second it is 41 microseconds. At 1,000 it is 16.4. For Qwen3-235B’s 94 layers at 1,000 it is 10.6.
NVIDIA’s long-context figure for LPX, a median of 3,431 tokens per second on Gemma 4 31B with 100K tokens of input context as measured by Artificial Analysis, comes from a dense model whose weights fit in one rack’s SRAM.
For calibration, here is what a crossing costs on an ordinary scale-out fabric. DeepSeek’s open DeepEP low-latency kernels report 77 µs for a dispatch and 114 µs for a combine at expert-parallel degree 8, with 128 tokens per GPU on H800s over 400 Gb/s InfiniBand. At degree 256 the pair costs 554 µs.
If a token had to wait for one such round trip per layer, a 61-layer model would top out near 86 tokens per second per user at degree 8, and near 30 at degree 128 and above. DeepSeek reported an average of 20 to 22 tokens per second per user for its own service, with decode spread over 18 nodes at EP144, which is consistent with per-token latency set by the fabric.
A February paper extended the roofline down to the communication level and found a dead zone on standard clusters, where adding FFN instances no longer raises hardware FLOPS utilization because scale-out bandwidth caps the work each instance can be fed.
Its authors also find that AFD’s discrete, node-level scaling costs more in load imbalance than expert parallelism’s continuous batch adjustment, and they conclude that AFD suits specific combinations of hardware and model, superpod-class interconnects and coarse-grained, less sparse experts among them, rather than serving as a general solution.
The Georgia Tech study handles the same problem by placement: the per-layer exchange, the most frequent traffic, goes on the NVLink domain inside a node, and prefill-to-decode KV transfers, which happen once per request, are deferred to InfiniBand between nodes.
NVIDIA’s description of LPX attacks the problem from the other side. On the LPU, the compiler schedules computation, data movement and synchronization ahead of time, and the chip-to-chip links are built for deterministic exchange, so the timing of a transfer is known before the workload runs.
One part of the path is not scheduled that way. According to SemiAnalysis, FPGAs in the LPX system convert the LPU’s chip-to-chip protocol to Ethernet so the LPUs can reach the GPUs over the Spectrum-X scale-out fabric. We have not seen a published latency for that hop.
NVIDIA’s August description of AFD on Vera Rubin has the NVL72 computing attention and holding the KV cache while Groq 3 LPX executes the FFN layers, and then adds: “Only intermediate tokens are sent between racks, once per full-attention layer.”
Once per full-attention layer, not once per layer. The post describes its model as a scaled-up version of gpt-oss-120b, in which half the layers use a 128-token sliding window.
If the 2-trillion-parameter version keeps that pattern, the most natural reading is that the sliding-window layers, whose cache never grows, stay on the LPU next to the FFN, and only the layers with a growing cache send activations to the GPU. That would halve the crossings and double the time available for each.
What the papers measured
The first results were clean wins. MegaScale-Infer reported up to 1.9 times the per-GPU throughput of state-of-the-art systems on homogeneous clusters, and AFD-Ledger reads its heterogeneous experiment, with H20s running attention and L40S cards running the experts, as 1.57 times a colocated H20 baseline.
FastAFD, on GB200 NVL72, reported 1.41 times the tokens per second per GPU of expert parallelism at degree 4 on Qwen3-235B.
Then AFD-Ledger asked a different question. With the model, workload, latency target, budget and hardware catalog held fixed, does the best AFD deployment beat the best colocated one? Its answer, validated against three physical LongCat 2.0 deployments on Ascend 910C SuperPods to within 6.6 to 9.6 percent, is mostly no.
Re-provisioned with both sides free to optimize, FastAFD’s 1.41 becomes 0.99, because the colocated side widens expert parallelism to 72 and catches up. MegaScale-Infer’s heterogeneous win becomes 0.95 once the colocated side may pick L40S cards too.
Across 36 homogeneous settings AFD wins 7, all on Qwen3-235B and none on DeepSeek-V3.2. The example we keep coming back to: on 72 H200s, AFD lifts FFN utilization from 27.9 to 84.3 percent and the deployment still serves 0.71 times as many tokens, because every GPU moved into the FFN pool stops holding requests. The authors call this the request-bearing-capacity tax.
The vLLM project’s AFD plugin, released in July as an experimental external plugin, found the same sensitivity on real hardware.
On DeepSeek-V3.2 on Ascend 910C, with expert routing forced into balance for the benchmark, 48 attention dies feeding 16 FFN dies were 5 to 10 percent slower per die than plain expert parallelism at 16K and 32K context, while 64 feeding 16 were 9 to 11 percent faster, with the model, hardware and plugin unchanged.
The authors read the trend as a sign that the FFN dies still had compute headroom at 64:16, which is what you would expect if the FFN pool was still short of its compute-bound batch.
The plugin also still loads the full weights on both roles, so the memory benefit has not reached the open stack yet.
The design-space study, previously cited, from Georgia Tech, Intel and Google draws the line that reconciles all of this.
On a 128-GPU B200 cluster across chat, coding and agentic workloads, aggregated serving with chunked prefill wins the throughput frontier on most panels, and AFD wins the latency end on every panel. Under strict latency targets on DeepSeek-V3.2, AFD sustained around 4,000 tokens per second of system throughput where no non-AFD deployment was feasible at all.
AFD also wins whenever a model and its cache stop fitting on one GPU: with a one-million-token prefix, the colocated layout needed about 298 GiB per GPU against a B200’s 180, and AFD brought that to about 165 by separating weights from caches.
Read this way, the results stop contradicting each other. AFD loses or ties when the goal is throughput under a relaxed deadline, and wins where the speed target or the memory footprint puts a colocated layout out of reach.
Where the cut goes
Everything so far treats the cut as attention on one side and the FFN on the other. The arithmetic suggests a different rule, which that cut approximates: separate the bytes that are shared from the bytes that are private, and among the private ones, separate those that grow with context from those that do not.
Attention-FFN disaggregation gets the first half mostly right. FFN and expert weights are shared, and moving them off the attention GPUs gives those GPUs their whole memory for caches, which is where AFD’s memory benefit comes from.
But the attention side also carries shared weights: the QKV and output projections and the LM head. In Qwen3-235B that is 7.3 billion parameters, next to the 14.2 billion FFN parameters a token touches; in DeepSeek-V3.2, 13.2 billion next to 24.2. Those matrix multiplies batch exactly like the FFN does. They sit on the attention side only because the data dependency ties them to the cache.
The second half is what the sparse and hybrid architectures of the past year change, and it is where a September paper moves the line. Tschand and colleagues argue that heterogeneous systems should split decode by quadratic versus subquadratic attention rather than by operator type, which in memory terms is the line between state that grows with context and state that does not.
In sparse-attention models the top-k selection has to index through the full cache, so it goes where the capacity is, while the top-k attention and the FFN have static memory footprints and can travel together.
In models with linear or sliding-window layers, the dense attention layers go one way and the subquadratic layers go with the FFN. In an analytical model of a Rubin plus LPX system under fixed power budgets, they report 1.2 to 1.5 times tighter achievable latency and up to 3.6 times the throughput of the best attention-FFN split, and, on an adjusted 8-GPU B200 proxy for the heterogeneous system, 31 to 56 percent more tokens per joule than the strongest GPU-only baselines. NVIDIA’s design for its scaled-up gpt-oss, as we read it, makes the same cut.
Bounded does not mean small, though. Qwen3-Next keeps 36 Gated DeltaNet states per request, each 32 heads of 128 by 128, which is 37.7 MB in BF16 plus 1.8 MB of convolution state: as much memory as 3,216 tokens of its own full-attention cache, read and rewritten at every step. A sliding-window layer is cheaper.
All of gpt-oss’s window layers together hold 2.4 MB per request in FP8, the cache of 128 tokens. On an LPX rack, after Qwen3-Next’s FP8 weights, the remaining SRAM would hold the state of about 1,200 concurrent requests. Bounded state can live on the SRAM side, and it is paid for by the gigabyte, like the weights.
Who pays for which pool
If the attention pool is mostly memory and the FFN pool is mostly weights, the economics of AFD come down to two prices: what a terabyte per second of HBM costs, and what a gigabyte of fast memory costs.
AFD-Ledger’s hardware catalog, built from RunPod’s hourly prices and normalized to an H200, makes the first one visible.
On that price list a B200 rents for 1.64 H200s. Per FP8 FLOP it is 28 percent cheaper than the H200. Per terabyte per second of HBM it costs 0.98 of an H200, the same within rounding, and per gigabyte of HBM it is 29 percent more expensive.
The B300 brings the gigabyte back to parity and makes the terabyte per second 20 percent dearer. The cheapest bandwidth and the cheapest capacity on the whole list are on A100s.
For a pool whose job is to hold caches and read them, two generations of NVIDIA silicon have not made the job cheaper on this list. What they have made cheaper is FLOPs, and attention at 16 to 32 FLOPs per byte uses very few of them.
This may also be why AFD-Ledger’s heterogeneous winners look backwards at first sight. Its best DeepSeek-V3.2 deployment put attention on H200s and the experts on A100 PCIe cards, for 1.82 times the best colocated throughput; for Qwen3-235B, H200 attention with L40S experts did the same.
Our reading is that under the latency targets behind those numbers, the expert pool needed cheap capacity and bandwidth to stream weights, not FLOPs. The same H200 and A100 pair ranged from 0.48 to 1.82 times the colocated baseline as budget and latency target moved, so a pairing that wins in one setting can lose badly in the next.
NVIDIA’s own choices point the same way. In September 2025 it announced Rubin CPX, a GDDR7 part to speed up prefill. At GTC in March 2026 it was missing from the keynote and the roadmap, in May a Taiwanese supply-chain report said the industry suspects it has been effectively cancelled, and NVIDIA has given no reason.
What the platform gained instead is a decode accelerator for the stateless half of the layer, built on technology NVIDIA licensed from Groq for a reported $20 billion, in a deal that also brought Groq’s team to NVIDIA. Prefill, increasingly served from prefix caches in agentic workloads, can stay on the general-purpose GPU.
Whether that pays depends on the price of speed, and that price has moved fast. When Anthropic introduced fast mode for Claude Opus 4.6 in February, it cost six times the standard rate, $150 per million output tokens, for about 2.5 times the output speed. By May, on Opus 4.8, the multiple was two.
Claude Opus 5.5’s fast mode, now generally available, lists $40 per million output tokens against $20 at standard speed, still two times, for up to 2.5 times the output tokens per second. OpenAI renamed its priority tier Fast mode on July 30 and promises up to 2.5 times the speed on its latest frontier model, at a premium over standard rates.
At GTC, Jensen Huang sketched a token market in tiers from free to an Ultra Fast tier near $150 per million, with a Fast tier near $45. If those two tiers were sold at 400 and 1,000 tokens per second, one stream would bring in $64.80 and $540 an hour.
The condition for a fast tier to make sense is simple. If speed sells at a multiple m of the standard price, the fast operating point has to keep at least 1/m of the standard point’s throughput per GPU.
At today’s posted multiple of 2, a provider can give up half its throughput per GPU to deliver 2.5 times the speed and still earn the same per GPU-hour. That is the trade the papers say AFD makes well: it loses or ties on raw throughput and wins at the interactive end.
The posted prices even sit inside NVIDIA’s range, since Anthropic’s fast mode launched at the price of the Ultra Fast tier and now sits just under the Fast one. But the speed on offer is up to 2.5 times standard, and its price has fallen from $150 to $40 per million output tokens since February.
NVIDIA’s case for LPX assumes buyers who will pay a premium for 400 to 1,000 tokens per second per user. Whether those buyers exist in volume is, as we see it, the main commercial question for LPX.
Where we might be wrong
Our model is a roofline. It ignores kernel inefficiency, scheduler overhead and network contention. In AFD-Ledger’s robustness case studies, about 10 percent of added runtime overhead left the decisions intact and about 20 percent flipped them to colocated serving, and our ratios share that blind spot.
We count the attention-side projections at FP8 peak, the large-batch limit. At 32 to 128 requests per attention GPU, re-reading those weights every step costs more than the math: 7 to 29 µs per request for Qwen3-235B and 13 to 52 µs for DeepSeek-V3.2 on a GB200 GPU. That makes the attention side heavier and the balanced ratios higher than our curves.
Our FFN streaming figures assume uniform routing. Skewed routing touches fewer experts per step and needs less bandwidth, while replicating hot experts for load balance needs more capacity.
DeepSeek-V3.2’s position depends on its kernels: BF16 math for FlashMLA and FP8 for the indexer, as published. A kernel that ran MLA in FP8 would move it relative to the ridge.
We assumed Gated DeltaNet state in BF16. Serving stacks that keep it in FP32 double those numbers.
The price list is one marketplace at one point in time. Contract and hyperscaler prices differ, and the flat price of bandwidth could be specific to that list.
The GPU-to-LPU path comes from SemiAnalysis, and the interpretation of NVIDIA’s crossing sentence is ours. It depends on the 2T model keeping gpt-oss-120b’s alternating windows, which NVIDIA does not say.
Rubin CPX’s cancellation is suspected, not confirmed. The keynote tiers carry no speeds, so pairing them with 400 and 1,000 tokens per second is our assumption, and most of the posted fast-mode prices come from one vendor.
Predictions
By the end of 2027, vLLM or SGLang will ship attention-FFN disaggregation in its main tree, with independently sized pools and without full weights on both roles.
By the end of 2027, someone will publicly report a production AFD deployment running ten or more attention devices per FFN device.
Through 2027, the fast-tier premium at Anthropic, OpenAI and Google will stay at or below three times the standard price, for speedups of up to about 2.5 times.
NVIDIA will not bring back a prefill-specialized part in the Rubin generation.
By the end of 2027, at least two major open-weight model reports will state an attention intensity target or an attention-to-FFN provisioning ratio as a design goal, as Step-3 did.
Confidence dossier
Sources
NVIDIA, Inside NVIDIA Groq 3 LPX: The Low-Latency Inference Accelerator for the NVIDIA Vera Rubin Platform, March 16, 2026, updated October 13, 2026.
NVIDIA, How NVIDIA Groq 3 LPX Unlocks Ultrafast Interactivity at Long Context on NVIDIA Vera Rubin, August 24, 2026.
SemiAnalysis, Nvidia: The Inference Kingdom Expands, March 24, 2026.
C. Qiu et al., AFD-Ledger: Deployment Provisioning for Attention-FFN Disaggregation, arXiv:2608.04502, August 2026.
H. Wu et al., How Far Can Disaggregation Go? A Design-Space Exploration of Attention-FFN Disaggregation for Efficient MoE LLM Serving, arXiv:2605.28302, May 2026.
G. Liu et al., Revealing the Challenges of Attention-FFN Disaggregation for Modern MoE Models and Hardware Systems, arXiv:2602.09721, February 2026.
A. Tschand et al., Rethinking Heterogeneous System Disaggregation for Subquadratic Attention, arXiv:2609.13134, September 2026.
R. Zhu et al., MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism, arXiv:2504.02263, SIGCOMM 2025.
StepFun, Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding, arXiv:2507.19427, July 2025.
vLLM project, Announcing vLLM AFD Plugin: Disaggregating Attention and FFN for Flexible MoE Serving, July 23, 2026.
DeepSeek, FlashMLA, README sections on sparse decoding and the FP8 KV cache format.
DeepSeek, DeepEP, README section on low-latency kernel performance.
DeepSeek, Day 6 inference system overview, March 2025, as summarized by Analytics Vidhya.
Model configurations: Qwen3-235B-A22B, gpt-oss-120b, DeepSeek-V3.2, Qwen3-Next-80B-A3B.
Tom’s Hardware, Nvidia removes Rubin CPX accelerators from its roadmap, March 17, 2026; TechNews on the CPX supply chain, May 28, 2026.
ASCII.jp, GTC 2026 analysis of Groq 3 and Rubin CPX, April 13, 2026.
Anthropic, Fast mode and Pricing documentation, read November 7, 2026; Simon Willison, note on Claude fast mode pricing, February 7, 2026; Opus 4.8 fast mode pricing, May 2026.
OpenAI, Fast mode for API customers.
eWeek, Everything Nvidia announced at GTC 2026; keynote transcript (ChainCatcher).
Our earlier pieces, How the Groq 3 LPX Actually Works and How the GPU Software Gap Actually Works (both August 2026), for the saturation context and the precision invariance of the critical batch.
By your Qwen3-235B numbers, requests at 32K context need about 70 attention GPUs per FFN GPU against about 18 at 8K. So a pool provisioned on last month's average context goes attention bound as soon as long agent sessions become a bigger share of traffic. Routing requests to separate pools by context bucket seems simpler than re-provisioning one fixed ratio.
One wrinkle in the shared/private split: KV for a common prefix isn't private. Shared-prefix kernels like Hydragen or FlashInfer's cascade attention read it once per batch, so it batches like weights. With agentic traffic carrying long shared system prompts, does that pull effective κ down enough to move the 1:1 context point?