Grouped-query attention does 16 to 32 FLOPs for every byte it reads. A Blackwell GPU needs more than 550 to stay busy. Splitting the transformer layer in two is the industry’s answer.
By your Qwen3-235B numbers, requests at 32K context need about 70 attention GPUs per FFN GPU against about 18 at 8K. So a pool provisioned on last month's average context goes attention bound as soon as long agent sessions become a bigger share of traffic. Routing requests to separate pools by context bucket seems simpler than re-provisioning one fixed ratio.
Thanks Abhishek, well….that’s a real risk, and the numbers back you up. The linearity does give you one small thing, though. What sets the ratio is the mean context of the requests in flight, because in general attention demand adds up linearly across requests and the FFN doesn't care about context at all. A( let’s call it a mix ) 50/50 mix of 8K and 32K needs about 44 attention GPUs per FFN GPU whether you run it as one pool or as two pools at 18 and 70. Buckets keep each pool's ratio stable, but when agent traffic grows you still have to move GPUs, between pools instead of within one.
What I'd try first is bucketing inside one pool. Keep a single FFN pool, since its work per token is the same at any context, and group requests onto attention instances by context. Then balance those instances by KV bytes read per step rather than by request count: at AFD-Ledger's 8K setup that's 122 requests per attention GPU, so roughly 30 at 32K. In a synchronized step the slowest attention instance sets the pace for everyone, which is why the balance has to be in bytes. When the mix shifts you add attention instances, which are the small unit, and leave the experts alone.
Where I would split pools is by speed tier, since the FFN is sized by compute at a relaxed deadline and by weight streaming at 400 tokens per second. On an SRAM machine an extra copy of the experts is also the expensive part, about 1.8 LPX racks for Qwen3-235B in FP8. I haven't run nor I have the possibility to do at scale, so if you've seen bucketed pools hold up better in practice I'd like to know how you sized the buckets.
One wrinkle in the shared/private split: KV for a common prefix isn't private. Shared-prefix kernels like Hydragen or FlashInfer's cascade attention read it once per batch, so it batches like weights. With agentic traffic carrying long shared system prompts, does that pull effective κ down enough to move the 1:1 context point?
By your Qwen3-235B numbers, requests at 32K context need about 70 attention GPUs per FFN GPU against about 18 at 8K. So a pool provisioned on last month's average context goes attention bound as soon as long agent sessions become a bigger share of traffic. Routing requests to separate pools by context bucket seems simpler than re-provisioning one fixed ratio.
Thanks Abhishek, well….that’s a real risk, and the numbers back you up. The linearity does give you one small thing, though. What sets the ratio is the mean context of the requests in flight, because in general attention demand adds up linearly across requests and the FFN doesn't care about context at all. A( let’s call it a mix ) 50/50 mix of 8K and 32K needs about 44 attention GPUs per FFN GPU whether you run it as one pool or as two pools at 18 and 70. Buckets keep each pool's ratio stable, but when agent traffic grows you still have to move GPUs, between pools instead of within one.
What I'd try first is bucketing inside one pool. Keep a single FFN pool, since its work per token is the same at any context, and group requests onto attention instances by context. Then balance those instances by KV bytes read per step rather than by request count: at AFD-Ledger's 8K setup that's 122 requests per attention GPU, so roughly 30 at 32K. In a synchronized step the slowest attention instance sets the pace for everyone, which is why the balance has to be in bytes. When the mix shifts you add attention instances, which are the small unit, and leave the experts alone.
Where I would split pools is by speed tier, since the FFN is sized by compute at a relaxed deadline and by weight streaming at 400 tokens per second. On an SRAM machine an extra copy of the experts is also the expensive part, about 1.8 LPX racks for Qwen3-235B in FP8. I haven't run nor I have the possibility to do at scale, so if you've seen bucketed pools hold up better in practice I'd like to know how you sized the buckets.
One wrinkle in the shared/private split: KV for a common prefix isn't private. Shared-prefix kernels like Hydragen or FlashInfer's cascade attention read it once per batch, so it batches like weights. With agentic traffic carrying long shared system prompts, does that pull effective κ down enough to move the 1:1 context point?