Discussion about this post

User's avatar
Abhishek Pan's avatar

The 0.43 ms plus 13.6 µs per request is the number that stuck with me. At 256 requests that's roughly 3.9 ms of Python scheduler per step, already longer than a Rubin step for that model. Is the scheduler next in line for Rust, or is the near-term fix just running more engines with smaller batches each?

Kanishk Patel's avatar

The 1 / (1 - h) line is the one I'll remember: a 95.7% cache hit saves the GPU's prefill but none of the host's template, tokenizer and block-hash work, so agent traffic makes the host's share bigger every turn. Since an agent loop resends a transcript that's mostly identical to the previous one, I keep wondering why the frontend doesn't cache token IDs per conversation the way the engine caches KV. The catch I'd expect is chat templates that rewrite earlier turns, like dropping old reasoning blocks, so the rendered prefix isn't stable. Did you check whether Qwen3's template output stays prefix-stable across turns?

1 more comment...

Ready for more?