At batch 256, a B200 serving an 8B model leaves the host about 28 microseconds per generated token. On our test core, vLLM’s Python frontend spent 75 and its new Rust frontend 24.
The 0.43 ms plus 13.6 µs per request is the number that stuck with me. At 256 requests that's roughly 3.9 ms of Python scheduler per step, already longer than a Rubin step for that model. Is the scheduler next in line for Rust, or is the near-term fix just running more engines with smaller batches each?
The 1 / (1 - h) line is the one I'll remember: a 95.7% cache hit saves the GPU's prefill but none of the host's template, tokenizer and block-hash work, so agent traffic makes the host's share bigger every turn. Since an agent loop resends a transcript that's mostly identical to the previous one, I keep wondering why the frontend doesn't cache token IDs per conversation the way the engine caches KV. The catch I'd expect is chat templates that rewrite earlier turns, like dropping old reasoning blocks, so the rendered prefix isn't stable. Did you check whether Qwen3's template output stays prefix-stable across turns?
"As GPUs get faster, the frontend has become a real share of CPU time" is the most polite way anyone has ever said the host code is now the bottleneck. Swapping the GPU for a timer is a great trick, because it's the only way to see the host tax nobody budgets for. Even if most of that 37.2 ms vs 2.63 ms gap is really a tokenizer swap, that's the number agent workloads with giant, prefix-heavy prompts actually feel all day. Is the engine core scheduler next in line for the Rust treatment, or does free-threaded Python buy it a reprieve?
The 0.43 ms plus 13.6 µs per request is the number that stuck with me. At 256 requests that's roughly 3.9 ms of Python scheduler per step, already longer than a Rubin step for that model. Is the scheduler next in line for Rust, or is the near-term fix just running more engines with smaller batches each?
The 1 / (1 - h) line is the one I'll remember: a 95.7% cache hit saves the GPU's prefill but none of the host's template, tokenizer and block-hash work, so agent traffic makes the host's share bigger every turn. Since an agent loop resends a transcript that's mostly identical to the previous one, I keep wondering why the frontend doesn't cache token IDs per conversation the way the engine caches KV. The catch I'd expect is chat templates that rewrite earlier turns, like dropping old reasoning blocks, so the rendered prefix isn't stable. Did you check whether Qwen3's template output stays prefix-stable across turns?
"As GPUs get faster, the frontend has become a real share of CPU time" is the most polite way anyone has ever said the host code is now the bottleneck. Swapping the GPU for a timer is a great trick, because it's the only way to see the host tax nobody budgets for. Even if most of that 37.2 ms vs 2.63 ms gap is really a tokenizer swap, that's the number agent workloads with giant, prefix-heavy prompts actually feel all day. Is the engine core scheduler next in line for the Rust treatment, or does free-threaded Python buy it a reprieve?