Introduction
In May, vLLM merged a frontend written in Rust. It is opt-in, it talks to the same Python engine through the same ZMQ socket as before, and the announcement contained one sentence that says more than the benchmark attached to it: “As GPUs get faster, the frontend has become a real share of CPU time.”
That sentence describes a budget. A GPU emits tokens at some rate, and each one passes through host code on its way out: an incremental detokenizer, a stop-string check, a JSON chunk, an HTTP write.
Each prompt passes through host code on its way in: a chat template, a tokenizer, and in vLLM a hash for every 16-token block so the prefix cache can find it. When the host needs more time per token than the GPU, the GPU waits. When the GPU gets faster, the host’s share grows, even if no line of host code changed.
I really wanted to see how big that budget is on current hardware and how much of it Python actually uses. So we ran stock vLLM 0.33 on a machine with no GPU, then we replaced the GPU with a timer, and measured everything above it, once with the Python frontend and once with the Rust one.
Streaming a token cost the Python frontend 75 µs of CPU on our core and the Rust frontend 24. Before the first token, a 10,693-token prompt cost the Python frontend 37.2 ms of CPU and the Rust frontend 2.63 ms, but as you may have imagined, most of that second gap has little to do with Python: it comes from replacing one Rust tokenizer with a faster Rust tokenizer.
What stays in Python after the frontend moves, the engine core’s scheduler and block bookkeeping, costs 0.43 ms plus 13.6 µs per request on every decode step. On a Rubin-class GPU serving an 8B model at full batch, that is longer than the step it has to hide behind.
P.S. If you haven’t done it yet, please take a look at our brand new LLM Systems Engineering guide, now available on gumroad.
The budget
Decode on a modern GPU is mostly memory traffic. Every step reads the weights once and each sequence’s KV cache once, so the GPU time behind one generated token, at batch size B and average context L, is close to
t_token = (W / B + L x k) / (eta x BW)where W is the bytes of weights read per step, k the bytes of KV per token of context, BW the HBM bandwidth and eta the fraction of it a real decode kernel achieves. That t_token is the host’s budget. If the host spends more than t_token per token on one core, one core is not enough.
Take Qwen3-8B with FP8 weights and an FP8 KV cache. Each step reads about 7.57 GB (6.95 billion non-embedding parameters plus the 0.62 billion of the output projection), and each token of context holds 73,728 bytes of KV (36 layers, keys and values, 8 KV heads of 128 values).
At 256 sequences of 2,048 tokens and eta = 0.8, a B200 spends 4.6 µs per token on weights and 23.6 µs on KV, 28.2 µs in total. The same arithmetic gives 67.4 µs on an H100, 47.0 on an H200, and 11.8 on Rubin at its currently listed 19.2 TB/s (Figure 1). NVIDIA’s product page listed Rubin at 22 TB/s until recently.

Two things make that number fall, and neither involves the host. HBM bandwidth grows about 2.4 times per generation: 3.35 TB/s on H100, 8 on B200, 19.2 on Rubin.
And the KV term, which is most of the budget, shrinks with every attention change the field has adopted: grouped-query attention, FP8 KV caches, DeepSeek’s latent attention (576 values per layer per token against 2,048 for Qwen3-8B’s eight KV heads), sparse attention. All of it makes the GPU’s time per token smaller and leaves the host’s time per token where it was.
The budget is per core, and a Python API server is roughly one core: the event loop, the HTTP server, the detokenizer and the response serialization all take turns on one interpreter lock. vLLM’s answer was to run several API server processes in front of the same engine.
The RFC that proposed the Rust frontend measured where that runs out, in a setup its authors call unrealistic and chose to show the ceiling of the Python implementation: four GB200s serving Qwen3-0.6B at 1,024 concurrent requests.
On chat prompts of about 10K tokens with a warm prefix cache, one Rust frontend process handled 837 requests per second, four Python processes 162, and thirty-two Python processes 786.
The problem is older than the fix. When SGLang shipped its overlap scheduler in December 2024, the blog post announcing it said an unoptimized engine can spend as much as half of its time on CPU overhead.
Overlap, preparing the next batch on the CPU while the GPU runs the current one, hides host time only as long as the host is shorter than the GPU step.
A GPU that is a timer
vLLM lets you plug in your own executor with --distributed-executor-backend, which accepts a class path. Ours emulates a GPU stream in about 150 lines of Python.
Each step occupies the “GPU” for a fixed time, then returns, for every request that finished its prefill, the next token of a pre-tokenized real text (docstrings and source from the CPython standard library), so the detokenizer sees real text.
It supports async scheduling, so the engine prepares step N+1 while step N “runs”. Everything above it is stock vLLM 0.33: the API server, the renderer, the tokenizer and detokenizer, the output processor, the engine core, the scheduler and the KV block manager.
Setting VLLM_USE_RUST_FRONTEND=1 swaps the Python API server for vllm-rs, a 61 MB binary that ships inside the vLLM wheel.
The model is a local stand-in for Qwen3-8B: the real architecture values, and the Qwen BPE vocabulary rebuilt from llama.cpp’s GGUF file and checked against llama.cpp’s 47 tokenizer test vectors, all of which it reproduces.
The host is two vCPUs of an Intel Xeon at 2.1 GHz under KVM, CPU family 6 model 207, which is Emerald Rapids. That’s the microarchitecture of the Xeon Platinum 8570 in a DGX B200, which boosts to 4.0 GHz.
We pinned the frontend to one vCPU, put the engine core and the load generator (vLLM’s own Rust benchmark client) on the other, and read each process’s CPU time from /proc before and after every run.
At 20 ms per step, the two frontends deliver about the same throughput, 8.2K and 8.4K tokens per second, with the same average batch of 194 requests. The distance to the grey line in Figure 2 comes from the workload: each run drains at the end, as the last requests of uneven length finish one by one.
From 10 ms down, the Python frontend sits at 92 to 99 percent of its core and stops following: 12.3K tokens per second at 10 ms, 16.5K at 5, 19.5K at 2.5. New requests wait in the frontend, so median time to first token climbs past one second and the engine’s average batch falls to 175, 143 and 97.
The Rust frontend reaches 16.2K, 26.8K and 32.5K with batches between 217 and 236, its median time to first token stays between 25 and 31 ms, and it never uses more than 57 percent of its core (Figure 2). At 2.5 ms what holds the Rust run back is the engine core, which is Python and shares its core with the load generator.

Streaming
Per streamed token, the Python frontend used 51 to 84 µs of CPU across the four step times, and the Rust frontend 18 to 26 (Figure 3). At the 10 ms point the figures are 75.0 and 24.2. Both include the kernel’s loopback TCP work for one server-sent event per chunk, which no language avoids.

vLLM has a knob that shows how much of this is per chunk rather than per token: --stream-interval, which coalesces several tokens into one event.
At four tokens per chunk the Python frontend dropped to 35.4 µs per token and the Rust frontend to 11.1. Fitting cost = a + b/k to the two points gives 22.2 µs per token plus 52.8 µs per chunk for Python, and 6.7 plus 17.5 for Rust.
Most of the cost is per chunk in both, and Python costs about three times as much in both terms. With four-token chunks, the Python frontend at the 10 ms step carried 16.0K tokens per second, about what the Rust frontend did, at 57 percent of its core and with a median time to first token of 47 ms instead of 1.7 s.
If your clients can live with chunkier streams, this is the cheapest fix available today.
To see what the language itself costs, we wrote the core of the streaming path twice, outside any server: an incremental detokenizer step, a stop-string check, an OpenAI chat-completion chunk, the SSE framing. 256 streams of 512 tokens, the same tokenizer.json, and byte-identical output, 29,384,583 bytes of SSE in both.
Python took 11.1 µs per token and Rust 0.86. Inside the real frontends the gap shrinks to three times, because most of the Rust frontend’s 24 µs is spent outside the per-token logic we rewrote: in the kernel’s socket calls, the HTTP stack, the async runtime and the link to the engine.
Where the microseconds go
We sampled the Python API server with py-spy during the 10 ms run (Figure 4). 43 percent of the samples landed in building and serializing response objects, in vLLM’s serving code and in pydantic.
20 percent landed in the HTTP server and the ASGI stack, 17 percent in output processing and detokenization, 11 percent in asyncio and threading, and 9 percent elsewhere.

The detokenizer is what people usually optimize first, and vLLM did that a while ago: since tokenizers 0.21.1 its fast incremental detokenizer is a thin wrapper around the DecodeStream state machine, which is Rust.
We also ran the Python frontend with VLLM_USE_FASTOKENS=1, which swaps in a faster Rust tokenizer, and the cost per streamed token went from 75.0 to 74.0 µs. When the cost is spread across every layer a token crosses, moving one function into Rust does not move the total.
The Rust frontend replaces the layers instead: same engine, same ZMQ boundary, and above it an axum HTTP server, a template renderer, a tokenizer and an incremental detokenizer in one process, with threads that do not share a lock.
The same boundary shows up elsewhere this year. The model gateway that started in SGLang, now called Shepherd Model Gateway, talks to engines over gRPC with preprocessed tokens in and generated tokens out, and its authors write that tokenization and detokenization had become production bottlenecks in both SGLang and vLLM.
NVIDIA’s Dynamo tokenizes in a Rust preprocessor by default and sends token IDs to the engine workers behind it; it needs the IDs there anyway, to compute the prefix hashes its KV-aware router matches on. None of this layout is new.
Hugging Face’s TGI put a Rust router in front of Python model servers from 2022, and it has been in maintenance mode since December 2025, while the engines that replaced it adopt the same split.
The tokenizer
Before the first token, the host renders the chat template over the whole conversation and tokenizes all of it, on every request, because the API is stateless.
We sent 384 unique prompts averaging 10,693 tokens, tool transcripts built from CPython source, with 16 output tokens each. The Python frontend spent 37.2 ms of CPU per request and saturated its core at 26.6 requests per second, with a median time to first token of 2.4 s.
The Rust frontend spent 2.63 ms and served 222.9 requests per second at 59 percent of its core (Figure 5). At 64,525 tokens per prompt the costs were 209.4 and 10.8 ms.

Most of that gap is not Python against Rust. vLLM’s Rust frontend loads tokenizer.json with fastokens, a BPE tokenizer written in Rust by Crusoe together with NVIDIA’s Dynamo team, while the Python frontend uses Hugging Face’s tokenizers, which is also Rust, through its Python binding.
Switching the Python frontend to fastokens brings it to 7.16 ms per request at 10,693 tokens and 51.3 ms at 64,525, a factor of five and four with no Rust frontend involved. The Rust frontend then takes another factor of 2.7 at 10K tokens and 4.7 at 64K.
On one core, outside any server, the same tokenizer.json gives identical token IDs three ways (Figure 6). Calling Hugging Face’s tokenizers from Rust instead of from Python is 1.1 to 2.7 times faster. Switching to fastokens is 9 to 13 times faster again.
For a 117,738-token prompt that is 230 ms through Python, 166 ms from Rust and 13.4 ms with fastokens. Crusoe reports a 9.1 times average speedup over Hugging Face across models and lengths, which is in line with what we see.

One detail about the binding matters if you write your own serving code. In the tokenizers Python API, encode holds the interpreter lock for the whole call: a thread that ticks every millisecond was frozen for 19.4 ms during a 7,715-token encode and for 335 ms during a 117,738-token one.
encode_batch releases the lock, with worst gaps of 1.4 and 7.0 ms. transformers calls encode_batch, so vLLM’s frontend is safe from this. A server that calls encode directly on a long prompt stops streaming to every other client while it tokenizes.
Prefix caching
Agentic traffic is mostly input. TraceLab, a trace of coding agents from the University of Washington, found 95.7 percent of input tokens served from the prefix cache, inputs outnumbering outputs 294 to 1, and typical prefixes of 32K to 256K tokens.
GitHub Copilot traces analyzed by Microsoft researchers show hit rates averaging 90 percent within a turn and 55 percent across turn boundaries. Inferact’s traces of Codex working on SWE-bench Pro show 94.2 percent.
A cache hit removes the GPU’s prefill work for the cached tokens. It removes none of the host’s: the template, the tokenizer and the block hashes still run over the whole prompt.
So the host cores that one prefill GPU needs grow like 1 / (1 - h), where h is the hit rate:
cores = host_us_per_prompt_token / ((1 - h) x gpu_us_per_new_token)For Qwen3-8B on a B200 at an assumed 50 percent of dense FP8 peak, with a 64,525-token context, each new token costs 51.96 GFLOP, 13.9 in the weights and 38.1 in attention over the context, which is 23.1 µs.
At a 95.7 percent hit rate, the Python frontend with Hugging Face’s tokenizers needs 3.3 cores per prefill GPU, with fastokens 0.8, and the Rust frontend 0.17 (Figure 7).
A DGX B200 has 14 host cores per GPU and a GB200 has 36, so Python survives this. It does so by spending cores and adding queueing delay to time to first token.

The engine core is the part that does not scale out.
It hashes every 16-token block of every prompt so the prefix cache can find the cached ones, by default with SHA-256 over a pickled tuple of the parent hash and the block’s token IDs, and during the 64K run py-spy found the block hasher in 40 percent of the engine core’s samples outside the stub.
Per request it cost 20.9 ms of CPU at 64,525 tokens, 0.32 µs per prompt token. One engine-core process per GPU then saturates above a 98.6 percent hit rate: past that point the Python bookkeeping for a prompt takes longer than the GPU’s prefill of the part that is not cached.
vLLM offers xxhash instead with --prefix-caching-hash-algo xxhash, which brought the cost to 16.8 ms per request and throughput from 38.5 to 44.4 requests per second in our run.
Its documentation is explicit that a non-cryptographic hash raises the collision risk, which in a multi-tenant deployment could leak private information between tenants.
The engine core
With the frontend in Rust and the model runner’s input preparation moved to the GPU, what Python still does on the critical path is decide: which requests run this step, which KV blocks they get, which have finished.
For the model runner, vLLM’s answer was not Rust: Model Runner V2 moved input preparation into Triton kernels and kept request state on the device, and on one GB200 serving Qwen3-0.6B it raised output throughput from 16K to 25K tokens per second.
The scheduler and the KV block manager are still Python, and they run once per step for the whole batch.
We measured what they cost. Across seven decode runs with average batches from 30 to 236 requests, engine-core CPU per step fits 0.43 ms plus 13.6 µs per request, with R² = 0.996.
Profiling at a batch of 236 put about a quarter of the samples outside the stub in the scheduler’s own functions, a fifth in the KV cache manager and block pool, and more than a third in the locks and condition variables the engine loop uses to hand work to its I/O threads. At 256 requests the fit gives 3.9 ms per step.
Async scheduling hides those 3.9 ms behind the GPU step, as long as the step is longer. In the scenario of Figure 1 the step is 17.2 ms on an H100, 12.0 on an H200 and 7.2 on a B200, so the engine core uses 23, 33 and 54 percent of the time it has.
On Rubin the step is 3.0 ms and the engine core needs 130 percent of it. Figure 8 shows why a smaller batch does not help: each extra request costs the engine core 13.6 µs and costs a Rubin 9.8 µs of KV reads at 2K context, so the scheduler is the longer of the two at every batch size from 16 requests up.
On a B200 each extra request costs the GPU 23.6 µs, and the engine core stays below it at every batch size.

Our core is slower than a production host, and a faster one moves the blue line down. A host core 1.4 times faster than ours would bring the engine core’s cost per request level with Rubin’s at 2K context.
On Rubin-class bandwidth with an 8B model, then, the Python scheduler is close enough to the GPU step that it should be measured on real hardware before anyone assumes it is hidden.
Kernels
Rust also reached the device this year. In September NVIDIA announced CUDA Rust in two tracks: cuda-oxide, a custom rustc code generation backend for SIMT kernels that needs a pinned nightly toolchain and is described as early alpha, and cutile-rs, a tile-based model on stable Rust 1.89 and CUDA 13.3, already used in Hugging Face’s Grout inference engine and in mistral.rs. NVIDIA says neither is production-ready.
The cuTile Rust paper reports about 2 PFLOP/s for a safe f16 GEMM on a B200, which it puts at 96 percent of cuBLAS in the abstract and 98 percent in the conclusion, and Grout decoding Qwen3-32B at 82 tokens per second at batch 1 on the same GPU.
The reason to want it is different from the host’s. On the host, Rust is bought for throughput: there is an interpreter and a lock between the token and the socket, and removing them is worth about three times per streamed token.
In a kernel there is no interpreter to remove, and a CUDA C++ kernel and a Rust one both end as PTX. What NVIDIA’s announcement offers is the borrow checker.
In both tracks the types give each thread, or each tile, its own part of the output, so two threads writing the same address in an order nobody chose becomes a compile error instead of a wrong result found weeks later; shared memory in the SIMT track still needs unsafe.
That is an argument about correctness, and with more kernels now written by models, I think it may turn out to be the stronger one.
If you run vLLM today
Measure your frontend’s CPU per streamed token and compare it with the budget of your GPU and model.
If one process cannot keep up, --stream-interval is the cheapest change, then VLLM_USE_FASTOKENS=1 for prompt-heavy traffic, then the Rust frontend, keeping in mind that it is still marked experimental.
For agentic traffic, watch the engine core’s CPU per request, which grows with prompt length whatever the cache hit rate; xxhash helps if you serve one tenant. On Rubin-class bandwidth with small and mid-size models, measure the engine core per step before assuming the GPU is the bottleneck.
Where we might be wrong
Our host core is a 2.1 GHz vCPU. Absolute costs on a production host will be lower and the crossovers in Figures 1, 7 and 8 will move. The ratios between the two frontends should travel better than the absolute numbers, but we measured them on this core only.
The stub is not a GPU. Real executors also spend host CPU below the line we drew: kernel launches, CUDA graph replay, sampling, and without Model Runner V2 the input preparation. The real host load is higher than what we measured.
The load generator ran on the same machine. Loopback TCP puts receive-side kernel work on the sender’s CPU, which adds the same absolute amount to both frontends’ cost per token and makes our three-times ratio an underestimate, if anything.
The GPU side of Figures 1, 7 and 8 is a model with assumed efficiency: 80 percent of HBM bandwidth in decode and 50 percent of FP8 peak in prefill.
The Rust frontend is a few months old and not feature-complete, according to vLLM’s own roadmap issue. We compared it with a Python path tuned for years, which is not a fair fight in either direction.
Not every layer needs Rust. In June, Ray Serve LLM matched the throughput of the Rust vllm-router by putting HAProxy, written in C, on the data path and asking its Ray Serve router only for the routing decision. Per-request decisions are cheap in any language; per-token and per-byte work is where the interpreter shows.
Free-threaded CPython is officially supported since 3.14. It could let one Python frontend use many cores, which addresses scale-out and not the cost per token we measured.
If you want the long version of how an engine core batches, pages and schedules, it is in LLM Systems Engineering, the handbook we wrote for The Software Frontier, at lorenzobrada.gumroad.com.
Method
Software. vLLM 0.33.0 from PyPI (CUDA 13.0 wheel, PyTorch 2.14.0, transformers 5.20.2, tokenizers 0.23.3), with the bundled vllm-rs frontend. fastokens 0.3.6 for the Python frontend and 0.2.1, the version vllm-rs links, for the Rust microbenchmarks, which use the tokenizers crate 0.23.2. Jinja2 3.1.6, minijinja 3.1.0, pydantic 2.14.3, Python 3.13.18, Rust 1.97.0.
Host. Two vCPUs of an Intel Xeon reporting 2.1 GHz, CPU family 6 model 207 (Emerald Rapids), KVM. VLLM_TARGET_DEVICE=cpu lets the engine start without CUDA; nothing below the executor runs.
Engine. 256 maximum sequences, 16,384 batched tokens per step, prefix caching on with the default SHA-256 unless stated, async scheduling on. The stub’s own time is measured inside it and subtracted from the engine core’s CPU. The trimmed core of the stub:
class FakeExecutor(Executor):
"""vLLM executor whose GPU is a timer. Load with
--distributed-executor-backend fakeexec.FakeExecutor"""
def execute_model(self, scheduler_output, non_block=False):
self.worker.execute_model(scheduler_output) # track prompt progress
start = max(time.perf_counter(), self.gpu_free_at)
self.step_done_at = start + STEP_S # occupy the "GPU"
self.gpu_free_at = self.step_done_at
return done_future(None) if non_block else None
def sample_tokens(self, grammar_output, non_block=False):
out = self.worker.sample_tokens() # next real token per
fut = GpuDone(self.step_done_at, out) # request past prefill
return fut if non_block else fut.result() # result() sleeps
@classmethod
def supports_async_scheduling(cls):
return True # step N+1 overlaps NWorkloads. Decode: 128-token random prompts, 512 output tokens plus or minus 50 percent, 256 concurrent, 768 to 3,072 requests per run after a warm-up. Prompt-heavy: tool-transcript prompts built from CPython source, 16 output tokens; 384 unique prompts at 64 concurrent for the 10K runs, 48 requests drawn from 24 prompts at 8 concurrent for the 64K runs. Where a run lasted long enough to log it, vLLM’s own counter reported prefix-cache hit rates of 1 to 2 percent in the 10K runs and up to 18 percent in the 64K runs; hits change the engine core’s work a little and the frontend’s not at all. The load generator is vllm-rs bench serve.
Accounting. CPU time is utime plus stime of each process from /proc/<pid>/stat, read before and after the measured run. The frontend is pinned to one vCPU; the engine core and the load generator share the other.
Checks. Rendered prompts are byte-identical between Jinja2 and minijinja, token IDs are identical across Hugging Face tokenizers from Python, from Rust and fastokens, and the bare streaming loops emit byte-identical SSE. Every measured number in this piece is recomputed from the raw results by a script that also checks it against the text.
Confidence
M: measured by us for this piece. A: primary source. B: credible secondary source or vendor claim. C: our inference from A or B. D: assumption.
M. Python frontend 51 to 84 µs of CPU per streamed token, Rust frontend 18 to 26; 75.0 and 24.2 at the 10 ms step.
M. With four-token chunks, 35.4 and 11.1 µs per token; split as 22.2 per token plus 52.8 per chunk (Python) and 6.7 plus 17.5 (Rust).
M. Bare streaming loop 11.1 µs per token in Python, 0.86 in Rust, byte-identical output.
M. Prompt-heavy load at 10,693 tokens: 37.2, 7.16 and 2.63 ms of frontend CPU per request (Python, Python with fastokens, Rust); 209.4, 51.3 and 10.8 ms at 64,525 tokens.
M. Engine core per decode step: 0.43 ms plus 13.6 µs per request, R² 0.996 over seven runs. Engine core per 64,525-token request: 20.9 ms with SHA-256, 16.8 with xxhash.
M. tokenizers
encodeholds the interpreter lock for the whole call;encode_batchdoes not.M, with a caveat. Profile shares come from py-spy in non-blocking mode and are shares of samples outside the stub. Samples in locks and condition variables may include time a thread spent runnable but waiting for the core or for the interpreter lock.
A. Qwen3-8B architecture (model card). DGX B200 host CPU and FP8 throughput (NVIDIA datasheet). GB200 has one 72-core Grace per two GPUs (NVIDIA). Rubin listed at 19.2 TB/s, down from 22 (NVIDIA page, reported by TechRadar).
A. Rust frontend RFC benchmark and setup, roadmap status, and the X announcement (vLLM). Model Runner V2 result (vLLM blog). Fast detokenizer built on DecodeStream since tokenizers 0.21.1 (vLLM source). SHA-256 default and the xxhash caveat (vLLM documentation).
A. CUDA Rust tracks, toolchains and maturity (NVIDIA Technical Blog). cuTile Rust GEMM and Grout numbers (arXiv 2606.15991).
A. Coding-agent cache hit rates: TraceLab (UW), GitHub Copilot traces (arXiv 2608.00101), Codex traces (Inferact dataset card).
B. Tokenization and detokenization as production bottlenecks, and the gRPC token boundary (Shepherd Model Gateway authors, PyTorch blog). fastokens 9.1 times average speedup (Crusoe).
B. Ray Serve LLM matching vllm-router (Anyscale and Google). TGI maintenance mode since December 2025 (Hugging Face documentation; date from secondary sources).
C. That faster host cores leave the Python to Rust ratios roughly unchanged. That the engine core is the next host bottleneck on Rubin-class bandwidth for small and mid-size models.
D. 80 percent of HBM bandwidth in decode, 50 percent of FP8 peak in prefill, and the 256 by 2K scenario.




The 0.43 ms plus 13.6 µs per request is the number that stuck with me. At 256 requests that's roughly 3.9 ms of Python scheduler per step, already longer than a Rubin step for that model. Is the scheduler next in line for Rust, or is the near-term fix just running more engines with smaller batches each?
The 1 / (1 - h) line is the one I'll remember: a 95.7% cache hit saves the GPU's prefill but none of the host's template, tokenizer and block-hash work, so agent traffic makes the host's share bigger every turn. Since an agent loop resends a transcript that's mostly identical to the previous one, I keep wondering why the frontend doesn't cache token IDs per conversation the way the engine caches KV. The catch I'd expect is chat templates that rewrite earlier turns, like dropping old reasoning blocks, so the rendered prefix isn't stable. Did you check whether Qwen3's template output stays prefix-stable across turns?