The Software Frontier
Subscribe
Sign in
Home
Notes
Chat
Start Here
Archive
Leaderboard
About
Latest
Top
Discussions
How Rust Moved Into the Inference Server
At batch 256, a B200 serving an 8B model leaves the host about 28 microseconds per generated token. On our test core, vLLM’s Python frontend spent 75…
Nov 17
•
Lorenzo Bradanini
and
Lorenzo Tettamanti
8
3
2
Attention-FFN Disaggregation: How it Works Under The Surface
Grouped-query attention does 16 to 32 FLOPs for every byte it reads. A Blackwell GPU needs more than 550 to stay busy. Splitting the transformer layer…
Nov 13
•
Lorenzo Bradanini
and
Lorenzo Tettamanti
9
5
2
Optimal LLM Batching Is NP-Hard, Even with Identical Prompts
As each request’s KV cache grows with generated tokens, scheduling becomes a problem of expanding footprints. We prove it remains strongly NP-hard even…
Nov 6
•
Lorenzo Bradanini
and
Lorenzo Tettamanti
11
1
2
October 2026
How to Serve an Open Model on Day Zero
The arithmetic we run before renting a GPU: KV cache from layer types, expert coverage, memory pools for hybrid attention, speculation, and cost per…
Oct 27
•
Lorenzo Bradanini
and
Lorenzo Tettamanti
10
4
2
LLM Inference Benchmarks: Seven Stopwatches, One Server
Same server, same timestamps, seven tools. The p99 time to first token moves twenty times, and the peak of a concurrency sweep is a rate the server…
Oct 13
•
Lorenzo Bradanini
and
Lorenzo Tettamanti
10
3
3
Inside Google’s TPU: How it works, and what it costs
We ran it against five TPU generations, from v4 to Ironwood, then followed the money: a rate card that prices bandwidth, an anchor tenant paying 13% of…
Oct 6
•
Lorenzo Bradanini
and
Lorenzo Tettamanti
11
2
2
A wonderful dive into how FP4 works
NVFP4 and MXFP4 multiply the same sixteen codes. The difference is one byte of scale per block, and it is worth 1.6 dB on Gaussian data, 4.2 dB on…
Oct 4
•
Lorenzo Bradanini
and
Lorenzo Tettamanti
6
2
September 2026
Why Cached Tokens Cost 10% and Vanish in 5 Minutes
Every agent resends its whole context at every step, and every provider bills the repeat at a discount with a timer attached. The discount and the timer…
Sep 29
•
Lorenzo Bradanini
and
Lorenzo Tettamanti
6
1
How Mojo Actually Compiles
Ask Mojo 1.1 for sm_90 and it quietly returns sm_90a, hinting that portability no longer lives in the binary. An H100 executable can carry kernels…
Sep 24
•
Lorenzo Bradanini
and
Lorenzo Tettamanti
6
1
How NVIDIA's Data Center Business Actually Works in 2026
NVIDIA booked $89.0 billion of data center revenue in its latest quarter. The company now reports about $530 billion of commitments and maximum…
Sep 20
•
Lorenzo Bradanini
and
Lorenzo Tettamanti
9
1
Unleashing OpenCL’s secrets
A technical account of the standard, its implementations, and where it actually stands in 2026.
Sep 10
•
Lorenzo Bradanini
and
Lorenzo Tettamanti
9
3
August 2026
When Batching Stops Working
Everyone says to raise the batch until you are compute bound. Solve for where that stops and every datacenter GPU of the last four generations goes…
Aug 28
•
Lorenzo Bradanini
and
Lorenzo Tettamanti
4
3
This site requires JavaScript to run correctly. Please
turn on JavaScript
or unblock scripts