Intro

The first essay in this series argued that distributed systems fail in the spaces between their components, and that those spaces are structurally opaque. This one argues something more uncomfortable.

Even if you accept that the system is opaque, you still have to operate it. You still have to debug it at 03:47 UTC. You still have to decide, in the next ninety seconds, whether to roll back, fail over, shed load, or page someone more senior.

To do any of that, you have to see the system.

This is the second structural problem, the one most production engineers learn about the hard way, usually during an incident that lasted longer than it should have because the dashboards stayed green until they didn’t.

The system you observe is not the system that exists. It is a projection of the system into a low-dimensional representation, built out of metrics, logs, traces, and the mental model that lives in the operator’s head.

The projection is incomplete by construction. It is delayed by the time it takes to collect, aggregate, and render. It is biased by what someone decided to instrument three years ago. And it is aggregated, often violently, in ways that destroy precisely the signal needed to recover from an incident.

This essay is about that gap. Not the gap between the system as designed and the system as composed (Part I), but the gap between the system as composed and the system as perceived. The two gaps are different, and they compound.

Three incidents, mechanically reconstructed: Slack’s autoscaler chasing metrics that had decoupled from the network failure underneath, while the dashboards that would have shown the decoupling failed at the same time as the system;

GitHub’s 43-second partition and the 24-hour reconciliation that followed; Roblox’s 73-hour outage, where the monitoring stack failed at the same time as the system it monitored, and engineers spent two days debugging a fully dark cluster.

The pattern beneath all three is the same: the operator’s view of the system, and the system itself, were two different things. The interesting question is why this is structural, not accidental.

The observer problem, mechanically

The canonical incident for this material is the Slack outage of 4 January 2021, the first business day after the holiday break, documented in detail in Slack’s post-mortem by Laura Nolan.

Slack runs on AWS, with services running in dedicated VPCs (Virtual Private Clouds) connected by AWS Transit Gateways (TGWs). TGWs are managed by AWS and intended to scale transparently. Slack’s traffic pattern is unusual: the platform is quiet over the holidays and then ramps to one of its biggest days of the year on the first Monday back, when clients reconnect with cold caches and pull down more data than usual.

On 4 January, the TGWs did not scale fast enough for that ramp. Around 6:00 AM PST, one of them began dropping packets. The packet loss caused widespread degradation in internal calls across Slack’s services, but the symptom was not yet visible to users.

Slack’s web tier autoscales on two signals: CPU utilisation and utilisation of available Apache worker threads. Here is where the failure begins. As packets dropped on the TGW, threads in the web tier spent more time waiting on slow or stalled backend calls.

Waiting threads do not burn CPU. So as the system became less able to serve users, CPU utilisation actually dropped. The autoscaler, looking at CPU, concluded the fleet was over-provisioned and downscaled the web tier.

Then the mini-peak at 7:00 AM PST arrived. Load increased against the now-smaller fleet on a degraded network. Apache worker thread utilisation climbed sharply, threads were waiting longer, and more of them were in use, and the thread-utilisation signal triggered aggressive upscaling. Slack attempted to add 1,200 servers between 7:01 and 7:15 AM PST.

The scale-up failed. New instances are configured by an internal service Slack calls provision-service, which talks to other Slack systems and to AWS APIs over the same degraded network. Under the sudden load of 1,200 simultaneous provisioning requests, with elevated latency on every dependency call, provision-service hit two resource ceilings: the Linux open files limit and an AWS quota limit.

Most of the 1,200 instances were created but never fully provisioned. They counted against the autoscaling-group size limit, blocking further scale-up, but they did not serve traffic.

And then the second layer of the observer problem revealed itself. Slack’s dashboarding and alerting service had failed during the early stages of triage. The reason was structural: the monitoring stack ran in a different VPC from its backend databases, and the same TGW that was dropping packets sat on the path between them. The failure that was breaking the web tier had also blinded the engineers trying to diagnose it.

For roughly the next hour, incident responders worked without dashboards. They had logs, command-line tools, and the ability to query metrics backends directly, but none of the pre-built queries that turn raw metrics into actionable views.

Some engineers were SSHed into production instances when the autoscaler deprovisioned them mid-investigation, abruptly ending their sessions. provision-service recovered around 8:15 AM PST; serving capacity reached a degraded-but-functional state by 9:15 AM PST; full recovery, after AWS engineers manually scaled TGW capacity, completed at 10:40 AM PST.

The whole sequence is the observer problem in compounded form. The autoscaler responded first to a metric (CPU) that did not represent the system state, then to a metric (thread utilisation) that drove the wrong action under the conditions.

The control loop did exactly what it was designed to do; the signals it was acting on had decoupled from the reality of the network underneath.

And the observation surface that operators would normally have used to see this decoupling was itself, by architectural coincidence, a casualty of the same failure.

This is the observer problem in its operational form, and it has three structural sources, each worth pulling apart.

Instrumentation lag

Every observability pipeline introduces delay between an event occurring in the system and that event being visible on a dashboard. The delay has multiple stages.

First, emission delay: the event happens, but the code that emits the metric or log runs after the event, and the emission itself takes some time, usually buffered behind a batching layer with a flush deadline (StatsD typical: 10s; OpenTelemetry batch span processor: 5s default, max queue 2048). Second, collection delay: the emitted data is scraped or pushed to a collector at fixed intervals (Prometheus default scrape interval: 15s, with scrape_timeout typically 10s).

Third, aggregation delay: the collector pre-computes summary statistics, often on a window ending some seconds in the past to allow late-arriving data. Fourth, render delay: the dashboard queries the storage layer and renders, typically on a 30s to 60s refresh.

End-to-end delay in a well-tuned production pipeline is typically 15 to 60 seconds. In many real pipelines it is several minutes.

This is not a defect. It is the cost of producing observations that are coherent across thousands of hosts. The delay is the price paid for the metric being computable at all.

But the delay has a consequence that classical control theory makes precise. Any closed-loop control system whose feedback path introduces delay τ experiences a phase lag of ωτ radians at frequency ω.

The Nyquist stability criterion says, in essence, that a feedback loop becomes unstable when its total phase lag approaches 180° at the frequency where the loop gain is unity; with even modest controller gain, sufficient phase lag turns negative feedback into positive feedback and the loop oscillates.

Concretely: for an autoscaler with a minute-scale measurement-to-action delay attempting to track load that varies on similar timescales, phase lag approaches the stability boundary. Aggressive scaling policies tip the loop into oscillation; this manifests operationally as the autoscaler over-provisioning, then over-scaling-down, then over-provisioning again, never settling.

Slack’s post-mortem describes a variant of this pattern: an initial downscale on CPU, followed by an aggressive upscale on thread utilisation, against a network problem the loop could not see at all.

The Shannon-Nyquist sampling theorem provides the converse bound: a control loop sampling at interval T cannot observe, and therefore cannot react to, disturbances faster than 2T. A 15-second Prometheus scrape interval is structurally blind to load dynamics on timescales below 30 seconds. The information about those dynamics has been aliased into the lower-frequency band, where it appears as noise.

Marc Brooker has written about this directly in the context of AWS load balancing: a control loop with delay longer than the time constant of the thing it controls cannot stabilise that thing. It can only chase it.

The Slack autoscaler chasing a CPU metric that had decoupled from real load (because waiting threads do not burn CPU) was operating in exactly this regime.

The mitigation is not to make the metrics faster, though that helps. The mitigation is to design control loops that do not depend on global metrics: rate-limiting at the boundary, admission control based on local queue depth observable in O(1) from inside the affected process, fallback to last-known-good rather than reactive scaling.

Local observations made by the component itself have zero collection delay because they bypass the pipeline entirely. Global observations always carry the pipeline’s phase lag.

This is the structural argument for autonomic behaviour at the component level. Components that defend themselves locally (with circuit breakers, backpressure signals propagated synchronously to upstream callers, and load-shedding triggered by their own queue depth) do not depend on a delayed control loop to survive.

Components that wait for the autoscaler to rescue them are operating inside a feedback loop whose stability margin is, almost always, narrower than anyone has measured.

Subscribe now

Aggregation destroys the signal

Every metric you look at on a production dashboard is an aggregate. A counter has been summed across hosts. A latency value has been percentiled across requests. A gauge has been averaged or maxed across a time window.

Aggregation is mathematically necessary; you cannot stare at every individual request. It is also, almost always, the thing that hides the failure.

The pathology is most visible in latency aggregation, and the canonical analysis is Gil Tene’s How NOT to Measure Latency (2015). The argument is worth deriving mechanically, because the conclusion is counter-intuitive and the mechanism is not.

Consider a load generator configured to issue requests at a rate of R per second; one request every 1/R seconds. The generator records, for each response, the time elapsed between send and receipt. Call this service_time. The reported P99 is the 99th percentile of the service_time distribution across some number of samples.

Now suppose the system stalls completely for T seconds, then recovers. During the stall, no responses arrive. The generator, in its naive form, has two possible behaviours.

In the coordinated form, the generator waits for each response before sending the next. During the T-second stall, exactly one request is in flight; its service_time is recorded as T. The other R·T requests that should have been sent during the stall are never sent. They do not appear in the histogram at all.

In the uncoordinated form, the generator sends on schedule regardless of whether responses arrive. During the stall, R·T requests pile up in the kernel’s socket buffer or in the generator’s own queue. When the system recovers, those requests are drained; each one’s service_time is measured from the moment it was sent, not the moment it was scheduled to be sent.

The bias is statistical. Let F_observed(t) be the empirical CDF of service_time the generator records, and F_true(t) the CDF of latency that a real user (arriving according to a Poisson process at rate R) would experience.

In the coordinated form, the missing samples should have been drawn from the slowest part of the latency distribution; their absence systematically truncates the right tail of F_observed. The quantile function Q_observed(0.99) = F_observed⁻¹(0.99) is, by construction, a lower bound on Q_true(0.99), with the gap widening as T grows.

The user who clicked at time t and got a response at time t + T + ε experienced a latency of T + ε. The histogram has either zero entries near T + ε (coordinated form) or entries clustered near ε (uncoordinated form). In neither case does the percentile reflect what the user felt.

The correction Tene proposes (HdrHistogram’s recordValueWithExpectedInterval) is to synthesise the missing samples: for each measured service_time exceeding the expected interval 1/R, insert additional samples at service_time − 1/R, service_time − 2/R, ..., down to 1/R.

These synthetic samples represent the users who would have arrived during the stall and would have experienced progressively shorter waits.

The implementation, in essence:

void recordValueWithExpectedInterval(long value, long expectedInterval) {
    recordValue(value);
    if (expectedInterval <= 0 || value <= expectedInterval) return;
    long missingValue = value - expectedInterval;
    while (missingValue >= expectedInterval) {
        recordValue(missingValue);
        missingValue -= expectedInterval;
    }
}

Six lines. One additional method call per recorded measurement. The result, when applied to real traces, is routinely an order-of-magnitude shift in the tail. A system reporting P99 = 200 ms under uncorrected measurement reports P99 = 2 to 4 seconds under coordinated-omission correction.

The dashboard was lying by a factor of ten, not by accident, but by construction of how percentiles are computed from a fixed-rate sampler against a non-stationary service-time distribution.

The same pathology appears, in different shapes, throughout the metric stack. Server-side latency histograms measure only requests the server got to process. Requests rejected at the load balancer, dropped at the TCP layer, or held in the kernel accept queue do not appear.

The server’s P99 can be excellent while the connection P99 (which the user actually experiences) is catastrophic. The Slack incident is exactly this: the web tier was reporting acceptable internal latencies for the requests it was handling, because the requests not being handled were not in the denominator. Survivorship bias, in observation form.

The same problem appears in averaging. A service that handles two classes of request, 99% of them fast and 1% of them slow, will show a mean latency dominated by the fast class. If the slow class gets ten times slower during an incident, the mean barely moves. The average is structurally insensitive to the tail, which is where outages live.

Percentiles are better than averages, but only if computed correctly, and only if the percentile you care about is in the data.

P99 across a million requests has ten thousand data points and is statistically reliable. P99 across a thousand requests has ten data points and is statistical noise. P99.9 across a million requests has a thousand data points, marginal. P99.99 across a million requests has a hundred data points, useless.

The deeper into the tail, the more samples needed to stabilise it, and the deeper into the tail is exactly where the interesting failures live.

The reason failures live in the tail is itself a queueing-theoretic result. For an M/M/1 queue (Poisson arrivals at rate λ, exponential service times at rate μ, single server) with utilisation ρ = λ/μ, the expected waiting time in the system is 1/(μ(1−ρ)). As ρ → 1, this diverges hyperbolically; the variance of waiting time grows even faster, as 1/(1−ρ)².

The practical consequence is that high-percentile latencies blow up far faster than utilisation increases: a service running near saturation has a P99 that is many multiples of its mean, and the ratio worsens sharply as you approach full utilisation.

Tails are not a measurement artefact; they are the physics of contention, and they are precisely what averages and low-resolution percentiles destroy.

The problem compounds when percentiles are themselves aggregated. The P99 of a service that is the union of ten hosts is not the average of those hosts’ P99s, nor their maximum, nor any function of them; it is a quantile of the underlying merged distribution, which is unrecoverable once each host has been independently percentiled.

Many monitoring systems do exactly this aggregation, producing a number labelled “P99” that is mathematically meaningless. Theo Schlossnagle and the Circonus team have written extensively on this; the correct primitive is to store the full histogram and percentile at query time, after merging across hosts.

Three production-grade histogram primitives dominate:

• HDR Histogram (Tene): fixed-precision logarithmic bucketing across many orders of magnitude (typically nanoseconds to hours at three significant digits), O(1) insert, mergeable across instances, widely used in the JVM ecosystem.

• t-digest (Dunning, 2013): centroid-based sketch with concentrated precision in the tails, useful when storage is constrained but tail accuracy matters; O(log n) insert in the worst case, mergeable with a controlled error bound.

• DDSketch (Masson, Rim, Lee, 2019): relative-error guarantee α, log-bucketed similarly to HDR but with provable tail accuracy, fully mergeable without error accumulation; used by Datadog as its native primitive.

All three solve the merge problem. None of them solve the storage problem: keeping a per-request histogram per dimension (customer, region, endpoint, version) at full cardinality costs roughly two orders of magnitude more than scalar metrics. Almost no organisation does it for everything.

Beyond latency, aggregation collapses cardinality. If a service is failing for one specific customer, on one specific endpoint, in one specific region, the aggregate error rate may show a 0.1% blip below any reasonable alerting threshold. The blip is the entire experience of that customer. Aggregation makes the rare invisible by averaging it against the common.

Charity Majors’s argument for high-cardinality observability, developed across her writing at Honeycomb and in Observability Engineering (O’Reilly, 2022, with Liz Fong-Jones and George Miranda), reduces to this: the questions that matter during an incident are almost always questions about specific slices of traffic, and any pre-aggregated metric has already destroyed the dimensions needed to slice on.

The pre-aggregation step is irreversible. Information theory enforces this; once you have computed the count of errors per minute, you cannot recover which customers produced those errors.

The cost of preserving cardinality is large; the cost of discarding it is invisible until the incident in which you need the dimension you discarded. The discipline is to know which dimensions are likely to matter, instrument those at full cardinality, accept that some incidents will be invisible until someone re-instruments after the fact.

Subscribe now

You can only see what you instrumented

This is the third and most permanent of the three sources, and the hardest to mitigate, because it is a statement about the closure of the observable space.

Every metric, log line, and trace span in your system exists because some engineer, at some point, decided it would be useful. The decision was made under one model of how the system would fail. The decision was made before the migration that changed the failure modes. The decision was made by someone who has since left.

The observation space of a production system is, in effect, a fossil record of past concerns. It captures the questions someone thought to ask. The questions that nobody thought to ask are invisible, by definition, and they remain invisible until an incident forces someone to add the instrumentation in the middle of the firefight.

This is the operational meaning of Cindy Sridharan’s distinction (in Distributed Systems Observability, O’Reilly, 2018) between monitoring and observability. Monitoring is the practice of watching known failure modes by means of pre-defined metrics and alerts.

Observability is the property of a system that allows you to ask questions about its state that you did not anticipate having to ask.

The two are different in kind. Monitoring is comprehensive at handling familiar failures and useless at handling novel ones. Observability is the opposite. A mature production environment requires both, but the second is much harder to achieve, because it requires preserving structured, high-cardinality data about every interesting event, in a form that can be queried after the fact along dimensions no one specified in advance.

The DynamoDB DNS race condition from Part I is a clean example of the limit. The plan generation system had monitoring: are the Enactors running, are they applying plans, is Route 53 returning valid responses? All of these were green throughout the incident.

The question that would have caught the failure, is there a window during which one Enactor has applied an older plan after another Enactor has deleted it, was a question nobody had thought to ask, because the failure mode it describes had never happened.

There was no monitoring for it, because there was no model of it. Observability, in Sridharan’s sense, might have caught it: if the raw event stream of Enactor operations, with full cardinality on plan version and Enactor identity, had been queryable, an engineer during the incident could have constructed the query that revealed the interleaving. Whether anyone would have thought to construct that query in the first ninety minutes of the outage is a separate question.

Ben Sigelman, who designed Google’s Dapper tracing system and later co-founded LightStep, has argued that the practical limit of observability is set by the cost of the questions you are not yet asking. Storing every span, log, and structured event at full cardinality is theoretically ideal and economically impossible.

Every organisation makes a choice about which dimensions to preserve, and that choice is, in retrospect, always slightly wrong, because the next incident is the one whose relevant dimension was sampled out.

The discipline is not to eliminate this gap, which cannot be done, but to acknowledge it: to recognise that your dashboards are a model, that the model is incomplete, and that the moments when the model and the system disagree are the moments that matter most.

The three frames of reference

The DynamoDB cascade in Part I introduced, almost in passing, an idea worth making explicit. During the outage, the system existed in three simultaneously valid states, depending on whose vantage point you took: the data plane saw a healthy service, the control plane saw correctly-served DNS, the client saw nothing.

This is not a metaphor. It is structural, and it has a theoretical grounding. Lamport’s 1978 paper Time, Clocks, and the Ordering of Events in a Distributed System established that in a distributed system without a shared global clock, the only well-defined ordering between events is the happens-before relation →, defined transitively by causal message passing. Events that are not connected by → are concurrent, and concurrent events have no meaningful temporal ordering across observers; each observer may legitimately see them in a different sequence. Each observer constructs a partial order from the messages it has received, and the partial orders need not agree.

Any distributed system at scale has at least three such frames, and they almost never agree.

The data plane frame is the view of the components doing the actual work. Storage nodes know whether they are reachable on their primary network interfaces and whether their disks are responding. Compute hosts know whether their workers are processing requests. From this frame, the system is described in terms of internal state: queue depths, lock contention, GC pauses, file descriptor counts.

The control plane frame is the view of the systems that manage the data plane. Schedulers, load balancers, service discovery, deployment systems, autoscalers. The control plane sees the data plane through its own observations, typically lagged metrics and periodic health checks. From this frame, the system is described in terms of declarative goals and reconciliation: how many instances should be running, how many are running, what is the gap?

The client frame is the view of whatever is trying to use the system. This includes external customers but also internal services that depend on the one in question. The client sees the system only through its responses: latency, error rate, correctness. From this frame, the system is described in terms of the service contract being honoured or not.

In a healthy system, all three frames produce consistent descriptions. The data plane is processing requests, the control plane sees that the data plane is processing requests, and the client receives correct responses. This consistency is what allows operators to use any one frame as a proxy for the other two.

In an unhealthy system, the frames diverge, and the pattern of divergence is diagnostic. Data plane healthy, client failing: the failure is in the path between them, usually DNS, routing, or load balancing. Control plane healthy, data plane degraded: the control plane is observing a stale or filtered view of the data plane.

All three frames disagreeing: the system has entered a regime its designers did not anticipate, and the on-call engineer is going to have a long night.

Frame divergence, mechanically: GitHub, 21 October 2018

The 2018 GitHub MySQL split-brain is the textbook case of frame divergence, and it is worth reconstructing because the mechanism turns on which observers were in which partition.

At 22:52 UTC on 21 October, routine maintenance to replace failing 100G optical equipment severed connectivity between GitHub’s US East Coast network hub and its primary US East Coast data centre. The break lasted 43 seconds. Not enough for a human to react. More than enough for everything that follows.

GitHub ran MySQL in a topology managed by Orchestrator, an open-source replication-topology manager that, in GitHub’s configuration, uses Raft consensus among its own nodes to decide when to promote replicas. The primary was in US East.

Replicas existed in US West and in a public-cloud region. Orchestrator nodes were distributed across all three. Crucially, Orchestrator’s automated failover was configured to promote across regional boundaries.

The Raft protocol requires a strict majority for any decision: for a cluster of n nodes, ⌊n/2⌋ + 1 must agree. When the East Coast data centre dropped off the network, the Orchestrator nodes inside it were partitioned with it. The remaining nodes (US West plus US East public cloud) retained a quorum.

From their frame, the primary had failed; the only action under Raft’s liveness assumption was to elect a new leader and promote. Within seconds of the partition forming, the Orchestrator quorum began the leadership deselection process, opened a new Raft term, and promoted a US West MySQL replica to primary. Application traffic in the unaffected regions began flowing to it.

This is, in the language of CAP, the choice MySQL replication and Orchestrator had been configured to make: availability over consistency. When the network partitioned, the system preserved availability (writes continued to be accepted somewhere) at the cost of consistency (writes accepted in one partition were unknown to the other). Daniel Abadi’s PACELC refinement (2010) makes the framing sharper: if Partitioned, choose between Availability and Consistency; Else, choose between Latency and Consistency. GitHub’s topology had chosen PA/EL: availability under partition, latency in the steady state. The cost of that choice was paid, in full, during the 43 seconds the partition was active and the 40 minutes that followed.

The partition healed 43 seconds later. From the East Coast frame, nothing had happened: the local MySQL primary had continued serving writes the whole time, because applications in East continued routing to it. From the West Coast frame, it was now the primary, and writes were flowing in. Both databases had accepted writes for the duration of the partition, neither aware of the other’s writes.

Cross-region MySQL replication carries some lag in steady state, which is the window during which writes acknowledged to clients on East had not yet reached West and were therefore not present on the newly-promoted West primary.

The trap closed in the next forty minutes. Once connectivity restored, GitHub’s application tier saw the new West Coast primary and began directing writes to it. For nearly 40 minutes, the West Coast accepted writes that the East Coast primary did not see. Meanwhile, the East Coast primary contained the few seconds of writes from the partition window that had never been replicated to West.

When engineers locked deployment tooling and assessed state, they found two databases with divergent histories. Reconciling by failing back to East would discard the 40 minutes of West Coast writes. Failing forward on West would discard the East Coast partition-window writes. Neither was acceptable.

GitHub chose to fail forward, preserving the 40 minutes of West Coast writes at the cost of consistency: applications in the East Coast now had to make a cross-country round trip for every database call, adding cross-country latency to operations that had been designed to complete in local-region time.

The site was effectively degraded for 24 hours and 11 minutes while data was restored from backups, replication was rebuilt, and the orphaned East Coast partition-window writes were manually reconciled from binary logs. As the post-mortem records: one of the busiest clusters in the partition window contained 954 writes that had to be reconciled by hand.

From the East Coast control plane’s frame, the system had operated correctly: writes were accepted, replication was healthy locally, no monitoring fired. From the Orchestrator quorum’s frame, failover happened exactly as designed when the primary became unreachable.

From the data plane’s frame, two independent write histories existed during a window neither side observed in full. From the client frame, some writes had succeeded that, in a consistent universe, would have been rejected.

The 43 seconds of partition were the trigger. The frame divergence was the failure. The trigger was unavoidable; physical networks partition.

The deeper unavoidability was theoretical: the FLP impossibility result (Fischer, Lynch, Paterson, 1985) proves that no asynchronous deterministic consensus protocol can guarantee both safety and liveness in the presence of even a single crash failure.

Every real consensus system, including Raft, breaks this tie pragmatically, in Raft’s case, by leaning on timing assumptions to maintain liveness. Those timing assumptions are exactly what fail during a network partition; the algorithm has no choice but to make progress on the basis of local observations that, during the partition, were sufficient for a local quorum decision but insufficient to determine the global state.

GitHub’s subsequent re-architecture eliminated cross-region automatic failover precisely because no observation surface available in real time was sufficient to detect the divergence as it was happening.

They moved the consistency/availability tradeoff from automatic to human-in-the-loop, accepting longer mean-time-to-recovery in exchange for not making this specific decision wrong again.

Subscribe now

The Heisenbug problem

There is a class of bug in distributed systems that disappears when you try to observe it. The folklore name is Heisenbug, after the uncertainty principle; the technical name is observation-dependent failure.

These bugs exist because adding observation to a system changes the system. Logging a value takes time, which changes the timing of the surrounding code, which changes the order in which concurrent operations interleave, which changes whether the race condition fires.

Capturing a stack trace acquires locks, which changes contention, which changes which thread reaches the critical section first. The act of looking at the system alters what the system does.

This is, again, not a metaphor. Modern observability tooling routinely takes 1-5% of a service’s CPU and a measurable fraction of its memory. eBPF-based profilers, distributed tracing, log aggregation, all of them consume resources that come from the same pool the application uses to do its work. In the steady state, the cost is acceptable.

In the regime where the system is already saturated, adding observation can be the perturbation that pushes the system from a marginal state into a failure state.

The mitigation is not to remove observation, which would leave the system unobservable. The mitigation is to design observation to be low-overhead and load-shedding: when the system is saturated, observation is the first thing to drop, not the last.

Sampling rather than full capture, head-based sampling rather than tail-based, structured events written to local buffers rather than synchronous network calls.

The modern technical answer to this is eBPF: in-kernel observation programs verified for safety at load time and executed in response to kernel events. Because the aggregation happens in kernel space, written into perf event arrays or BPF ring buffers shared with user-space readers via mmap, the observation path bypasses the syscall boundary entirely.

The cost of recording an event collapses to a few cache-line writes, with no context switch and no allocator pressure on the application path.

The Linux kernel’s eBPF verifier statically proves termination and memory safety at program load time, which means an observation program cannot crash the kernel even if it has a bug; bpftrace, BCC, and Cilium’s Hubble all build on this substrate.

The implication for the Heisenbug problem is that, for many workloads, eBPF-based observation has overhead small enough that observing the system no longer meaningfully alters the system’s behaviour.

Each of these decisions trades observability for the ability to keep the system running while it is being observed. The trade is acceptable only if you have decided it deliberately.

In most systems, it has been decided by accident, by whatever the default configuration of the tracing library is, set by whichever engineer integrated it.

But the deepest version of the observation-disturbing-the-system problem is not about overhead. It is about circular dependency between the observation surface and the thing being observed.

When the monitoring system depends on the system it is monitoring, a failure in the monitored system blinds the monitoring system, and the operator loses access to the diagnostic data at precisely the moment they need it most.

There is no better illustration of this than the 73 hours that began at 13:37 PDT on 28 October 2021.

The Roblox outage, mechanically

Roblox at the time ran more than 18,000 servers and 170,000 containers across its own data centres, orchestrated using the HashiCorp stack: Nomad for scheduling, Vault for secrets, and Consul for service discovery, health checks, session locking, and as a KV store.

Consul was the central nervous system. Every service depended on it to find its peers.

A single Consul cluster supported the entire backend: 5 voter nodes plus 5 non-voter read replicas. This was, as the post-mortem would later note, a single point of failure of a kind that violated every textbook lesson about blast-radius isolation.

In the months leading up to October, Roblox had upgraded from Consul 1.9 to 1.10 to take advantage of a new streaming feature designed to reduce CPU and network bandwidth on large clusters. The feature had been incrementally enabled across services without incident.

On 27 October at 14:00, the day before the outage, it was enabled on the traffic routing service, and the number of routing nodes was increased by 50% in anticipation of end-of-year traffic.

At 13:37 PDT on 28 October, Vault performance began to degrade and a single Consul server began exhibiting high CPU load. Engineers began to investigate; users were not yet impacted. The first signal was unusual write latency on Consul’s underlying KV store: the 50th percentile, normally under 300 ms, had climbed to 2 seconds.

The cluster was failing for two reasons that interacted, neither of which engineers identified for days.

The first was the streaming feature itself. HashiCorp would later explain that streaming, while overall more efficient than long polling, used fewer concurrency control elements (Go channels) in its implementation.

Under very high read and very high write load, the design exacerbated contention on a single Go channel, blocking writes and consuming CPU in kernel spin locks along the streaming subscription code path. This pathology had not appeared in HashiCorp’s pre-release benchmarks because it required the specific combination of large stream count and high churn rate that Roblox’s workload produced.

The second was buried inside Consul’s persistence layer. Consul uses BoltDB, an embedded Go key-value store inspired by LMDB’s memory-mapped design, to persist its Raft write-ahead log.

BoltDB’s design is a single memory-mapped file organised as a copy-on-write B+tree: every write transaction allocates new pages, never modifies existing ones, and commits by atomically swapping a single root pointer. This gives crash-safety, at the cost of page churn.

When pages become unreachable, BoltDB does not release them to the OS. Instead it tracks them in a freelist of free page IDs, which is rewritten in its entirety on every transaction commit. At normal scale, freelist maintenance is negligible. At Roblox’s scale, after months of accumulated Raft log writes, the freelist had grown pathologically.

The post-mortem provides the actual numbers, taken from a Consul server during the incident: the 4.2 GB Raft log store contained only 489 MB of actual data. The remaining 3.8 GB was empty space, tracked as free pages.

The freelist tracking those pages had grown to 7.8 MB, containing nearly a million free page IDs. For every Raft log append, with all the batching Consul applies, a write of 16 KB or less was triggering a rewrite of the entire 7.8 MB freelist to disk.

This is the pathology. Each transaction commit performed: a search of the million-entry freelist for free pages; an update to that freelist; serialisation of the entire 7.8 MB freelist to disk; and an fsync(2) whose cost was dominated by the size of the dirty page set, which was now dominated by the freelist itself.

The work scaled linearly with freelist length, and the freelist length grew with every snapshot the system performed to keep itself trim.

Raft, sitting on top of BoltDB, has timing assumptions. The leader replicates log entries to followers and must commit them durably.

When BoltDB commit latency entered the multi-second range, leaders could not durably persist log entries fast enough; followers timed out, started elections, and a new leader was chosen, which inherited the same BoltDB file, performed the same expensive freelist operations, became slow, lost its leadership in turn, and triggered another election.

The cluster was alive but unable to make progress: a Raft cluster trapped in a leader-flap loop, with each leader’s presence too brief to commit meaningful work.

This was an internal failure at the level of database page management, observed at the level of cluster leadership stability. The two layers were not connected in anyone’s mental model of the system. They were connected through a freelist data structure most engineers did not know existed.

The team’s first hypotheses, in order, were the ones the operator’s model suggested. They suspected degraded hardware and replaced a Consul node. Performance continued to suffer.

They suspected capacity, and replaced all the Consul nodes with new machines: 128 cores (up from 64) on faster NVME SSDs. As the post-mortem would later document, this made things worse: the new servers were dual-socket NUMA architectures, and the additional cores meant additional concurrent goroutines contending on the same Go channel in the streaming code path.

Cross-socket memory access added latency to operations that had been local on the old 64-core single-socket machines.

By 16:35 PDT on the 28th, concurrent users had dropped to 50% of normal. Subsequent attempts, resetting the cluster from a snapshot, blocking incoming traffic with iptables to bring it back under controlled conditions, reducing health-check frequency from 60 seconds to 10 minutes to give the cluster breathing room, all stabilised the system briefly and then returned it to the same 2-second KV write latency.

None of these interventions worked because none of them addressed the actual mechanism. The post-mortem is explicit: engineers did not identify the BoltDB freelist issue during the incident. HashiCorp engineers determined the root cause in the days after the outage ended.

This is where the observer problem becomes operationally devastating. Roblox’s monitoring infrastructure depended on Consul. When Consul was unhealthy, the dashboards that would have shown engineers what was happening inside Consul were themselves unable to report. The post-mortem describes this directly:

There was a circular dependency between our telemetry systems and Consul, which meant that when Consul was unhealthy, we lacked the telemetry data that would have made it easier for us to figure out what was wrong.

The diagnostic question the operator needed to ask, what is the actual state of the BoltDB file inside the affected Consul instances, required telemetry that the affected systems were supposed to provide. They could not.

Engineers were debugging a system whose internal state was now unobservable, against a failure mode buried two software layers deep in an open-source dependency that the affected engineers had not personally written.

The breakthrough came at 15:51 PDT on 30 October, roughly 50 hours after the outage began, when engineers disabled the streaming feature across all Consul systems. KV write latency immediately returned to 300 ms.

The Heisenbug-disguised streaming contention had been suppressed; the underlying BoltDB freelist problem was still there, manifesting as a “slow leader” symptom in which certain leaders inherited the worst of the freelist state.

The team pragmatically worked around it by preventing those leaders from staying elected, and continued the long process of repopulating caches and restarting services.

Total downtime: 73 hours, from 13:37 PDT on 28 October to 16:45 PDT on 31 October, when 100% of players were given access.

The trigger was the interaction between Consul streaming and BoltDB’s freelist; both bugs, both fixable, both ultimately fixed (the BoltDB freelist issue was resolved by migration to bbolt, the etcd-io fork of BoltDB, which uses a hashmap-based freelist).

The outage duration was a property of the observation surface. With working telemetry into Consul’s internal state, the freelist issue would have been visible in hours, not days. Without it, engineers were solving a Heisenbug with their eyes closed.

The lesson Roblox drew, encoded explicitly in their post-mortem and in the architectural changes that followed, was that observation surfaces must be independent of the systems they observe.

Telemetry must run on infrastructure that does not depend on the thing being measured. If the monitoring stack and the production stack share a common substrate, a failure in that substrate blinds both at once, and the operator is left to debug a fully dark system.

This is, in the language Part I used, an invariant at the boundary between the observation system and the production system: the observability stack must function independently of any system whose state it reports.

The invariant is rarely enforced. It is rarely even written down. It is one of the assumptions the operator does not know they have made until the day Consul stops responding and the dashboards go dark with it.

The operator’s simulation

Every operator, when debugging a system, is in fact debugging a model of the system that lives in their own head.

The model was built from architecture documents, code reading, prior incidents, and conversation with colleagues. The model is a simplification, by necessity, because the system is too complex to fit in one head.

The model is also wrong, in specific ways, and the operator does not know which specific ways until the model and the system diverge.

This determines whether an incident is resolved in twenty minutes or six hours. During an outage, the operator performs inference: symptoms are observed, hypotheses are generated from the model, tests of those hypotheses are designed and executed. The hypothesis space the operator can explore is bounded by the model the operator has.

If the failure mode lives outside the model, the operator cannot generate the hypothesis that would lead to it. They will iterate, increasingly desperately, on hypotheses inside the model, none of which fit the symptoms, until either someone with a better model joins the call or the system recovers on its own.

The Cloudflare incident from Part I is partly an instance of this. The first ninety minutes were spent on the hypothesis that the failure was an external DDoS attack, because the oscillation between good and bad states matched the signature of intermittent external pressure better than it matched the signature of anything Cloudflare’s engineers had a model for.

The model said “oscillation means external adversary.“ The reality was that oscillation meant gradual rollout against a five-minute regeneration cycle, but that failure mode was not in anyone’s model until the post-mortem.

Richard Cook’s How Complex Systems Fail (1998) names this directly: every operator’s view of the system is a practitioner-constructed simulation, assembled from training, experience, and the artefacts of past incidents.

The simulation diverges from reality continuously and silently. The role of the practitioner during an incident is to detect the divergence, update the simulation, and act on the updated version, all in real time, under pressure, with incomplete information.

The systems that recover quickly from incidents are not the systems with the best dashboards. They are the systems whose operators have the best simulations, and whose dashboards expose enough raw data that the simulation can be corrected in flight.

The discipline of seeing what is there

The compressed form of this essay, the operational counterpart to Part I’s diagnostic question, is also a question, asked of every observation surface in the system: what is this metric not telling me, and what would I look at if it were lying?

Every dashboard has a set of failure modes for which it is the right view, and a set of failure modes for which it is misleading. Both sets are large. The first is documented (it is the reason the dashboard was built). The second is almost never documented, because the failure modes it contains are, by definition, the ones nobody anticipated.

The discipline is to know, for every observation, what it is showing and what it is hiding.

The metric is an aggregate over what dimensions? Over what time window? With what sampling?

Does the observation surface share a substrate with the system it observes?

What would a failure that is invisible on this metric look like, and what other view would catch it?

When the metric and the user experience disagree, which one should be believed?

The answer to the last question is always: the user experience. The metric is a model. The user is in the data plane. The data plane is the system.

The discipline of observation-aware engineering is not to build dashboards that show everything; that is impossible, and the attempt produces dashboards that show nothing useful.

It is to know, for every dashboard, what it cannot show, to keep the raw event stream queryable for the moments when the dashboard and the system disagree, and to keep the observation stack architecturally independent of the systems it watches.

This is what separates the systems where the operator spends the first hour of an incident narrowing the hypothesis from the systems where the operator spends the first hour arguing about whether the dashboards are correct.

Part III will move from observation to action: what it takes to operate a system whose state you cannot fully see, whose feedback loops fight you, and whose composition you do not control. The technical name for this is control theory under uncertainty. The operational name is on-call.

One more thing…

The reason these failures keep happening is that engineers are trained to reason about the logical structure of their systems, not the physical dynamics of how those systems execute under load.

The same gap exists, in concentrated form, in GPU programming. CUDA correctness is necessary but not sufficient. Performance lives entirely in how memory traffic, warp scheduling, instruction issue, and inter-block synchronisation interact under realistic workloads: the same composition-and-observation problem, compressed into a single die.

I wrote a deep guide on CUDA from this perspective: not isolated tricks, but how to reason about the GPU as a coupled dynamical system whose performance regimes are as discontinuous as any distributed system’s.

Read the CUDA Guide on Gumroad

Intro

There is a version of distributed systems that exists in textbooks, RFCs, and the architecture diagrams that get drawn during the first month of a new project.

In that version, failures are discrete events: a node dies, a network partitions, a disk fills. Each event has a name. Each name has a mitigation. The mitigations compose. The composition is correct.

Then there is the version of distributed systems that exists in production at 03:47 UTC, when the on-call engineer is staring at a dashboard that shows everything green except for a customer-impact metric that has been climbing for nine minutes.

The runbook does not apply because it was written for the system that existed before the migration, the last engineer who understood the offending subsystem left the company seven months ago, and the only documentation is a Confluence page from 2023 that contradicts itself in the third paragraph.

This series is about the second version.

It is not about how to design distributed systems. There are good books for that. It is about what happens to those designs after they meet reality: after the load grows by a factor of fifty, after three reorgs change the ownership of half the services, after the configuration file that was supposed to be immutable acquires a small permissions change on a Tuesday morning in November.

It is about the failure modes that emerge not from broken components but from the interaction between working ones. It is about why debugging at scale is not a technical activity but an epistemic one.

And it is about the design decisions, often made years before the outage, that determine whether a system has a fighting chance when the failure arrives.

Five essays. Each stands alone. They share a thesis: the gap between how engineers reason about systems and how systems actually behave is not a knowledge problem. It is a structural property of complexity.

The faster you accept this, the better your systems will be.

Subscribe now

The Composition Problem

On Monday, 17 November 2025, an engineer at Cloudflare merged a change to a permissions policy on the company’s ClickHouse database clusters.

The change was part of a long-running effort to migrate distributed queries from a shared system account to per-user authentication, so that query limits and access grants could be evaluated at finer granularity. It was the right kind of change. Reviewed, staged, rolled out gradually across cluster nodes, exactly as a careful operator would do it.

At 11:05 UTC the following morning, the rollout reached a critical threshold. Twenty-three minutes later, the internet broke.

At 11:28 UTC, Cloudflare’s network, which fronts roughly 20% of the websites on the public internet, began returning HTTP 5xx errors at scale. ChatGPT failed. X failed. Spotify, Discord, Canva, Figma, 1Password, Trello.

The outage lasted until 14:30 UTC for core traffic, with full restoration at 17:06 UTC. Matthew Prince, Cloudflare’s CEO, would later describe it as the worst outage since 2019. Estimated revenue loss across the affected ecosystem ran into the hundreds of millions of dollars.

The chain of causation, once it was understood, fits in a paragraph.

Cloudflare’s Bot Management module runs inside its core proxy (a system called FL, with a newer version FL2). The module scores every request as bot-or-human using a machine-learning model.

That model takes as input a “feature configuration file”, a list of per-request features, which is regenerated every five minutes by a query against a ClickHouse cluster. The regeneration query reads from system.columns, ClickHouse’s metadata table:

SELECT name, type
FROM system.columns
WHERE table = 'http_requests_features'
ORDER BY name;

Note what is not in this query: a filter on the database name. The query implicitly assumed that system.columns would only return columns from the default database, because before the permissions migration users only had visibility into default.

ClickHouse’s distributed table engine actually stores shards in an underlying physical schema named r0. The new permissions policy granted explicit access to r0. After the change, the same query returned columns from both default and r0, roughly doubling the row count.

That row count was used directly to construct the feature file. The file had previously contained around 60 features. It now contained more than 200.

Downstream, in the Rust code that loaded the file into the FL2 proxy, there was a preallocated array sized for a hard ceiling of exactly 200 features: a performance optimisation so that runtime feature lookups would never allocate.

When the oversized file arrived, the load path returned Err(_). The calling code, written under the assumption that this could not happen, called .unwrap() on the Result.

The worker thread panicked with the now-public string:

thread fl2_worker_thread panicked: called Result::unwrap() on an Err value

Every request routed through that worker returned 5xx.

The damage was amplified by a second-order property. ClickHouse was being rolled out gradually, so for nearly an hour only some cluster nodes returned the duplicated result.

The feature file regenerated every five minutes, and whether the run hit an upgraded node or a non-upgraded node was effectively random.

The file therefore alternated, every five minutes, between “good” and “bad,” and the proxy fleet oscillated between recovery and failure on a five-minute cycle.

From the dashboards, this looked exactly like an active DDoS attack, an external adversary probing the network with intermittent pressure.

The incident commander spent the first two hours of the outage investigating that hypothesis, because the signature of the failure mimicked a known threat.

Read this again. Notice what is not in it.

There was no bug in the database. The new permissions behaviour was correct ClickHouse semantics. There was no bug in the query; it executed exactly as written.

There was no bug in the feature file format; it stored what it was given. There was no bug in the Rust proxy, its bounds check correctly refused to process malformed input rather than corrupting state.

There also was no bug in the deployment process, gradual rollout to a database cluster is exactly how you mitigate rollout risk. Every component, examined in isolation, behaved as designed, as documented, as code-reviewed.

The outage existed in the spaces between the components. It existed in an unwritten assumption, that the cardinality of the metadata query was bounded by the schema layout.

It was present in the gap between the team that owned the permissions migration and the team that owned the feature pipeline.

It even existed in the asymmetry between data Cloudflare treated as “trusted” (internally generated configuration) and data it treated as “untrusted” (everything from outside).

The failure was not a property of any component. It was a property of the system.

This is the central, uncomfortable fact about distributed systems: their failure modes are not documented because they cannot be documented.

They emerge from the composition of components, and the space of possible compositions grows faster than anyone can enumerate it.

Subscribe now

Why decomposition breaks down

Software engineering is almost entirely built on decomposition.

You take a hard problem, split it into smaller problems, solve each, and compose the solutions. The discipline assumes (implicitly, almost religiously) that the behaviour of the whole can be derived from the behaviour of the parts.

This is the foundation of modular design, encapsulation, microservices, contracts, type systems. It is what allows ten thousand engineers to build a system no one of them understands in full.

The assumption is wrong, or rather: it holds only within a regime, and the regime ends somewhere around the scale where a system has enough components, enough state, and enough concurrency that the interactions between components become a richer source of behaviour than the components themselves.

The formal version of this argument is older than computer science. Herbert Simon, in The Architecture of Complexity (Proc. Am. Phil. Soc., 1962), distinguished between decomposable and nearly-decomposable systems.

In a decomposable system, interactions between subsystems are negligible compared to interactions within them, and the whole behaves like the sum of independent parts.

In a nearly-decomposable system, this is approximately true on short timescales but not on long ones, the weak inter-subsystem couplings accumulate into qualitatively different behaviour.

Simon’s claim, which has held up for sixty years across biology, economics, and engineering, is that all real systems of significant size are nearly-decomposable, not decomposable.

Distributed systems are an extreme case. The components have clean interfaces and look decomposable on a diagram.

But the interactions are mediated by shared resources (networks, clocks, storage, control planes) and those shared resources transmit perturbations between components in ways the diagram does not show.

A change in one component changes the load profile on the shared network, which changes the queueing behaviour at a different component, which changes the timing of its responses, which changes the retry behaviour of yet another component. The composition is opaque because the couplings are invisible.

Distributed systems theory has known a version of this for forty years. Fischer, Lynch, and Paterson (JACM 1985) proved that consensus is impossible in a purely asynchronous system with even one faulty process; a result that, properly understood, is not about consensus algorithms but about the impossibility of producing globally consistent system behaviour from locally correct components under partial failure.

Brewer’s CAP conjecture (PODC 2000) and the Gilbert-Lynch proof (ACM SIGACT 2002) formalised the same point at the level of state.

Lamport’s “Time, Clocks, and the Ordering of Events in a Distributed System” (CACM 1978) showed that there is no observer-independent simultaneity in a distributed system without explicit synchronisation, meaning every “global view” of the system is a stitched-together fiction.

The classical literature focused on discrete failures: a node dies, a message is lost, a clock drifts. The modern failures are stranger.

They are failures of coupling: moments when two pieces of working software, communicating through an interface both implement correctly, produce a behaviour neither would produce alone.

The Cloudflare incident is one. The DynamoDB DNS race condition that took down AWS US-EAST-1 on 19–20 October 2025 is a more elaborate example of the same pattern, and it is worth reconstructing mechanically because it shows how thoroughly the composition can betray its components.

Subscribe now

The AWS DynamoDB cascade, mechanically

DynamoDB’s regional endpoint, dynamodb.us-east-1.amazonaws.com, is served by an internal DNS management system that exists because DynamoDB runs on hundreds of thousands of load balancers, and the DNS records pointing clients at those load balancers must be updated continuously as capacity is added, removed, and rebalanced.

The system has two logical components. The DNS Planner monitors load-balancer health and produces “DNS plans”, versioned snapshots of which load balancers receive which fraction of regional traffic.

The DNS Enactor reads plans and applies them to Route 53, AWS’s DNS service. For availability, three Enactors run in parallel, one per availability zone. They operate concurrently and independently; no distributed lock, no leader election, no coordination protocol.

The system was designed this way deliberately, so a single Enactor crashing mid-run would not stall propagation; the other two would simply pick up subsequent plans and continue.

To prevent stale plans from overwriting newer ones, each Enactor performs a freshness check before applying a plan.

To prevent unbounded growth of historical plans, each Enactor also performs a cleanup pass after applying its current plan, deleting plans significantly older than the current one.

The freshness check happens once, at the start of the application phase. The cleanup happens once, at the end.

This is, again, the kind of design that gets praised in code review. Independent. Stateless. Fault-tolerant. Each component does one well-bounded job.

Now consider what actually happened. At 23:48 PDT on 19 October (06:48 UTC on 20 October), Enactor A read plan #N−1 from the Planner and began applying it to Route 53.

For reasons AWS’s post-mortem describes as “unusual delays”, likely network-mediated queueing inside Route 53’s control plane, Enactor A’s update run took longer than normal.

In the meantime, the Planner produced plan #N. Enactor B picked up plan #N, performed its freshness check (newer than the currently applied plan: pass), and began its own update run.

Enactor B finished first, applying #N to Route 53. It then began its cleanup pass, scanning for plans significantly older than #N and deleting them.

By the time Enactor A finished its delayed run and went to apply the last few records of plan #N−1, Enactor B had already applied #N to those same records.

Enactor A’s freshness check, made at the start of its run, had not detected this; the check was made when #N−1 was still the freshest plan, and that result was now stale. Enactor A overwrote those records with #N−1.

Now Enactor B’s cleanup pass arrived at plan #N−1. By Enactor B’s bookkeeping, #N−1 was significantly older than #N. Enactor B deleted plan #N−1. But Enactor A had just applied #N−1 to the regional endpoint records.

The records now pointed at a plan that did not exist. Route 53 dutifully served what it had: an empty answer set for dynamodb.us-east-1.amazonaws.com.

This is the worst possible DNS response. It is not NXDOMAIN, which clients treat as transient and retry. It is NOERROR with an empty ANSWER section; semantically “this name exists, intentionally, with zero addresses.” Compliant clients stop. There is no answer to retry.

Within seconds, every system inside and outside AWS that wanted to talk to DynamoDB in us-east-1 began failing to resolve its address. From the DynamoDB control plane’s view, the service was healthy: load balancers up, storage reachable, request handlers idle.

From Route 53’s view, the service was healthy: DNS was returning valid authoritative responses. From clients’ view, the service had ceased to exist. Three different frames of reference, three different “states” of the same service, all simultaneously true within their own frame. The mismatch between them was the outage.

It took manual intervention from on-call engineers to identify the empty record, repair Route 53 by hand, and re-enable normal automation. DynamoDB DNS recovered in approximately three hours.

The cascade that followed lasted ten more hours, and is the second composition failure embedded inside the first. EC2’s DropletWorkflow Manager (DWFM), the system that maintains operational leases on the physical hypervisors hosting customer EC2 instances, stores its lease state in DynamoDB.

While DynamoDB was unreachable, DWFM could not renew leases. Existing leases expired silently. When DynamoDB recovered, DWFM woke up to discover that essentially every hypervisor in the region needed a fresh lease, and tried to issue them all at once.

The lease-renewal subsystem entered what AWS’s post-mortem calls “congestive collapse”, a regime where throughput of useful work approaches zero because the system is spending all its time servicing retries of work that has already timed out.

Network Load Balancer health checks began failing en masse. New EC2 launches were impossible. The region was effectively down for production workloads until late that evening. Every design decision in this chain was defensible. Three Enactors instead of one, for availability.

Freshness check, to prevent old plans winning. Cleanup pass, to prevent unbounded growth. No distributed lock, to avoid coordination overhead and tolerate Enactor failures. DWFM storing state in DynamoDB, because what else would you use for a high-availability lease manager.

Each decision is the textbook answer to a specific risk.

The composition of all those textbook answers produced fifteen hours of regional unavailability and an industry-wide impact measured in hundreds of millions of dollars.

Share

Why documentation cannot close the gap

The instinct, after this kind of incident, is to write better documentation. Add the failure mode to the runbook. Update the architecture diagram. Note the implicit assumption in a comment. Surely, next time, we will know.

We will not. The reason is not laziness; it is combinatorial.

Consider a system with N components, each with a small number of internal states, dependencies, and inputs. The number of pairwise interactions grows as O(N²).

The number of trajectories (sequences of states the system can traverse) grows much faster: for any reasonable model of state and concurrency, at least exponential in N. By the time N is in the low thousands (a serious production system), the trajectory space is unbounded for practical purposes.

Documentation is a linear medium. It can describe a finite number of states, interactions, and failure modes. The space of actual failure modes is not finite in any meaningful sense.

What documentation actually captures, in practice, is the failure modes that have already happened; the ones recovered from, written up, discussed in architecture review.

This is useful, but it is fundamentally backward-looking. The next outage is, almost by definition, the one not yet documented. It lives in some currently-undocumented region of the trajectory space, which the system will enter for the first time when some perturbation pushes it there.

This is not an indictment of documentation. Runbooks save lives. Post-mortems compound institutional knowledge. The point is that no quantity of documentation, however thorough, can close the gap between the system as designed and the system as composed.

The gap is structural. It widens deeply with scale.

The pattern beneath the patterns

If you read enough post-mortems (Dan Luu’s catalogue on GitHub remains the best free education in this material) a pattern emerges.

The triggers vary wildly: a permissions change, a DNS update, a config push, a deploy, a hardware failure, a thundering herd. The shape of the failure is often the same.

Nathan Bronson and his collaborators, in a 2021 HotOS paper, gave this shape a name: metastable failure. The framing has become foundational, and is worth restating precisely because it is the closest the field has come to a formal theory of why composition produces outages.

A metastable failure occurs in an open system with an uncontrolled load source. The system has at least two stable operating regimes: a stable regime, in which a transient perturbation decays back to equilibrium, and a metastable failure regime.

In that case, the system is functioning (consuming CPU, processing messages, producing output) but its useful throughput, what Bronson precisely terms with the word goodput, has collapsed.

The system transitions between regimes via a trigger: a load spike, a deploy, a partial failure, a configuration change.

What keeps the system in the failure regime, even after the trigger is removed, is a sustaining effect: a positive feedback loop, usually involving work amplification, in which the system’s response to its own degraded state increases the load on itself further.

The canonical example, paraphrased from the paper:

A web tier calls a database tier through a connection pool. Database latency is normally well below the client’s request timeout. A brief perturbation, like a network blip, a slow GC pause, causes some requests to exceed the timeout.

The client retries. The retry is a new request, added to the existing load. Database queue depths grow. Latency increases, pushing more requests past the timeout. More retries fire. Each timed-out request still consumed full database work to compute its answer, but no client ever saw it; that work was wasted.

The system is now processing 3× its normal request volume (originals plus retries), succeeding in completing them all, but every client is timing out before the answer arrives. Goodput is zero. Throughput is at saturation. The trigger (the original network blip) is long gone. The retry storm is sustaining the failure regime on its own.

The key insight is that the root cause of a metastable failure is the sustaining loop, not the trigger. Triggers are infinitely various and mostly cannot be prevented.

Sustaining loops are finite and identifiable, and if you eliminate them, the same trigger fails to produce the same outcome.

A follow-up paper, Metastable Failures in the Wild (Huang et al., OSDI 2022), examined 22 publicly disclosed incidents at 11 major organisations and concluded that at least 4 of the previous 15 major AWS outages fit the metastable pattern.

The October 2025 DynamoDB incident makes 5. The EC2 cascade after DynamoDB recovered is the metastable pattern in textbook form: the trigger (DynamoDB DNS being empty) was resolved in three hours; the sustaining loop (every hypervisor in the region simultaneously demanding lease renewal from a system that could not handle the surge) took ten more hours to break, and only broke when AWS manually rate-limited the work.

Marc Brooker, a principal engineer at AWS who has written extensively on this material, has pointed out that the appropriate intellectual framework here is not algorithms-and-data-structures but control theory and dynamical systems.

A metastable failure is, in dynamical-systems terms, a system with two stable attractors, where the perturbation required to push the system from the desirable attractor into the undesirable one is much smaller than the perturbation required to push it back.

The state-space geometry is asymmetric. Most production engineers have never thought about their systems this way, because computer science is taught around discrete models. The systems are continuous and dynamical, whether we model them that way or not.

Subscribe now

Invariants and the cardinality contract

The implication is not that distributed systems are unbuildable. They obviously are.

The true implication is that the mental model under which most distributed systems get built (components compose, contracts compose, correctness composes) is wrong in a way that matters for production behaviour.

The discipline that replaces this mental model is the explicit enforcement of invariants at every component boundary, including internal ones.

An invariant, in this context, is a property of a value that the consumer’s correctness depends on, but that the producer is not contractually obligated to maintain. The Cloudflare feature file had at least three such invariants, none enforced by any check at the boundary:

1. A cardinality bound. The Rust consumer required n_features ≤ 200. The ClickHouse query had no LIMIT, no WHERE on database, and no schema constraint preventing growth.

2. A schema invariant. The consumer assumed columns came from default only. The query implicitly assumed the same via the permissions model. Neither stated the invariant in code.

3. A monotonicity invariant. A doubling of feature count between two consecutive runs is, on its face, anomalous. No alarm fired on that delta.

Each invariant was true for years. Each became false silently when an upstream change reshaped the world. The boundary between producer and consumer had no formal contract; the contract lived in the heads of engineers, some of whom had left the company.

The discipline that prevents this is not “validation” in the loose sense. It is the explicit, in-code, enforced declaration of every cardinality, ordering, schema, and freshness constraint that the consumer relies on, with explicit handling of violation: typically degradation to last-known-good rather than panic.

The Rust idiom for this is the difference between .unwrap() and explicit pattern matching on Result; the operational idiom is the difference between trusting upstream data and treating every input as adversarial regardless of source.

The cost of the former is a few additional lines per consumer boundary. The cost of the latter is, occasionally, six hours of global downtime.

Sustaining loops and characteristic metrics

The second formal property the Cloudflare and DynamoDB incidents share is the presence of sustaining loops; control loops whose response to system degradation increases the load on the system rather than decreasing it.

The discipline for finding these before they fire is to enumerate every feedback loop in the system and classify each one’s stability properties.

A feedback loop is stable if, when perturbed from equilibrium by a small amount ε, it returns to equilibrium with error decaying as some function f(t,ε) that approaches zero.

A feedback loop is sustaining if the same perturbation produces error that grows or stays bounded away from zero.

The distinction is mathematically standard (Lyapunov stability) but is almost never applied to production systems, because engineers do not model their systems as dynamical systems.

The catalogue of loops in any non-trivial production system:

• retry policies (timeout → retry → load → timeout amplification);

• autoscaling (latency → scale-up → cold-start latency → more scale-up);

• lease renewal (load → renewal delay → lease expiry → mass renewal storm);

• connection pooling (failure → reconnect → handshake load → failure);

• cache warming (cold cache → DB load → DB slow → cache cannot warm);

• health checks (slow response → marked unhealthy → traffic shifted to fewer hosts → those hosts slower).

Each is a control loop. Each can be classified. The classification is rarely written down.

The observability counterpart of this classification is what Bronson calls characteristic metrics: observations of the loop state itself, not of the loop’s inputs or outputs.

Queue depth is a loop-state observable; request rate is not. Retry rate is a loop-state observable; error rate is not. Lease renewal latency is a loop-state observable; lease expiry rate is not.

The relationship between loop-state metrics and incident causality is direct: when a sustaining loop activates, its characteristic metric crosses out of its historical operating envelope before the user-facing symptom appears.

Instrumenting characteristic metrics is the difference between detecting a metastable failure during its inflation phase (when mitigation is cheap) and detecting it after it has saturated (when mitigation requires load-shedding the user-facing service).

Subscribe now

The diagnostic question

The compressed form of the entire discipline reduces to a single question, asked of every component boundary in the system: what am I assuming about my input that is not enforced by a check in this code?

Every such unenforced assumption is a future incident. The space of unenforced assumptions is large but finite, and it can be enumerated. Most engineering organisations have never done this enumeration.

The ones that have produce systems that fail in less catastrophic ways; not because they fail less often, but because the failures that occur are caught at the boundary where the assumption was violated, rather than three layers downstream after corruption has propagated.

The system you have is not the system you designed. The system you have is the composition. The composition is opaque, and the opacity is permanent, but the opacity at every individual boundary is not permanent.

Each boundary is a place where assumptions can be made explicit and enforced. The discipline of composition-aware engineering is not to make the whole transparent.

It is to make every boundary honest about what it requires from its neighbours, and to refuse to operate when those requirements are not met.

This is what separates the systems that fail loudly at the seams from the systems that fail catastrophically in the centre.

One more thing…

Modern systems rarely fail because of a single broken component.

They fail because interactions between correct components create behaviours nobody explicitly designed for. The same thing happens in high-performance GPU systems.

Most CUDA optimisation is not about isolated tricks. It is about understanding how kernels, memory hierarchies, scheduling, communication, and throughput constraints interact under load.

I wrote a deep guide on CUDA from exactly this perspective: systems-level performance engineering, bottlenecks, hidden coupling, and why many “optimisations” simply move the problem elsewhere.

Read the CUDA Guide on Gumroad

Intro

For the past few days we have been quiet here. Not because the newsletter slowed down, but because we were building something underneath it.

Today we are publishing what came out of that work: CUDA Mastery 2026, The Definitive Engineer’s Reference for Hopper, Blackwell, and Beyond.

Twenty-seven chapters, five appendices, fact-checked end to end against NVIDIA’s own documentation, the PTX ISA 8.7, and primary architecture whitepapers.

It covers CUDA Toolkit 13.0, 13.1, and 13.2, compute capabilities 7.5 through 12.1, WMMA, WGMMA, UMMA (tcgen05), TMA, thread block clusters, Tensor Memory, CUDA Tile and cuTile Python, CUTLASS 4 / CuTe, NCCL 2.30, and Nsight 2025.4.

It is on Gumroad. The price is $89. If you have been following The Software Frontier, you already know whether this is for you. The rest of this post is for everyone who is on the fence.

Subscribe now

Why we wrote this

There is a strange gap in CUDA literature.

On one side, you have the official NVIDIA programming guide: dense, accurate, and written for people who already know what they are looking for.

On the other side, you have an ocean of blog posts and YouTube tutorials that stop at vector addition and matrix multiplication, repeating the same surface level explanations of threads, blocks, and grids.

What sits in the middle, the part that actually matters when you are writing production code or debugging a kernel that runs at 30 percent of peak, is mostly missing.

Or rather, it exists, but it is scattered across NVIDIA whitepapers, GTC talks from 2018, PTX ISA documentation, decompiled SASS dumps, the Hopper and Blackwell Tuning Guides, the Microbenchmarking Hopper and Microbenchmarking Blackwell arXiv papers, the CUTLASS source, and Stack Overflow threads from people who clearly know more than they are saying.

We have been reading and writing about this gap for months on the newsletter. The articles on the A100 memory hierarchy, on cp.async semantics, on scoreboard mechanics, on the submission pipeline, all of them came from the same frustration. Every time we wanted to explain something properly, we had to do the archaeology ourselves.

So we decided to do the archaeology once, in a single document, and structure it the way we wish someone had structured it for us when we started.

Share

What is in the guide

Twenty-seven chapters across eleven parts, plus five appendices. Four chapters were rewritten end-to-end at handbook depth. Those are the PREMIUM chapters: the SM, the memory system, tensor cores, and the SGEMM walkthrough. The rest of the structure looks like this.

Foundations. The GPU as a throughput machine, the CUDA programming model, and the memory hierarchy at a glance. This is the vocabulary layer. A senior engineer can skim it in an afternoon; a new graduate can use it as their entry point and grow into the rest of the book.

The Streaming Multiprocessor in mechanical detail. The SM is the unit of concurrency, the unit of resource accounting, and the unit at which every meaningful CUDA performance argument must eventually be made.

We walk through the four near-independent partitions, the operand collector and its bank conflicts, the short and long scoreboards, the quantitative arithmetic of latency hiding via Little’s law, the full Nsight Compute stall taxonomy, and the structural deltas across Volta, Turing, Ampere, Ada, Hopper, and Blackwell.

The chapter ends with an end-to-end walkthrough of a single warp executing a wgmma.mma_async on a Hopper SM, stage by stage.

The memory system mechanically. The 32-byte sector model. The L1TEX path and every cache modifier you can attach to a global load. The L2 partitioning on H100 and the access policy window. HBM3 in three numbers and why the practical roofline is 70 to 85 percent of the headline.

Shared memory banks, swizzle modes, and the descriptor-encoded layout that WGMMA actually expects. cp.async mechanics on Ampere. TMA on Hopper and Blackwell, including descriptors, transaction barriers, phase parity, and cluster multicast. The mbarrier family and how warp-specialized GEMM mainloops use it.

Synchronization and concurrency. The CUDA memory model with its scopes and orderings. Cooperative Groups including cluster.sync on Hopper and beyond. Streams, events, and CUDA Graphs for launch-overhead amortization in inference and physics workloads.

Tensor cores mechanically. The hardware origin of the unit, the generation-by-generation shape and precision progression, the per-lane fragment ownership for mma.sync, the bit-level layout of the WGMMA matrix descriptor, and the structural transition to UMMA with the accumulator living in Tensor Memory.

A full section on the new numerical formats: FP6, FP4, and the MX wrapper that makes FP4 inference near-lossless on trained transformer weights.

Modern hardware. A Hopper deep dive on sm_90 / sm_90a. A Blackwell deep dive on sm_100 / sm_100a / sm_120. A chapter on Blackwell Ultra (B300, compute capability 10.3) and the trajectory toward Rubin.

Performance engineering. The roofline model in practice, with the second roofline for shared-memory bandwidth on tile-based kernels. Profiling with Nsight Systems and Nsight Compute, including the four-section workflow and the new tile-kernel statistics added in CUDA 13.1. Numerics and reproducibility, including the TF32 trap that silently downgrades FP32 GEMMs.

Multi-GPU and distributed. NVLink 5 and the NVL72 domain. SHARP v4 in-network reductions. NCCL internals across Ring, Tree, NVLS, and PAT. NVSHMEM and PGAS for sparse all-to-all in MoE training.

Libraries and toolchain. cuBLAS, cuBLASLt epilogue fusion, cuDNN, cuFFT, cuSPARSE. CUTLASS 4 and CuTe for hand-written tensor-core mainloops. CCCL (Thrust + CUB + libcu++). A full chapter on CUDA Tile and cuTile Python, the largest single addition to the CUDA programming model since cooperative groups. A chapter on nvcc, PTX, SASS, the fatbinary, and inline PTX as an escape hatch.

Capstone kernels. The SGEMM walkthrough from v1 to v6, with numbers. Most treatments stop at register tiling and gesture vaguely at “tensor cores make it faster.”

Ours follows the bottleneck through six versions on a 4096³ FP32 problem on H100 SXM5, names the architectural feature that breaks each ceiling, and gives the analytical bound. v1 reaches 0.2 percent of peak. v6, on Blackwell with UMMA + TMEM at FP8, reaches 88 to 95 percent of peak.

The chapter exists to teach what every transition costs and what it buys. Reductions and scans with single-pass decoupled lookback. Flash Attention 2, 3, and 4 / 5 including the online softmax derivation and the WGMMA / UMMA mainloop. Sort, hash, and graph primitives built on CUB.

Appendices. Compute capability quick reference. Architecture spec sheet from A100 through B300 with verified numbers from each part’s datasheet. PTX quick reference. Glossary. And a full bibliography of the primary sources every claim in the book was checked against.

Every claim in the guide has been fact checked against primary sources. Where we had to infer something from SASS or from the behavior of the hardware rather than from a published spec, we say so explicitly.

Subscribe now

What was truly missing

Maxwell, Pascal, and Volta offline-compilation material was retired, in line with CUDA 13.0 dropping pre-Turing offline compilation in August 2025. The tensor-core chapter was rewritten around UMMA, which supersedes Hopper’s wgmma.mma_async.

New material on CUDA Tile and cuTile Python, both introduced in CUDA 13.1 in December 2025 and extended in 13.2. New material on Tensor Memory.

The four PREMIUM chapters are new from the ground up. Numbers verified against NVIDIA’s Hopper, Blackwell, and Blackwell Ultra public datasheets at print time.

Who this is for

If you are writing CUDA professionally, in HPC, in ML systems, in inference engines, or in any context where kernel performance is part of your job, this guide is calibrated for you.

If you are a senior engineer transitioning into GPU work and you want one document that takes you from competent to dangerous without 200 hours of fragmented reading, this is the document.

If you are deep into the PTX weeds already, writing your own warp-specialized WGMMA mainloops and tuning CuTe layouts, you probably know a lot of what is in here.

You will still find the SM and memory chapters useful as a reference, and the SGEMM walkthrough is one of the few places where the v5-to-v6 transition is laid out in full. But we would not pretend to teach you something you do not already know.

If you are completely new to CUDA, with no parallel programming background, this is not the right starting point.

The guide assumes you can read C++ and that you have at least written a few kernels before. We would not want you to spend $89 and feel lost on chapter four.

Subscribe now

Why $89

Because the alternative is reading what we read, in the order we read it, over the same number of months.

We are not pricing this against tutorials. We are pricing it against the time of an engineer who bills somewhere between $80 and $200 an hour and needs to be productive on Hopper and Blackwell GPU code by next quarter.

If the guide saves you a single afternoon of debugging a kernel that turns out to be limited by operand collector bank conflicts that no public documentation describes, it has paid for itself.

There is no DRM, no expiration, no upsell. You buy the PDF, you own it. As future architectures change material details, we will publish updates to buyers at no additional cost.

The next edition is already scheduled to cover Rubin when its public specifications stabilize.

What happens next

The newsletter continues. The Mastering CUDA series is not over, and there are several articles already in draft on topics that did not fit cleanly into the guide.

If you buy the guide and you have feedback, send it. We read every email. The first revision is going out within thirty days based on what readers tell us, and the people who bought early get it first.

You can find the guide here: CUDA Mastery

Thank you for reading. Thank you for being here while this was being built. The next article goes out as scheduled.

Lorenzo and Lorenzo

Where Part IX Left Us

Part IX ended with a provocation dressed as a summary. Training, we said, is a one-time cost. Inference is the workload that runs forever.

That sentence deserves to be interrogated before we accept it as a frame, because it contains a hidden asymmetry that shapes everything that follows.

Training a frontier model is an event: it happens once, or perhaps a handful of times with different hyperparameters, and then it stops. The cost is large and bounded.

Inference is a process: it happens billions of times per day, across hardware that may or may not resemble the training cluster, under latency constraints that the training job never had to respect, serving users who have no patience for pipeline bubbles and no interest in MFU.

The engineering discipline of inference optimization is therefore a different subject from the engineering discipline of training optimization, not merely a scaled-down version of it. The bottlenecks are different in kind. The metrics are different. The vocabulary is different. The hardware choices are sometimes deliberately different.

But the physics is the same, because physics does not have a training mode and an inference mode. Compute and memory bandwidth are always the two resources, and every optimization in this space is, at root, a claim about which of the two you are spending and whether you are spending it wisely.

What we will do in this part is work through the inference problem with the same level of precision we brought to the training problem.

We will derive the arithmetic of autoregressive decoding from first principles, establish exactly why the decode phase of transformer inference is memory-bandwidth-bound by construction, explain what that means for hardware selection and batching strategy, and then examine the tools that practitioners have developed to recover the compute utilization that the memory-bound regime takes away.

We will go into detail that most treatments of this subject avoid, because the details are where the engineering actually lives.

Subscribe now

The two phases of transformer inference

Transformer inference for a generative model consists of two distinct phases that are so different in their computational character that they might as well be different workloads running on different hardware.

The prefill phase processes the input prompt. Given a prompt of length S tokens, the model performs a forward pass over all S tokens simultaneously.

This is a dense matrix multiply of shape [S, d_model] against the weight matrices, which is computationally equivalent to a training forward pass on a batch of S examples, with the important difference that no gradient computation happens.

The arithmetic intensity of prefill is high: the GEMM is large, the compute-to-memory ratio is favorable, and for sufficiently long prompts, prefill saturates tensor core utilization. Prefill is compute-bound.

The decode phase generates the output, one token at a time. At each step, the model processes a single new token and uses the key-value (KV) cache, which stores the key and value projections for all previously seen tokens, to compute attention over the full context without recomputing those projections.

The new token produces one row of the Q matrix, one row of the K matrix (appended to the KV cache), one row of the V matrix (also appended), and one output token.

The matrix multiply that dominates decode is therefore a matrix-vector product: a single vector of shape [1, d_model] multiplied against weight matrices of shape [d_model, d_model].

For an H100 at 494 TFLOP/s peak BF16, and a weight matrix that requires 2 × d_model² bytes to read from HBM (one load), the arithmetic intensity of this operation is:

flops = 2 × d_model² bytes = 2 × d_model² arithmetic intensity = 1 FLOP/byte

The H100’s ridge point, the arithmetic intensity at which the machine transitions from memory-bandwidth-bound to compute-bound, is approximately 494 TFLOP/s divided by 3.35 TB/s HBM3 bandwidth, which equals roughly 147 FLOP/byte.

Single-token decode has an arithmetic intensity of 1 FLOP/byte. The ridge point is at 147 FLOP/byte.

The gap is not a small inefficiency to be engineered away. It is two orders of magnitude. It is structural.

A matrix-vector product with batch size 1 will always be memory-bandwidth-bound on any hardware where compute throughput scales faster than memory bandwidth, which is every piece of hardware available today and likely every piece available for the next several years.

The H100’s tensor cores sit at 99.3% utilization waiting for data that cannot arrive fast enough. This is the inference problem, stated precisely.

Subscribe now

What the arithmetic intensity gap actually costs

Before we can appreciate why batching is not a simple fix, we need to quantify what the gap costs in concrete terms.

Consider the weight matrices of a 70B parameter model. They occupy 140 GB in BF16. To generate a single output token, the decode phase must read essentially all of the weight matrices from HBM: every attention projection, every MLP layer, every embedding lookup.

(The KV cache is also read, but its size is proportional to context length and sequence position, not model size.) At the H100’s HBM3 bandwidth of 3.35 TB/s, reading 140 GB takes approximately 42 milliseconds.

In 42 milliseconds, a single token is produced. That is approximately 24 tokens per second per H100 for a 70B model at batch size 1.

Now read that sentence again: 24 tokens per second per H100, a machine that costs tens of thousands of dollars and can perform 494 trillion floating-point operations per second, is producing tokens at roughly the rate that a person reads them.

The 494 TFLOP/s are not being used. The H100 is acting as a very expensive, very fast HBM3 reader. The silicon that took years to design and billions of dollars to fabricate is waiting for data.

This is the central pathology of autoregressive decode, and it motivates every technique we will discuss in this part.

Batching as arithmetic intensity recovery

The solution that every inference practitioner reaches for first is batching: if you run decode for multiple requests simultaneously, the weight reads are shared across the batch, and the arithmetic intensity increases proportionally.

The arithmetic is clean. For a batch of B requests, the matrix multiply in decode is no longer a matrix-vector product but a matrix-matrix product: [B, d_model] × [d_model, d_model].

The flop count scales as B × 2 × d_model², while the bytes for the weight matrix remain 2 × d_model² (the weights are read once, regardless of batch size). Arithmetic intensity is now B FLOP/byte.

To reach the ridge point at 147 FLOP/byte, you need a batch of 147 requests running simultaneously on the same H100. At batch size 147, the tensor cores begin to saturate and further increasing the batch does not change the arithmetic intensity (you are now compute-bound, and more requests means more total compute, not more memory reads per unit time).

The batch size at which you saturate the machine is the target operating point for maximum throughput per GPU. Everything below this point is wasted hardware.

But batch size is not free. Each request in the batch has its own KV cache, and the KV cache size is proportional to the sequence length of that request. For a context length of 8192 tokens, a model with 80 layers, 8 KV heads, a head dimension of 128, and BF16 storage, the KV cache size per request is:

8192 × 80 × 2 × 8 × 128 × 2 bytes = 26,843,545,600 bytes ≈ 25 GB

Twenty-five gigabytes per request, on a GPU with 80 GB of HBM. You can serve at most three concurrent requests at 8192-token context length before HBM is exhausted and you cannot increase the batch further.

The tension is fundamental: to achieve good arithmetic intensity you need large batches, but large batches require large KV caches, and large KV caches consume the memory that large batches require.

This is the central resource allocation problem of transformer inference, and it is more constrained than it appears, because KV cache memory is not static. It grows with sequence length.

A request that has generated 100 tokens has a small KV cache; the same request after generating 4000 tokens has a KV cache that is 40× larger. The memory footprint of the batch changes continuously as generation proceeds.

Subscribe now

Continuous batching and the end of static padding

The naive approach to batched inference is static batching: collect B requests, pad them all to the same sequence length, run them as a batch, return all B results when the longest sequence finishes.

Static batching is deeply inefficient. Consider a batch of 8 requests where one request will generate 2000 tokens and the others will generate 20 tokens each.

After the short requests finish at step 20, 7 of the 8 slots in the batch are empty, but the batch continues until step 2000 to service the one long request. The GPU is running at 1/8 occupancy for 99% of the wallclock time.

Continuous batching (also called iteration-level batching or in-flight batching), implemented in systems like vLLM, Orca, and TensorRT-LLM, solves this by removing the assumption that all requests in a batch start and finish together.

Instead, the batch is managed at the per-decoding-step level: at each step, the set of active requests is the set of requests that have not yet finished and for which memory is available.

When a request completes (generates a stop token or reaches the maximum length), its slot in the batch is immediately freed. A new waiting request is inserted into the freed slot and begins generating from its first decode step. There is no waiting for the batch to drain. The batch is always as full as memory permits.

The implementation requires that the CUDA kernels for attention and the MLP can handle variable-length sequences within a single kernel invocation, which is non-trivial because standard GEMM implementations assume a fixed batch dimension.

Paged attention (discussed shortly) is the memory management technique that makes this practical; the PagedAttention kernel from vLLM and the FlashAttention variants for variable-length sequences are the implementations that make it fast.

Continuous batching does not increase the maximum batch size (which is still limited by KV cache memory). What it does is ensure that the batch is always at or near the maximum size, eliminating the idling that static batching induces.

A system running continuous batching with maximum batch size 64 will achieve dramatically higher throughput than a system running static batching with the same maximum batch size, because the latter is almost never actually running 64 requests simultaneously.

PagedAttention and the memory management revolution

The KV cache memory problem has an analogy so precise it deserves to be stated explicitly: the KV cache is to inference systems what physical memory is to operating systems.

In an operating system, multiple processes compete for a fixed physical memory. Processes do not know their memory needs in advance (a process may allocate more memory as it runs).

Memory fragmentation is a real cost: even if the total free memory is sufficient, if it is not contiguous, an allocation may fail.

The solution that operating systems developed is virtual memory with paging: memory is divided into fixed-size pages, processes address a virtual space that the OS maps to physical pages on demand, and fragmentation is eliminated because non-contiguous physical pages can be mapped to a contiguous virtual space.

PagedAttention, introduced by Kwon et al. (2023) and implemented in vLLM, applies exactly this insight to KV cache management.

In a naive KV cache implementation, each request’s KV cache is a contiguous block of GPU memory allocated at request arrival. The maximum context length is reserved at allocation time (because the request might generate that many tokens), even if the actual generation is much shorter.

Fragmentation is severe: the gap between reserved and used memory across all requests is wasted, and new requests cannot use it.

PagedAttention divides the KV cache into fixed-size physical blocks (pages), where each block stores the keys and values for a fixed number of tokens (the block size, typically 16 or 32 tokens). When a request needs more KV cache space, it is allocated additional pages from a free pool.

Pages for a single request need not be contiguous in physical GPU memory; the PagedAttention kernel uses a block table (a small integer array per request mapping logical page indices to physical page indices) to find the right physical memory at attention computation time.

The consequences are significant. First, fragmentation falls from potentially 50% (if requests reserve maximum-length buffers but generate much shorter sequences) to under 5% (only the last partially-filled page of each sequence wastes space).

Second, sequences can share physical pages: if two requests have identical prompt prefixes (common in chat applications with a fixed system prompt), the KV cache pages for the shared prefix can be physically shared between them, eliminating redundant computation and halving the memory footprint of the prefix.

Third, memory allocation is lazy: pages are allocated only as tokens are generated, not at request arrival, which means a request does not consume its full potential KV cache until it actually generates enough tokens to need it.

The prefix sharing (also called KV cache sharing or prompt caching) deserves additional attention because its impact at scale is large. Consider a chat application where every request is prefixed with a 2000-token system prompt.

Without prefix sharing, each request independently computes and stores the KV cache for those 2000 tokens. With prefix sharing, the 2000-token prefix KV cache is computed once and shared across all concurrent requests.

For a batch of 64 requests, this eliminates 63 redundant prefill computations and reduces KV cache memory by 2000 × 64 tokens worth of activations, freeing space for a larger batch.

Speculative decoding and the bandwidth wall

Continuous batching with PagedAttention brings the inference system to the state of efficiently using available hardware at the maximum batch size the KV cache permits.

But we have still not escaped the fundamental constraint: at maximum batch size, the system is compute-bound, but getting to maximum batch size requires enough concurrent requests, which requires enough users, which is not always the case in low-traffic scenarios. At low batch sizes, we are still memory-bandwidth-bound, still producing tokens at the same rate that memory bandwidth permits.

Speculative decoding attacks this from a different angle. The observation is this: for a memory-bandwidth-bound system, the cost of generating one token and the cost of generating K tokens simultaneously in a single forward pass are approximately equal, because the bottleneck is reading the weight matrices from memory, and reading them once versus K times is the same operation if the K proposals can be evaluated in a single forward pass.

The mechanism: a small draft model (or a non-autoregressive heuristic, or a retrieval system) proposes a sequence of K candidate continuation tokens in a single forward pass.

The large target model then verifies this proposal in a single forward pass over the K tokens simultaneously. If the target model accepts all K tokens, K tokens have been generated at the cost of one large model forward pass plus one small model forward pass.

If the target model rejects some tokens, the generation rewinds to the first rejection and continues from there, having wasted some small model computation but no large model computation beyond what was necessary.

The analysis of when speculative decoding accelerates inference requires understanding the acceptance rate, which is the fraction of proposed tokens that the target model accepts.

For a good draft model generating from a similar distribution, acceptance rates of 70-90% are achievable. With an acceptance rate of α and a proposal length of K, the expected number of tokens accepted per large model forward pass is:

E[accepted] = sum_{k=0}^{K} (k+1) × αᵏ × (1 − α) + (K+1) × αᴷ

For α = 0.8 and K = 4: E[accepted] ≈ 3.3 tokens per large model forward pass, compared to 1 token per forward pass without speculation. Speedup is approximately 3.3×, reduced by the overhead of the draft model, typically 0.2-0.3 of a large model forward pass for a draft model that is 10-15× smaller.

Net speedup: approximately 3.3 / 1.2 ≈ 2.7×. This is not free, but it is substantial, and it is available even at batch size 1.

The reason speculative decoding works, and why it does not violate the memory-bandwidth constraint, is subtle. The verification pass is a prefill operation over K+1 tokens, which has higher arithmetic intensity than single-token decode.

For K=4, the verification pass processes 5 tokens simultaneously, which is 5× the arithmetic intensity of single-token decode. It is still memory-bandwidth-bound for small K, but the per-accepted-token cost of the memory reads is reduced because multiple tokens share the weight reads.

The deeper insight is that speculative decoding converts memory-bandwidth-bound decode operations into a mixture of memory-bandwidth-bound draft decode and slightly-less-memory-bandwidth-bound verification, and the mixture achieves better tokens-per-second because the draft model is smaller and therefore faster per forward pass.

The pathological case is when the draft model produces tokens that the target model almost never accepts, in which case the overhead of draft generation and failed verification exceeds the benefit.

Acceptance rates below approximately 0.5 make speculative decoding harmful, not helpful. This is why the choice of draft model matters: it should be distilled from or aligned with the target model, not simply chosen to be the fastest available.

Self-speculative decoding, where the model speculates with its own early layers (exiting at an intermediate layer for draft generation and running the full forward pass for verification) eliminates the draft model requirement at the cost of some architectural complexity.

Medusa, a multi-head speculative decoding method that adds dedicated draft heads to the target model, is a variant that achieves similar benefits with a different implementation strategy.

Subscribe now

KV cache quantization and the memory tradeoff

Even with PagedAttention and prefix sharing, the KV cache is often the binding constraint on batch size for long-context models. A 70B model with a 128K context length and a batch of 8 requests generates a KV cache of:

128,000 × 80 × 2 × 8 × 128 × 2 bytes × 8 requests = 3.3 TB

Three terabytes for 8 requests. No H100 cluster of reasonable size holds this in HBM. The only responses are to reduce context length (not always possible), reduce batch size (reduces throughput), or reduce the precision of KV cache values.

KV cache quantization stores keys and values in INT8 or FP8 rather than BF16, halving the memory requirement at the cost of approximation error. The question is whether that approximation error materially affects output quality.

The answer, empirically, is that keys and values are more quantization-sensitive than weights, because the attention score computation amplifies outliers in the key matrix, and those outliers are common and important.

Naive INT8 quantization of the KV cache causes measurable quality degradation on tasks requiring precise retrieval over long contexts. The degradation is smaller for short contexts where fewer keys compete for attention.

Techniques that address this include grouped-query attention (GQA), which reduces the number of KV heads (and therefore the KV cache size) without proportionally reducing the expressivity of attention by sharing keys and values across groups of query heads; and mixed-precision KV caching, which stores frequently-accessed (recent) tokens in higher precision and distant-context tokens in lower precision, exploiting the empirical observation that attention weights are concentrated on nearby and highly salient tokens.

GQA deserves detailed treatment because it has become standard in essentially all modern large language models: Llama 3, Mistral, Gemma, Qwen, and their successors all use it.

The mechanism is to reduce the number of KV heads from H to H/G for a group size of G, where typically G = 8. Each KV head is shared by G query heads during attention computation. The KV cache size shrinks by a factor of G, and with G=8 on a 128-head attention, the cache is 8× smaller.

The expressivity cost is small for typical values of G because attention heads tend to specialize into redundant groups anyway: the empirical evidence across many model evaluations suggests that the quality loss from GQA at G=8 is negligible relative to the memory benefit.

The prefill-decode disaggregation architecture

We have established that prefill is compute-bound and decode is memory-bandwidth-bound. The hardware that is optimal for one is different from the hardware that is optimal for the other.

High-compute GPUs (H100, B200) with large HBM capacity are appropriate for both prefill and decode, but they are expensive. For decode in particular, the binding resource is memory bandwidth, not compute throughput. A GPU with lower compute throughput but the same memory bandwidth would serve the decode phase equally well at lower cost.

This observation motivates prefill-decode disaggregation: running prefill and decode on separate hardware pools, each sized for its actual bottleneck.

The architecture is approximately as follows: a request arrives at a scheduler, which assigns it to a prefill worker. The prefill worker (a high-compute GPU) processes the prompt and produces the initial KV cache.

That KV cache is then transferred to a decode worker (which may be a different, possibly cheaper GPU). The decode worker generates tokens autoregressively, managing the KV cache in its memory, and streams tokens back to the user.

The KV cache transfer itself is non-trivial: for a long prompt on a large model, the KV cache may be tens of gigabytes, and transferring it across a network link between prefill and decode workers takes time proportional to size. The transfer must complete before the first decode token can be generated, which adds latency to the time-to-first-token (TTFT) metric.

For applications where throughput matters more than latency, this tradeoff is acceptable: the disaggregated system produces more tokens per dollar per second because each hardware type is used for the phase it is efficient at.

For applications where TTFT is critical (real-time conversational AI), the added transfer latency may be unacceptable.

The engineering tension here is real, and production systems navigate it differently depending on workload: Splitwise (from Microsoft Research), DistServe (from Peking University and others), and production implementations at major AI serving providers all make different tradeoffs along the latency-throughput frontier.

Subscribe now

Flash decoding and the attention bottleneck

We have spent considerable time discussing the weight matrix reads as the memory bottleneck for decode, but for long-context inference, a second bottleneck emerges: the KV cache reads during attention computation.

The attention operation during decode requires computing, for the new token’s query vector q of shape [d_head], an attention score against every key in the KV cache of shape [context_length, d_head], and then a weighted sum of every value in the KV cache of shape [context_length, d_head].

The KV cache for a single layer, a single head, at context length L is 2 × L × d_head × 2 bytes (keys and values, BF16). For L = 128,000, d_head = 128, this is 64 MB per head per layer per request.

For a model with 80 layers and 8 KV heads, the total KV cache read per decode step is 64 MB × 80 × 8 = 40,960 MB ≈ 40 GB.

At 3.35 TB/s HBM3 bandwidth, reading 40 GB takes approximately 12 milliseconds per decode step. The weight matrix reads (140 GB at 3.35 TB/s) take approximately 42 milliseconds. At 128K context length, KV cache reads are therefore about 22% of the per-step memory read time, a non-trivial contribution.

At context lengths beyond 128K (1M tokens is now a research target), KV cache reads can dominate weight reads entirely. The bottleneck for very-long-context inference is not the model weights but the growing KV cache.

Flash Decoding, introduced by Dao et al. and integrated into FlashAttention-3, addresses this by parallelizing the KV cache reads across multiple warps in a different pattern from standard FlashAttention.

In standard FlashAttention, which is designed for prefill (where Q, K, and V all have sequence dimension S), the parallelism is along the query sequence dimension.

In Flash Decoding, where the query has sequence dimension 1 but the KV cache has sequence dimension L, the parallelism instead decomposes along the key/value sequence dimension, allowing multiple warps to read different segments of the KV cache simultaneously and combine their partial softmax results using a numerically stable reduction.

The speedup from Flash Decoding is most pronounced when the KV cache length is large and the batch size is small, precisely the regime of long-context single-request inference where standard attention is most bottlenecked.

For L = 64K and batch size 1, Flash Decoding achieves 4-8× speedup over a naive attention implementation, which translates directly to 4-8× faster decode throughput for long-context requests.

The production inference stack

The mechanisms discussed above do not exist in isolation; they compose into a production inference serving system.

It is worth describing what that stack looks like end-to-end, because the interactions between components create optimization opportunities that are invisible when examining each component individually.

A production inference server for a 70B model on an 8-GPU node, circa mid-2026, runs approximately as follows.Tensor parallelism at degree 8 distributes the weight matrices across all 8 GPUs in the node, using NVLink for the all-reduce at each layer boundary.

The effective model per GPU is 70B / 8 = 8.75B parameters, requiring approximately 17.5 GB of HBM for weights in BF16. With 80 GB per GPU, this leaves 62.5 GB per GPU for KV cache.

The KV cache is managed by PagedAttention with a block size of 16 tokens. The pool of free blocks is divided among the 8 GPUs (with TP, the KV cache is also sharded, since each GPU handles a subset of the KV heads under GQA).

For a model with 8 total KV heads under 8-way TP, each GPU handles exactly 1 KV head, and the per-GPU KV cache is accordingly 1/8 of the total.

The scheduler runs continuous batching, maintaining a queue of waiting requests and a set of active requests. At each step, it admits new requests into the batch as KV cache pages become available. It preempts requests (evicts their KV cache pages, returning them to the free pool) when memory pressure is high, re-scheduling preempted requests from the beginning (or from a checkpoint) later.

Speculative decoding is enabled with a draft model of approximately 7B parameters, contributing an additional 14 GB of weight memory across the 8 GPUs (1.75 GB per GPU) and an additional 14 GB KV cache (1.75 GB per GPU), net of which there remains approximately 60 GB per GPU for the target model KV cache.

The prefill of long prompts is chunked (also called chunked prefill): instead of processing a 64K-token prompt as a single prefill operation that saturates compute for a long time and blocks decode requests from running, the prompt is processed in chunks of, say, 2048 tokens per step, interleaved with decode steps.

This trades slightly higher TTFT for dramatically better decode latency for concurrent users, a tradeoff that is almost always correct in a multi-user serving scenario.

Subscribe now

What the series has built

Ten parts. From SMs to NVSwitch. From warp scheduling to speculative decoding. From single-GPU kernels to multi-node inference stacks.

Across all of it, one idea stays invariant: a GPU is not a “compute machine” in the naive sense. It is a latency-hiding system. Every layer of the stack exists to keep data moving while something else is waiting; warps hiding memory latency, NVLink hiding interconnect latency, batching hiding kernel inefficiency, and scheduling hiding autoregressive seriality.

At scale, the same structure repeats. Training systems hide communication under compute. Inference systems hide serial generation under parallel requests. Serving stacks hide memory fragmentation under virtualized allocation.

The details change, but the pattern does not: identify the bottleneck, then restructure the system so that bottleneck is no longer visible to the critical path.

What emerges is not a collection of optimizations, but a consistent way of thinking about hardware systems. Everything reduces to one question:

what resource is constraining progress right now (compute, memory bandwidth, or communication) and how do we prevent it from sitting idle?

The specific techniques will evolve. NVLink will be replaced, attention kernels will be rewritten, new quantization schemes will appear, and hardware will continue to shift the ridge points we computed throughout this series.

But the underlying structure will not change, because it is not a property of transformers: it is a property of physics.

That is the real object we have been studying. Not GPUs. Not transformers. But the constraints that govern how any system can compute under finite bandwidth and finite time.

Share

Where Part VIII Left Us

Part VIII ended at something close to a philosophical statement: the SMSP on a well-tuned Hopper GEMM kernel is a machine that does one thing. Everything else has been delegated.

That is true, and it is beautiful, and it is also irrelevant the moment the model you are trying to train does not fit in 80 GB of HBM3.

GPT-3 has 175 billion parameters. In BF16 that is 350 GB of weights alone, before you add optimizer state, activations, and gradients. A single H100 has 80 GB. You need at least five of them just to hold the parameters, and in practice you need significantly more to make training feasible.

At this point the single-GPU roofline model, with its ridge points, its arithmetic intensity calculations, its tensor core utilization percentages, becomes necessary but not sufficient.

You need a new abstraction layer that sits above the GPU and treats a rack, or a pod, or a datacenter, as the compute substrate.

This part is about that layer. We will cover tensor parallelism, pipeline parallelism, data parallelism, and the collective communication primitives that tie them together.

We will look at NCCL, at NVLink topology and how it interacts with bandwidth requirements, and at the specific arithmetic of why certain parallelism strategies work and others do not at scale.

We will go into great detail. Tighten your seatbelts.

The memory wall has not gone away

Before we talk about parallelism strategies, we need to internalize what “model doesn’t fit” actually means, quantitatively.

A transformer with P parameters trained in mixed precision requires, at minimum:

• 2P bytes for the model weights in BF16

• 4P bytes for the master weights in FP32 (kept by the optimizer for numerical stability)

• 8P bytes for the Adam optimizer states (m and v vectors, both FP32)

• 2P bytes for the gradients in BF16

Total: 16P bytes in the steady state, not counting activations.

For a 70B parameter model (Llama 3 scale), this is 1,120 GB; fourteen H100s worth of HBM just for the optimizer state. This is not a pathological edge case; this is the routine reality of training frontier models.

Inference is cheaper (you do not need optimizer state) but for a 405B parameter model in FP8, you are still looking at 405 GB, spread across at minimum six H100 80GB instances, with careful attention to how the tensor operations are partitioned so that no single GPU computes a matrix multiply that requires moving activations larger than the HBM capacity.

The problem is therefore not just “how do we make one GPU fast” but “how do we decompose a computation that is too large for one GPU into pieces that run efficiently on many GPUs, with the communication overhead between those pieces small enough that the multi-GPU system achieves a meaningful fraction of the theoretical sum of its parts.”

That fraction has a name: parallel efficiency. Getting it above 0.5 for a thousand-GPU training run is hard. Getting it above 0.8 is a research problem.

Getting it above 0.9 is what separates companies that can train frontier models economically from companies that cannot.

Subscribe now

Three orthogonal dimensions of parallelism

The standard taxonomy, established empirically by the Megatron-LM work at NVIDIA and subsequently refined, identifies three orthogonal axes along which a transformer training job can be parallelized.

Data Parallelism (DP): Replicate the model across N GPUs, partition the training batch into N micro-batches, run a forward and backward pass independently on each GPU, and then average the gradients across all N replicas. Every GPU holds the full model. The communication pattern is a single all-reduce over the gradient tensors after each backward pass.

Tensor Parallelism (TP): Partition individual weight matrices across N GPUs, so that each GPU holds a 1/N shard of each matrix. A single matrix multiply that would require, say, a 4096×16384 GEMM on one GPU instead requires a 4096×(16384/N) GEMM on each of the N GPUs, followed by a collective to reassemble the result.

The communication pattern is tightly coupled to the forward pass; an all-reduce (or all-gather + reduce-scatter) at every layer boundary.

Pipeline Parallelism (PP): Partition the layers of the model across N GPUs, so that GPU 0 holds layers 1–L/N, GPU 1 holds layers L/N+1 through 2L/N, and so on. A micro-batch flows through the pipeline sequentially.

The communication pattern is a point-to-point activation transfer between adjacent stages, one per micro-batch per layer boundary.

These three dimensions compose. The Megatron-LM paper that trained GPT-3-scale models on A100 clusters used all three simultaneously: 8-way TP within a node (exploiting NVSwitch), 4-way PP across nodes, and data parallelism across node groups.

The product is the total GPU count: 8 × 4 × D = total GPUs, where D is the data parallel degree.

Understanding why this particular combination was chosen, and not, say, 64-way TP with no PP, requires understanding the communication topology and the arithmetic of collective operations.

That is what the rest of this part is about.

NVLink and NVSwitch

Communication between GPUs can happen over two physical fabrics: PCIe and NVLink. The performance difference between them is not small.

PCIe 4.0 x16 provides 32 GB/s unidirectional bandwidth. PCIe 5.0 x16 doubles that to 64 GB/s. These numbers sound reasonable until you compare them to what you actually need during all-reduce.

NVLink 4.0 (H100) provides 900 GB/s bidirectional bandwidth per GPU in NVLink-connected configurations: that is 450 GB/s in each direction. This is roughly 7× better than PCIe 5.0 in each direction, and the real-world benefit is larger because NVLink latency is also significantly lower.

But “NVLink-connected” hides a critical topological detail. NVLink connects pairs of GPUs (or GPUs through an NVSwitch). The DGX H100 system has 8 GPUs connected through NVSwitch 3.0, which provides full all-to-all connectivity at 900 GB/s per GPU.

This means any GPU can communicate with any other GPU in the same node at full bandwidth simultaneously. The NVSwitch acts as a non-blocking switch fabric.

Across nodes, the picture changes entirely. Multi-node communication happens over InfiniBand (HDR or NDR), with typical all-reduce bandwidth of 25–50 GB/s per GPU depending on topology and rail configuration: roughly 10–20× slower than intra-node NVLink.

This 10–20× bandwidth gap between intra-node and inter-node communication is the single most important physical fact for understanding why multi-GPU parallelism is structured the way it is.

The implication is immediate: communication-heavy parallelism strategies (like tensor parallelism, which requires an all-reduce at every layer) should be confined within a node, where NVLink bandwidth makes the overhead acceptable.

Communication-light parallelism strategies (like pipeline parallelism, which only requires activation transfers at layer boundaries) can span node boundaries.

This is exactly what Megatron-LM does, and the reason is physics, not convention.

Subscribe now

Data parallelism in depth

Data parallelism is the simplest strategy and the one that scales best in terms of implementation complexity. It is also the one where the communication overhead is most amenable to hiding behind computation, given careful engineering.

The communication requirement for data parallelism is an all-reduce over the gradient tensors after each backward pass. For a model with P parameters in BF16, this all-reduce moves 2P bytes of data through the network.

For a 70B model, that is 140 GB per all-reduce. At 25 GB/s inter-node InfiniBand, a naive all-reduce would take approximately 5.6 seconds. For a training step that takes 2–3 seconds of compute, this is catastrophically inefficient. The GPU would be idle for twice as long as it was computing.

The solution is to overlap gradient communication with the backward pass computation. As each layer’s gradients are computed during the backward pass, those gradients can be immediately all-reduced while the backward pass continues computing the gradients of earlier layers.

This is called gradient overlap, and it is implemented in PyTorch via the DistributedDataParallel (DDP) bucket mechanism: gradients are grouped into buckets of approximately 25 MB, and an all-reduce is launched for each bucket as soon as it fills, overlapping with the backward computation of earlier layers.

The efficiency of gradient overlap depends on the ratio of compute time to communication time per layer.

For large models with large batch sizes, this ratio is favorable: the layers are compute-heavy, and there is always useful computation happening while the all-reduce for a previous layer’s gradients is in flight.

For small models or small batch sizes, the backward computation per layer is short and the all-reduce cannot be fully hidden.

This is one reason why very large batch sizes are computationally efficient beyond the obvious “more samples per step” benefit: larger batches mean longer per-layer compute time, which means more time to hide communication.

ZeRO: when the model doesn’t fit, but you want data parallelism anyway

Vanilla data parallelism replicates the full model on every GPU. For a 70B model, this requires every GPU to have 1,120 GB of memory (with optimizer state), which is physically impossible today and will remain so for some time.

ZeRO (Zero Redundancy Optimizer), developed by Microsoft DeepSpeed, addresses this by partitioning the model state across the data parallel group rather than replicating it.

ZeRO comes in three stages of increasing memory savings and communication cost:

ZeRO-1: Partition the optimizer state (m, v vectors) across the DP group. Each GPU holds a 1/N shard of the optimizer state. Communication overhead is unchanged (gradients are still all-reduced). Memory savings: up to 4× for Adam (8P → 2P per GPU for optimizer state).

ZeRO-2: Partition the gradients in addition to the optimizer state. After the all-reduce, each GPU keeps only its 1/N shard of the gradients (the portion it needs for its optimizer state shard). Memory savings: up to 8× (12P → 1.5P per GPU for gradients + optimizer state). Communication overhead: unchanged.

ZeRO-3: Partition the model parameters as well. Each GPU holds only 1/N of the model weights at any given time. During the forward and backward pass, the needed weight shards are all-gathered from the DP group just-in-time. Memory savings: up to 64× for a large DP degree. Communication overhead: increases by 1.5× compared to vanilla DDP (due to the all-gather operations for parameters).

ZeRO-3 with a DP degree of 64 reduces the per-GPU memory for a 70B model from 1,120 GB to approximately 17.5 GB: comfortably fitting on a single H100. The tradeoff is the 1.5× increase in communication volume, which must be weighed against the larger batch sizes that ZeRO-3 enables.

The engineering implementation of ZeRO-3 is non-trivial: parameters must be gathered before each layer’s forward pass and immediately freed afterward (unless gradient checkpointing is also active, in which case they must be re-gathered during the backward pass as well).

The memory allocator must be aware of these temporary parameter buffers and free them aggressively.

DeepSpeed’s implementation of ZeRO-3 does this with a parameter fetch prefetch buffer: as layer i is executing its forward pass, layer i+1‘s parameters are being all-gathered in the background, overlapping communication with computation at the layer level rather than the bucket level.

Tensor parallelism in depth

Tensor parallelism (TP), as formalized in the Megatron-LM paper, exploits the specific structure of transformer layers to split individual matrix multiplications across multiple GPUs.

Consider a single transformer MLP layer with weight matrices W1 of shape [d_model, d_ffn] and W2 of shape [d_ffn, d_model], where d_ffn = 4 × d_model. For a 70B model, d_model ≈ 8192, so d_ffn ≈ 32768.

The output of the MLP layer is: Y = GeLU(XW1)W2

With 8-way tensor parallelism, W1 is split column-wise across 8 GPUs, each holding W1_i of shape [d_model, d_ffn/8]. W2 is split row-wise across 8 GPUs, each holding W2_i of shape [d_ffn/8, d_model].

The computation on GPU i becomes:

1. Y1_i = X × W1_i (local GEMM, shape [batch, d_ffn/8])

2. Z_i = GeLU(Y1_i) (local elementwise)

3. Y2_i = Z_i × W2_i (local GEMM, shape [batch, d_model])

4. Y = AllReduce(Y2_i) (sum partial results across 8 GPUs, shape [batch, d_model])

The all-reduce at step 4 is the communication bottleneck. Its cost is:

2 × (N-1)/N × |Y2| bytes

For a batch of 2048 tokens, d_model = 8192, BF16: |Y2| = 2048 × 8192 × 2 bytes ≈ 32 MB. At 450 GB/s NVLink bandwidth, this all-reduce takes approximately 32 MB / 450 GB/s ≈ 71 microseconds.

The compute time for the local GEMMs is: 2 × (2048 × 8192 × 32768/8) × 2 ops ÷ (494 TFLOP/s per GPU) ≈ 2 × 2048 × 8192 × 4096 × 2 / 494e12 ≈ 1.1 milliseconds per GEMM, so approximately 2.2 ms for both GEMMs.

Communication (71 µs) is roughly 3% of compute (2200 µs). This is an excellent ratio; the all-reduce is effectively free.

But now notice what happens if you push TP degree from 8 to 64 over InfiniBand at 25 GB/s instead of NVLink at 450 GB/s: the all-reduce takes 32 MB / 25 GB/s × (2 × 63/64) ≈ 2.5 ms, which is now comparable to the compute time.

The compute per GPU has also dropped by 8× (fewer flops per GPU due to more partitioning), so each GEMM takes 2.2ms / 8 ≈ 275 µs.

The all-reduce (2.5 ms) now takes 9× longer than the compute it is supposed to overlap with. You are GPU-idle for 90% of the time. This is why tensor parallelism over InfiniBand at high TP degrees is a terrible idea, not a theoretical concern but an arithmetic certainty.

The intra-node NVLink constraint on TP degree is therefore typically TP ≤ 8 for an 8-GPU node, precisely because that is the point where NVLink bandwidth makes the communication overhead negligible.

Self-attention and sequence parallelism

The attention mechanism has a different structure than the MLP, but the Megatron-LM approach handles it symmetrically: Q, K, and V projection matrices are split column-wise (head-parallel), and the output projection is split row-wise. Each GPU handles a subset of attention heads.

For H total heads and TP degree N, each GPU computes H/N heads independently. No communication is needed within the attention computation itself; only the output projection requires an all-reduce.

This works cleanly as long as H is divisible by N, which is why transformer architectures are almost always designed with head counts that are powers of 2 or multiples of 8.

There is an additional subtlety for attention: LayerNorm and dropout require the full activation tensor, not a sharded one.

In vanilla TP, these operations run on the full (all-reduced) activations, which means they see the full sequence length at full d_model dimension: they are not distributed and do not benefit from the TP decomposition.

Sequence Parallelism (SP), introduced as an extension to Megatron-LM TP, addresses this by replacing the all-reduce at layer boundaries with an all-gather + reduce-scatter pattern, and distributing the non-tensor-parallel operations (LayerNorm, dropout) across the sequence dimension rather than the model dimension.

In SP, the activation between transformer layers is sharded across the TP group along the sequence dimension: each GPU holds [batch, seq_len/N, d_model] instead of [batch, seq_len, d_model].

Before entering the tensor-parallel MLP, an all-gather reconstructs the full [batch, seq_len, d_model] tensor. After the MLP’s row-parallel W2 partial sum, a reduce-scatter simultaneously sums the partial results and re-shards the output along the sequence dimension.

The communication volume is identical to the all-reduce, but the memory advantage is significant: activations are now partitioned across the TP group, reducing the peak memory per GPU by a factor of N for the activation tensors.

At large sequence lengths (128K tokens, as in recent long-context models), this memory saving is the difference between fitting in HBM and catastrophic OOM.

Subscribe now

Pipeline Parallelism

Pipeline parallelism is the ugliest of the three strategies. This is a statement of fact, not an aesthetic judgment.

It introduces pipeline bubbles (periods where some GPUs are idle because they are waiting for activations from the previous stage) and managing those bubbles is the central engineering challenge.

In the simplest pipeline schedule (GPipe), a batch of M micro-batches flows through P pipeline stages sequentially. The forward pass processes micro-batch 0 through stage 0, then stage 1, ..., then stage P-1. Then micro-batch 1 flows through. Then micro-batch 2. And so on.

The backward pass happens in reverse, with a full flush between the forward and backward sweeps. The pipeline bubble fraction (time wasted on idle GPUs as a fraction of total time) is approximately:

bubble fraction ≈ (P − 1) / (M + P − 1)

For P = 4 stages and M = 8 micro-batches: bubble fraction ≈ 3/11 ≈ 27%. More than a quarter of GPU time is wasted.

To reduce the bubble, you increase M. For M = 32: bubble fraction ≈ 3/35 ≈ 8.5%. For M → ∞, bubble fraction → 0.

But increasing M increases the memory required to store the activations of all in-flight micro-batches during the forward pass before the backward pass can begin (GPipe requires storing all activations). For M micro-batches each with activation size A, the activation memory is M × A, and this grows linearly with M.

1F1B (One Forward One Backward) scheduling, introduced in PipeDream and used by Megatron-LM, breaks this linear activation scaling. In 1F1B, each pipeline stage interleaves one forward pass and one backward pass for different micro-batches, rather than running all forwards before any backwards.

The pipeline still has a bubble at startup and drain, but the steady-state memory is bounded: at any point, a stage has at most P micro-batches’ activations in flight (not M). The bubble fraction is the same as GPipe ((P−1)/(M+P−1)), but the activation memory is P × A instead of M × A.

For large models where M must be large to amortize the bubble, this is a crucial difference. With M = 64 and P = 8, GPipe stores 64 micro-batches of activations; 1F1B stores at most 8.

Interleaved 1F1B goes further: each GPU holds V virtual pipeline stages (chunks of layers) instead of one contiguous block, enabling the bubble fraction to be reduced to:

bubble fraction ≈ (P − 1) / (V × M + P − 1)

with V times the point-to-point communication per step (because activation tensors must be sent between non-adjacent GPUs). This is the schedule used by Megatron-LM for the largest training jobs, with V=2 or V=4 providing a 2-4× reduction in bubble at a 2-4× increase in inter-stage communication.

The Activation Recomputation Tradeoff

One more tool for managing activation memory in pipeline parallelism (and, frankly, in any large training run): gradient checkpointing, also called activation recomputation.

The idea is simple: instead of storing the full activation tensor for every layer during the forward pass (needed for the backward pass), you store only the activations at checkpoint boundaries (e.g., every transformer block) and recompute the intermediate activations on-demand during the backward pass by running the forward computation again.

The memory cost is reduced from O(L × A) to O(√L × A) for optimal checkpoint placement (checkpoint every √L layers). The compute cost increases by approximately 30-40% (one extra forward pass per layer, amortized).

For training at the frontier, this tradeoff is almost always worth it: compute is more abundant than HBM capacity, and the alternative is buying more GPUs (or, equivalently, using more PP stages, which increases the bubble).

The interaction between activation recomputation and pipeline parallelism is non-trivial: if you are recomputing activations, the backward pass must re-run the forward computation for each stage, which means each stage must still have access to the input activations from the forward pass checkpoint.

This constrains how aggressively you can combine PP and recomputation, and the Megatron-LM codebase has explicit logic for managing which activations are stored versus recomputed across pipeline stages.

Subscribe now

The library that makes multi-GPU work

Everything discussed above (all-reduce, all-gather, reduce-scatter) is implemented in practice by NCCL (NVIDIA Collective Communications Library).

Understanding what NCCL does, and more importantly how it does it, is necessary for diagnosing performance problems at scale.

NCCL implements the standard collective communication operations (AllReduce, AllGather, ReduceScatter, Broadcast, Reduce, Barrier) on NVIDIA GPUs, with topology-aware algorithms that exploit NVLink, NVSwitch, and InfiniBand according to the detected hardware configuration.

Ring-AllReduce vs Tree-AllReduce

The canonical AllReduce algorithm for a ring of N GPUs is ring-allreduce, introduced in the deep learning context by Baidu Research and later made famous by the Horovod library.

In ring-allreduce, GPUs are arranged in a logical ring. AllReduce is decomposed into two phases:

Reduce-Scatter: Each GPU sends a chunk of its data to the next GPU in the ring, while simultaneously receiving and accumulating a chunk from the previous GPU. After N−1 steps, each GPU holds the fully reduced value for one chunk (its 1/N shard of the full tensor).

AllGather: Each GPU broadcasts its reduced chunk to all others by rotating it around the ring. After N−1 more steps, every GPU holds the fully reduced tensor.

Total data sent per GPU: 2 × (N−1)/N × |data| ≈ 2 × |data| for large N. This is bandwidth-optimal: any AllReduce algorithm must send at least 2 × (N−1)/N × |data| bytes per GPU, so ring-allreduce achieves the theoretical minimum.

The latency of ring-allreduce scales as 2(N−1)α + 2(N−1)/N × |data|/β, where α is the point-to-point latency and β is the bandwidth.

For large data (|data| >> α/β × N²), the bandwidth term dominates and the algorithm is efficient. For small data, the latency term (proportional to N) dominates and ring-allreduce becomes expensive.

This is why AllReduce for gradient synchronization (large tensors, many GB) works well with ring-allreduce, but AllReduce for small tensors (like the normalization statistics in LayerNorm) can benefit from tree-based algorithms that have O(log N) latency scaling at the cost of non-optimal bandwidth utilization.

NCCL automatically selects the algorithm based on message size, topology, and a heuristic tuning table, but understanding the underlying tradeoff is necessary when the automatic selection is suboptimal for your specific workload.

NCCL and topology awareness

NCCL’s topology detection is worth examining in detail because it directly determines which algorithm it selects for intra-node vs inter-node operations.

On startup, NCCL probes the system topology using the CUDA device properties API and, where available, NVML (NVIDIA Management Library) topology information.

It constructs an internal graph where GPUs are nodes and NVLink/PCIe/InfiniBand connections are edges with associated bandwidths.

For an 8-GPU DGX H100 node with NVSwitch, NCCL detects a fully connected graph with 900 GB/s bidirectional bandwidth.

For multi-node communication over InfiniBand with a single SHARP switch, NCCL can use SHARP (Scalable Hierarchical Aggregation and Reduction Protocol), an in-network computing feature of modern InfiniBand switches that performs the AllReduce reduction inside the switch fabric rather than at the endpoint GPUs.

SHARP effectively moves the bandwidth bottleneck from the GPU NICs to the switch, and for large-scale clusters with SHARP-capable HDR/NDR InfiniBand, it can reduce AllReduce latency by 2-3× compared to standard ring-allreduce.

For hierarchical topologies (NVLink within nodes, InfiniBand between nodes), NCCL uses a two-level algorithm: a ReduceScatter within each node using the fast NVLink fabric, followed by an AllReduce across nodes over InfiniBand, followed by an AllGather within each node.

This correctly exploits the bandwidth hierarchy: the slow inter-node link sees only the partially reduced results, not the full gradient tensor from every GPU.

The implementation detail that matters for practitioners: the NCCL_SOCKET_NTHREADS and NCCL_BUFFSIZE environment variables, along with the NCCL_P2P_LEVEL setting, are often the first things to tune when AllReduce performance is below theoretical bandwidth.

NCCL’s defaults are conservative for stability across diverse hardware configurations.

The NCCL + CUDA stream interaction

NCCL operations run on CUDA streams, and correct stream management is the source of many subtle performance bugs in multi-GPU training code.

The critical invariant: NCCL operations on the same communicator object are serialized in NCCL’s internal queue, but operations on different communicators are independent.

The PyTorch DDP implementation creates one NCCL communicator per process group, and all AllReduce operations for gradient synchronization go through this communicator.

When gradient overlap is active (AllReduce launched for bucket i while the backward pass continues computing gradients for bucket i-1), PyTorch launches the AllReduce on a separate CUDA stream from the backward pass computation. The backward computation stream and the AllReduce stream run concurrently on the GPU.

The synchronization at the end of the backward pass (before the optimizer step) ensures that all AllReduce streams have completed. Failure to synchronize here is a common bug that manifests as non-deterministic gradient corruption: the optimizer sees a partially all-reduced gradient tensor.

The interaction with tensor parallelism adds another layer: the AllReduce within a TP group happens on a separate communicator from the AllReduce within the DP group.

These two communicators must be correctly ordered in the forward/backward pass: TP AllReduce happens at each layer boundary (blocking for the next layer to begin), while DP AllReduce happens after the full backward pass.

Subscribe now

The arithmetic of parallelism efficiency

With the mechanisms in place, we can now put numbers on the parallel efficiency achievable under different configurations. This is where the theoretical discussion becomes practically useful.

For data parallelism, the relevant ratio is Rᴅᴘ = T_compute / T_allreduce. For gradient overlap to be effective, we need Rᴅᴘ >> 1. For a 70B model over InfiniBand at 25 GB/s, T_allreduce ≈ 5.6 seconds. For a per-GPU batch of 512 tokens at 50% H100 utilization, T_compute ≈ 0.58 seconds. Rᴅᴘ ≈ 0.1.

The compute is an order of magnitude faster than the all-reduce, and no amount of overlap engineering fixes a ratio that is inverted.

The resolution is gradient accumulation: run multiple micro-batches locally before triggering the all-reduce, increasing the logical batch size without increasing activation memory.

This is not a workaround. It is the correct operating point, because convergence is measured in tokens seen, not steps taken.

For tensor parallelism, the critical TP degree N* is the point above which T_allreduce_TP > T_GEMM_local. On NVLink at 450 GB/s, N* ≈ 8. On InfiniBand at 25 GB/s, N* ≈ 1 to 2. Tensor parallelism over InfiniBand is not a suboptimal choice; it is an arithmetic mistake.

The practical ceiling for a well-tuned thousand-GPU training job with Megatron-LM or DeepSpeed on H100 hardware is approximately 40-60% MFU.

The gap from 100% is accounted for by pipeline bubbles (~15%), communication overhead (~10-15%), activation recomputation (~5-10%), and the residual from kernel inefficiency on non-GEMM operations and host-side Python overhead.

Getting from 40% to 60% MFU is worth approximately a 50% reduction in training cost. Every major AI lab has engineers whose entire job is that gap.

What the three strategies actually buy you

It is worth stepping back and stating plainly what each parallelism axis is for, now that we have done the arithmetic.

Data parallelism buys throughput. More DP replicas means more tokens per second, with communication overhead that is manageable at any scale where the per-GPU batch is large enough.

ZeRO makes DP viable even when the model does not fit on a single GPU, at the cost of increased communication volume that must be weighed against the larger batch sizes it unlocks.

DP is the outer loop of every large training job; the other strategies are refinements that make DP feasible at model sizes and cluster scales where it would otherwise be communication-bound.

Tensor parallelism buys memory capacity per layer, by distributing the weight matrices and activations of individual layers across multiple GPUs.

Its cost is a mandatory all-reduce at every layer boundary, which makes it viable only where NVLink bandwidth makes that all-reduce cheap.

TP is fundamentally an intra-node strategy on current hardware. Using it otherwise is trading compute for communication at an exchange rate that is never favorable.

Pipeline parallelism buys the ability to cross node boundaries without paying the tensor-parallel penalty. Its cost is the pipeline bubble, which is a tax on idleness that shrinks with micro-batch count and with interleaved scheduling.

PP is the mechanism by which you scale beyond a single node when the model is too large for TP alone to partition. It is ugly, and it is necessary, and the 1F1B schedule and its interleaved variants exist specifically to make that ugliness manageable.

The three-dimensional parallelism strategy (TP within a node, PP across nodes, DP across node groups) is not a design choice that someone made. It is the unique solution that the bandwidth hierarchy of current hardware enforces.

NVLink makes TP cheap within a node. InfiniBand makes PP the only viable strategy across nodes. Gradient accumulation and ZeRO make DP efficient across the full cluster. Change the hardware, and the optimal strategy shifts.

NVL72 on Blackwell, by extending the NVLink fabric to 72 GPUs, shifts the TP/PP boundary outward.

InfiniBand NDR at 400 Gb/s per link, as it becomes more widely deployed, shifts the point at which inter-node DP communication becomes the bottleneck. The strategies are not timeless; the physics that motivates them is.

Subscribe now

What actually limits you at scale

The question that every large-scale training practitioner eventually asks is: why isn’t my thousand-GPU job achieving 70% MFU? The answer is rarely a single cause.

It is a stack of inefficiencies, each modest in isolation, compounding into the gap between theoretical and measured throughput.

Pipeline bubbles are usually the largest single contributor at high PP degrees. For P=16 and M=32, the bubble fraction is 32% before any other inefficiency is counted.

Interleaved 1F1B at V=2 halves this, at the cost of doubled inter-stage communication. The right V depends on whether communication or compute is the limiting resource at the PP boundary, which varies by model size, node count, and InfiniBand configuration.

Inter-node AllReduce tail latency is the second major contributor and the hardest to reason about, because it is statistical rather than deterministic.

The average InfiniBand bandwidth may be 25 GB/s, but the 99th-percentile latency can be 3-5× higher due to switch congestion, adaptive routing jitter, and multi-tenancy.

The all-reduce waits for the slowest link. At scale, the slowest link is almost always slower than the average, which means the effective all-reduce bandwidth is consistently worse than the number on the datasheet.

RDMA configuration, static routing, and dedicated rail topology are the levers. They require infrastructure access that not every team has.

Host-side Python overhead is the most embarrassing source of inefficiency, because it is entirely self-inflicted.

PyTorch’s dispatcher, the GIL, and the overhead of the training loop’s Python logic can appear as measurable GPU idle time for models where per-step compute is short. CUDA graph capture eliminates per-operation launch overhead for the inner loop.

Careful pipelining of data loading and logging eliminates it for the outer loop. Teams that have done this work carefully report 5-10% improvement in effective MFU from host-side optimizations alone, which is not a small number when the training run costs millions of dollars.

Conclusion

The trajectory from single-GPU to multi-GPU mirrors the trajectory within the single GPU: find the bottleneck, route around it, measure again.

On a single GPU, the bottleneck moved from arithmetic units (tensor cores solved it) to memory bandwidth (cp.async attacked it) to instruction overhead (TMA eliminated it).

The SMSP on a well-tuned Hopper GEMM kernel is a machine that does one thing, because every other thing it used to do has been delegated to dedicated hardware.

At the multi-GPU level, the bottleneck is the communication fabric, and the solution follows the same logic: match the parallelism strategy to the bandwidth available at each level of the hierarchy.

TP over NVLink, PP over InfiniBand, DP across the full cluster. The ring-allreduce, the 1F1B schedule, the ZeRO partitioner: these are the mechanisms by which a thousand-GPU cluster achieves 40-60% of the theoretical sum of its parts.

40-60% of a thousand H100s is still an extraordinary amount of compute. Whether it is enough to train the next frontier model is left as an exercise for the reader’s infrastructure budget.

Part X will close in on a corner of this picture we have deliberately deferred: inference. Training is a one-time cost; inference is the workload that runs forever, at a scale that dwarfs training once a model is deployed.

The optimization challenges at inference time are different in kind: speculative decoding, KV cache management, continuous batching, and the specific arithmetic of when the prefill and decode phases are bottlenecked by entirely different resources.

The tools change; the principle does not.

Share

Where Part VII Left Us

Part VII ended with a promise and an architectural cliffhanger.

The promise: on Hopper, the compute-to-load instruction ratio in a GEMM inner loop approaches infinity from the SMSP’s perspective.

The cliffhanger: one instruction moves a 128×128 BF16 tile, the TMA unit generates all the addresses, and something called an mbarrier replaces the __syncthreads() you have been writing since your first CUDA “hello world”.

Let us unpack exactly what that means, why NVIDIA made those choices, and what you have to understand to write, read, or debug CUTLASS 3.x kernels without feeling like you are reading a foreign language.

We will go very deep. There is no other way.

The Problem cp.async Did Not Fully Solve

Part VII established that cp.async is superior to the conventional LDG → STS path because it removes the destination registers from the scoreboard. The SMSP issues the copy, hands it off to the Async Copy Engine, and is immediately free to issue the next instruction.

This is genuinely great. But it has a hidden cost that only becomes visible when you look at the SMSP instruction stream of a real GEMM kernel.

Consider a 128×128×32 BF16 tile. Loading that tile requires 128 × 32 BF16 elements = 4096 BF16 = 8 KB. At 16 bytes per cp.async, that is 512 individual CP.ASYNC.CA.SHARED.GLOBAL instructions.

Those 512 instructions have to be fetched from the instruction cache, decoded, dispatched through the MIO unit, and tracked by the hardware. They consume SMSP instruction bandwidth even though they produce no register results.

On Ampere, the SMSP can issue roughly one 128-bit cp.async every 4 cycles per SMSP. For 512 instructions, that is approximately 2048 SMSP cycles per tile load, just for the instruction overhead. The actual data movement happens asynchronously, but the instruction stream is not free.

For large tiles this is manageable. For smaller tiles, or for architectures where you want the SMSP to spend every cycle on tensor core instructions, it is a ceiling.

Hopper (SM90, H100) was designed to remove that ceiling entirely. The answer is the Tensor Memory Accelerator.

Tensor Memory Accelerator

The TMA is a hardware unit introduced in Hopper that performs multi-dimensional tensor copies between global memory and shared memory (or distributed shared memory across a cluster, but we will get to clusters).

It accepts a tensor descriptor computed on the host and a set of coordinates computed on the device, and it handles everything else: address computation, striding, data type conversion, out-of-bounds clamping, cache policy, and transaction completion signaling.

Let us be concrete about what “everything else” means.

In a conventional tiled GEMM, for every tile you load, every thread in the warp must compute its portion of the global memory address.

That address computation involves the block index, the thread index, the tile dimensions, the matrix stride, and the element size. It is entirely deterministic arithmetic that produces the same result every time you execute the same tile iteration.

It is also arithmetic that the SMSP has to execute. On Ampere with cp.async, that arithmetic still happens in the SMSP even though the subsequent memory transaction is asynchronous.

The TMA eliminates that arithmetic from the SMSP. One thread issues one instruction with a tensor descriptor handle and a pair of (y, x) coordinates.

The TMA unit uses those coordinates and the descriptor’s metadata to compute every address needed for the entire tile transfer, scatter or gather the data, and write it to shared memory. The SMSP emitted one instruction. One.

This is not a minor optimization. It is a qualitative change in what the SMSP does during a GEMM kernel. On Hopper, the SMSP’s job is to run WGMMA.MMA_ASYNC instructions.

The TMA’s job is to move data. These two jobs happen simultaneously, on separate hardware units, and the only communication between them is an mbarrier synchronization object.

Subscribe now

The Tensor descriptor

Before a Hopper kernel runs, the host must create a tensor descriptor using cuTensorMapEncodeIm2col or, more commonly for GEMM, cuTensorMapEncodeTiled. This is a 128-byte opaque structure stored in constant memory (or passed through a register and loaded into the L1).

The descriptor encodes:

Base pointer: the global memory address of tensor element [0, 0, 0, ...].

Global dimensions: the actual size of each dimension in the full tensor, in elements. For an M×K matrix A, this is {M, K} (or {K, M} if column-major).

Global strides: the byte stride between consecutive elements in each dimension. For a row-major matrix with K columns and BF16 elements, the stride between row i and row i+1 is K × 2 bytes. These strides allow arbitrary non-contiguous tensors.

Box dimensions: the size of the tile to be transferred in each dimension. For a 128×32 BF16 tile, this is {128, 32}.

Interleave and swizzle mode: how data should be rearranged during the transfer to produce a shared memory layout that avoids bank conflicts. This is the part that replaces all the padding arithmetic from Part VII.

Element stride and data type: how to interpret the raw bytes.

The descriptor is created once on the CPU and passed to the kernel. On the device, a single warp or even a single thread can then use this descriptor to initiate a full tile transfer with one instruction, because all the per-tile invariant information is already encoded.

This is a deliberate design choice: move the expensive computation (descriptor creation) to the host, where latency is irrelevant relative to the kernel launch overhead, so that the device-side instruction can be as cheap as possible.

The TMA instruction itself

The PTX for a 2D TMA load looks like this:

cp.async.bulk.tensor.2d.shared::cluster.global.mbarrier::complete_tx::bytes
    [smem_dst], [gmem_desc, {coord_y, coord_x}], [mbar];

Let us parse every token.

cp.async.bulk means this is an asynchronous bulk copy; “bulk” distinguishes it from scalar cp.async. The transfer size is determined by the descriptor, not encoded in the instruction.

tensor.2d means the TMA will interpret the coordinates as a 2D tensor access. There are variants for 1D through 5D tensors.

shared::cluster is the destination scope: shared memory that is visible to the entire thread block cluster (more on clusters shortly). For single-CTA kernels this is simply shared memory.

global is the source: global memory, indexed via the descriptor.

mbarrier::complete_tx::bytes is the completion signaling mechanism. When the transfer completes, the TMA will signal a mbarrier object, decrementing its transaction count.

When the count reaches zero, threads waiting on the barrier are unblocked. This replaces consumer_wait() and __syncthreads() in the sense that the barrier itself tracks both the data arrival and the thread synchronization in a single primitive.

[smem_dst] is the destination address in shared memory.

[gmem_desc, {coord_y, coord_x}] is the descriptor plus coordinates. The TMA extracts the base pointer, strides, and box dimensions from the descriptor, applies the coordinates, and generates the full address range.

[mbar] is a pointer to the mbarrier object in shared memory.

In CUDA C++, the cuda::experimental::tma:: API (or __pipeline_memcpy_async for simpler cases) generates this instruction. The canonical production path is through CUTLASS 3.x’s cute::copy with a TMA copy atom, which we will examine in the CUTLASS section.

Subscribe now

A synchronization primitive you have not seen before

__syncthreads() is a full thread block barrier. Every thread in the block must arrive before any thread proceeds.

It is implemented via a shared counter that is decremented by each arriving thread and checked by a hardware barrier mechanism. Its cost is proportional to thread block size, and it cannot distinguish between “I’m done computing” and “my data has arrived from the DMA engine”.

mbarrier (memory barrier, or more precisely, the Hopper barrier object) solves both of those problems.

An mbarrier object is a 64-bit value stored in shared memory. It has two phases, expect and arrive, and it tracks two distinct counts:

The arrival count is decremented by threads calling mbarrier.arrive or mbarrier.arrive_drop. When this count hits zero, the barrier phase flips.

The transaction count is decremented by the TMA engine itself when a bulk copy completes. This is the complete_tx::bytes in the PTX instruction above. The programmer initializes this count to the expected number of bytes that the TMA will deliver.

The barrier is “complete” when both counts reach zero: all participating threads have arrived, and all expected TMA transactions have completed.

This means you can have a consumer wait on a barrier that is signaled partly by threads and partly by hardware DMA engines, with no polling loop, no atomics in the critical path, and no __syncthreads() that serializes all 128 threads in the block.

The setup looks like this in CUDA C++:

__shared__ cuda::barrier<cuda::thread_scope_block> mbar;

// One thread initializes the barrier for N_THREADS participants
if (thread_rank == 0) {
    init(&mbar, N_THREADS);
    // Tell the barrier to also expect TMA_BYTES bytes of async data
    cuda::device::barrier_native_handle(mbar).arrive_tx(TMA_BYTES);
}
__syncthreads();  // This syncthreads is to publish the initialized mbar

// Producer thread issues TMA
if (thread_rank == 0) {
    tma_load(&mbar, smem_A, gmem_desc_A, tile_coord_m, tile_coord_k);
}

// All threads arrive at the barrier (decrement arrival count)
auto token = cuda::device::barrier_native_handle(mbar).arrive();

// Wait for both arrival count and transaction count to reach zero
cuda::device::barrier_native_handle(mbar).wait(std::move(token));

Note the asymmetry: one thread issues the TMA, all threads participate in the barrier synchronization. This is not a bug; it is the design.

The TMA is a singleton operation that one thread initiates, but the data it delivers is consumed by all threads, so all threads must synchronize on its completion.

The arrive_tx call informs the barrier that TMA bytes are expected. Without it, the barrier would complete as soon as all threads arrived, regardless of whether the DMA data had landed in shared memory. That would be a race condition.

The token returned by arrive is a phase token. mbarrier operates in alternating phases (like a double buffer at the synchronization level), and the token ensures that wait waits on the correct phase.

This is how Hopper avoids the ABA problem in barrier reuse: you cannot accidentally wait on a barrier phase that already completed in a previous iteration.

Subscribe now

Warpgroup MMA

Part VII did not cover the compute side of Hopper in depth because the memory side was already enough to digest. Now we need to talk about WGMMA, and it is equally radical.

On Ampere, tensor core instructions are issued per-warp: HMMA.1688 or the PTX mma.sync.aligned operates on 16×8×16 tiles with 32 threads participating. Each warp independently executes its tile of the matrix multiply.

Warp-level tensor core instructions were already a significant departure from SIMT, since all 32 threads in a warp cooperate to produce a single 16×8 output tile. But the warp is still the unit of scheduling and the unit of tensor core execution.

On Hopper, the tensor core instruction is warpgroup-level: WGMMA.MMA_ASYNC operates on a group of 4 warps (128 threads) simultaneously. The input tile dimensions for BF16 are:

• A: 64×16 per warpgroup (contributed from registers or shared memory)

• B: 16×256 per warpgroup (always from shared memory)

• C/D: 64×256 accumulator (in registers, split across the 128 threads)

A single WGMMA.MMA_ASYNC instruction computes a 64×256×16 BFGEMM, producing 64×256 = 16,384 output elements in one instruction.

For comparison, an Ampere mma.sync.aligned with the largest BF16 shape produces 16×8×16 BFGEMM, 128 output elements.

The output volume ratio is 128:1. This is what “approaching infinite compute-to-load ratio” means in practice.

The _ASYNC suffix is critical: WGMMA.MMA_ASYNC does not complete synchronously. The 4 warps issue the instruction and the result is not guaranteed to be in the accumulator registers until a WGMMA.WAIT_GROUP instruction is executed.

The hardware can overlap multiple WGMMA operations in flight simultaneously, and the programmer must insert explicit waits before reading the accumulators.

The programming model therefore looks like this at the instruction level:

WGMMA.MMA_ASYNC D, A, B   ; issue tile multiply k=0
WGMMA.MMA_ASYNC D, A, B   ; issue tile multiply k=1
WGMMA.MMA_ASYNC D, A, B   ; issue tile multiply k=2
...
WGMMA.WAIT_GROUP 0         ; wait for all outstanding WGMMAs
; D accumulator registers now hold valid results

In CUDA C++, this is exposed through the cute::wgmma abstractions in CUTLASS 3.x, or through the lower-level cuda::wgmma:: namespace. Direct PTX is also possible but strongly inadvisable outside of research contexts.

The reason B must always come from shared memory (not registers) is a hardware constraint. The tensor core units on Hopper are wired directly to the shared memory arrays.

The B operand is fetched directly from the shared memory banks by the tensor core datapath, without going through the register file.

This is why the TMA delivering B into shared memory is on the critical path, but there is no “load B from shared memory to registers” step. The tensor core reads shared memory directly.

A can come from either registers or shared memory. For the highest-performance kernels, A also comes from shared memory, which means both operands bypass the register file entirely on the compute side. The register file holds only the C/D accumulator.

Subscribe now

Thread Block Clusters

Hopper introduced a new level of the GPU hierarchy between the thread block and the grid: the thread block cluster.

A cluster is a group of up to 8 thread blocks that are guaranteed to be co-scheduled on the same GPC (Graphics Processing Context, a group of SMs sharing an L2 slice).

Thread blocks within a cluster can access each other’s shared memory via the Distributed Shared Memory (DSMEM) mechanism, using TMA to move data between SMs without going through L2.

The PTX instruction for a cross-SM TMA transfer is:

cp.async.bulk.tensor.2d.shared::cluster.global.mbarrier::complete_tx::bytes
    [smem_dst], [gmem_desc, {coord_y, coord_x}], [mbar];

This is the same instruction as a regular TMA load, with the shared::cluster scope indicating that the destination is visible cluster-wide. The TMA unit manages the inter-SM data movement transparently.

Why does this matter for GEMM? Consider a cluster of 2 CTAs, each responsible for a different row block of C. Both need access to the same columns of B.

With clusters, CTA 0 loads B into its shared memory via TMA, and CTA 1 can read CTA 0’s shared memory directly via DSMEM. B is loaded once and consumed by two CTAs. This effectively doubles the B reuse without doubling the shared memory per CTA.

For an N=8 cluster, 8 CTAs share the B tile load, amortizing the HBM bandwidth for B across 8x more compute.

This is the mechanism by which Hopper GEMM kernels approach hardware peak on large problem sizes: the cluster architecture allows the working set of the entire computation to be held in distributed shared memory, with HBM touched only once per element.

The cluster size is specified at kernel launch:

cudaLaunchConfig_t config = {};
config.gridDim = grid;
config.blockDim = block;
cudaLaunchAttribute attr;
attr.id = cudaLaunchAttributeClusterDimension;
attr.val.clusterDim.x = 2; // 2 CTAs per cluster
attr.val.clusterDim.y = 1;
attr.val.clusterDim.z = 1;
config.attrs = &attr;
config.numAttrs = 1;
cudaLaunchKernelEx(&config, my_kernel, args...);

Cluster scheduling is cooperative: the hardware will attempt to co-locate the CTAs of a cluster on the same GPC, but this is a hint, not a guarantee for clusters larger than what fits on one GPC.

On H100 SXM5 with 132 SMs organized into 7 GPCs, clusters of up to 8 are always satisfied within a single GPC.

The Persistent Kernel Model

On Ampere, a typical GEMM kernel is a “grid kernel”: each thread block handles one (M_TILE, N_TILE) output tile and exits. The CUDA runtime schedules new thread blocks as soon as SM capacity becomes available.

For large matrices this is fine: there are enough tiles that the SM scheduler is always busy.

For smaller matrices, the overhead of launching and retiring thread blocks dominates. Each thread block must load its A and B tiles from scratch, write its C tile to global memory, and terminate. The shared memory state is not reused across thread blocks.

Hopper’s memory hierarchy and cluster model make a different approach attractive: persistent kernels.

In a persistent kernel, a thread block (or warpgroup) does not terminate after processing one tile.

Instead, it loops over multiple output tiles, maintaining the A and B tiles in shared memory between iterations where the tile is reused, and fetching new tiles via TMA only when necessary. The kernel terminates only after all output tiles in its assigned partition are complete.

CUTLASS 3.x implements this via the Tile Scheduler, a device-side component that manages the assignment of output tiles to persistent CTAs.

The scheduler atomically increments a work counter stored in global memory, assigning the next available (m_tile, n_tile) pair to the requesting CTA. When all tiles are assigned, the scheduler signals completion and the CTA exits the work loop.

The advantages are concrete:

L2 reuse improves because the same CTA processes multiple adjacent tiles, and the A or B tiles they share remain in L2 (or even in shared memory) between iterations.

Thread block launch overhead is amortized: the GPU launches one wave of persistent CTAs and they run to completion, rather than launching thousands of transient blocks.

Irregular problem sizes are handled more gracefully: the final partial tile is processed by whichever CTA happens to claim it, without requiring separate epilogue kernel launches.

The disadvantage is programming complexity: you are writing a software scheduler inside a CUDA kernel, with all the attendant concerns about correctness under concurrent access and load balancing across heterogeneous tile work.

CUTLASS handles this for you, which is one reason the library exists.

Subscribe now

The CUTLASS 3.x Architecture

CUTLASS 3.x is a complete rewrite of CUTLASS 2.x, built on a new abstraction layer called CuTe (CUDA Template library).

Understanding CUTLASS 3.x requires understanding CuTe, because CUTLASS 3.x is essentially CuTe plus a set of kernel templates that use it.

CuTe: Layouts as First-Class Objects

CuTe’s central idea is that a layout is a function from a logical coordinate space to a physical offset in memory. A layout encodes both shape (the extents of each dimension) and stride (the distance in elements between consecutive elements along each dimension).

In CuTe, a layout is written as Shape:Stride. For example, a 4×8 row-major matrix with elements of size 2 bytes has layout (4,8):(8,1), meaning: the outer dimension (rows) has stride 8 (each row is 8 elements apart), and the inner dimension (columns) has stride 1. A column-major version of the same matrix would be (4,8):(1,4).

The power of this representation is that it composes. A tiling operation is just a layout composition. A swizzle (bit permutation of addresses to avoid bank conflicts) is a layout transformation that permutes the address bits in a specific pattern.

The entire address computation for a tiled, swizzled, transposed tensor is expressed as a sequence of layout compositions that the compiler evaluates at compile time, producing a single address formula.

This is why CUTLASS 3.x can express complex access patterns without any runtime branching in the address computation.

using LayoutA = Layout<Shape<_128, _32>, Stride<_32, _1>>;  // 128x32 row-major
using LayoutA_Swizzled = ComposedLayout<Swizzle<3,3,3>, LayoutA>;

The Swizzle<B,M,S> template encodes a specific XOR-based address permutation. B bits are permuted with S bits, offset by M bits.

For BF16 with 32 banks of 4 bytes each, the correct swizzle eliminates all bank conflicts without any padding. CUTLASS ships with the correct swizzle parameters for every element type and tile dimension it supports.

The MMA Atom and Copy Atom

In CUTLASS 3.x, a tensor core instruction is an MMA atom: a typed object that describes the input/output shapes, thread-to-data mapping, and instruction to emit. The canonical Hopper MMA atom for BF16 is:

using MMA_Atom = MMA_Atom<SM90_64x256x16_F32BF16BF16F32_SS>;

The name encodes: SM90 (Hopper), 64×256×16 tile dimensions, F32 accumulator, BF16 A and B inputs, F32 output, SS meaning both A and B come from shared memory.

A TMA copy is a copy atom:

using Copy_Atom_A = Copy_Atom<SM90_TMA_LOAD, bfloat16_t>;

The CUTLASS kernel template composes these atoms with tile dimensions, cluster shapes, and pipeline stages into a complete kernel:

using CollectiveMainloop = cutlass::gemm::collective::CollectiveMma<
    cutlass::gemm::MainloopSm90TmaGmmaRmemAAccumulator<3>,  // 3-stage pipeline
    Shape<_128, _256, _64>,                                   // tile MxNxK
    bfloat16_t, LayoutA,
    bfloat16_t, LayoutB,
    TiledMma,
    GmemTiledCopyA,
    SmemLayoutA,
    SmemCopyAtomA,
    cute::identity,
    GmemTiledCopyB,
    SmemLayoutB,
    SmemCopyAtomB,
    cute::identity
>;

This is verbose, but every template parameter maps to a concrete hardware mechanism: MainloopSm90TmaGmmaRmemAAccumulator<3> means “use TMA for loads, use WGMMA for compute, keep the accumulator in registers, with 3 pipeline stages”.

The compiler resolves all of this into a kernel where the main loop body is a tight sequence of WGMMA.MMA_ASYNC instructions, interrupted only by TMA-initiated mbarrier waits at stage boundaries.

The address computation for the TMA loads is essentially absent from the device code, having been moved to the descriptor construction on the host.

The Producer-Consumer Warpgroup Model

CUTLASS 3.x on Hopper adopts a warpgroup specialization model within each CTA. A thread block of 128 threads (one warpgroup) is divided at compile time into a producer warpgroup and one or more consumer warpgroups.

The producer warpgroup is responsible for issuing TMA loads (one thread per load, the others arrive at barriers). The consumer warpgroups are responsible for issuing WGMMA.MMA_ASYNC instructions and running the epilogue (writing C to global memory via the output TMA store).

This specialization is explicit:

if (warpgroup_id == 0) {
    // Producer: issue TMA loads into shared memory stages
    collective_mainloop.load(params, smem_tensors, pipeline, pipeline_state, k_tile_count);
} else {
    // Consumer: issue WGMMA instructions, run epilogue
    collective_mainloop.mma(params, smem_tensors, accumulators, pipeline, pipeline_state, k_tile_count);
    collective_epilogue.store(params, accumulators, ...);
}

The producer and consumer warpgroups communicate exclusively through the mbarrier-protected shared memory pipeline. There is no __syncthreads() between them in steady state. The barriers are sufficient.

This is architecturally important: __syncthreads() is a full CTA barrier. In a producer-consumer model where the producer and consumer have different amounts of work to do per iteration, a full CTA barrier would force the faster group to wait for the slower one on every iteration.

The mbarrier primitive allows asymmetric synchronization: the consumer waits only for the data it needs, not for the producer to reach any particular point in its control flow.

Subscribe now

The N-Stage Pipeline on Hopper

Part VII described double buffering (2 stages) on Ampere. On Hopper, CUTLASS uses 3 to 8 stages by default, with the optimal stage count depending on the tile size, problem size, and occupancy target.

The pipeline state machine on Hopper manages N shared memory stages, N producer mbarriers (one per stage, signaling data arrival), and N consumer mbarriers (one per stage, signaling that the consumer is done reading and the stage can be reused).

The steady-state loop looks like this conceptually:

Stage 0: [TMA load A0, B0] → [mbar_full[0] signaled] → [WGMMA on A0,B0] → [mbar_empty[0] signaled]
Stage 1: [TMA load A1, B1] → [mbar_full[1] signaled] → [WGMMA on A1,B1] → [mbar_empty[1] signaled]
Stage 2: [TMA load A2, B2] → [mbar_full[2] signaled] → [WGMMA on A2,B2] → [mbar_empty[2] signaled]
Stage 0: [TMA load A3, B3] → ...

The producer issues TMA loads into stage i and signals mbar_full[i]. The consumer waits on mbar_full[i], runs WGMMA, signals mbar_empty[i], and moves to stage (i+1) % N.

The producer waits on mbar_empty[i] before reusing that stage for the next load. This circular buffer in shared memory, managed by mbarrier pairs, is the fundamental data structure of a Hopper GEMM kernel.

The prologue loads N-1 tiles before the main loop begins (same invariant as Part VII’s double buffer prologue, just with more stages). The epilogue drains the remaining in-flight tiles after the k loop exits.

With 3 stages on an H100 with 228 KB of shared memory per SM (up from Ampere’s 192 KB), a 128×256 BF16 tile pair consumes approximately:

• A tile: 128 × 64 × 2 bytes = 16 KB

• B tile: 64 × 256 × 2 bytes = 32 KB

• Per stage: 48 KB

• 3 stages: 144 KB

• Remaining for mbarriers and accumulator spills: 84 KB

At 3 stages and a 128×256 tile, one CTA per SM is feasible. Two CTAs would require 288 KB, which exceeds the 228 KB shared memory limit.

Occupancy is therefore 1 CTA per SM, which is fine on Hopper because the single CTA fills the SM with WGMMA instructions and the TMA unit is fully occupied.

This is a fundamentally different occupancy philosophy from Ampere. On Ampere, you often needed 2-4 CTAs per SM to hide memory latency through warp-switching.

On Hopper, one CTA with TMA and WGMMA already achieves near-peak throughput on large tiles, because the hardware units that matter (TMA, tensor cores) are all fully occupied.

Subscribe now

What the Profiler Shows You on Hopper

The Nsight Compute metrics shift dramatically compared to Ampere.

smsp__warp_issue_stalled_long_scoreboard approaches zero. Not because the memory is fast, but because TMA loads do not involve the scoreboard at all. The SMSP is not waiting for memory; it is not the unit that issued the memory request.

smsp__warp_issue_stalled_mio_throttle is also low. The single TMA instruction per tile barely loads the MIO unit.

smsp__warp_issue_stalled_wgmma_global_wait is the new dominant stall: this is the SMSP waiting for a WGMMA.WAIT_GROUP to complete so it can read the accumulator registers.

This stall is unavoidable for kernels that read their accumulators between WGMMA groups (e.g., for split-K partial reductions). For kernels with long K dimensions, the WGMMA pipeline fills up and this stall disappears.

sm__pipe_tensor_op_hmma_cycles_active should be 80-95% for a well-tuned Hopper GEMM. Anything below 70% suggests either a pipeline depth problem (too few stages) or a cluster scheduling problem (the GPC is not scheduling the cluster CTAs together).

l1tex__data_pipe_lsu_wavefronts_mem_shared_op_ld counts shared memory read operations. For a kernel where both A and B are read from shared memory by WGMMA (SS variant), this metric reflects tensor core throughput, not programmer-visible loads. The tensor cores are reading shared memory directly, and this shows up in the LSU metrics.

The TMA throughput metrics are in the tma namespace: tma__read_bytes and tma__read_transactions. A kernel that is achieving peak TMA throughput will show TMA bandwidth close to the theoretical HBM bandwidth, because TMA is the only thing accessing HBM.

The key diagnostic insight on Hopper: if your WGMMA utilization is high and your TMA bandwidth is high, the kernel is good. The two hardware units are the bottleneck by design. Everything else should be idle or near-idle.

Subscribe now

The Roofline on Hopper, revisited

Part VII introduced the roofline model and noted that the useful diagnosis is hierarchical: not “memory-bound” but “memory-bound at the L2 level, achieving 60% of L2 peak”. On Hopper the hierarchy has the same levels (L1, L2, HBM) but new slopes.

H100 SXM5 roofline parameters:

• HBM3 peak bandwidth: 3.35 TB/s

• L2 peak bandwidth: approximately 12 TB/s (across 50 MB of L2, two slices)

• Shared memory peak bandwidth: approximately 33 TB/s aggregate (SM-local)

• Tensor core peak (dense BF16): 494 TFLOP/s

Ridge points:

• HBM ridge: 494 / 3.35 ≈ 147 FLOP/byte

• L2 ridge: 494 / 12 ≈ 41 FLOP/byte

• Shared memory ridge: 494 / 33 ≈ 15 FLOP/byte

For a GEMM with arithmetic intensity of 147 FLOP/byte or above, the kernel should be compute-bound assuming the memory hierarchy is properly utilized. Below 147 FLOP/byte, it is HBM-bandwidth-bound.

Below 41, even a perfect L2 hit rate cannot save you. Below 15, the tensor core throughput is limited by shared memory bandwidth, which means either bank conflicts or tile sizes that do not saturate the WGMMA datapath.

The key new insight on Hopper: TMA changes the shape of the memory hierarchy’s contribution. The SMSP instruction bandwidth, which was a secondary bottleneck on Ampere (and a primary bottleneck for small tiles), is effectively removed from the HBM bandwidth calculation.

The raw bandwidth to shared memory is now limited only by the TMA unit’s throughput, which the H100 documentation lists at approximately 900 GB/s aggregate (across all TMA units on all SMs).

This is below the HBM bandwidth of 3.35 TB/s, so for kernels that are purely bandwidth-limited (not compute-bound), TMA is not the constraint; HBM is.

For compute-bound kernels with large tiles, TMA’s instruction offloading is what enables the SMSP to run WGMMA at full throughput.

A Brief Look at Blackwell

Blackwell (SM100, B100/B200) was announced in March 2024 and began shipping to hyperscalers in late 2024. The architectural trajectory established by Hopper continues and accelerates.

The Blackwell tensor core introduces a 5th generation MMA with FP4 support (MXFP4 and NF4 formats), enabling 20 PFLOP/s peak at the full B200 system level (dual-die). The FP8 dense throughput is approximately 9 PFLOP/s per chip.

TMA on Blackwell gains native support for im2col pattern transforms (relevant for convolutions) and transposed stores, reducing the need for separate transpose kernels.

The cluster size limit increases to 16 CTAs (from 8 on Hopper), further amortizing B tile loads across more compute.

A new fifth-generation NVLink provides 1.8 TB/s bidirectional bandwidth per GPU in NVLink-connected systems (NVL72 rack), enabling multi-GPU kernels where the “global memory” seen by a TMA operation is distributed across 72 GPUs. This is the level at which the distinction between a single-GPU kernel and a distributed compute graph begins to blur.

CUTLASS 3.x supports Blackwell through new SM100 collective templates. The programming model is the same; the numbers are larger.

Subscribe now

Conclusion

The trajectory from Volta through Ampere to Hopper is a coherent story: every generation pushes more of the data movement machinery off the SMSP and onto dedicated hardware.

Volta gave you tensor cores, so the SMSP stopped doing the arithmetic. Ampere gave you cp.async, so the SMSP stopped waiting for loads. Hopper gave you TMA, so the SMSP stopped issuing loads entirely.

The SMSP on a well-tuned Hopper GEMM kernel is a machine that does one thing: issue WGMMA.MMA_ASYNC. Everything else has been delegated.

This is not an accident. It is the logical endpoint of the observation that matrix multiply is the kernel that matters most for modern ML workloads, and the most efficient hardware for matrix multiply is hardware where the compute units are never idle.

Every architectural innovation from 2017 onwards has been an attack on a different reason why the compute units were idle: arithmetic latency (tensor cores), memory latency (cp.async), instruction bandwidth (TMA), inter-SM bandwidth (clusters, NVLink).

The mbarrier, the tensor descriptor, the warpgroup specialization, the producer-consumer pipeline, the tile scheduler: these are not ornamental complexity.

They are the mechanisms by which a 2024 GPU running a 2024 kernel achieves 80-90% of theoretical peak on matrix multiply, a number that would have seemed implausible to practitioners writing hand-tuned BLAS routines a decade ago.

Part IX will step back from the single-GPU picture and look at multi-GPU parallelism: tensor parallelism, pipeline parallelism, NCCL, and the question of how NVLink bandwidth interacts with the per-GPU compute performance we have spent eight parts building up.

The tools change; the principle does not: find the bottleneck, route around it, measure again.

Where Part VI Left Us

Part VI ended with a sentence that deserves to be unpacked:

cp.async instructions do not set the long scoreboard.

The register file is not involved, so no register’s bit is marked pending.

The SMSP issues the cp.async, the copy engine takes it, and the SMSP is immediately free to issue the next instruction for that warp.

This is not a minor optimization note.

It is a description of a fundamentally different execution model: one that requires you to abandon the mental model of “instruction issues, result arrives, next instruction proceeds”, and replace it with something more like a production pipeline in a factory:

stages overlap, buffers exist between them, and throughput is determined by the slowest stage, not the sum of all stage latencies.

Before we can make cp.async do useful work, we need an accurate model of what it is hiding from: the memory hierarchy.

Subscribe now

The Memory Hierarchy of the A100

The A100 SXM4 has six levels of memory that matter to kernel programmers. They are not equally documented, and the numbers in marketing materials are frequently not the numbers in production code.

Registers

Each SM on Ampere has a 256 KB register file, shared across the four SMSPs: 64 KB per SMSP, with a 256-bit read port per cycle.

Register file access latency is effectively 0 cycles in the bypass case; for non-bypassed reads the cost is absorbed into the 4-cycle FMA pipeline. Registers are not a latency source. They are a capacity and bandwidth source.

The capacity limit is the one that matters: each thread can use at most 255 registers.

Pressure above this causes the compiler to spill values to local memory; a per-thread private region mapped to L1/L2/DRAM.

Spills are indistinguishable from any other global memory access at the hardware level: they go through the MIO unit, set the long scoreboard, and wait 400+ cycles for DRAM. Every spilled register costs two MIO operations.

Shared Memory / L1 Cache

Ampere’s per-SM L1 is a 192 KB pool partitioned between shared memory and the hardware L1 data cache.

The split is configurable (0/192, 32/160, 64/128, 100/92, 132/60, 160/32 (shared/cache, in KB)) via cudaFuncSetAttribute with cudaFuncAttributePreferredSharedMemoryCarveout.

Shared memory has 32 banks, each 4 bytes wide.

Bank index for a byte address:

bank = (address >> 2) & 31

Access patterns where multiple threads in a warp access different addresses in the same bank serialize.

One 4-bank conflict causes 4× the latency of the conflict-free case. The conflict-free latency is approximately 23 cycles; a 4-bank conflict extends this to ~35 cycles; an 8-bank conflict to ~51 cycles. The penalty scales linearly.

The broadcast exception: if all threads in a warp access the exact same address within a bank, the hardware services this as a single read and broadcasts the result.

Thirty-two threads accessing thirty-two different addresses that all map to the same bank is not a broadcast. It is a 32-way serialization.

L2 Cache

The A100 has 40 MB of L2 cache, split into two 20 MB slices. L2 hit latency: approximately 180–200 cycles, higher than most documentation implies.

Accesses to the local slice are ~160–180 cycles; accesses to the remote slice (requiring crossbar traversal) are ~200–230 cycles.

L2 bandwidth is approximately 4 TB/s aggregate. The ratio of L2 bandwidth to HBM bandwidth is approximately 15:1. Fitting a working set in L2 is qualitatively different from spilling it to HBM.

HBM2e

The A100 SXM4 has six HBM2e stacks providing a peak theoretical bandwidth of 2 TB/s. In practice: a kernel with access pattern regularity sufficient to saturate all channels achieves 1.6–1.9 TB/s.

Irregular access patterns with row buffer conflicts: 800 GB/s–1.2 TB/s. Random byte-granularity reads: tens of GB/s, due to cache line waste.

HBM2e latency, measured with L1 and L2 bypassed: approximately 450–600 cycles at 1410 MHz. Row buffer hits land around 300–350 cycles; misses around 550–650 cycles.

The consequence at 1410 MHz: 500 cycles × 0.71 ns/cycle ≈ 355 nanoseconds of stall per warp. In that window, 500 instruction issue slots across the SM go dark.

If every resident warp has issued an HBM load and is waiting, you have a 500-cycle stall with no eligible warp to rescue you.

This is the memory wall in concrete form. The solution is not a faster memory: it is to restructure data movement so that HBM latency is overlapped with computation.

Subscribe now

The cp.async Instruction

cp.async was introduced in Ampere (sm_80). It performs a direct DMA-like transfer from global memory to shared memory, bypassing the register file entirely:

cp.async.ca.shared.global [dst], [src], size;
cp.async.cg.shared.global [dst], [src], size;   // bypass L1

The size parameter is 4, 8, or 16 bytes. The 16-byte variant is the most important: it issues a vectorized LDG.128, achieving maximum memory interface utilization.

What “bypassing the register file” actually means

The conventional load path:

LDG.128 R4, [R2]         ; → long scoreboard set for R4,R5,R6,R7
                         ; → warp stalls on any read of R4-R7
                         ; → 450-600 cycles later, HBM returns data
STS.128 [smem_ptr], R4   ; store registers → shared memory

This requires 4 registers in transit. The load sets four long scoreboard bits. The warp is ineligible for any instruction reading R4–R7 until the HBM transaction completes.

The cp.async path:

CP.ASYNC.CA.SHARED.GLOBAL [smem_dst], [R2], 0x10
; → no scoreboard bits set (no destination register)
; → warp immediately eligible to issue next instruction
; → data arrives in shared memory asynchronously

A dedicated Ampere Asynchronous Copy Engine receives the request via the MIO unit, takes ownership of the transaction, and performs the HBM load and shared memory write independently of the SMSP. The MIO unit is freed immediately after handoff.

The commit/wait mechanism

Commit (CP.ASYNC.COMMIT_GROUP): marks all preceding cp.async instructions as a commit group. Bookkeeping only,does not wait for anything.

Wait (CP.ASYNC.WAIT_GROUP N): stalls until at most N commit groups remain pending. N=0 is complete synchronization.

N=1 allows one in-flight group to remain outstanding while you compute on the previous.

auto pipe = cuda::make_pipeline();

for (int i = 0; i < BATCH_SIZE; i++)
    cuda::memcpy_async(smem[0][i], &gmem[base + i], sizeof(float4), pipe);
pipe.producer_commit();

for (int i = 0; i < BATCH_SIZE; i++)
    cuda::memcpy_async(smem[1][i], &gmem[base + BATCH_SIZE + i], sizeof(float4), pipe);
pipe.producer_commit();

pipe.consumer_wait();   // CP.ASYNC.WAIT_GROUP 1
__syncthreads();        // mandatory: propagates visibility to all threads

compute(smem[0]);

The __syncthreads() after consumer_wait is mandatory. consumer_wait ensures the data is in shared memory from the perspective of this warp.

Other warps in the thread block may not see the writes until __syncthreads() propagates them through the SM’s coherence domain.

Omitting it is a race condition: one that produces correct results most of the time and incorrect results unpredictably under heavy memory pressure.

The Double Buffer Pattern

A standard tiled GEMM loop is fully sequential: load tile, sync, compute, sync, repeat. The timeline is a flat chain of dependencies. For smaller problems or thinner tiles where T_load / T_compute > 1, the kernel is memory-bound.

The double buffer pattern breaks that chain:

Iter k:   |-- cp.async A[k] --|-- cp.async B[k] --|-- commit --|
                                                                |-- wait(k-1) --|-- compute(k-1) --|
Iter k+1: |-- cp.async A[k+1] --|-- cp.async B[k+1] --|-- commit --|
                                                                    |-- wait(k) --|-- compute(k) --|

Loads for iteration k+1 overlap with computation of iteration k. Memory latency is hidden as long as T_load(k+1) < T_compute(k). The pipeline then runs at the compute rate with zero memory stall.

This requires two ping-pong buffers in shared memory, doubling the shared memory requirement.

Doubling shared memory per thread block halves the maximum resident thread blocks per SM, reducing occupancy. The trade-off is explicit and computable.

Diagnostic signal: if smsp__warp_issue_stalled_long_scoreboard.avg.pct_of_peak_sustained_active exceeds 20%, memory latency is not being hidden. The first intervention is higher occupancy.

The second, when occupancy is already near maximum, is cp.async pipelining, which removes the long scoreboard from the equation entirely.

The Full Kernel Pattern

constexpr int TILE_M = 128, TILE_N = 128, TILE_K = 32;
constexpr int NUM_STAGES = 2;

__global__ void gemm_async_kernel(
    const __nv_bfloat16* __restrict__ A,
    const __nv_bfloat16* __restrict__ B,
    float* __restrict__ C,
    int M, int N, int K
) {
    __shared__ __nv_bfloat16 smem_A[NUM_STAGES][TILE_M][TILE_K];
    __shared__ __nv_bfloat16 smem_B[NUM_STAGES][TILE_K][TILE_N];
    float acc[4][4] = {};

    auto pipe = cuda::make_pipeline();
    const int k_tiles = K / TILE_K;

    // PROLOGUE: issue tile 0 before the main loop
    if (k_tiles > 0) {
        int row_a = threadIdx.x / TILE_K, col_a = threadIdx.x % TILE_K;
        if (row_a < TILE_M)
            cuda::memcpy_async(&smem_A[0][row_a][col_a],
                               &A[(blockIdx.y * TILE_M + row_a) * K + col_a],
                               sizeof(__nv_bfloat16), pipe);
        pipe.producer_commit();
    }

    // MAIN LOOP
    for (int k = 1; k < k_tiles; k++) {
        const int sw = k % 2, sr = (k - 1) % 2;

        int row_a = threadIdx.x / TILE_K, col_a = threadIdx.x % TILE_K;
        if (row_a < TILE_M)
            cuda::memcpy_async(&smem_A[sw][row_a][col_a],
                               &A[(blockIdx.y * TILE_M + row_a) * K + (k * TILE_K + col_a)],
                               sizeof(__nv_bfloat16), pipe);
        pipe.producer_commit();

        pipe.consumer_wait();   // CP.ASYNC.WAIT_GROUP 1
        __syncthreads();

        for (int ki = 0; ki < TILE_K; ki++)
            for (int i = 0; i < 4; i++)
                for (int j = 0; j < 4; j++)
                    acc[i][j] += __bfloat162float(smem_A[sr][threadIdx.y*4+i][ki])
                               * __bfloat162float(smem_B[sr][ki][threadIdx.x*4+j]);
        __syncthreads();
    }

    // EPILOGUE
    pipe.consumer_wait();   // CP.ASYNC.WAIT_GROUP 0
    __syncthreads();
}

Three things to internalize about this structure:

The prologue is not optional. Without issuing tile 0 before the loop, the first consumer_wait blocks on a commit group that doesn’t exist. Undefined behavior. The prologue establishes the “one stage ahead” invariant that the loop depends on.

Both synchronization primitives are required. consumer_wait ensures the DMA engine has written the data to shared memory for this warp. __syncthreads() ensures all threads in the block have reached this point before any thread reads.

They solve different problems. Neither substitutes for the other.

Stage read and stage write are never equal. The modular arithmetic guarantees sw ≠ sr for NUM_STAGES = 2. The DMA engine writes to one buffer while threads read from the other.

With NUM_STAGES ≥ 3 you deepen the pipeline, more latency hidden, more shared memory consumed.

N-Stage Generalization

With N stages, you issue N tiles’ worth of cp.async before the first computation begins. The latency is hidden when T_compute(tile) > T_HBM_load / N.

CUTLASS implements up to 5-stage pipelines for its Ampere GEMM kernels, with stage count as a compile-time template parameter swept by the profiler at tuning time. The shared memory cost scales linearly with stage count.

At some crossover point the shared memory requirement forces an occupancy reduction that exceeds the pipelining benefit.

This crossover depends on the specific kernel and problem size, which is why CUTLASS exposes the parameter rather than hardcoding it.

What the Profiler Shows You

Before pipelining (conventional LDG loads):

• smsp__warp_issue_stalled_long_scoreboard — 40–70%, dominant stall

• smsp__pipe_fma_cycles_active — 30–60%, computation starved

After pipelining (cp.async, double buffer):

• smsp__warp_issue_stalled_long_scoreboard. <5%, cp.async sets no scoreboard bits

• smsp__pipe_fma_cycles_active. 70–90% for a well-tuned kernel

• Watch for smsp__warp_issue_stalled_mio_throttle; if you issue cp.async faster than the MIO unit can service them (~1 per 4 cycles per SMSP for 128-bit transfers), this stall replaces the scoreboard stall.

  The fix is larger tiles or accepting the throttle if MIO throughput still exceeds compute throughput.

Bank Conflicts

The 32-bank model is documented. The practical implications for matrix access patterns are not.

In a tiled GEMM, tile A is loaded into shared memory in row-major layout, then read column-wise during the multiply.

For TILE_K = 32 and BF16 elements (2 bytes each), element [j][i] sits at byte offset j × 64 + i × 2. Bank index: (j × 16 + i/2) & 31.

For a warp reading column i (i fixed, j running 0..31) every pair of threads maps to the same bank. This is a 2-way bank conflict on every column read.

The fix is padding:

__shared__ __nv_bfloat16 smem_A[TILE_M][TILE_K + 2];   // +2 BF16 = +4 bytes per row

With the pad, element [j][i] is at byte offset j × 68 + i × 2. Bank index: (j × 17 + i/2) & 31. Since gcd(17, 32) = 1, the bank indices as j runs 0..31 form a complete permutation of 0..31. Zero conflicts.

The shared memory overhead is TILE_M × 4 bytes per buffer: 512 bytes for TILE_M = 128, trivial against the 8 KB tile.

CUTLASS’s Swizzle technique achieves the same result via address bit permutation rather than linear padding, which handles non-power-of-two tile sizes cleanly.

The arithmetic underneath is identical.

L1 Cache Policy

Cache behavior on Ampere is configurable at the instruction level:

Qualifier Behavior LDG.CA Cache in L1 (default) LDG.CG Bypass L1, go to L2 LDG.CS Streaming: insert at LRU position LDG.CV Bypass all caches (almost never correct)

In CUDA: __ldg() for L1-cached, __ldcg() / __ldcs() for the bypass variants. The compiler defaults to LDG.CA when uncertain.

For kernels that process each input element exactly once, elementwise operations, reductions, anything with no reuse, __ldcg() eliminates L1 pollution and preserves L1 capacity for data that does benefit from caching.

The effect in the profiler: lower L1 hit rate, unchanged L2 hit rate. The data skips one cache level without reducing effective bandwidth at the level where reuse actually exists.

The Roofline Model

The roofline model (Williams, Waterman, Patterson, 2009) plots FLOP/s against arithmetic intensity (FLOP/byte of DRAM traffic). For the A100 in FP32:

• Peak compute: ~19.5 TFLOP/s

• Peak HBM bandwidth: ~2 TB/s

• Ridge point: ~9.75 FLOP/byte

Below the ridge: memory-bound. Above: compute-bound. The common mistake is treating DRAM bandwidth as the only line that matters.

The L2-based roofline has a ridge at ~4.9 FLOP/byte. The L1-based roofline has a ridge at ~1 FLOP/byte.

A kernel with strong L1 reuse can be compute-bound at an arithmetic intensity that looks memory-bound on the DRAM roofline.

A kernel that thrashes L2 will underperform the DRAM roofline because its effective bandwidth is below the theoretical peak. NCU’s roofline chart shows all three simultaneously.

The correct first diagnostic is hierarchical bandwidth analysis. Not “it’s memory-bound”; that’s a category.

The useful diagnosis is “it’s memory-bound at the L2 level, achieving 60% of L2 peak, because 40% of L2 bandwidth is wasted on non-reused data evicted before second use.” That tells you the fix.

Subscribe now

The Tensor Memory Accelerator

Ampere introduced cp.async. Hopper (sm_90, H100) introduced the Tensor Memory Accelerator (TMA), the same idea taken to its logical conclusion.

With cp.async, the programmer still computes every element’s global memory address and constructs the instruction stream.

For a 128×128 BF16 tile, that is 512 vectorized 128-bit cp.async instructions consuming SMSP instruction bandwidth, even though the transfers are asynchronous.

TMA accepts a tensor descriptor (base address, dimensions, strides, element type) and issues a single instruction:

cp.async.bulk.tensor.2d.shared::cluster.global.mbarrier::complete_tx::bytes
    [smem_dst], [gmem_desc, {coord_y, coord_x}], [mbar];

One instruction. One 128×128 BF16 tile. The TMA unit generates all the addresses, manages all the transactions, and signals completion via the mbarrier primitive;

a synchronization mechanism lighter than __syncthreads(), designed for producer-consumer coordination without a full SM barrier.

The consequence: on Hopper, the compute-to-load instruction ratio in a GEMM inner loop approaches ∞ from the SMSP’s perspective.

The SMSPs run wgmma.mma_async continuously; the TMA unit handles all data movement independently. CUTLASS 3.x is designed around this model. Part VIII will cover it in full.

Subscribe now

Conclusion

The line from Part VI, “the SMSP is immediately free to issue the next instruction for that warp”, is the hinge on which this article turns.

The memory hierarchy imposes latencies that are not negotiable in nanoseconds: 23 cycles for shared memory, 180 for L2, 500 for HBM. These numbers do not change by complaining about them.

They change by structuring code so that the latency is incurred before the result is needed: issuing the memory request while computing on previously loaded data.

cp.async is the mechanism. Software pipelining is the pattern. Double buffering is the minimum viable instance. The commit/wait protocol maintains correctness while the DMA engine and the compute engine run simultaneously.

The bank conflict analysis and the L1 bypass discussion are extensions of the same idea: minimize latency and maximize effective bandwidth at every level of the hierarchy, so that by the time data arrives at the computation, it has traveled through the hardware as efficiently as physics allows.

The limits of this approach on Ampere are what motivate TMA on Hopper: an architecture where the gap between what the programmer expresses and what the hardware executes narrows further, approaching the regime where the programmer describes what should move and the hardware decides when.

Part VIII begins there.

Share

The Pipeline’s One Promise, and How It Fails

The A100 SM runs at a base clock of approximately 765 MHz, boost to ~1410 MHz. At boost, one clock cycle is ~0.71 nanoseconds. The SM has four SMSPs.

Each SMSP has four warp schedulers (confirmed in NVIDIA’s Ampere whitepaper and independently via microbenchmarks by Jia et al. and the work of Markidis, Larsson et al.).

Each scheduler attempts to issue one instruction per cycle to one eligible warp.

At full throughput (all four schedulers in all four SMSPs issuing every cycle) a single A100 SM issues 16 instructions per cycle.

Across 108 SMs at 1410 MHz, peak issue rate is roughly 2.4 trillion instructions per second. This is the theoretical ceiling. You will never reach it. The question is why, and by how much.

An instruction issues in a given cycle when three conditions are simultaneously true:

1. The warp is eligible: it has been selected by the round-robin/priority scheduler, it is not stalled on a scoreboard dependency, and it has not exceeded the warp’s instruction buffer depth.

2. The execution unit is available: the target pipe (FMA, SFU, MIO, LSU...) has a free slot.

3. All operands are ready: every source register’s scoreboard bit has been cleared by its producing instruction.

When any of these three conditions fails, the scheduler increments a stall counter and moves to another warp.

The beauty of the GPU microarchitecture, and the central insight of GPU optimization, is that condition (3) failing for warp A doesn’t stall the SM; it just causes the scheduler to attempt warp B instead.

The SM stalls only when no warp satisfies all three conditions simultaneously. That’s the failure mode we are trying to prevent.

ncu --metrics l1tex__t_sectors_pipe_lsu_mem_global_op_ld.sum is how you measure condition (3) failures for memory. smsp__warp_issue_stalled_* counters measure them by category.

We will use both throughout.

Subscribe now

What lives inside one SMSP

Before discussing stalls, you need an accurate map of what execution units exist and what their throughput and latency look like.

Much of the confusion in GPU optimization literature stems from people using “the FP32 pipe” as a monolith when it is not.

One Ampere SMSP contains, per NVIDIA’s Ampere Architecture whitepaper and corroborating microbenchmark work:

A few notes on the accuracy of this table:

The FMA latency of 4 cycles is confirmed by the CUDA C Programming Guide and by numerous independent microbenchmarks.

It is not 1 cycle. It is not 2 cycles. It is 4, and every serial dependency chain in your kernel pays it in full.

The SFU latency of 16 cycles is confirmed by microbenchmarks. Throughput of 4 cycles/instruction means one MUFU occupies the single SFU for 4 cycles; other warps’ MUFU instructions queue behind it.

Since there is only one SFU per SMSP and the SMSP has 16 warps maximum, a warp issuing a MUFU must wait for the SFU to become free if another warp issued one within the last 4 cycles.

The FP64 throughput asymmetry is critical for A100 versus A30/A10 hardware: the A100 has full-rate FP64 (2 cycles per DFMA per SMSP), while the A10 has 1/16th the FP64 throughput (DFMA at 32 cycles per instruction).

Running FP64 code on an A10 is not slower: it is catastrophically slower. Verify your hardware before benchmarking.

Shared memory load latency of 23 cycles is confirmed by microbenchmarks (Luitjens 2011, Volkov 2016, and more recently by Yan et al. in their SM scheduling simulator).

The official CUDA documentation says “~20 cycles” without precision; 23 cycles is the empirically correct number for Ampere under normal bank-conflict-free access.

With 4-way bank conflicts the effective latency compounds because the MIO pipe is occupied for additional cycles while the bank serialization completes.

The scoreboards in detail

Each SMSP has two scoreboards, as described in Part V. Understanding their interaction with execution units is worth revisiting with more precision:

Short scoreboard: covers arithmetic results from the FMA pipe, INT32 ALU, and SFU. Latency tracked: 4 cycles (FMA/INT32) and 16 cycles (SFU). The scoreboard has one bit per register per warp.

When an FFMA issues with destination R4, bit R4 for that warp is set in the short scoreboard. It is cleared 4 cycles later (for FMA results) by the pipeline’s bypass network.

An instruction in another warp that reads R4 of the issuing warp is unaffected: scoreboards are per-warp, not global.

Long scoreboard: covers memory results: any instruction that issues to the MIO unit (loads from global/shared/local memory, atomic operations).

The long scoreboard bit is set when the load issues and is not cleared until the data physically arrives and is written to the register file.

For an HBM access this can be 400+ cycles. The SMSP does not know in advance how long an HBM access will take (it depends on DRAM row buffer state, competing traffic, etc.); it just waits for the completion signal from the memory system.

An important subtlety: cp.async instructions do not set the long scoreboard. This is the mechanism by which they achieve asynchrony.

The register file is not involved, so no register’s bit is marked pending.

The SMSP issues the cp.async, the copy engine takes it, and the SMSP is immediately free to issue the next instruction for that warp.

We will return to the exact implications of this in future posts.

Subscribe now

The SFU, quantified completely

The Special Function Unit executes MUFU instructions. Let’s be more precise than “the SFU is slow.”

On Ampere, one SMSP has one SFU. The SFU pipeline is 4 stages deep; this is why it has a 4-cycle throughput: one new instruction can enter the pipeline every 4 cycles (this is the initiation interval, II), and the result is available 16 cycles after issue (this is the latency, L = 4 × II).

This is a reasonable structural design for a unit that computes hardware approximations to transcendentals: the underlying Newton-Raphson iterations take multiple stages.

The key distinction: throughput (4 cycles/instruction) limits how often you can issue MUFU instructions from the same SMSP.

Latency (16 cycles) limits how soon a downstream instruction can use the MUFU result.

Both matter; they fail you in different scenarios.

What MUFU opcodes actually compute

MUFU is not one instruction. It is one instruction format with an operation selector.

The compiler maps standard C math functions to MUFU as follows.

This mapping is important because the additional FFMA instructions required for argument scaling and result scaling come for free (they don’t touch the SFU), but they do consume FP32 pipe cycles:

expf(x) → __expf(x) (with -use_fast_math):

// Argument reduction: convert from base-e to base-2
// exp(x) = 2^(x * log2(e)) = 2^(x * 1.44269504...)
FMUL   R1, R0, 1.44269502f          ; x * log2(e), FP32 pipe, 0.25 cycles throughput
MUFU.EX2  R2, R1                    ; 2^(x*log2e), SFU, 4 cycles throughput

expf(x) (IEEE-compliant, without fast-math):

// Range check and reduction (compiler-generated, varies)
FMNMX  R1, R0, 88.722839f, ...      ; clamp to avoid overflow  
FFMA   R2, R1, 1.44269502f, ...     ; argument reduction with correction term
MUFU.EX2  R3, R2                    ; core computation
FMUL   R4, R3, ...                  ; reconstruction (potentially)
// Plus additional corrections for subnormals, NaN, INF

The IEEE-compliant version may issue conditional branches for edge cases. When your kernel has inputs that might be NaN, INF, or very large/small, the compiler generates defensive code.

__expf() removes these guards entirely: it is undefined behavior for inputs outside [−87.3, 88.7] (the approximate FP32 range of exp before overflow/underflow).

If you know your inputs are bounded, and in softmax after max-subtraction they are, since all values are ≤ 0, __expf() is always the correct choice.

tanhf(x) (any mode):

tanh has no single MUFU opcode. The compiler implements it using the identity: tanh(x) = 1 - 2/(exp(2x)+1).

The resulting SASS (approximately, varies by compiler version) includes:

FMUL   R1, R0, 2.0f                 ; 2x
FMUL   R2, R1, 1.44269502f          ; 2x * log2(e)
MUFU.EX2  R3, R2                    ; 2^(2x * log2e) = exp(2x)
FADD   R4, R3, 1.0f                 ; exp(2x) + 1
MUFU.RCP  R5, R4                    ; 1/(exp(2x)+1)
FMUL   R6, R5, 2.0f                 ; 2/(exp(2x)+1)
FADD   R7, -R6, 1.0f                ; 1 - 2/(exp(2x)+1) = tanh(x)

That’s two MUFU instructions (one EX2, one RCP) per tanhf call; 8 cycles of SFU pipe occupied per call in throughput terms.

For GELU, which uses tanhf internally (the fast approximation 0.5x(1+tanh(√(2/π)(x+0.044715x³)))), you have additional FFMAs on top.

GELU activation in a fused kernel is expensive in SFU terms, which is one motivation for the simpler SiLU activation (x * σ(x) = x / (1 + exp(-x))): it requires one MUFU.EX2 plus a few FMAs versus two MUFUs for tanh.

Throughput model for an SFU-bottlenecked loop

Suppose your kernel’s inner loop body, after compilation, contains:

• 1× MUFU.EX2 (4-cycle throughput, SFU pipe)

• 3× FFMA (0.25-cycle throughput each, FMA pipe)

• 2× FADD (0.25-cycle throughput each, FMA pipe)

• 1× LDS (shared memory load, ~1-cycle throughput assuming no bank conflict)

Total FMA pipe demand: 5 × 0.25 = 1.25 cycles
Total SFU pipe demand: 1 × 4.0 = 4.0 cycles
Total MIO demand: 1 × ~1.0 = ~1.0 cycles

The SFU is the bottleneck: the loop cannot issue faster than 4.0 cycles per iteration.

The FMA pipe is occupied 1.25/4.0 = 31% of the time. The remaining 69% of FMA pipe capacity is wasted, waiting for the SFU to finish so the next iteration can begin.

You can fill this gap in two ways: more ILP within the loop (unroll and issue multiple independent MUFU calls, keeping both the SFU and FMA pipe busier) or replace MUFU with FMA-pipe arithmetic.

The first approach doesn’t change the SFU ceiling; it just makes better use of the FMA pipe in parallel.

The second moves the ceiling.

Polynomial exp replacement: the real implementation

The “4th order polynomial” approach described in the previous version of this article is plausible but underspecified.

Here is a properly validated implementation using a piecewise approach compatible with softmax use cases:

// Fast exp2f approximation — pure FP32, no SFU
// Maps to ~6 FFMAs in SASS
// Valid for x ∈ [-126, 127] (FP32 normal range for 2^x)
// Error: < 2^-23 relative for x ∈ [-16, 16] (sufficient for softmax)
__device__ __forceinline__ float fast_exp2f_fma(float x) {
    // Decompose x = n + f where n is integer, f ∈ [-0.5, 0.5]
    float n = __float2int_rn(x);      // round to nearest int — FMUL-based  
    float f = x - n;                  // fractional part

    // Minimax polynomial for 2^f over [-0.5, 0.5]
    // Coefficients: Sollya minimax degree-4 in Horner form
    // 2^f ≈ 1 + f*(0.693147 + f*(0.240227 + f*(0.055504 + f*0.009618)))
    float p = 0.009618f;
    p = fmaf(p, f, 0.055504f);
    p = fmaf(p, f, 0.240227f);
    p = fmaf(p, f, 0.693147f);
    p = fmaf(p, f, 1.0f);             // 2^f approximation

    // Reconstruct 2^x = 2^n * 2^f via integer exponent manipulation
    // Pack n into FP32 exponent bits: (int)(n + 127) << 23
    int e = __float2int_rn(n) + 127;
    float scale = __int_as_float(e << 23); // exact power of 2, no error

    return p * scale;
}

// For expf(x): expf(x) = exp2f(x * log2(e))
__device__ __forceinline__ float fast_expf_fma(float x) {
    return fast_exp2f_fma(x * 1.4426950408889634f);
}

SASS output for fast_expf_fma: approximately 8 FFMAs, 1 F2I, 1 I2F, 1 integer SHL, 1 FMUL. No MUFU.

Throughput: ~2–2.5 cycles per call on the FMA pipe. Versus MUFU.EX2 at 4 cycles: a genuine 1.6–2× throughput improvement for softmax inner loops on Ampere.

The catch: verify SASS output yourself. The compiler has latitude with __int_as_float and __float2int_rn.

Confirm with nvdisasm that no MUFU instructions appear in the compiled output.

Measuring SFU utilization precisely

The two relevant Nsight Compute metrics:

smsp__pipe_fma_cycles_active.avg.pct_of_peak_sustained_active
smsp__pipe_xu_cycles_active.avg.pct_of_peak_sustained_active

On an SFU-bottlenecked kernel, xu (XU = execution unit, NVIDIA’s internal name for the SFU pipe) will be near 100% and fma will be proportionally lower.

The ratio xu_cycles / fma_cycles tells you the SFU/FMA throughput imbalance directly.

Also useful: smsp__average_warp_latency_per_inst_executed.ratio; if this is high while xu_cycles_active is also high, the warp latency is being driven by MUFU’s 16-cycle result latency, not just its 4-cycle throughput.

Both cost you, via different mechanisms.

Subscribe now

The L0 Instruction cache

Each SMSP on Ampere has a dedicated 32 KB L0 instruction cache (also referred to as the I-cache in some microarchitecture literature).

This is physically separate from the unified L1 data/shared memory: it is not carved from the 192 KB L1 pool.

The L0 is private to each SMSP; four SMSPs per SM means four independent L0 caches per SM.

Instructions on Ampere are 128 bits (16 bytes) wide. The L0 holds 32 KB / 16 B = 2048 instructions.

A typical kernel loop body of 100–300 instructions fits comfortably; the L0 warms up on the first iteration and subsequent fetches are essentially free (one cycle or less).

The exception: kernels generated from heavily templated C++ code (think Thrust or hand-unrolled matrix multiplication with large tile sizes) can have loop bodies exceeding 500–1000 instructions.

A kernel that fully unrolls a 256-wide loop body with 8 FMAs per iteration emits 2048 instructions for that loop; exactly filling the L0 and leaving nothing for the rest of the kernel.

Add one more instruction and you start thrashing.

When the L0 misses, the SMSP must fetch from L1 instruction cache (shared with data traffic, with associated latency) or, worse, from L2.

L1 instruction fetch latency is approximately 20–30 cycles. The miss is captured by:

smsp__pcsamp_warps_issue_stalled_imc_miss.sum

Values above 2% indicate a structural code-size problem. Values above 10% are severe.

There is no runtime mechanism to manage L0 occupancy. The only intervention is compile-time code size reduction:

• Replace #pragma unroll N with smaller N or #pragma unroll 1 for large N

• Mark non-critical helper functions with __noinline__

• Split large kernels into kernel launch sequences (costs launch overhead; evaluate the trade-off)

• Use --maxrregcount to limit register count, which sometimes causes the compiler to generate shorter instruction sequences

Instruction decode bandwidth

Decoded instructions are held in per-warp instruction buffers before issue.

On Ampere, these buffers are approximately 2 entries deep per warp (this is not officially documented; it is reverse-engineered from microbenchmarks.

Specifically, from observing that back-to-back dependent instructions with 1-cycle-latency arithmetic operations still issue without stall, implying at least 2-deep pre-decoding).

The decoder can process approximately 1 instruction per cycle per SMSP (across all warps).

This exceeds the issue rate for any single warp (maximum 1 instruction every 4 cycles for a compute-bound warp at peak), so the decoder runs ahead and the per-warp instruction buffer is almost always populated.

The pathological case: a kernel at very high occupancy (32 warps per SMSP, the A100 maximum) with a simple loop body of 3 instructions (say, a vectorized element-wise operation: LDG.128, FFMA.x4, STG.128).

All 32 warps are eligible every cycle. The decoder must keep all 32 instruction buffers populated.

At 1 decode per cycle and 32 warps each needing fresh instructions, the decoder is stretched.

If the instruction stream is not in L0 (forcing L1 fetch at 20+ cycle latency), the buffers drain and the schedulers stall even though 32 eligible warps exist.

This is rare but real. It manifests as a high smsp__pcsamp_warps_issue_stalled_imc_miss combined with near-100% occupancy;

confusing until you understand that 32 resident warps generates 32× the instruction fetch pressure of 1 warp.

Predicated execution

Each thread on Ampere has 7 predicate registers (P0 through P6).

These are separate from the 255 available scalar registers (R0–R254, with R255 reserved as the zero register).

Predicate registers are 1-bit values set by comparison instructions:

// Source: if (a > b) { ... }
FSETP.GT.AND P0, PT, R0, R1, PT   ; set P0 = (R0 > R1), unconditional (PT = true predicate)

FSETP.GT.AND P0, PT, R0, R1, PT reads as: “set predicate P0 to (R0 > R1) AND PT, and set the complement predicate (implicit) to the inverse, and all of this unconditionally (final PT).”

The AND/OR suffix specifies the combining mode for nested predicate logic.

This instruction issues on the FP32 pipe, costs 4-cycle latency, and produces a predicate bit, not a register value.

An instruction with a predicate prefix:

@P0    FFMA  R3, R1, R2, R3    ; execute FFMA only if P0 is true
@!P0   FFMA  R5, R1, R2, R5   ; execute FFMA only if P0 is false

The semantics at the hardware level: all lanes in the warp issue the instruction. The instruction traverses the pipeline.

When the result write-back occurs, it is gated by the predicate: lanes where the predicate is true write their result; lanes where it is false suppress the write-back.

No branch. No warp divergence. No reconvergence stack manipulation.

Consequence: predicated instructions consume throughput proportional to the total number of instructions, not proportional to the number of active lanes.

A 32-thread warp where 16 threads have P0=true and 16 have P0=false, executing @P0 FFMA R3, R1, R2, R3, consumes exactly the same FFMA pipe resources as all 32 threads having P0=true.

The 16 non-writing threads waste their execution slots.

This is the precise definition of “predicated execution trades throughput for divergence avoidance.”

The compiler’s branch/predicate decision heuristic

The CUDA compiler (nvcc, using the LLVM PTX backend) uses a cost model to decide between a BRA (branch) and predication. The model is approximately:

Predication is chosen when:

• The combined instruction count of both branch arms is ≤ ~8–12 instructions total

• OR the divergence probability is estimated to be high (many warps will have mixed predicate values)

• OR the branch target is not cache-resident (branch prediction overhead is higher)

Branch is chosen when:

• One arm is long (> ~6 instructions) and the other is short

• The compiler can estimate that the majority of warps will take one branch uniformly

• The branch condition is amenable to warp-uniform evaluation (all threads agree)

The threshold is not a hard constant: it depends on the compiler version, optimization level, and the surrounding code structure.

The reliable way to check what the compiler chose is to inspect SASS:

# Disassemble a compiled kernel to SASS
nvdisasm --print-instruction-types mykernel.cubin | grep -E "BRA|@P[0-9]"

Or, within Nsight Compute:
Source tab → enable “Source Counters” → switch to “SASS” view → look for @P0 prefixes versus BRA instructions in the hot loop.

When to override the compiler’s choice

The compiler is generally right. The cases where it is wrong:

Case 1: Long rare branch incorrectly predicated. If your hot loop has a condition triggered 1% of the time (e.g., an overflow check, a boundary condition), and the expensive handler is 15 instructions, the compiler might still predicate if the loop body is otherwise short and the combined instruction count falls under the threshold.

But 15 instructions × 32 threads × 1% frequency = the equivalent of 0.15 × 32 = ~5 instructions of wasted throughput per loop iteration, running at full throughput instead of 1% of it. Branch would cost nothing for the 99% case.

Fix: restructure the code to make the “expensive” path obviously large and separated: e.g., a function call rather than inlined code, which the compiler treats as a definite branch site.

Case 2: Warp-uniform condition incorrectly compiled as branch. If every thread in a warp evaluates the same condition (e.g., based on blockIdx.x or a value loaded from constant memory that all threads share), the warp takes the branch uniformly and pays zero divergence cost.

The compiler sometimes generates a branch here and sometimes predicates. When the branch is warp-uniform and the body is long, you want a branch (all threads skip the long body together); predication would execute the long body for all threads on every iteration.

You can encourage warp-uniform branch treatment by computing the predicate with __all_sync(__activemask(), condition) when you know it’s warp-uniform — this makes the intent explicit.

Conclusion

Predication is not a free lunch, and it is not free branching.

It is a specific trade: you pay throughput for all threads to avoid the divergence tax of splitting and reconverging a warp.

That trade is profitable when the branch body is short and divergence is likely.

It is catastrophically unprofitable when the branch body is long and most threads would have skipped it entirely.

The compiler’s heuristic gets this right most of the time, because most conditionals in well-written kernels are short.

The cases where it fails: a rare overflow handler that gets predicated, a warp-uniform load flag that gets branched, are invisible at the source level and only show up as unexplained throughput loss in the profiler.

The smsp__warp_issue_stalled_not_selected stall counter rising without a corresponding increase in occupancy is one signal; anomalously low FMA pipe utilization relative to the instruction count is another.

The discipline is the same as everywhere else in this series: don’t assume the compiler made the optimal choice.

Inspect the SASS, verify the @P prefixes are where you expect them and absent where you don’t, and use __all_sync to make warp-uniform conditions structurally explicit rather than relying on the compiler to infer them.

A predicate register costs nothing. A 15-instruction predicated block running at full throughput for 99% of warps that didn’t need it costs exactly as much as running it unconditionally; which is, in fact, what you did.​​​​​​​​​​​​​​​​

The instruction pipeline

There is a moment in GPU optimization work that comes after the first profiling session, after you’ve internalized roofline arithmetic and stopped chasing occupancy for its own sake.

You’ve fixed the obvious things. Coalescing is clean. Shared memory is tiled. Bank conflicts are gone. Register pressure is measured and tolerated. Arithmetic intensity sits above the ridge point. You run the kernel.

The profiler still shows stalls.

Different stalls. The memory stalls are reduced to acceptable levels, but something else is happening.

The SM is doing work, not waiting on DRAM, and yet it is slower than the theoretical ceiling by a factor the memory model doesn’t explain.

This is when the focus shifts from data movement to instruction flow. From where data lives to how instructions are scheduled, issued, and retired.

The second real lesson of GPU programming is this: the instruction pipeline is not transparent.

It has depth, hazards, and throughput ceilings that are entirely independent of memory bandwidth. It can be the bottleneck even when memory is not.

Understanding it requires going deeper into the SM than most CUDA tutorials ever go.

Subscribe now

The instruction stream beneath the kernel

Every CUDA kernel you write is compiled to PTX (NVIDIA’s Parallel Thread eXecution intermediate representation) and from there to SASS: Streaming ASSembler, the native instruction set of the GPU.

PTX is portable across GPU generations within broad families. SASS is not. It is tied to a specific architecture, encodes pipeline-specific timing constraints directly into instruction control bits, and is the only thing that actually executes on the hardware.

Most CUDA developers never look at SASS. This is a mistake: not because you need to write it, but because it is the only place you can verify what the compiler actually produced, confirm that instruction selection matches your intent, and identify pipeline bottlenecks that the high-level model cannot expose.

The translation path:

CUDA source (.cu)
     ↓  [nvcc / clang front-end: C++ parsing, template instantiation]
PTX  (target-independent virtual ISA; open, documented)
     ↓  [ptxas: machine-specific optimization, register allocation, SASS generation]
SASS (cubin; architecture-locked, closed binary)
     ↓  [CUDA runtime: kernel loading, parameter binding, SM dispatch]
SM execution units

To inspect SASS for any compiled binary, use:

# From a compiled CUDA binary
cuobjdump --dump-sass my_kernel.cubin

# Or directly from a PTX file compiled for a target
ptxas -arch=sm_80 kernel.ptx -o kernel.cubin
cuobjdump --dump-sass kernel.cubin

When you write float x = a * b + c, the compiler emits a single FFMA instruction (fused multiply-add) which performs both operations in one pipeline pass with a single rounding step.

This is semantically different from two separate operations: (a * b) + c computed with FFMA rounds once at the end; float t = a * b; t = t + c; rounds twice.

The compiler fuses by default. Disable with -fmad=false if numerical reproducibility across implementations matters, accepting roughly halved throughput on compute-bound code.

The SASS instruction stream is not a one-to-one transcription of your source. It is the compiler’s best attempt to map your intent to the hardware’s actual execution model.

Understanding that model is what makes the difference between reading SASS as noise and reading it as a diagnostic.

Subscribe now

Latency versus throughput

The single most important conceptual distinction in pipeline-level reasoning is between instruction latency and instruction throughput. They govern different bottlenecks, require different fixes, and are frequently confused.

Latency is the number of clock cycles between when an instruction is issued and when its result is available to a subsequent instruction that depends on it. A dependent instruction cannot issue until this interval has elapsed.

Throughput is the inverse of the rate at which independent instructions can flow through an execution unit, expressed as cycles per instruction. It represents the pipeline’s steady-state capacity, ignoring data dependencies entirely.

The following table gives measured values for the Ampere architecture (A100, sm_80), derived from microbenchmarking work by Abdelkhalik et al. (2022) and corroborated across multiple independent sources:

Instruction              Latency (cycles)   Throughput (cyc/instr, per SMSP)
─────────────────────────────────────────────────────────────────────────────
FFMA (FP32)                     4                  0.25  (4 FMAs/cycle)
FADD / FMUL (FP32)              4                  0.25
IMAD / IADD3 (INT32)            4                  0.25
DFMA (FP64)                     8                  2.0
MUFU.RCP / RSQ                 16                  4.0
MUFU.SIN / COS                 16                  4.0
MUFU.EX2 / LG2                 16                  4.0
HFMA2 (FP16×2)                  4                  0.25
─────────────────────────────────────────────────────────────────────────────
Shared memory load             23                  ~1.0
Shared memory store            19                  ~1.0
L1 cache hit (LDG)            ~33                  ~1.0
L2 cache hit (LDG)           ~200                  ~1.0
HBM uncached (LDG)         ~290–566†               ~1.0

†The range reflects two different measurement methodologies: pointer-chasing through fully resident arrays (290 cycles, Abdelkhalik et al.) versus pointer-chasing through a working set larger than L2, which forces full HBM round-trips and measured ~566 cycles on A100 (Shi et al., 2025). The lower number approximates L2-miss latency; the upper number is closer to the true cold-DRAM round-trip experienced in bandwidth-saturated conditions.

The MUFU throughput of 4 cycles per instruction requires emphasis. MUFU executes transcendental functions (sinf, cosf, expf, logf, rsqrtf, rcpf) via the Special Function Unit, a separate pipeline from the FP32 FMA units.

On A100, each SM sub-partition has one SFU. That SFU can issue one MUFU instruction every 4 cycles, while the FP32 pipe can issue four FMAs per cycle.

A kernel that mixes heavy transcendental usage with FP32 arithmetic will hit SFU throughput long before it approaches FP32 throughput. This matters for ML activation functions (tanhf, expf in softmax) and any scientific kernel using trigonometry.

Now the fundamental insight: FFMA latency is 4 cycles, but FFMA throughput is 1 instruction per 0.25 cycles (4 per cycle). Four independent FP32 FMAs can enter the pipeline simultaneously every cycle.

If no dependency links them, all four execute in parallel, and the pipeline issues one per cycle. If every instruction reads the result of the previous one, the pipeline must wait 4 cycles between each issue. The throughput ceiling goes unrealized.

This is not hypothetical. It is the dominant performance regime for naive accumulation loops.

The SMSP: the real unit of execution

A critical detail that the previous part’s SM diagram abstracts away: on Ampere (and Volta and Turing), the SM is not monolithic. It is divided into four SM sub-partitions, each designated SMSP in the Nsight Compute metric namespace.

Each SMSP contains:

• One warp scheduler (one instruction issued per cycle)

• One dispatch unit

• An L0 instruction cache (private to the SMSP)

• A 16K×32-bit register file (64 KB, 16,384 32-bit registers per SMSP; four SMSPs yield 65,536 total per SM)

• 32 FP32 CUDA cores (on GA100; gaming Ampere GA10x uses a different split)

• 16 INT32 cores

• 8 FP64 cores

• 1 third-generation Tensor Core

• 8 Load/Store Units

The SMSP, not the SM, is the scheduling atom. A warp is assigned to a specific SMSP at launch and remains there for its entire lifetime. It does not migrate between SMSPs.

All the warp stall metrics in Nsight Compute that carry the smsp__ prefix are per-SMSP counters; sm__ metrics aggregate across all four. This subdivision matters for two reasons.

First, it explains the per-SMSP warp pool size. On Ampere, each SMSP can host up to 16 warps. Four SMSPs yield the 64-warp-per-SM maximum. The scheduler operates per SMSP, picking one eligible warp per cycle from its local pool of 16.

The practical ceiling for latency hiding is thus determined per SMSP, not per SM. A kernel with 32 resident warps per SM has 8 per SMSP, which is often sufficient for latency hiding when HBM latency is the bottleneck.

Second, it exposes the register file partitioning. The 65,536 registers per SM are physically distributed across four SMSPs, 16,384 each.

When you compute that 32 registers per thread allows 2,048 threads per SM at full occupancy, those 2,048 threads are physically spread 512 per SMSP, each holding 32 registers, fully consuming the 16,384 available.

The constraint is per-SMSP, and violating it in one SMSP limits the entire SM.

The Nsight Compute metric smsp__warps_active.avg.pct_of_peak_sustained_active reports active warp fraction per SMSP averaged over time.

It is more informative than sm__warps_active for diagnosing occupancy limits because it reflects the actual scheduling capacity of the unit that does the scheduling.

Subscribe now

The hardware that enforces the dependency graph

Before a warp can issue its next instruction, the warp scheduler must verify that all source operands for that instruction are available.

The hardware mechanism for this is the scoreboard, a per-SMSP register file that tracks which registers have outstanding writes from in-flight instructions.

Every issued instruction that writes to a register marks that register as “pending” in the scoreboard. When the instruction completes and the result is written to the register file, the mark is cleared.

If the scheduler selects a warp to issue and the warp’s next instruction reads a register that is still marked pending, the warp is stalled. It is not eligible. The scheduler moves to the next warp.

CUDA distinguishes two scoreboard domains based on the source of the pending result:

Short scoreboard (Nsight metric: smsp__pcsamp_warps_issue_stalled_short_scoreboard): tracks instructions with latency short enough that the hardware uses a fixed countdown timer rather than a completion signal.

This covers: FP32/INT32/FP16 arithmetic from the FMA pipe, shared memory loads (23-cycle latency), SFU/MUFU results (16 cycles), indexed constant loads, and warp-level vote instructions.

The hardware knows the exact cycle at which the result will be ready and unblocks the scoreboard entry on that cycle.

Long scoreboard (Nsight metric: smsp__pcsamp_warps_issue_stalled_long_scoreboard): tracks instructions whose completion time is not fixed in advance.

This covers all loads from global memory ( L1 hits (~33 cycles), L2 hits (~200 cycles), HBM (~290–566 cycles)) and anything that crosses the L1TEX pipeline.

The hardware cannot predict when the data will arrive; it waits for an explicit writeback signal from the memory subsystem.

This split has a crucial diagnostic implication. High stall_long_scoreboard means threads are waiting for data from DRAM. The fix is occupancy (more warps to swap in while waiting), prefetching, or restructured data layout.

High stall_short_scoreboard means threads are waiting on arithmetic results or shared memory: a dependency bottleneck in the instruction stream itself. The fix is instruction-level parallelism, not occupancy.

A third stall class completes the picture: MIO throttle (stall_mio_throttle). This appears when the input FIFO to the Memory I/O pipeline is full: too many outstanding memory requests are already in flight and new ones cannot be accepted.

It is distinct from stall_long_scoreboard. The latter means a warp is waiting for a specific result. The former means a warp cannot even submit a new request yet.

MIO throttle is the signature of heavy but poorly coalesced global memory access, where many independent 32-transaction memory operations are flooding the queue simultaneously.

And the fourth: math pipe throttle (stall_math_pipe_throttle). This appears when a warp is ready to issue an FP32 (or FP64, or tensor) instruction but the execution pipeline is already occupied by instructions from other warps.

This is the good stall: it means arithmetic throughput, not memory or dependencies, is the actual ceiling. On a well-tuned compute-bound kernel, stall_math_pipe_throttle should dominate the stall breakdown.

The differential diagnosis in Nsight Compute’s Warp State Statistics section is the most powerful single analytical tool available after the roofline. Reading the dominant stall reason maps directly onto the class of optimization required:

• stall_long_scoreboard dominant → memory latency. Fix: more warps, better tiling, async prefetch.

• stall_short_scoreboard dominant → arithmetic dependency chain. Fix: ILP, loop unrolling, independent accumulators.

• stall_mio_throttle dominant → memory request queue saturation. Fix: coalescing, vectorized loads, reduced memory instruction count.

• stall_math_pipe_throttle dominant → compute-bound. Fix: tensor cores if using scalar FP16/FP32, mixed-precision, or just accept that you’ve reached peak.

• stall_barrier dominant → synchronization structure. Fix: reduced barrier scope, better work distribution, cooperative groups.

• stall_not_selected dominant → too many eligible warps, not enough issue bandwidth. Fine above ~20%; unusually high means you’re register-rich with very high ILP and can potentially increase complexity per thread.

  Subscribe now

The dependency chain

Consider a canonical example: a dot product over a fixed array, unrolled.

float acc = 0.0f;
for (int i = 0; i < N; i++) {
    acc += a[i] * b[i];
}

After loading all a[i] and b[i] into registers (ignoring the loads themselves for now), the inner loop body compiles to a sequence of FFMA instructions, each writing to acc and reading the result of the previous one:

FFMA R4, R6, R8, R4    // acc += a[0]*b[0]; R4 depends on R4
FFMA R4, R10, R12, R4  // acc += a[1]*b[1]; R4 depends on R4 from prior
FFMA R4, R14, R16, R4  // acc += a[2]*b[2]; R4 depends on R4 from prior
...

Each FFMA writes to R4 and reads from R4. The short scoreboard marks R4 as pending for 4 cycles after each issue. The next instruction reads R4, finds it pending, and stalls. The effective issue rate is 1 FFMA per 4 cycles.

The FP32 pipeline on an A100 SMSP can issue 4 FMAs per cycle when fed independent work. This naive accumulation uses 1/16 of that capacity.

The fix is independent accumulators, which break the serial chain into parallel chains that the hardware can interleave:

float acc0 = 0.0f, acc1 = 0.0f, acc2 = 0.0f, acc3 = 0.0f;
for (int i = 0; i < N; i += 4) {
    acc0 += a[i+0] * b[i+0];
    acc1 += a[i+1] * b[i+1];
    acc2 += a[i+2] * b[i+2];
    acc3 += a[i+3] * b[i+3];
}
float acc = (acc0 + acc1) + (acc2 + acc3);

The SASS now has four independent dependency chains. After FFMA R4, ..., R4, the scheduler can immediately issue FFMA R5, ..., R5, FFMA R6, ..., R6, and FFMA R7, ..., R7.

When the first FFMA’s 4-cycle latency expires and R4 is ready, the next FFMA R4 instruction can issue. The pipeline runs at near peak throughput.

This is not a micro-optimization on the margin. On compute-bound kernels with long accumulation chains (reductions, dot products, small GEMMs written without tensor cores) the difference between serial and independent-accumulator form is frequently 4–8× in throughput.

The compiler performs this unrolling automatically in many cases when #pragma unroll is used or when the loop trip count is known at compile time and is small.

It does not reliably unroll across complex loop bodies, through function call boundaries, or when accumulators are accessed via pointers (which the compiler may assume alias).

To verify: inspect the SASS. If you see FFMA R4, ..., R4 repeating with the same destination register, the dependency chain is serialized. If you see FFMA R4, FFMA R5, FFMA R6, FFMA R7 cycling, the compiler found the independent accumulator form.

The profiler can confirm via smsp__pcsamp_warps_issue_stalled_short_scoreboard — if this metric is non-trivial on a compute-bound kernel, the dependency graph is restricting throughput.

Register bank conflicts

The register file on each SMSP is a four-banked SRAM, where each bank is 4 bytes wide. An FFMA instruction reads up to three source registers (two multiplicands and an addend) and one destination.

If two or more source registers of a single instruction map to the same bank, the reads serialize. Each register at index R maps to bank R % 4 (on Ampere; earlier architectures used different moduli and widths).

For an FFMA with sources R4, R8, R12: banks are 0, 0, 0. All three reads conflict. The register file must issue three sequential read operations to bank 0, adding latency to the instruction even if no data dependency exists.

For an FFMA with sources R4, R5, R6: banks are 0, 1, 2. No conflict. All three reads issue in parallel.

This has a direct consequence for accumulation loop unrolling. Consider four accumulators in R4, R5, R6, R7. If the multiplicands for each land in:

FFMA R4, R8,  R12, R4   // banks: 0, 0, 0, 0 → conflict on R8, R12, R4
FFMA R5, R9,  R13, R5   // banks: 1, 1, 1, 1 → conflict

All three sources of each FFMA map to the same bank (since 8%4=0, 12%4=0, 4%4=0). Every instruction stalls internally. The throughput benefit of ILP is partially eaten by register bank conflicts.

The compiler’s register allocator is aware of this and attempts to assign registers to avoid conflicts in hot instruction sequences. But it operates under register pressure constraints and cannot always do so.

When manual register assignment is needed for peak performance (as in hand-tuned sgemm kernels) this requires explicit attention to bank distribution.

To check: in SASS output from Nsight Compute’s Source view, bank conflict indicators appear per instruction when hardware counter smsp__sass_data_bank_conflicts_pipe_fma_cycles_active is nonzero.

Values above 5% on a compute-bound kernel suggest register allocation is constraining pipeline throughput.

Subscribe now

Control divergence

Divergence is well-understood at the conceptual level but frequently mis-modeled quantitatively.

The SIMT execution model on post-Pascal NVIDIA GPUs (Volta and later) uses independent thread scheduling: each thread has its own program counter and stack, and the hardware can reconverge threads that diverged.

This replaced the pre-Volta model where threads were locked to warp-level lockstep with explicit SIMD masking.

What independent thread scheduling provides:

• Threads can diverge without being permanently trapped in separate execution paths until an explicit reconvergence point.

• The scheduler can interleave instructions from different sub-groups within a warp to improve utilization.

What it does not provide:

• SIMT execution is still 32-wide. When a warp diverges, the hardware executes the taken path and the not-taken path serially, masking inactive threads. Independent thread scheduling changes when reconvergence can happen, not whether both paths must execute.

The true cost of a divergent conditional is not “50% efficiency if threads split 50/50.” It is the sum of execution time for all distinct paths through the conditional, not the maximum.

If half the threads take path A (10 instructions) and half take path B (20 instructions), total warp execution time is ~30 instruction-cycles, not ~20.

For nested conditionals, the cost compounds. A kernel with a two-level nested conditional where each level has 50/50 divergence may see a 4× slowdown compared to fully converged execution.

The SASS opcode BRA (branch) is preceded by a predicate evaluation. All threads evaluate the predicate; the warp then issues along the taken path with non-predicated threads masked.

The reconvergence point is encoded in SSY (set synchronize) and SYNC instructions in the SASS, inserted by the compiler.

The diagnostic metric is smsp__sass_thread_inst_executed_op_control.sum relative to total instructions: high control overhead relative to arithmetic instructions indicates either heavy divergence or loop overhead.

Nsight Compute’s Source Counters section shows per-instruction thread execution counts; instructions in the taken path of a divergent branch show fewer thread executions than instructions outside the branch.

The practical implication: for kernels with data-dependent branching (parsing, tree traversal, sparse format processing), minimizing the number of distinct per-warp execution paths matters more than minimizing the total number of conditionals.

One 32-way divergent conditional is better than eight 2-way ones.

Share The Software Frontier

Conclusion

After roofline analysis, memory coalescing, and careful tiling, most GPU kernels are no longer limited by bandwidth: they are limited by the instruction pipeline itself.

Understanding the pipeline means looking beyond PTX and into SASS: the real instruction stream executed by the hardware, with its latencies, throughput ceilings, and resource partitions.

Instruction-level bottlenecks (dependency chains, register bank conflicts and control divergence) often dominate performance even when memory stalls are minimal.

Metrics like short and long scoreboard stalls, math pipe utilization, and SMSP-level warp activity provide the only reliable window into these hidden limits.

The practical lesson is simple but profound: to push a compute-bound kernel to its theoretical peak, you must treat the instruction pipeline as first-class terrain.

Unroll loops into independent accumulators, balance register allocation to avoid bank conflicts, minimize divergent paths, and account for specialized units like the SFU.

Only by reasoning at the level of instruction flow, rather than just data movement, can you approach the true limits of GPU performance.

Share

The moment arithmetic stops mattering

Every CUDA program eventually reaches the same reckoning.

You’ve written the kernel. You’ve structured the grid. The launch configuration looks reasonable. The arithmetic is correct. You submit it, measure wall time, and feel like a competent engineer.

Then you open a profiler.

Nsight Compute loads the metrics. You look at the first column: SM utilization. You expect something respectable: 70%, 80%, maybe more. What you see instead is a number that doesn’t make sense.

12%. Sometimes a little lower.

The tensor cores are almost entirely idle. The floating-point pipelines are stalled. A chip whose transistor count would have represented the entire global semiconductor output of the mid-1990s is sitting mostly quiet, doing almost nothing, waiting.

Waiting for data.

This is the first real lesson of GPU programming, and it is one of the most counterintuitive in all of systems engineering: modern GPUs are not, in practice, compute-limited. They are memory-limited. The bottleneck isn’t the arithmetic: it’s the infrastructure that feeds the arithmetic.

And that infrastructure is governed not by logic design, not by clock frequency, but by the literal physics of moving electric signals through metal on a substrate the width of a few dozen silicon atoms.

The GPU was designed to compute at enormous scale. But before it can compute anything, it needs data. And moving that data, from DRAM, through cache hierarchies, across the chip, into registers, costs time. Often far more time than the computation itself.

Understanding why, and what to do about it, is the entire subject of this article.

The physics of moving bits through silicon

Let’s start with something that almost never appears in CUDA tutorials but underpins everything else: the actual physical constraints on data movement inside a chip.

An electric signal traveling through a metal interconnect on silicon propagates at roughly two-thirds the speed of light in vacuum. That sounds fast. It is fast. But it’s not the bottleneck.

The bottleneck is capacitance.

Every wire connecting two points on a chip behaves as a tiny capacitor. Moving a signal through that wire requires charging or discharging that capacitance. That costs energy, and more importantly for our purposes, it takes time.

The longer and wider the wire, the greater the capacitance, the slower the signal edges, the more time the receiving circuit spends waiting for the voltage to settle.

At 1 GHz, one clock cycle is 1 nanosecond. Light travels about 30 cm in that time, which is longer than most GPU dies. Signal propagation across the chip isn’t the problem.

The problem is the full pipeline required to actually deliver data: address generation, routing through the memory subsystem, cache tag lookup, DRAM row activation, column access, data return bus, write to register file. Each of these steps adds cycles. Latencies compound.

When you add them all up, reaching off-chip DRAM from an execution unit costs between 400 and 800 clock cycles on a modern GPU.

That number is worth sitting with.

At 1.5 GHz, 600 cycles is 400 nanoseconds. In that same window, a modern NVIDIA SM could theoretically issue hundreds of independent arithmetic instructions.

Instead, if it’s waiting for DRAM, it issues zero. The arithmetic units sit idle. The watt-hours tick over. Nothing useful happens.

This gap (the chasm between compute speed and memory latency) is the central engineering problem of GPU design. Everything else in GPU architecture exists to paper over this gap.

The warp scheduler, the cache hierarchy, shared memory, asynchronous pipelines, tensor cores: all of it is infrastructure built around one uncomfortable physical fact: moving data costs time, and that time is long.

DRAM hasn’t gotten meaningfully faster in latency terms for decades. It has gotten wider, more parallel, higher bandwidth.

But the fundamental latency of a DRAM access (row activation, column select, sense amplifiers settling) is governed by the same physics it always was.

HBM2e has extraordinary bandwidth. Its latency is still measured in hundreds of nanoseconds.

You cannot optimize your way out of physics.

Subscribe now

Inside the Streaming Multiprocessor

Modern GPUs solve this problem through a specific architectural pattern: massive parallelism layered over a deep memory hierarchy, orchestrated by a scheduling engine designed to hide latency rather than eliminate it.

To understand how, we need to look at the machine’s basic unit of execution. The SM is, in fact, the atom of GPU execution. Every thread you launch lands in one, executes inside one, and exits from one.

Everything else (grid geometry, block dimensions, global memory layout) is scaffolding around the SM.

Understanding what’s physically inside is the prerequisite for reasoning about performance. Here is a schematic of an Ampere SM:

┌──────────────────────────────────────────────────────────────────┐
│                     Streaming Multiprocessor (SM)                │
│                                                                  │
│   ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  ┌────┐  │
│   │ Warp Sched 0 │  │ Warp Sched 1 │  │ Warp Sched 2 │  │ W3 │  │
│   └──────┬───────┘  └──────┬───────┘  └──────┬───────┘  └─┬──┘  │
│          │                 │                 │             │      │
│   ┌──────▼─────────────────▼─────────────────▼─────────────▼──┐  │
│   │                   Dispatch / Issue Logic                    │  │
│   └────────────────────────────────────────────────────────────┘  │
│                                                                  │
│   ┌─────────────────┐  ┌─────────────────┐  ┌──────────────────┐ │
│   │  FP32 Cores     │  │  INT32 Units    │  │  Tensor Cores    │ │
│   │  (128 per SM)   │  │  (128 per SM)   │  │  (4 per SM)      │ │
│   └─────────────────┘  └─────────────────┘  └──────────────────┘ │
│                                                                  │
│   ┌─────────────────┐  ┌─────────────────┐                       │
│   │  FP64 Cores     │  │  Special Func.  │                       │
│   │  (64 per SM)    │  │  Units (SFU)    │                       │
│   └─────────────────┘  └─────────────────┘                       │
│                                                                  │
│   ┌──────────────────────────────────────────────────────────┐   │
│   │              Load / Store Units (32 LSU)                  │   │
│   └──────────────────────────────────────────────────────────┘   │
│                                                                  │
│   ┌──────────────────────────────────────────────────────────┐   │
│   │              Register File (~256 KB)                      │   │
│   │              (65,536 × 32-bit registers)                  │   │
│   └──────────────────────────────────────────────────────────┘   │
│                                                                  │
│   ┌──────────────────────────────────────────────────────────┐   │
│   │       L1 Cache / Shared Memory (192 KB unified)          │   │
│   └──────────────────────────────────────────────────────────┘   │
└──────────────────────────────────────────────────────────────────┘

The arithmetic units (FP32 cores, INT32 units, tensor cores) get all the attention in marketing materials. They are not where performance is determined.

Performance is determined by the four warp schedulers at the top.

The warp scheduler’s decision tree

Every cycle, each of the four warp schedulers examines its pool of eligible warps and must select one to issue. A warp is eligible if it satisfies four conditions simultaneously:

1. It has a valid next instruction

2. All source operands for that instruction are available (no outstanding data dependency)

3. The required execution unit is not occupied by another instruction

4. No synchronization barrier is blocking it

If a warp fails any condition, it is ineligible. The scheduler ignores it and checks the next one. The scheduler doesn’t understand priority, criticality, or deadlines. It is a simple priority-free selector: find the first eligible warp, issue it, repeat.

If no warp is eligible (all are waiting for memory, or all have unresolved register dependencies) the SM stalls. Not one warp stalls. The entire SM stalls. All four schedulers sit idle.

The execution pipelines emit nothing. Every watt powering the chip produces zero useful work. This is the stall you see in the profiler as “no eligible warp selected.”

On Ampere, the theoretical maximum is 64 resident warps per SM, 4 schedulers each issuing 1 warp per cycle. In the best case, the SM issues 4 independent instructions simultaneously from 4 different warps. In the worst case, when all 64 warps are blocked on DRAM, it issues zero.

The entire architecture of GPU programming is a battle to keep this scheduler fed. Everything else (tiles, pipelines, occupancy tuning, coalescing) is in service of that one goal.

Subscribe now

The memory hierarchy

Each layer of the GPU memory hierarchy represents a different engineering tradeoff between latency, bandwidth, capacity, and programmer control.

Understanding each layer’s character (not just its numbers) determines whether you can make intelligent decisions about where data should live at any moment during execution.

Registers: the fastest storage on the chip

Registers are physically distributed across the SM register file — on Ampere, a 256 KB SRAM array containing 65,536 32-bit registers per SM.

Access latency is 1 clock cycle. Bandwidth between the register file and execution units is effectively unlimited for the purposes of performance analysis.

Registers are not a cache. They do not hold data that might be needed. They hold data that is actively being computed on.

The compiler allocates registers deterministically at compile time. Every local variable, intermediate computation, and loop accumulator that doesn’t get spilled lives in the register file.

But registers introduce the most important constraint in CUDA performance engineering: their consumption directly limits occupancy.

The register file must be divided among all threads resident on the SM simultaneously. If each thread requires 64 registers, the 65,536 available registers accommodate at most 1,024 threads: 32 warps, half the Ampere maximum.

With 32 warps instead of 64, the scheduler has half as many options when a warp stalls. Latency hiding degrades.

This is why --maxrregcount exists in the CUDA compiler toolchain. Capping register usage forces spilling to local memory which is DRAM, cached through L1/L2, with all the attendant latency.

The tradeoff is: more concurrent warps (better latency hiding) at the cost of some extra memory traffic (more memory pressure). Whether it helps depends entirely on the kernel’s specific balance of compute and memory.

The profiler metric to monitor is sm__warps_active.avg.pct_of_peak_sustained_active: the fraction of cycles during which at least one warp was active on the SM.

Values below 50% on memory-bound kernels usually indicate occupancy is limiting performance. Values above 75% suggest you’re probably fine.

Shared memory: the programmable cache

Shared memory is physically the same SRAM as L1 cache; on Ampere, a unified 192 KB block that software partitions between the two.

Typical configurations: 128 KB shared / 64 KB L1, or 64 KB shared / 128 KB L1, selectable per kernel.

The critical property that distinguishes shared memory from every other memory type on the GPU: it is explicitly managed by software. L1 is automatic. L2 is automatic. HBM is automatic.

Shared memory requires the programmer to decide what to load, when to load it, and when to synchronize threads after loading. The hardware does exactly what the programmer specifies and nothing more.

This explicitness is simultaneously the source of its power and its most common source of subtle bugs.

Latency is roughly 20 clock cycles, an order of magnitude faster than L2, 20–40× faster than DRAM. Intra-SM bandwidth reaches several terabytes per second.

For workloads with structured access patterns and significant data reuse within a thread block, shared memory is the most important performance tool available.

Bank conflicts: the hidden serialization

Shared memory is internally divided into 32 banks, each 4 bytes wide. In a single clock cycle, the hardware can service 32 simultaneous 4-byte accesses (one per bank) as long as no two accesses target the same bank.

This design provides very high bandwidth when accesses are distributed. The bank index for a 32-bit word at byte address addr is:

bank = (addr / 4) % 32

If two or more threads in the same warp access different addresses mapping to the same bank, those accesses serialize. This is a bank conflict. The hardware issues them sequentially, one per cycle, multiplying the effective latency by the conflict degree.

A 32-way bank conflict, 32 threads hitting the same bank simultaneously, effectively transforms a 20-cycle operation into a 640-cycle one. On shared memory.

The canonical example is column access in a row-major 2D shared memory array:

__shared__ float tile[32][32];
float val = tile[threadIdx.x][threadIdx.y];  // accessing column threadIdx.y

In a row-major layout, tile[row][col] is stored at byte offset (row * 32 + col) * 4. For a fixed column c, the elements tile[0][c], tile[1][c], tile[2][c]... are stored at offsets c*4, (32+c)*4, (64+c)*4...

The bank index for tile[i][c] is (i*32 + c) % 32 = c. Every row has the same bank for column c.

If 32 threads simultaneously access column c, all 32 accesses hit bank c. Complete serialization. 32× slowdown. The fix is padding by one element:

__shared__ float tile[32][33];  // one extra float per row

Now tile[i][j] is at byte offset (i*33 + j) * 4. Bank index is (i*33 + j) % 32. For column j, bank = (i*33 + j) % 32. Since 33 is coprime to 32, incrementing i by 1 increments the bank by 33 % 32 = 1. The 32 threads access 32 different banks. No conflicts.

One unused float per row, 128 bytes of wasted SRAM, eliminates a 32× serialization penalty. This is one of the cheapest performance wins in CUDA.

L1 cache: automatic but controllable

The portion of the unified SRAM not allocated to shared memory functions as an automatic L1 data cache for global memory accesses. Access latency is 28–33 cycles on Ampere.

Unlike shared memory, L1 is not explicitly managed. The hardware decides what to cache based on access patterns. For streaming workloads with no temporal reuse, L1 caching is actively harmful; it evicts potentially useful data to cache data that will never be accessed again.

CUDA provides tools to limit this damage. The __ldg() intrinsic routes loads through the read-only texture cache, bypassing L1 entirely and preserving it for data with genuine reuse.

Cache-bypass load modifiers (.cs for streaming, .cg for L2-only) are available in PTX and accessible via __builtin_nontemporal_load variants. Using them correctly on streaming data can meaningfully improve L1 hit rates for other kernel data.

L1 is most valuable for irregular access patterns with temporal reuse: hash table lookups, graph traversals, sparse matrix operations, embedding lookups.

For structured compute kernels with predictable access patterns, explicit shared memory almost always dominates.

L2 cache: the shared reservoir

L2 is a resource shared across all SMs: on the A100, a 40 MB unified L2 with approximately 4 TB/s aggregate read bandwidth. Access latency is roughly 200 cycles.

The bandwidth matters more than the latency for most workloads. If data fits in L2 and is accessed repeatedly across different thread blocks, L2 reuse can dramatically reduce DRAM traffic without any explicit shared memory management.

This is the main performance lever for workloads with inter-block reuse: embedding tables, small lookup matrices, bias vectors applied to many thread blocks.

CUDA 11.1 and Ampere introduced explicit L2 residency controls via cudaStreamSetAttribute with cudaStreamAttrAccessPolicyWindow.

This allows developers to mark a specific memory region as high-priority for L2 retention: the hardware will attempt to keep it resident across thread block launches.

For embedding lookups or frequently-accessed read-only tables, this can reduce DRAM bandwidth consumption by an order of magnitude.

HBM: the distant reservoir

High Bandwidth Memory sits physically separate from the SM die, stacked on the GPU package via silicon interposer.

Access requires leaving the SM die entirely, traversing the memory controller, and accessing DRAM cells across the interposer.

Current numbers:

• V100 HBM2, 32 GB, 900 GB/s

• A100 HBM2e, 80 GB, 2.0 TB/s

• H100 SXM HBM3, 80 GB, 3.35 TB/s

• H100 NVL HBM3e, 188 GB, 3.9 TB/s

These bandwidth numbers are genuinely impressive. They are still not enough.

The A100’s peak FP16 tensor core throughput is 312 TFLOPS. A typical deep learning layer performs approximately 2 FLOPs per weight element loaded (one multiply, one accumulate).

To keep tensor cores saturated, you’d need to deliver 156 TB/s of weight data; it’s 78× more than HBM2e can provide.

This isn’t a design failure. It’s physics. The solution is arithmetic intensity: load each weight once, perform many operations on it before the data expires from registers or shared memory.

The entire science of GPU kernel optimization is the science of achieving sufficient arithmetic intensity to bridge this 78× gap.

Subscribe now

Warps and coalescing

Everything we’ve discussed about the memory hierarchy becomes concrete in the behavior of individual memory instructions.

The anatomy of a memory transaction

When a warp executes a load instruction:

float x = data[idx];

The hardware doesn’t see 32 independent loads. It sees 32 addresses simultaneously, one per thread, and must service them with as few memory transactions as possible.

The memory controller coalesces these 32 addresses into the minimum set of cache line requests that covers all of them.

DRAM is accessed in 128-byte cache lines on current NVIDIA GPUs. L2 sector granularity is 32 bytes. The hardware merges warp addresses into the minimum number of 128-byte requests covering all requested bytes.

Coalesced access: ideal

Thirty-two threads accessing consecutive floats:

Thread  0 → data[base + 0]
Thread  1 → data[base + 1]
...
Thread 31 → data[base + 31]

Total footprint: 128 bytes, exactly one cache line. One transaction. Every byte fetched is used. Memory efficiency: 100%.

This is exactly the access pattern the hardware was designed for.

Strided access: degraded

Stride-2 access:

Thread  0 → data[base + 0]
Thread  1 → data[base + 2]
...
Thread 31 → data[base + 62]

Address range: 252 bytes, spanning two 128-byte cache lines. Two transactions. 256 bytes transferred, 128 bytes used. Memory efficiency: 50%.

Stride 4: four cache lines, 25% efficiency. With stride 32, each thread’s access can fall in a different cache line, up to 32 transactions, 3% efficiency. The kernel is now burning 32× the memory bandwidth for the same amount of useful data delivered.

Random access

Scatter/gather, hash lookups, pointer chasing, patterns where 32 thread addresses bear no spatial relationship to each other.

The worst case: one transaction per thread. 32 transactions, each returning 128 bytes to deliver 4 bytes.

Bytes-transferred-to-bytes-used ratio: 32:1. The kernel consumes 97% of its memory bandwidth fetching data it will immediately discard.

This doesn’t just hurt the kernel itself. It saturates the L2 and memory controllers, degrading bandwidth for every other kernel running concurrently on the chip.

The profiler metric to inspect is the ratio of l1tex__t_sectors_pipe_lsu_mem_global_op_ld.sum to l1tex__t_requests_pipe_lsu_mem_global_op_ld.sum, sectors per request.

Perfectly coalesced: 1. Mild inefficiency: 2–4. Significant inefficiency: 4–16. “Please reconsider your life choices”: above 16.

Occupancy: the arithmetic of latency hiding

The GPU hides memory latency by switching between warps when one stalls. This only works if there are enough warps to switch to. If every warp is stalled waiting for DRAM, the scheduler has nothing to do. The SM stalls. Occupancy is the metric that captures this.

Occupancy is the ratio of active warps on an SM to the architectural maximum. On Ampere: maximum 64 warps, 2,048 threads. At 50% occupancy: 32 warps. At 25%: 16 warps.

Occupancy is constrained by three physical resources, all of which must simultaneously fit on the SM:

1. Registers. If each thread uses R registers, maximum concurrent threads = 65,536 / R. For full occupancy (2,048 threads), each thread can use at most 32 registers. Kernels routinely use 64–128. At 64 registers: 50% occupancy. At 128: 25%.

2. Shared memory. If a kernel uses S bytes of shared memory per block with T threads per block, maximum concurrent blocks = floor(192 KB / S). Maximum warps = floor(192 KB / S) × (T / 32). A kernel using 96 KB per block with 256 threads per block: 2 concurrent blocks × 8 warps = 16 warps. 25% occupancy.

3. Block limits. Ampere supports at most 32 concurrent blocks per SM. A kernel with 32-thread blocks (1 warp per block) hits this limit at 32 warps, 50% occupancy, regardless of register or shared memory pressure.

The occupancy-performance relationship

High occupancy is not synonymous with high performance. A compute-bound kernel with very few memory instructions doesn’t need 64 warps to keep the scheduler busy; 4 ready warps might be sufficient if they never stall on memory.

Chasing occupancy for its own sake can force the compiler to spill registers to DRAM, adding memory traffic that makes performance worse.

The right mental model: occupancy matters proportionally to memory latency exposure. The more frequently your kernel stalls on DRAM, the more warps you need to hide that latency. If your kernel rarely touches memory, occupancy barely matters.

cudaOccupancyMaxActiveBlocksPerMultiprocessor gives you the theoretical maximum for a given kernel. The gap between that theoretical maximum and what you observe in the profiler (sm__warps_active) tells you how much the hardware can actually hide.

A kernel at 25% theoretical occupancy but 24% active occupancy is fine. A kernel at 75% theoretical occupancy but 20% active occupancy has a structural stall problem.

Putting hard numbers on constraints

The roofline model is the most useful analytical tool in GPU performance engineering, and it is underused.

It doesn’t tell you how to optimize. It tells you what optimization is even possible, which is more valuable.

The central quantity is arithmetic intensity: floating-point operations performed per byte transferred from main memory.

Arithmetic Intensity (I) = FLOPs / bytes_transferred

Performance is bounded by the minimum of two constraints:

Attainable Performance = min(Peak_FLOPs, Peak_Bandwidth × I)

For the A100 FP16:

• Peak tensor throughput: 312 TFLOPS

• Peak HBM bandwidth: 2 TB/s

The ridge point, minimum arithmetic intensity required to be compute-bound rather than memory-bound:

I_ridge = 312 × 10¹² / 2 × 10¹² = 156 FLOPs/byte

To be compute-bound on the A100, every byte you load from HBM must be used for at least 156 floating-point operations. Most CUDA kernels don’t come close.

Consider a naive vector addition:

FLOPs:        1 (one addition)
Bytes moved:  12 (two float32 reads + one write)
I =           1/12 ≈ 0.08 FLOPs/byte

This kernel sits roughly 1,875× below the ridge point. Not 10% below. Not 50% below. Nearly 2,000× below. No amount of launch configuration tuning, thread count adjustment, or arithmetic reorganization will move the needle.

The kernel is physically limited by how fast you can move 12 bytes per FLOP through the memory hierarchy. That’s a fundamental property of the algorithm.

Dense matrix multiplication is different. A square matrix multiply of dimension N performs 2N³ FLOPs and reads 3N² elements (A, B, and C matrices). Arithmetic intensity grows as 2N³ / (3N² × 4 bytes) ≈ N/6.

For N=1,024: roughly 170 FLOPs/byte, above the A100 ridge point. Large matrix multiply on Ampere is compute-bound. This is why it saturates tensor cores; there’s enough arithmetic per byte to keep them fed.

The roofline tells you which side of the ridge your kernel sits on, and therefore which type of optimization is worth pursuing. Memory-bound kernels benefit from better coalescing, data reuse, and smaller datatypes.

Compute-bound kernels benefit from better instruction throughput, occupancy, and reduced arithmetic latency.

Applying compute-bound optimizations to a memory-bound kernel is how engineers spend weeks achieving nothing.

Subscribe now

The architecture’s logic, finally visible

After tracing the memory hierarchy from physics to TMA, a coherent picture emerges.

The GPU is not a calculator that operates on data. It is a machine for orchestrating the movement of data at massive scale.

It is just fast enough, and in just the right form, for computation to occur at the rate the arithmetic units can sustain. Everything in the architecture, the four warp schedulers, the unified 192 KB SRAM, cp.async, TMA, WGMMA, exists in service of that orchestration.

The fastest GPU programs are not the ones that perform the most arithmetic per line of code. They are the ones that most efficiently move data through the hierarchy:

HBM (2 TB/s, ~600 ns)
  ↓  [TMA or cp.async]
Shared Memory (several TB/s, ~20 ns)
  ↓  [warp-level loads]
Register File (~1 ns)
  ↓  [tensor core instructions]
Accumulators
  ↓  [store]
HBM

Each arrow is a potential bottleneck. Each transition must be managed so that the layer above never waits for the layer below.

There is a mental model shift that separates engineers who write fast GPU code from those who don’t. It isn’t knowledge of specific APIs or familiarity with PTX. It’s the habit of thinking about data location.

At every point during kernel execution:

where is this data right now?

Where does it need to be in three instructions?

How many cycles will it cost to move it there?

Is there computation I can usefully perform in the meantime?

This is how the FlashAttention team found their key insight: the bottleneck in naive attention wasn’t the matrix multiplies, it was repeated HBM round-trips for the attention matrix.

The arithmetic didn’t change. The data choreography did. That’s the entire optimization.

Once the hardware model is genuinely internalized, the techniques follow naturally. The padding that eliminates bank conflicts isn’t a trick you memorize, it falls directly out of understanding how the banking hardware works.

The cp.async pipeline isn’t a template you copy; it’s the obvious solution once you understand that synchronous loads are serializing your kernel for no reason.

That’s the real skill. Not writing fast arithmetic. Writing fast data movement, and just enough arithmetic to justify it.

Subscribe now

The system contract beneath the kernel

In Part I, we descended from C++ into LLVM IR, tracing each loop, phi node, and virtual register.

In Part II, we dissolved PTX into warps, registers, and latency-hiding strategies, exploring how threads collide and cooperate in the Streaming Multiprocessor pipelines.

While I was studying all of this, a few months ago, I genuinely thought I understood the atomic unit of performance. I was totally wrong.

The atomic unit is not the kernel. A kernel is not computation. It is a submission: a meticulously packaged descriptor, assembled by the driver, serialized over the PCIe or NVLink bus, and handed to a firmware-managed command processor.

This processor does not execute “kernels” the way you imagine. It schedules thread blocks onto independently clocked Streaming Multiprocessors, arbitrates shared registers and memory banks, manages multiple DMA engines, and interleaves thousands of warps to hide latency.

Every instruction, every FMA, every predicated branch in your kernel is meaningless until the submission reaches the scheduler. Until you reason at this level, you are tuning the wrong layer.

Optimizing PTX, unrolling loops, or balancing registers matters only after the submission has carved a feasible execution plan through firmware, SM resources, and memory subsystems.

This is where performance truly lives:

at the intersection of host, bus, firmware, and microarchitecture, where the invisible choreography of scheduling, arbitration, and allocation transforms a static descriptor into teraflops of actual computation.

Subscribe now

The atomic unit of GPU execution

When you write a kernel with the triple-chevron syntax, you are not “launching computation”. You are calling into the CUDA Runtime API, specifically cudaLaunchKernel.

Consider what that call site looks like after the compiler lowers it:

// Source-level kernel invocation
myKernel<<<gridDim, blockDim, sharedMemBytes, stream>>>(arg0, arg1, arg2);

// What the compiler actually emits (simplified)
void* args[] = { &arg0, &arg1, &arg2 };
cudaLaunchKernel(
    (const void*)&myKernel,  // fat binary function handle
    gridDim,                 // dim3
    blockDim,                // dim3
    args,                    // pointer to argument array
    sharedMemBytes,          // dynamic shared memory in bytes
    stream                   // cudaStream_t
);

That call is deceptively simple. Beneath it lies a cascade of complex, orchestrated operations that transform your high-level kernel into a GPU-executable submission.

The runtime layer, libcudart, performs several critical tasks before the GPU ever sees a single instruction. First, it resolves the function pointer for your kernel. This pointer does not point to machine code directly.

It points into a fat binary (fatbin) section embedded in the host ELF executable under the .nv_fatbin section, a container that may hold multiple artifacts. You can inspect this directly:

# Dump all ELF sections to verify the fat binary is embedded
readelf -S ./my_binary | grep nv_fatbin

# Or use cuobjdump to list all embedded architectures
cuobjdump --list-elf ./my_binary
# Output example:
# ELF file    1: my_kernel.sm_80.cubin
# ELF file    2: my_kernel.sm_86.cubin

# Disassemble the SASS for a specific architecture
cuobjdump --dump-sass --gpu-architecture sm_80 ./my_binary

The fat binary container holds one or more cubin objects, each containing SASS (Shader ASSembly, the actual machine-level ISA) for a specific compute capability, and possibly PTX (Parallel Thread Execution, a stable virtual ISA) for forward compatibility.

SASS is versioned by SM architecture: sm_80 targets Ampere A100, sm_86 targets Ampere RTX 30-series, sm_90 targets Hopper H100.

If a cubin matches the active device’s compute capability, the driver loads it directly into device memory.

Otherwise, the embedded JIT compiler in the driver compiles PTX into SASS tailored to the exact SM microarchitecture, applying instruction scheduling, register allocation, and latency-hiding optimizations on-the-fly, then caches the result.

The cache lives at:

# Default JIT cache location on Linux
~/.cache/nvidia/ComputeCache/

# Inspect the cache
ls -lh ~/.cache/nvidia/ComputeCache/

# Force JIT recompilation by clearing the cache
rm -rf ~/.cache/nvidia/ComputeCache/

# Disable the cache entirely (forces JIT on every launch)
export CUDA_CACHE_DISABLE=1

# Inspect PTX embedded in the binary (requires PTX to have been included)
cuobjdump --dump-ptx ./my_binary

Next, the runtime constructs the parameter buffer. All kernel arguments are serialized into a contiguous memory region, respecting ABI alignment and padding rules.

The CUDA ABI mandates that arguments are packed in declaration order, each aligned to its own size, up to a maximum natural alignment of 8 bytes. This buffer is critical: it decouples the host representation of arguments from the GPU’s execution context.

The parameter buffer, along with metadata about grid dimensions, block dimensions, shared memory allocation, and the kernel entry point, forms the launch descriptor.

Control now transitions from libcudart into the CUDA Driver API, libcuda. At this point, you leave the comfort of user-space abstractions. The driver constructs a command packet fully describing the submission.

You can see the Driver API equivalents directly, bypassing the Runtime API entirely:

// Equivalent kernel launch using the Driver API directly
CUfunction kernel;
CUmodule module;

// Load a cubin directly, skipping the fat binary resolution step
cuModuleLoad(&module, "my_kernel.sm_80.cubin");
cuModuleGetFunction(&kernel, module, "myKernel");

// Pack arguments manually
void* kernelArgs[] = { &arg0, &arg1, &arg2 };

// Launch via Driver API
cuLaunchKernel(
    kernel,
    gridDim.x,  gridDim.y,  gridDim.z,   // grid dimensions
    blockDim.x, blockDim.y, blockDim.z,   // block dimensions
    sharedMemBytes,                        // dynamic shared memory
    stream,                                // CUstream
    kernelArgs,                            // argument array
    NULL                                   // extra options (NULL = unused)
);

The driver then executes a system call into kernel mode, a user-to-kernel privilege transition. This step is measurable. You can observe it with strace on Linux:

# Trace system calls during a CUDA kernel launch
strace -e trace=ioctl ./my_cuda_binary 2>&1 | grep -A2 "ioctl.*NVOS"
# You will see a sequence of ioctl() calls into /dev/nvidia0
# Each call corresponds to a driver operation: context management,
# command buffer allocation, or doorbell signaling

# Profile the full launch overhead including driver roundtrip
nvprof --print-gpu-trace ./my_cuda_binary
# Or with Nsight Systems (preferred for modern CUDA)
nsys profile --trace=cuda,nvtx ./my_cuda_binary
nsys stats report.nsys-rep

Inside the kernel-mode driver (nvidia.ko on Linux), the GPU submission is materialized. A GPU command buffer entry is created, encapsulating all launch metadata.

This entry is written into a memory region visible to the GPU, either pinned host memory accessible via PCIe DMA or device BAR (Base Address Register) mapped space, a CPU-accessible window into device memory through MMIO.

A memory-mapped I/O (MMIO) write to the GPU’s doorbell register signals that new work is ready.

This write traverses the interconnect, typically PCI Express or NVLink, crossing from CPU memory controllers to the GPU’s front-end hardware queue manager.

From this moment, the CPU relinquishes control. The GPU’s front-end processor fetches the command packet from the submission queue, decodes it, and begins orchestrating thread blocks across SMs.

Launch latency, typically 5 to 15 microseconds on a modern discrete GPU, is dominated not by grid size but by runtime and driver overhead, mode switch cost, command serialization, and MMIO signaling. You can verify this invariance empirically:

// Microbenchmark: measure launch latency vs. grid size
cudaEvent_t start, stop;
cudaEventCreate(&start);
cudaEventCreate(&stop);

// Launch a trivially empty kernel
__global__ void emptyKernel() {}

for (int blocks : {1, 256, 65535}) {
    cudaEventRecord(start);
    emptyKernel<<<blocks, 256>>>();
    cudaEventRecord(stop);
    cudaEventSynchronize(stop);

    float ms;
    cudaEventElapsedTime(&ms, start, stop);
    printf("blocks=%d  latency=%.3f ms\n", blocks, ms);
    // You will observe near-constant latency across all grid sizes
    // The dominant cost is the submission path, not the computation
}

This fact fundamentally shifts how you reason about GPU performance: kernels are not units of computation. They are descriptors.

Your optimization focus must move away from the code inside the kernel and toward the orchestration of submissions, streams, and resource contention at the firmware and microarchitectural level.

Subscribe now

Inside the GPU front-end

GPUs are not passive execution units. They are orchestrators of massive parallelism, with hardware designed to manage, schedule, and feed thousands of threads simultaneously.

At the apex of this architecture lies the front-end command processor (sometimes called the FECS, Front-End Command Streamer on Ampere and later architectures), a microarchitectural state machine tasked with transforming host-submitted work into actionable instructions for Streaming Multiprocessors.

When the CPU writes to the doorbell register via MMIO, the signal propagates across the PCIe interconnect into the GPU’s front-end.

The command processor wakes, fetches the command packet from its pushbuffer (the ring buffer used for command submission, a concept originating in GPU architecture from the early 2000s and still present in modern CUDA), and begins parsing the launch descriptor.

This descriptor is more than metadata. It encodes the grid dimensions, block dimensions, parameter buffer pointers, kernel entry points, shared memory allocation sizes, and stream association.

Crucially, the front-end does not immediately partition work across SMs. There is no pre-distribution.

Instead, the GigaThread Engine (NVIDIA’s term for the global thread block scheduler present since Fermi) maintains internal hardware work queues, constantly tracking SM occupancy, shared memory usage, register pressure, and warp slot availability.

Thread blocks are dynamically dispatched to SMs only when resources allow. This is why occupancy and resource usage matter at launch: a block cannot begin execution until the SM has enough free registers, shared memory bytes, and warp slots.

You can query the resource limits of any SM directly:

# Query device properties relevant to block scheduling
nvidia-smi --query-gpu=name,compute_cap --format=csv

# Or from CUDA code
cudaDeviceProp prop;
cudaGetDeviceProperties(&prop, 0);
printf("Max threads per SM:       %d\n", prop.maxThreadsPerMultiProcessor);
printf("Max blocks per SM:        %d\n", prop.maxBlocksPerMultiProcessor);
printf("Shared memory per SM:     %zu bytes\n", prop.sharedMemPerMultiprocessor);
printf("Registers per SM:         %d\n", prop.regsPerMultiprocessor);
printf("Warp size:                %d\n", prop.warpSize);
printf("Number of SMs:            %d\n", prop.multiProcessorCount);
// On an A100 (sm_80):
// Max threads per SM:       2048
// Max blocks per SM:        32
// Shared memory per SM:     167936 bytes (with carveout config)
// Registers per SM:         65536
// Warp size:                32
// Number of SMs:            108

This dynamic dispatch has profound architectural implications. Each block must be fully independent: it cannot rely on any global synchronization that spans other blocks.

If it did, the GigaThread Engine would risk deadlock whenever the number of dispatched-but-not-yet-scheduled blocks exceeded the number of concurrently resident blocks across all SMs. CUDA eliminates this hazard by design.

Blocks are the unit of forward progress, and cross-block synchronization is limited to device-wide barriers using Cooperative Groups with grid.sync(), which requires explicit opt-in via cudaLaunchCooperativeKernel and hardware support (Pascal and later):

#include <cooperative_groups.h>
namespace cg = cooperative_groups;

__global__ void cooperativeKernel(float* data) {
    cg::grid_group grid = cg::this_grid();

    // Phase 1: every block writes
    data[blockIdx.x * blockDim.x + threadIdx.x] = threadIdx.x;

    // Device-wide barrier: ALL blocks must reach this point
    // Only safe when the kernel is launched with cudaLaunchCooperativeKernel
    // and grid size <= number of SMs * max resident blocks per SM
    grid.sync();

    // Phase 2: every block reads from another block's data
    float val = data[(blockIdx.x + 1) % gridDim.x * blockDim.x + threadIdx.x];
}

// Launch must use cudaLaunchCooperativeKernel, not the chevron syntax
void* args[] = { &data };
cudaLaunchCooperativeKernel(
    (void*)cooperativeKernel,
    gridDim, blockDim,
    args, 0, stream
);

Within the front-end, multiple execution engines work in parallel.

One engine fetches blocks from the queue, another evaluates resource availability per SM, a third schedules warps to the SM instruction dispatch units (each SM on Ampere has four warp schedulers, each capable of issuing one instruction per clock to its assigned warp pool).

All of this happens in tens of nanoseconds, invisible to the CPU but essential to sustaining throughput.

Subscribe now

Where kernels meet silicon

I used to think a kernel was the atomic unit of performance. I was wrong.

A kernel is not computation. It is a descriptor, a carefully structured packet, handed from the CPU, through a driver, across a bus, and into a firmware-managed execution engine that schedules thread blocks onto independently clocked Streaming Multiprocessors sharing a memory subsystem with multiple DMA engines.

That sentence is not dramatic. It is mechanically accurate. And until you reason at that level, you are optimizing the wrong layer.

The interconnect is the immutable physical boundary that every kernel submission must cross. Most discrete GPUs connect over PCI Express, a packetized, credit-based, arbitrated serial fabric.

PCIe 4.0 x16 (the most common configuration as of Ada Lovelace) provides approximately 32 GB/s bidirectional bandwidth with roughly 1 to 3 microseconds of round-trip latency per transaction.

PCIe 5.0 x16 doubles the bandwidth to 64 GB/s while keeping similar latency characteristics. You can observe the actual topology and bandwidth on your system:

# Inspect PCIe topology and bandwidth
nvidia-smi topo --matrix
# Output shows P2P access type between GPUs and CPU:
# NV# = NVLink (higher bandwidth), PIX = PCIe same switch, etc.

# Benchmark PCIe bandwidth directly
# (from CUDA samples: bandwidthTest)
./bandwidthTest --mode=shmoo --memory=pinned
# Example output on PCIe 4.0 x16:
# Host to Device: ~25 GB/s
# Device to Host: ~26 GB/s
# (theoretical max 32 GB/s; overhead from protocol reduces effective bandwidth)

Systems with NVLink (A100 uses NVLink 3.0, H100 uses NVLink 4.0) fundamentally change this picture.

NVLink 3.0 provides 600 GB/s total bidirectional GPU-to-GPU bandwidth across 12 links on the A100, with sub-microsecond latency. NVLink 4.0 on H100 scales to 900 GB/s.

The principle remains the same: data movement occurs over a finite physical fabric. Move 20 GB per iteration over a 25 GB/s effective PCIe bandwidth, and the lower bound is 0.8 seconds regardless of what your kernels compute.

No amount of register blocking or shared memory tiling can bypass that limit. Physics dominates.

To address this, GPUs expose dedicated asynchronous copy engines, entirely separate from the SM compute pipelines.

On the A100, there are two copy engines capable of overlapping host-to-device (H2D) and device-to-host (D2H) transfers simultaneously with kernel execution.

On the H100, the copy engine count increases to three. When you call cudaMemcpyAsync with pinned memory in a non-default stream, the driver programs a DMA descriptor and the copy engine pulls data directly from host memory using bus-mastering DMA while SMs continue executing kernels on a separate stream:

// Correct pattern for overlapping compute and transfer
// Requires: pinned memory, separate streams, no implicit sync

float *h_input, *h_output, *d_input, *d_output;
const size_t N = 1 << 24;
const size_t bytes = N * sizeof(float);

// Allocate pinned (page-locked) host memory
cudaMallocHost(&h_input,  bytes);  // pinned
cudaMallocHost(&h_output, bytes);  // pinned
cudaMalloc(&d_input,  bytes);
cudaMalloc(&d_output, bytes);

cudaStream_t computeStream, transferStream;
cudaStreamCreate(&computeStream);
cudaStreamCreate(&transferStream);

// Issue H2D copy on transferStream (handled by copy engine)
cudaMemcpyAsync(d_input, h_input, bytes, cudaMemcpyHostToDevice, transferStream);

// Issue kernel on computeStream (handled by SMs, independently)
processKernel<<<N/256, 256, 0, computeStream>>>(d_output, previousData);

// Insert event-based dependency: computeStream waits for transferStream
cudaEvent_t transferDone;
cudaEventCreate(&transferDone);
cudaEventRecord(transferDone, transferStream);
cudaStreamWaitEvent(computeStream, transferDone, 0);

// Now safe to launch the kernel that depends on d_input
dependentKernel<<<N/256, 256, 0, computeStream>>>(d_input, d_output);

// DO NOT call cudaDeviceSynchronize() here unless absolutely necessary.
// It drains ALL streams and ALL engines, collapsing the pipeline to serial.

The distinction between cudaMalloc for device memory and cudaMallocHost for pinned host memory is not stylistic. It is architectural.

The copy engine requires physically contiguous, page-locked pages to issue DMA transfers without CPU intervention.

Pageable memory allocated with standard malloc forces the driver to stage through a temporary pinned bounce buffer first, adding one full extra copy and destroying any possibility of true overlap.

Subscribe now

Pageable vs. page-locked memory and the orchestration of streams

I used to think memory transfers were trivial: just a cudaMemcpy and the GPU would magically have the data. That is not the case.

At the hardware level, not all host memory is created equal. Pageable memory, the default for every allocation from malloc, new, or std::vector, cannot be accessed directly by a GPU DMA engine.

The OS may migrate, swap, or remap pageable pages at any moment. The DMA engine requires a physically stable, contiguous range of pages with a fixed physical address to issue a transaction, because the DMA controller programs the physical address into the PCIe packet directly.

There is no page table walk on the GPU side during a DMA transfer.

When you pass a pointer to pageable memory to any cudaMemcpy variant, the driver silently performs a three-step process: it first allocates a temporary pinned staging buffer in driver-managed memory.

Second, it performs a CPU-side memcpy from your pageable buffer into the pinned staging buffer, then programs the DMA engine to transfer from the pinned staging buffer to device memory.

Two copies of the entire dataset, two chances for latency to accumulate, two pressure events on the CPU’s memory subsystem, multiplied by every transfer in your pipeline.

Pinning host memory eliminates this staging copy entirely. You can pin memory in several ways:

// Method 1: allocate pinned memory from the start (preferred)
float* h_data;
cudaMallocHost(&h_data, bytes);          // pinned, CUDA-managed
// or equivalently:
cudaHostAlloc(&h_data, bytes, cudaHostAllocDefault);

// Method 2: pin an existing pageable allocation (useful for legacy code)
float* existing_ptr = (float*)malloc(bytes);
cudaHostRegister(existing_ptr, bytes, cudaHostRegisterDefault);
// ... use existing_ptr in cudaMemcpyAsync ...
cudaHostUnregister(existing_ptr);        // must unpin before free
free(existing_ptr);

// Method 3: write-combined memory (good for H2D only, uncached on CPU side)
float* h_wc;
cudaHostAlloc(&h_wc, bytes, cudaHostAllocWriteCombined);
// Write-combined memory bypasses the CPU cache hierarchy,
// reducing cache pollution on the host but making CPU reads very slow.
// Only use when the CPU writes sequentially and the GPU reads.

// Verify a pointer is pinned and get the device-mapped address
cudaPointerAttributes attr;
cudaPointerGetAttributes(&attr, h_data);
printf("Memory type: %d (2 = cudaMemoryTypeHost/pinned)\n", attr.type);

But pinning comes at a real OS-level cost. Each pinned page is wired: the OS cannot reclaim it for paging, cannot map it to another physical address, and cannot swap it to disk.

On Linux, wired pages count against the locked memory limit (RLIMIT_MEMLOCK). High-performance systems typically raise this limit:

# Check current locked memory limit
ulimit -l
# Default is often 64 KB (far too low for GPU workloads)

# Raise the limit for the current session
ulimit -l unlimited

# Or permanently via /etc/security/limits.conf
echo "* hard memlock unlimited" >> /etc/security/limits.conf
echo "* soft memlock unlimited" >> /etc/security/limits.conf

# Verify pinned memory usage on the system
cat /proc/meminfo | grep -i locked
# Mlocked: shows currently wired pages in KB

Even with pinned buffers, transferring data efficiently requires orchestrated submission. Streams are the mechanism.

A CUDA stream is an ordered sequence of operations: kernel launches, memory copies, or event markers. Within a stream, operations execute in issue order, serialized by the hardware.

Across streams, no ordering is guaranteed unless the programmer explicitly inserts synchronization via cudaEvent or cudaStreamWaitEvent.

Internally, each stream maps to a logical command queue in the driver, which maps to a hardware channel on the GPU front-end.

The GPU scheduler interprets these queues at the hardware front-end, deciding dynamically which operation to issue to which engine.

The concurrency decisions depend on SM capacity (how many blocks can reside simultaneously on each SM given register and shared memory consumption), warp slot availability (each SM on Ampere supports up to 64 resident warps regardless of block count).

After that, it copy engine availability (two engines on A100, three on H100, each capable of one direction of transfer at a time), and memory bandwidth headroom (HBM2e on A100 provides 2 TB/s; GDDR6X on RTX 4090 provides 1 TB/s).

Streams are not independent parallel lanes. They represent potential parallelism, a promise the hardware can choose to fulfill when resources allow.

You can use Nsight Systems to visualize whether your multi-stream design is actually achieving overlap:

# Profile with stream-level visibility
nsys profile \
    --trace=cuda \
    --cuda-memory-usage=true \
    --output=timeline \
    ./my_application

# Open the report
nsys-ui timeline.nsys-rep
# The timeline view will show whether kernels and memcpy operations
# from different streams truly overlap, or serialize due to resource contention

Resource-constrained block residency

I used to think that once a block is launched, it simply executes. That is not how the hardware sees it. The Streaming Multiprocessor treats blocks as units of resource allocation, not just threads to execute.

When a block is assigned to an SM, its residency, the number of blocks and warps that can coexist, is determined by the most constrained of four interlocking resources.

• First, register consumption: each SM has 65,536 registers on Ampere (the register file is physically 32-bit wide). If your kernel uses 64 registers per thread and launches 256-thread blocks, that block consumes 256 × 64 = 16,384 registers, allowing at most 65,536 / 16,384 = 4 blocks to reside simultaneously.

• Second, static and dynamic shared memory: an Ampere SM has up to 164 KB of configurable shared memory (in a combined L1/shared memory array of 192 KB).

• Third, the hardware-imposed maximum of 2,048 resident threads per SM.

• Fourth, the architectural maximum of 32 resident blocks per SM on Ampere.

The CUDA Occupancy Calculator (available as both a spreadsheet and as API calls) computes the binding constraint:

// Query theoretical occupancy from the runtime
int minGridSize, blockSize;

// Let the runtime choose an optimal block size to maximize occupancy
cudaOccupancyMaxPotentialBlockSize(
    &minGridSize,
    &blockSize,
    myKernel,
    0,        // dynamic shared memory per block
    0         // block size limit (0 = no limit)
);
printf("Suggested block size: %d, min grid size: %d\n", blockSize, minGridSize);

// Compute occupancy for a specific configuration
int numBlocks;
cudaOccupancyMaxActiveBlocksPerMultiprocessor(
    &numBlocks,
    myKernel,
    256,   // threads per block
    0      // dynamic shared memory per block
);
printf("Max active blocks per SM: %d\n", numBlocks);
// occupancy = numBlocks * threadsPerBlock / maxThreadsPerSM

// Inspect register and shared memory usage of a compiled kernel
// (from command line)
// nvcc --ptxas-options=-v mykernel.cu
// Output:
// ptxas info: Used 32 registers, 4096 bytes smem, 400 bytes cmem[0]

You can also inspect register usage directly in the compiled binary:

# Check register usage per thread in a compiled cubin
cuobjdump --dump-sass ./my_binary | grep -A5 "Function : myKernel"
# Look for the .regcount field or count the register usage in SASS

# Alternatively, pass verbose flags to ptxas at compile time
nvcc -Xptxas -v -arch=sm_80 mykernel.cu -o mykernel
# ptxas info: compiling entry function 'myKernel' for 'sm_80'
# ptxas info: Function properties for myKernel:
#     0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
# ptxas info: Used 48 registers, 8192 bytes smem, 400 bytes cmem[0]

Register spilling, when the compiler cannot fit all live variables into the 65,536 register file and must evict some to local memory (a per-thread region in global DRAM), is one of the most expensive occupancy failures because local memory accesses are cached in L1/L2 but still require DRAM bandwidth when they miss.

You can force a register cap and observe the effect:

// Annotate a kernel to cap its register usage
// Forces the compiler to spill anything above the cap into local memory
__global__ void __launch_bounds__(256, 4) myKernel(float* data) {
    // 256 = max threads per block (helps compiler optimize)
    // 4   = minimum blocks per SM (compiler will spill registers to meet this)
    ...
}

// Or via nvcc flag (applies globally to all kernels in the translation unit)
// nvcc --maxrregcount=32 mykernel.cu

Residency is not just about thread count. It directly determines the SM’s ability to hide memory latency via warp-level latency hiding. An Ampere SM has four warp schedulers. Each scheduler selects a ready warp every clock cycle.

If a warp issues a global memory load (with a latency of roughly 290 clock cycles to HBM on A100), the warp scheduler immediately switches to another ready warp. To fully hide that 290-cycle latency, you need enough resident warps to keep all four schedulers busy during the wait.

With too few resident warps due to register or shared memory pressure, stalls become visible in the instruction pipeline, and throughput collapses.

Low residency at the block level also reduces system-level concurrency. A single resource-heavy kernel can monopolize SMs, eliminating the scheduler’s ability to overlap operations from multiple streams.

Kernel design cascades upward through the entire submission pipeline.

Subscribe now

The anatomy of GPU execution

Looking back across this journey, from kernel launches to PTX, from SM occupancy to DMA engines, from streams to page migration, the lesson is clear: the GPU is a tightly orchestrated ecosystem, not a black box that executes kernels on demand.

A kernel is not pure computation. It is a descriptor, a submission packet traveling from cudaLaunchKernel through libcudart, across the Driver API privilege boundary, into nvidia.ko, across the PCIe fabric or NVLink interconnect, into the GigaThread Engine’s pushbuffer, through the front-end command processor, and only then into the SM warp schedulers.

Execution is layered: submission queues, DMA engines, front-end command processors, SM schedulers, warp instruction buffers, and physical register files all contribute to the effective performance of even a trivially simple operation.

Hardware constraints propagate upward across the entire stack. Register pressure on a single kernel reduces SM residency, which reduces warp count, which reduces latency-hiding capability, which reduces SM throughput, which reduces the scheduler’s ability to overlap streams, which reduces system-level concurrency.

Memory is not homogeneous: pageable versus pinned allocation, Unified Memory page fault overhead, DMA copy engine count, and HBM bandwidth all determine whether your kernels can achieve real parallel throughput.

Every microsecond is measurable. Mode switches, MMIO signaling, JIT compilation, and page migrations all introduce latency invisible in source code but entirely observable with nsys, nvprof, cuobjdump, and /proc/driver/nvidia-uvm/stats.

Optimizing a GPU is not a matter of loop unrolling, warp-level FMA scheduling, or shared memory bank conflict elimination in isolation.

It is engineering at multiple layers simultaneously, understanding how software abstractions translate to hardware realities at every level of the submission path.

Only by reasoning across the entire execution path, from host-side memory allocation strategy to SM warp scheduler residency, can one design systems that realize the full potential of GPU hardware.

The atomic unit of GPU performance is submission. Everything else, threads, blocks, warps, registers, is consequence, not cause.

Every optimization, every kernel redesign, every memory allocation decision must respect the architecture-wide resource constraints, the interconnect physics, and the hardware scheduler’s discretion.

Share

Introduction

For a long time, I approached CUDA and GPU performance the way most engineers do: through fragments.

I read the CUDA Programming Guide, the PTX ISA specification, sections of the LLVM Language Reference, scattered forum posts, backend source files, and many whitepapers describing streaming multiprocessors and warp schedulers.

Each document was precise, internally consistent, and technically rigorous; but none of them, in isolation, explained the system. The PTX manual described instructions but not why they existed.

The CUDA guide described memory hierarchies but not how code became those memory accesses. LLVM documentation described passes and SSA form, but without grounding them in the physical constraints of a GPU executing 64 warps simultaneously.

I could see every layer, but I could not see how they connected. Performance tuning still felt empirical: inspect PTX, adjust launch parameters, add __restrict__, benchmark again.

The compiler remained a black box that emitted artifacts, not a system whose behavior I could predict.

The shift came when I stopped treating these as independent specifications and instead focused on the underlying invariants:

how computation is represented

how hardware executes dependency graphs

how those representations evolved alongside processor architecture.

I began reading compiler design notes, backend implementations, and especially the writings and technical explanations of Chris Lattner, tracing how LLVM’s SSA model, register allocation, and instruction selection were explicitly designed to map abstract programs onto finite physical machines.

In parallel, I studied the evolution of hardware itself: from in-order scalar pipelines to superscalar out-of-order execution, and eventually to GPUs, where thousands of threads exist not as independent programs but as replicated instances of the same dependency graph.

It became clear that hardware had stopped becoming fundamentally more complex in its execution model; instead, compilers had absorbed that complexity.

The compiler was no longer just translating syntax: it was actively restructuring computation to satisfy register file limits, memory latency constraints, and instruction throughput requirements.

Modern performance was no longer determined solely by hardware capability, but by how effectively the compiler could expose parallelism within the constraints of that hardware.

That realization reframed everything. CUDA was not a separate programming model layered on top of GPUs; it was a frontend into LLVM’s transformation pipeline, and LLVM itself was the mechanism that reshaped high-level intent into a form the GPU could physically sustain.

The compiler had become the critical mediator between software and silicon, encoding assumptions about latency, bandwidth, register pressure, and execution width directly into the structure of the program.

Understanding CUDA performance therefore required understanding LLVM: not just its syntax, but its internal passes, its SSA graph semantics, and its register allocation strategies.

Only then did the abstraction barrier dissolve, revealing that what ultimately runs on a GPU is not the original kernel, nor even its PTX representation, but the final physical realization of an LLVM-optimized dependency graph constrained by the realities of hardware.

Subscribe now

From CUDA kernels to silicon

I remember the exact moment the abstraction barrier finally dissolved into dust.

Until then, CUDA performance tuning felt like an endless cycle of micro-optimizations: insert a __restrict__ here, manually unroll a loop there, inspect the PTX output, rerun benchmarks, adjust launch parameters, and hope for better occupancy.

PTX felt like the canonical truth: the visible, low-level artifact you could reason about, but it was only the tip of the iceberg.

Reading the design notes and deep technical writings of Chris Lattner, combined with the LLVM Language Reference, the NVVM backend specification, and numerous LLVM source dives, revealed a far more rigorous reality: CUDA kernels never become GPU programs directly.

They first exist as fully functional LLVM SSA graphs. Every subsequent transformation, like for example NVVM lowering, PTX emission, instruction scheduling, register allocation, is a constrained projection of that SSA graph onto the finite hardware resources of an SM.

When Clang compiles a CUDA kernel, the “threads” you write in source code are abstracted away; the compiler produces a single SSA function parameterized by the thread index.

Execution is purely functional, and dependencies are fully explicit. Consider a canonical SAXPY kernel:

__global__ void saxpy(float a, float* __restrict x,
                      float* __restrict y,
                      float* __restrict out) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    out[i] = a * x[i] + y[i];
}

Clang then lowers this into an SSA function resembling:

define void @saxpy(float %a,
                   float addrspace(1)* noalias %x,
                   float addrspace(1)* noalias %y,
                   float addrspace(1)* noalias %out) {

entry:
  %tid   = call i32 @llvm.nvvm.read.ptx.sreg.tid.x()
  %ctaid = call i32 @llvm.nvvm.read.ptx.sreg.ctaid.x()
  %ntid  = call i32 @llvm.nvvm.read.ptx.sreg.ntid.x()

  %block_offset = mul i32 %ctaid, %ntid
  %i32          = add i32 %block_offset, %tid
  %idx          = sext i32 %i32 to i64

  %x_ptr   = getelementptr float, float addrspace(1)* %x, i64 %idx
  %y_ptr   = getelementptr float, float addrspace(1)* %y, i64 %idx
  %out_ptr = getelementptr float, float addrspace(1)* %out, i64 %idx

  %vx = load float, float addrspace(1)* %x_ptr, !noalias !0
  %vy = load float, float addrspace(1)* %y_ptr, !noalias !1

  %mul = fmul contract float %a, %vx
  %sum = fadd contract float %mul, %vy

  store float %sum, float addrspace(1)* %out_ptr, !noalias !2
  ret void
}

Several structural observations are immediately apparent:

1. SSA nodes encode all dependencies explicitly. Every intermediate %value, like %vx, %mul, %sum, occupies a virtual register that is immutable. Lifetimes are explicit; LLVM can compute dead values and reuse registers globally, long before physical registers exist.

2. Thread identity is parameterized, not instantiated. %tid, %ctaid, %ntid are just SSA inputs. The compiler sees a single, deterministic function, not thousands of concurrently executing threads. Hardware parallelism emerges only when the same SSA graph is instantiated across SM warps.

3. Memory hierarchy is embedded in the type system. addrspace(1) for global memory, addrspace(3) for shared memory, addrspace(5) for local/thread memory. This distinction allows alias analysis and load-store motion passes to apply different reordering guarantees depending on latency and side effects.

The role of __restrict__ is now immediately visible in LLVM IR:

load float, float addrspace(1)* %a, !noalias !0

This metadata allows the compiler to disambiguate memory locations, enabling Global Value Numbering (GVN), load forwarding, and redundant load elimination.

On a GPU, the difference is dramatic: a single unprovable alias in global memory could serialize hundreds of cycles of latency across a warp.

Another early and profound transformation is the mem2reg pass. Stack allocations introduced in IR construction are converted into pure SSA values:

Before mem2reg:

%tmp = alloca float
store float %vx, float* %tmp
%val = load float, float* %tmp
%mul = fmul float %val, %val

After mem2reg:

%mul = fmul float %vx, %vx

The load and store vanish entirely. On GPUs, this avoids global memory instructions that would have cost hundreds of cycles per thread. SSA immutability and live-range analysis now become the primary mechanism controlling register pressure downstream.

The next critical stage is SelectionDAG, where LLVM converts SSA IR into a directed acyclic graph of operations:

        fadd
       /    \
     fmul    load y
     /  \
    a   load x

This graph is target-independent but already encodes all data dependencies. DAG nodes are matched against target-specific instruction patterns using heuristics that account for latency, throughput, and register use. For example, the LLVM IR:

%offset = mul i64 %i, 4
%addr   = add i64 %base, %offset

becomes PTX:

mul.lo.s64 %rd2, %rd1, 4;
add.s64 %rd3, %rdBase, %rd2;

but at the SASS stage, NVIDIA’s backend can fold this into a single scaled addressing instruction.

Instruction count is not the primary metric; minimizing dependency depth and live-range length dominates warp occupancy and latency hiding.

Machine IR (MIR) after instruction selection but before register allocation exposes these constraints:

%3:gpr64 = IMUL64ri32 %1, 4
%4:gpr64 = ADD64rr %base, %3

Here, %gpr64 are still virtual. LLVM builds an interference graph and maps virtual registers onto finite physical registers.

On an Ampere SM, 65536 registers per SM dictate occupancy: a kernel requiring 32 registers per thread allows 2048 threads, but 96 registers per thread drops occupancy to 682.

Correlating LLVM live-range dumps with ptxas --verbose reports confirmed that register pressure originates entirely at the LLVM IR level, long before PTX emission.

Finally, loop transformations illustrate LLVM’s deterministic orchestration of latency hiding. Consider a reduction:

for (int i = 0; i < 1024; i++)
    sum += x[i];

The initial IR contains a loop-carried dependency:

%sum_next = fadd float %sum, %val

which enforces strict serialization. After partial unrolling:

%v0 = load float, ptr
%v1 = load float, ptr+4
%v2 = load float, ptr+8
%v3 = load float, ptr+12

%s1 = fadd float %sum, %v0
%s2 = fadd float %s1, %v1
%s3 = fadd float %s2, %v2
%s4 = fadd float %s3, %v3

LLVM’s scheduler can now interleave loads and arithmetic:

load x[i]
load x[i+1]
fmul previous
fadd previous

The scheduler can interleave arithmetic with independent loads. By the time PTX is emitted, all parallelism, instruction ordering, and memory dependencies have already been determined; the warp scheduler is merely executing what LLVM has exposed.

Global memory latency is hidden not by runtime heuristics, but by compiler-scheduled instruction independence.

Ultimately, CUDA performance is a function of SSA graph transformations, not PTX heuristics. Every critical metric (register pressure, warp occupancy, memory coalescing, instruction-level parallelism) originates in LLVM IR.

Once I understood this, CUDA tuning ceased being black magic: by inspecting IR and tracing passes, I could predict register allocation, spilling, latency hiding, and throughput entirely before compiling to PTX or running on hardware.

The abstraction barrier had vanished, and performance became a deterministic, analyzable function of compiler-driven SSA transformations.

Subscribe now

Tracing LLVM Transformations to Hardware

Once I understood that CUDA kernels first exist as LLVM SSA graphs, the next natural question emerged: how exactly do these immutable, dependency-annotated IR values map all the way down to warp execution on the SM?

To answer this, I traced a simple kernel (again, SAXPY) through every stage: SSA → NVVM → PTX → SASS → warp execution.

LLVM SSA Graph

The SSA graph encodes pure computation and memory dependencies:

%vx  = load float, float addrspace(1)* %x_ptr, !noalias !0
%vy  = load float, float addrspace(1)* %y_ptr, !noalias !1
%mul = fmul contract float %a, %vx
%sum = fadd contract float %mul, %vy
store float %sum, float addrspace(1)* %out_ptr, !noalias !2

Key points:

• Each %value is immutable. There is no “overwriting” of registers; instead, values flow along edges in a DAG.

• Dependencies are explicit. %sum depends on %mul, which depends on %vx and %a.

• Memory type is explicit. addrspace(1) informs alias analysis, reordering, and coalescing passes.

LLVM passes operate entirely on this DAG:

• mem2reg removes unnecessary stack loads/stores.

• GVN eliminates redundant calculations.

• LICM hoists loop-invariant loads into shared memory.

• LoopUnroll exposes independent operations for the scheduler.

• SLPVectorizer packs independent arithmetic into wider SIMD instructions (or pseudo-SIMT operations for the GPU).

At this stage, parallelism is implicit, coming from parameterization over %tid and %ctaid, not from threads or warps.

NVVM Lowering and PTX Generation

LLVM’s NVVM backend converts SSA into PTX pseudo-assembly, a virtual instruction set for NVIDIA GPUs:

// Global memory load
ld.global.f32 %f1, [%x+%idx];
// Arithmetic
mul.f32 %f2, %a, %f1
add.f32 %f3, %f2, %y[%idx]
// Store
st.global.f32 [%out+%idx], %f3

Observations:

• SSA immutability is preserved: %f1, %f2, %f3 correspond 1:1 with SSA nodes.

• Memory hierarchy is respected. Loads/stores from addrspace(1) are global, addrspace(3) would be shared.

• Instruction reordering is limited only by LLVM metadata and alias analysis, not PTX syntax. PTX reflects already scheduled operations.

PTX is still virtual: the actual scheduling, register allocation, and instruction fusion are deferred until SASS.

SASS: Physical Instruction Selection

The NVIDIA SASS (microarchitecture ISA) stage maps PTX to hardware instructions for the SM:

IMAD R3, R1, 4, RBase    // Compute address
LD.E R4, [R3]            // Load x[i]
LD.E R5, [R3_y]          // Load y[i]
FMAD R6, R4, Ra, R5      // Multiply-add
ST.E [R3_out], R6        // Store result

Key points:

• LLVM’s DAG scheduling determines the order of these operations. Instruction count is not minimized; dependency depth and live-range length are.

• Register pressure is now concrete: each %value is mapped to a physical register. The interference graph built in MIR ensures no two simultaneously live values share the same physical register.

• Instruction fusion happens here (FMAD); an optimization LLVM hinted at by exposing independent arithmetic in SSA.

By now, warp-level execution semantics are encoded: independent instructions are consecutive, loads can hide arithmetic latency, and dependency chains are minimized.

Warp Execution

Finally, the SM executes multiple threads (warps) of the same SASS code:

Thread 0: load x[0], load y[0], fma, store
Thread 1: load x[1], load y[1], fma, store
...
Thread 31: load x[31], load y[31], fma, store

• Warp scheduler selects ready instructions from the DAG instantiated across 32 threads.

• Global memory latency hiding emerges naturally: because LLVM already separated independent loads from dependent arithmetic, the scheduler always finds instructions to issue, overlapping memory and compute.

• Register pressure limits occupancy: if LLVM IR produced too many live %values per thread, fewer threads can execute concurrently.

The crucial insight: hardware doesn’t create parallelism. It executes what LLVM has already structurally exposed.

Warp scheduling, latency hiding, and coalescing are emergent properties of SSA DAG transformations combined with physical constraints.

Loop Unrolling in SSA → PTX → SASS

Consider a reduction:

for (int i=0; i<4; i++)
    sum += x[i];

SSA after partial unroll:

%v0 = load float, %x+0
%v1 = load float, %x+1
%v2 = load float, %x+2
%v3 = load float, %x+3

%s1 = fadd %sum, %v0
%s2 = fadd %s1, %v1
%s3 = fadd %s2, %v2
%s4 = fadd %s3, %v3

PTX:

ld.global.f32 %f0, [%x]
ld.global.f32 %f1, [%x+4]
ld.global.f32 %f2, [%x+8]
ld.global.f32 %f3, [%x+12]

add.f32 %sum1, %sum, %f0
add.f32 %sum2, %sum1, %f1
add.f32 %sum3, %sum2, %f2
add.f32 %sum4, %sum3, %f3

SASS:

LD.E R0, [R1]
LD.E R2, [R1+4]
LD.E R4, [R1+8]
LD.E R6, [R1+12]

FADD R8, Rsum, R0
FADD R10, R8, R2
FADD R12, R10, R4
FADD R14, R12, R6
ST.E [Rout], R14

• Instruction interleaving: the SM scheduler can issue independent loads before dependent FADDs.

• Latency hiding: memory loads from global memory overlap with prior arithmetic.

• Predictable register pressure: each %vN maps to R0–R6; FADDs reuse registers, keeping occupancy within hardware limits.

  Subscribe now

Merging History, Architecture, and Software

I remember stepping a little bit back and seeing the full picture: GPU evolution, compiler transformations, and kernel performance are inseparable.

Every decision in a CUDA kernel exists in the context of decades of architectural evolution.

In the 1990s, GPUs were fixed-function pipelines: vertices and pixels flowed through rigid stages, parallelism implicit but inaccessible.

Programmable shaders in the early 2000s allowed tiny per-pixel or per-vertex programs (SIMD-style warps, predication, and texture-aware memory) but general-purpose computation was still a hack.

Then CUDA arrived in 2006, exposing threads, blocks, and explicit memory hierarchies: registers, shared memory, global memory. For the first time, developers could directly reason about latency, coalescing, and warp-level execution.

Yet kernels were still only the tip of the iceberg: true performance lives in LLVM IR, long before PTX or hardware sees it.

Every kernel begins as an SSA graph: pure, immutable, fully explicit in dependencies. %tid, %ctaid, %ntid are function parameters, not threads.

Address spaces encode memory hierarchy: addrspace(1) for global, addrspace(3) for shared, addrspace(5) for local. __restrict__ metadata allows disambiguation, enabling GVN, load forwarding, and aggressive reordering.

Loop unrolling, scalar replacement, and DAG scheduling expose independent operations, determining instruction-level parallelism and register pressure.

When LLVM lowers to PTX and eventually SASS, the graph’s structure dictates occupancy, warp scheduling, and latency hiding. The GPU does not invent parallelism; it realizes the parallelism the compiler has already exposed.

Every optimization (register allocation, memory coalescing, interleaved arithmetic) is preordained by SSA transformations.

Subscribe now

Compiler, Hardware, and the Deterministic Birth of Performance

Stepping back, it becomes clear that GPU performance is never accidental.

Every evolution, starting from fixed-function pipelines to shaders, from CUDA to modern heterogeneous SMs, is a layer that shaped how we write, reason about, and execute kernels.

But the real magic lies in the compiler, particularly LLVM: it transforms human code into SSA graphs where dependencies, memory hierarchy, and live ranges are explicit, deterministic, and fully analyzable.

PTX and SASS are not optimization stages; they are projections of these decisions onto silicon.

Performance emerges not at runtime but at compile time, where LLVM orchestrates instruction scheduling, loop transformations, memory disambiguation, and register allocation.

The GPU simply executes what LLVM has already exposed: warps, latency hiding, coalesced accesses, and parallelism are consequences of carefully structured SSA graphs.

Every tweak in IR ripples through PTX, affects occupancy, and determines throughput. Understanding this chain (history, architecture, compiler) is what allows a developer to predict, reason about, and ultimately master GPU performance.

Share

A Personal Journey into GPU Computing

I still remember November 12, 2024, with a clarity that surprises even me to this day, even though more than a year has already passed.

I had spent the evening lost in LLVM internals, tracing the intricate dance between front-end parsing, IR transformations, optimization passes, and backend code generation.

I explored FunctionPassManager sequences, observed GVN, SROA, LoopVectorization, Instruction Combining, and Dead Code Elimination, and tried to correlate these transformations with register pressure, live interval analysis, and instruction scheduling heuristics.

Every detail, every pass, seemed like a miniature universe of logic, optimization, and constraint satisfaction.

I had always been fascinated by the intersection of software, hardware, and system infrastructure; but until that night, I had never realized just how foundational and deep that connection could be.

I was thinking about compilers, IR, and abstract execution models, but I had no sense yet of how these abstractions directly map to silicon at massive scale.

Then I stumbled upon an article about DeepSeek R1, the PTX intermediate representation, and the orchestration of thousands of threads across NVIDIA GPUs’ Streaming Multiprocessors (SMs).

It felt like stepping through a door into a world I had only glimpsed. Almost instantly, I found myself in the middle of a global conversation: for days, forums, blogs, and even financial news outlets buzzed about DeepSeek.

Developers dissected PTX, debated its scheduling optimizations, speculated about the next wave of AI workloads, and NVIDIA’s stock price swung dramatically.

While most people were caught up in hype, speculation, or high-level geopolitical and financial implications, I became obsessed with the core layer: the base of the software, the invisible threads connecting high-level code to the hardware executing it.

I devoured every whitepaper, blog post, SDK guide, and forum discussion I could find, trying to understand exactly how PTX bridges high-level CUDA kernels with the underlying SM pipelines, how execution is scheduled across warps, and how memory hierarchies (register files, shared memory banks, L1/L2 caches, and DRAM) are orchestrated at a microscopic level.

At that time, I barely understood what a GPU did beyond simple graphics acceleration or using high-level CUDA frameworks. But the article illuminated something crucial: modern computational performance isn’t just about clever algorithms.

It is about how software maps onto hardware through multiple layers of compilation and execution, aspects that most programmers, including myself until that night, rarely consider.

I realized that to truly understand high-performance GPU computing, I needed to trace the entire execution path: from CUDA C++ kernels, through LLVM IR transformations, into PTX, and finally into the Streaming Multiprocessor pipelines themselves.

Suddenly, my abstract fascination with compilers and infrastructure became a tangible, almost physical journey.

Every kernel, every loop, every thread had a story. I could begin to see how thread indexing (threadIdx, blockIdx, blockDim) maps logically to warps, how memory coalescing affects throughput, how shared memory bank conflicts serialize execution, and how predication avoids warp divergence penalties.

I was no longer just reading about GPUs; I was entering their world thread by thread, instruction by instruction, tracing the invisible logic that transforms high-level abstractions into thousands of coordinated instruction streams flowing through silicon.

That night marked the start of a transformation in my understanding. It wasn’t merely academic curiosity anymore; it was a journey into the layered reality of modern computing, where compiler theory, PTX abstractions, and microarchitectural details converge to define what is actually possible on a GPU.

And it was the beginning of the series you’re reading: a deep dive into CUDA, LLVM IR, PTX, and SM execution, told from the perspective of someone who has traced every layer, experimentally and obsessively, to understand not just how GPUs compute, but why.

Subscribe now

My LLVM odyssey

Before GPUs became the core focus of my studies, I was already deeply immersed in software infrastructure, with a singular fascination: compilers.

Their ability to transform high-level code into highly optimized machine instructions, managing dependencies, scheduling, and register allocation, had always captivated me: almost like engineering alchemy, where abstract logic crystallizes into finely tuned execution on silicon.

I spent months dissecting LLVM’s internals, tracing each stage of the pipeline.

At the front-end, I studied clang’s parsing, type-checking, and LLVM IR generation, observing how control flow graphs, SSA form, and metadata are preserved to represent program semantics accurately.

LLVM IR became a playground of possibilities, allowing multiple layers of analysis, transformation, and optimization without committing to a target architecture.

Next came the optimization passes. I followed the FunctionPassManager, analyzing pass ordering and dependencies.

I explored Global Value Numbering (GVN) to identify equivalent computations, Scalar Replacement of Aggregates (SROA) to break down complex structures, and LoopVectorization to exploit SIMD execution patterns.

Instruction Combining and Dead Code Elimination (DCE) revealed how small IR-level transformations propagate through instruction scheduling, register pressure, and memory accesses.

Diving deeper, I mapped IR onto hardware targets through InstructionSelector DAGs, examining how LLVM matches abstract operations to target-specific opcodes, considers RegisterClass constraints, and queries TargetTransformInfo to evaluate instruction latency, throughput, and memory cost models.

I traced live interval analysis in relation to register allocation heuristics, noting how spilling, rematerialization, and coalescing decisions affect execution efficiency: critical insights for architectures with thousands of concurrent threads.

Predication and control-flow lowering fascinated me: conditional branches in IR could be transformed into predicated instructions, minimizing pipeline stalls.

Loop transformations (unrolling, interchange, fusion, and vectorization) demonstrated the delicate balance between instruction-level parallelism, memory alignment, and cache behavior.

Each IR pass could increase or decrease register pressure, affect instruction scheduling, or modify memory footprint, directly influencing runtime performance.

Even then, I did not anticipate how this knowledge would later intersect with GPU architectures.

LLVM IR transformations, optimization passes, and backend heuristics were abstract exercises; but they had prepared me to reason about CUDA kernels, PTX intermediate representation, warp scheduling, shared memory layouts, and SM pipelines.

Every abstraction, every instruction, and every scheduling decision became a lens through which I could understand how software maps efficiently onto massively parallel hardware.

Subscribe now

The evening of discovery: PTX and DeepSeek

January 20, 2025. That evening remains etched in my memory. After hours lost in LLVM’s inner workings, by tracing IR transformations, exploring FunctionPassManager sequences, and correlating register allocation with live interval analysis, I stumbled upon the DeepSeek R1 article.

The moment felt surreal: everything I had been exploring in software infrastructure suddenly collided with the raw power of hardware parallelism.

The article unpacked PTX, NVIDIA’s virtual ISA, and detailed how thousands of threads execute across warps on a Streaming Multiprocessor (SM).

Each warp, a bundle of 32 threads in SIMT execution, follows the instruction scheduler, hiding latency by rapidly switching between warps.

PTX abstracts this complexity: developers write kernels in CUDA C++, compile them with NVCC, which emits PTX, and the NVIDIA driver JIT-compiles the PTX into device-specific cubins streamed into the SM pipelines.

PTX fascinated me for several reasons:

• Forward compatibility: A PTX kernel compiled today could run on next-generation GPUs via JIT compilation, bridging software longevity and hardware evolution.

• Hardware abstraction: Kernels could be written without intimate knowledge of specific SM layouts, register files, or shared memory bank conflicts.

• Optimization opportunities: Despite abstraction, peak performance still demanded deep awareness of occupancy, memory coalescing, shared memory conflicts, warp divergence, and pipeline latencies.

The parallels with LLVM were striking. LLVM transforms high-level C++ into IR, applies passes like GVN, SROA, LoopVectorization, Instruction Combining, and Dead Code Elimination, and lowers code to target-specific instructions.

PTX, on the other hand, expresses parallel execution semantics independently of final hardware mapping. Both systems separate abstraction from execution yet require deep knowledge to optimize performance.

I sat tracing mental parallels between LLVM’s DAG-based instruction selection and PTX’s warp-scheduled execution. LLVM must reason about register pressure, instruction latency, predication, and control-flow lowering.

PTX, in turn, demands understanding warp occupancy, coalesced memory accesses, shared memory bank conflicts, and L1/L2 cache interplay with DRAM.

That night, DeepSeek and PTX became more than concepts. They bridged my compiler obsession with high-performance GPU computing.

I realized that to truly understand CUDA, I needed to follow the full path: high-level CUDA kernels → NVCC → LLVM IR → PTX → cubins → SM pipelines, where thousands of threads coordinate in lockstep to perform computations that would overwhelm any CPU.

Subscribe now

Tracing CUDA Kernels, PTX, and the Hardware Dance

When I first started writing CUDA kernels, my experiments were deliberately simple: matrix addition, vector scaling, small convolution operations. I wanted to see, concretely, how high-level code transformed as it moved down the compilation chain.

I compiled these kernels and examined the PTX emitted by NVCC. The experience was revelatory, almost like peeling back a layer of abstraction to reveal a hidden parallel universe of computation.

PTX exposed the fundamental building blocks of GPU execution:

• Thread indexing: Registers like threadIdx, blockIdx, and blockDim define each thread’s unique identity in a multidimensional grid. Observing PTX code, I could see how each thread calculates its global index using arithmetic derived from block and grid dimensions.

• Memory coalescing: PTX made it clear how critical memory alignment is. Threads within a warp that access contiguous addresses can combine requests into a single transaction, dramatically increasing throughput. Misalignment, however, causes serialized transactions, stalls, and pipeline bubbles.

• Shared memory and bank conflicts: Threads in a block share memory, divided into banks. I could observe how multiple threads attempting to access the same bank would serialize operations, a subtle yet crucial bottleneck.

• Predication: Divergent branches are another warp-level concern. PTX can transform conditional execution into predicated instructions, effectively executing both paths but masking results for non-taken threads. Seeing divergent loops lowered into predicated instructions showed me how compiler heuristics, IR transforms, and PTX scheduling work together for correct, parallel execution.

I remember embedding inline PTX inside simple kernels, controlling instruction ordering, memory patterns, and even warp-level operations that CUDA C++ itself could not expose.

It was a hands-on lesson: the GPU is not just a black box for parallelism; it is a layered ecosystem of instruction streams, memory banks, and scheduling heuristics, all orchestrated to maximize throughput.

Matrix Multiplication and Warp-Level Optimization

After mastering vector addition, I wanted a more challenging experiment — something that truly tested the GPU’s compute and memory subsystems. I chose matrix multiplication, the classic computational kernel in high-performance computing, AI, and scientific simulations.

The first naïve implementation looked like this:

__global__ void matMulNaive(float* A, float* B, float* C, int N) {
    int row = threadIdx.y + blockIdx.y * blockDim.y;
    int col = threadIdx.x + blockIdx.x * blockDim.x;

    if(row < N && col < N){
        float sum = 0.0f;
        for(int k = 0; k < N; ++k){
            sum += A[row * N + k] * B[k * N + col];
        }
        C[row * N + col] = sum;
    }
}

At a high level, this seems straightforward: each thread computes a single element of the result matrix.

But my experience with vector addition had taught me to look deeper, beyond correctness, into how this kernel would map onto PTX and SM pipelines.

LLVM IR insights

I compiled the kernel with NVCC targeting PTX and examined the LLVM IR output. Immediately, several patterns emerged:

• Nested loops are represented as IR loops: The outer loops over rows and columns map directly to thread indexing arithmetic. The inner loop over k is a canonical loop with phi nodes tracking the accumulator sum.

• SSA form and virtual registers: Every variable (row, col, k, sum) exists as a virtual register, yet LLVM’s optimization passes can combine, eliminate, or spill them based on usage patterns.

• Load/store separation: Every access to A[row * N + k] and B[k * N + col] generates explicit load instructions. LLVM can apply scalar replacement or loop-invariant code motion, lifting certain computations out of loops to reduce redundant instructions.

I remember one night sitting with my IR and a notebook, tracing the phi nodes and loop unrolling transformations. I observed how a small tweak in IR, say unrolling the inner loop by 2, could double the number of live registers per thread.

I immediately understood that naive unrolling without accounting for register pressure could reduce occupancy, a lesson painfully learned with vector addition.

Subscribe now

PTX, a parallel symphony

After IR, I looked at the generated PTX. The arithmetic, loads, and stores became explicit thread-local instructions:

mul.lo.u32 %row_idx, %blockIdx.y, %blockDim.y
add.u32 %row, %row_idx, %threadIdx.y

mul.lo.u32 %col_idx, %blockIdx.x, %blockDim.x
add.u32 %col, %col_idx, %threadIdx.x

setp.ge.u32 %p0, %row, %N
setp.ge.u32 %p1, %col, %N
or.pred %p2, %p0, %p1
@%p2 bra DONE

mov.f32 %sum, 0.0
LOOP:
ld.global.f32 %a, [A + %row*N + %k]
ld.global.f32 %b, [B + %k*N + %col]
fma.rn.f32 %sum, %a, %b, %sum
add.u32 %k, %k, 1
setp.lt.u32 %cond, %k, %N
@%cond bra LOOP
st.global.f32 [C + %row*N + %col], %sum
DONE:

I was fascinated by several features:

• Fused multiply-add (FMA) instructions: PTX automatically uses FMA when possible, maximizing FLOPs per cycle.

• Predicated branches: Out-of-bounds threads are masked off, avoiding unnecessary execution while keeping warps aligned.

• Thread indexing arithmetic: Derived directly from threadIdx, blockIdx, and blockDim. Every PTX instruction respects these indices to maintain correctness across thousands of threads.

But even this PTX was naive. Each thread individually loaded elements of A and B from global memory. I knew from my vector addition experiments that misaligned global accesses and low arithmetic intensity would throttle performance.

Shared memory tiling

The next step was introducing shared memory tiling, a standard GPU optimization. I rewrote the kernel to load tiles of A and B into shared memory, perform the multiply-accumulate locally, and then write back the result:

__global__ void matMulTiled(float* A, float* B, float* C, int N){
    __shared__ float As[32][32];
    __shared__ float Bs[32][32];

    int row = threadIdx.y + blockIdx.y*blockDim.y;
    int col = threadIdx.x + blockIdx.x*blockDim.x;
    float sum = 0.0f;

    for(int tile=0; tile < N/32; ++tile){
        As[threadIdx.y][threadIdx.x] = A[row*N + tile*32 + threadIdx.x];
        Bs[threadIdx.y][threadIdx.x] = B[(tile*32 + threadIdx.y)*N + col];
        __syncthreads();

        for(int k=0; k<32; ++k)
            sum += As[threadIdx.y][k] * Bs[k][threadIdx.x];
        __syncthreads();
    }
    C[row*N + col] = sum;
}

Here, I could directly control shared memory layout, a factor invisible at the high-level IR stage. I experimented with bank conflicts, adjusting tile strides to avoid multiple threads accessing the same bank simultaneously.

The performance gains were immediate: bandwidth utilization improved, and warp stalls due to serialized accesses dropped dramatically.

Subscribe now

Warp scheduling, occupancy, and register pressure

Next, I traced the PTX generated by the tiled kernel. The number of live registers increased because each thread now held sum, multiple shared memory references, and loop counters. I correlated this with SM occupancy:

• Registers per thread: Increasing them beyond ~64 per thread reduced active warps per SM from 64 to 32.

• Warp interleaving: Fewer active warps reduced the hardware’s ability to hide memory latency.

• Shared memory allocation: Large tiles per block reduced the number of blocks that could reside concurrently on an SM.

These experiments mirrored what I had seen in LLVM: IR-level decisions cascade all the way to warp execution and memory latency hiding.

Inline PTX Experiments in Matrix Multiplication

I pushed further, embedding inline PTX inside tiled kernels:

asm volatile(
    "ld.shared.f32 %fA, [As + %threadIdx.y*32 + %k];\n\t"
    "ld.shared.f32 %fB, [Bs + %k*32 + %threadIdx.x];\n\t"
    "fma.rn.f32 %sum, %fA, %fB, %sum;\n\t"
);

This allowed me to:

• Test instruction ordering and latency hiding manually.

• Experiment with predication and divergent threads, observing how small PTX-level changes could improve warp execution.

• Measure shared memory bank conflicts in real time, adjusting indexing to maximize throughput.

I could see the subtle interplay between LLVM IR choices, PTX transformations, and hardware execution, solidifying my understanding that GPU performance is a multi-layered orchestration.

Subscribe now

My deep experiments

I approached CUDA like a lab. I experimented relentlessly:

• Observing how LLVM IR loop unrolling affects PTX instruction counts and warp efficiency.

• Measuring occupancy changes as kernel launch parameters and register usage were modified.

• Embedding inline PTX to explore instruction scheduling, shared memory bank conflicts, and predication.

• Mapping thread blocks to SMs to study warp interleaving and latency hiding.

These experiments made abstract compiler and microarchitectural concepts tangible. Every IR pass, PTX instruction, and memory layout decision was observable in execution behavior.

I saw clearly how software abstractions orchestrate silicon, and how performance is a delicate balance of register allocation, memory coalescing, and warp-level scheduling.

Why this series exists

By the end of this journey, I knew why I had to write this series. I wanted more than a cursory “launch a kernel” tutorial. My goal is a narrative-driven, technically rigorous exploration of CUDA:

• LLVM IR transformations and compiler mechanics.

• PTX intermediate representation and optimization strategies.

• CUDA kernel design, thread blocks, and warps.

• GPU microarchitecture, SM pipelines, and memory hierarchy.

• Real-world performance strategies, from occupancy tuning to memory coalescing.

This series is my attempt to bridge software and hardware, guiding readers through the journey I experienced.

Curiosity sparked by DeepSeek, insights crystallized by LLVM, experiments refined in PTX, and understanding solidified by GPU microarchitecture.

Readers will trace kernels from CUDA C++ to LLVM IR, into PTX, and finally to SM execution, experiencing the choreography of parallel computation firsthand.

You will see not only how to write CUDA code, but why each instruction executes as it does, how threads collaborate within warps, and how to extract maximum performance by understanding the deep interplay of compiler optimizations, PTX abstractions, and hardware realities.

Enjoy this incredible journey!

Share

Introduction

Engineers who have spent many years in the trenches of distributed systems develop an instinctive radar for complexity.

It’s a radar that basically never sleeps. It picks up on the faintest signs that a system is about to balloon beyond comprehension: the silent growth of memory tables, the small code change that cascades into unpredictable failure, the queues that inflate exponentially under the first unexpected burst.

At small or medium scale, these things are manageable. You add retries, per-flow connection tracking, dynamic allocation: you feel just a bit more clever. But scale up to planetary levels, and these “clever” additions become the system’s Achilles’ heel.

Every bit of mutable state, every dynamic structure, every per-flow table is a ticking time bomb. And this is precisely where Katran finds its elegance.

Katran strips away all unnecessary complexity. It reduces load balancing to pure, deterministic computation, executed at the first possible moment a packet enters the kernel.

There are no per-flow tables. There are no loops. There are no dynamic allocations. There is no memory of the past. Every packet is an independent, self-contained entity, evaluated as a function: headers in, backend out, no storage, no cleanup, no drama.

It is, in a sense, load balancing as mathematics, executed at line rate.

Subscribe now

Encountering the XDP Program

When you first lay eyes on Katran’s XDP program, disbelief is natural.

You instinctively start looking for the infrastructure you expect from a “normal” load balancer: socket managers, memory pools, per-flow hash tables, connection cleanup threads.

Where are the timers that expire idle flows?

Where is the heap allocation for buffering partial packets?

Where is the messy user-space coordination?

The answer, surprisingly, is: none of that exists. Every omission is intentional, dictated by the eBPF verifier’s strict execution model and the program’s architectural goal: stateless, line-rate forwarding.

The eBPF verifier acts as a pedantic static analyzer for the kernel. At load time, it simulates every possible execution path through the program, tracking all pointers, offsets, and integers.

It ensures that no memory access exceeds the bounds of the packet buffer (ctx->data to ctx->data_end), no pointer arithmetic overflows, and no uninitialized value is ever dereferenced.

If a program could potentially write past data_end or access memory outside a BPF map safely, the verifier refuses to load it.

This is why XDP code is highly defensive: explicit casts, pointer comparisons, early exits, and carefully sequenced checks are everywhere. In short, you can’t crash the kernel because the verifier won’t allow it.

Katran leverages this discipline elegantly. Every packet is validated before it touches the forwarding logic. Parsing begins at the Ethernet layer:

struct ethhdr *eth = data;
if ((void *)(eth + 1) > data_end)
    return XDP_DROP;

if (eth->h_proto != htons(ETH_P_IP))
    return XDP_PASS;

Here, (eth + 1) calculates the memory address immediately after the Ethernet header. Comparing this against data_end ensures that the packet buffer fully contains the Ethernet header.

If not, the packet is dropped (XDP_DROP). Non-IPv4 traffic is passed upstream (XDP_PASS) because Katran only routes IPv4 packets.

Notice that no memory allocation occurs, and no state is stored; the check is purely functional, evaluating the packet as it exists in memory.

Next, the program parses the IP header:

struct iphdr *ip = data + sizeof(*eth);
if ((void *)(ip + 1) > data_end)
    return XDP_DROP;

if (ip->protocol != IPPROTO_TCP)
    return XDP_PASS;

Pointer arithmetic ensures that ip points exactly at the start of the IP header. The verifier confirms that (ip + 1) does not exceed data_end. Any malformed or truncated packet is dropped immediately.

Non-TCP packets are passed upstream. Again, no dynamic structures are created, no per-flow tables are touched, no queues are used. Every packet is an independent entity, evaluated purely by its content.

The final layer of parsing handles TCP headers:

struct tcphdr *tcp = (void *)ip + sizeof(*ip);
if ((void *)(tcp + 1) > data_end)
    return XDP_DROP;

The program performs the same bounds validation at the TCP layer.

The cast (void *)ip is required because pointer arithmetic on struct iphdr * could otherwise introduce type-dependent scaling that violates verifier expectations.

Every field access is verified at load time, ensuring memory safety.

By the time the packet reaches this stage, Katran has guaranteed that Ethernet, IP, and TCP headers are complete, aligned, and accessible: all without allocating a single byte of memory.

Deterministic Hashing

With headers validated, Katran computes a deterministic 5-tuple hash:

__u64 hash = hash_5tuple(ip->saddr, ip->daddr,
                         tcp->source, tcp->dest,
                         ip->protocol);

This hash forms the cornerstone of the system. It alone determines the backend to which the packet will be forwarded.

The kernel holds no history of past flows; there is no connection table. There are no retries, no queues, no cleanup tasks. Every packet is a self-contained function:

Packet headers → 5-tuple hash → ring map lookup → backend map → packet rewrite → XDP_TX

eBPF Map Lookups

The program then queries two BPF maps: ring_map and backend_map. The ring_map implements a Maglev-style consistent hash ring.

Conceptually, it’s a fixed-size array where each slot points to a backend index, precomputed in the control plane. The backend_map holds the IP and port for each backend.

Both maps are read-only from the kernel’s perspective during packet processing; the control plane updates them asynchronously.

__u32 *backend_idx = bpf_map_lookup_elem(&ring_map, &hash);
if (!backend_idx)
    return XDP_DROP;

struct backend *b = bpf_map_lookup_elem(&backend_map, backend_idx);
if (!b)
    return XDP_DROP;

ip->daddr = b->ip;
tcp->dest = b->port;

recalc_ip_checksum(ip);
recalc_tcp_checksum(ip, tcp);

return XDP_TX;

Notice the simplicity and determinism. The kernel reads the slot atomically, rewrites the packet headers, recalculates checksums, and transmits.

No per-flow state, no locks, no dynamic weighting — all O(1), purely functional. The XDP_TX return code transmits directly back to the NIC TX ring, bypassing sockets entirely.

The packet never enters the kernel networking stack, avoiding context switches, scheduling delays, or buffer copies.

Line-Rate Safety and Verifier Compliance

At first glance, Katran’s code looks almost painfully minimalist. But every defensive check, every explicit cast, and every early return exists because the eBPF verifier demands total memory safety.

The verifier effectively simulates the NIC’s RX ring in software, reasoning about each possible execution path. If any access could exceed data_end or touch uninitialized memory, the program would fail to load.

This ensures that Katran’s XDP program is guaranteed to execute safely at line rate, independent of traffic patterns, packet sizes, or maliciously malformed headers.

By combining careful pointer arithmetic, deterministic 5-tuple hashing, and atomic BPF map reads, Katran transforms each packet into a stateless, functional computation.

No connections are remembered. No resources are dynamically allocated. No loops or locks slow the datapath.

This is minimalism enforced by both design and verifier; the kernel’s version of “if it isn’t necessary, it doesn’t exist”.

Subscribe now

Maglev-Style Hash Ring Construction

Katran’s stateless elegance rests on one deceptively simple concept: a precomputed Maglev-style consistent hash ring.

At first glance, it’s just an array, a circle of RING_SIZE slots. But in reality, it is the linchpin that allows packets to be routed deterministically, without storing per-flow state, and without any runtime iteration over backends.

Each backend server occupies multiple slots on this ring, proportionally to its assigned weight. Packets are mapped to slots via the 5-tuple hash computed in the kernel.

Once the hash is computed, the kernel does nothing more than perform a BPF map lookup: the heavy lifting has already been done by the control plane.

The XDP program never recomputes the ring, never tracks the flow history, and never allocates memory: it simply consults the map, rewrites headers, and transmits.

Virtual Nodes and Weighted Distribution

The magic lies in virtual nodes. Each backend is assigned a number of virtual nodes proportional to its weight.

A backend that is twice as capable as another receives twice as many virtual nodes, which in turn means it will receive roughly twice the traffic. The control plane distributes these virtual nodes pseudo-randomly but deterministically across the ring.

Deterministic here means that given the same backend configuration and weight, every calculation produces the same ring layout: a property essential for consistent flow mapping and minimal disruption during updates.

When a backend fails, only the slots associated with its virtual nodes are removed. The remaining slots for other backends are untouched.

This minimal disruption ensures that the vast majority of traffic continues to flow to existing backends without rerouting, avoiding cache thrashing, queue spikes, or packet reordering at the network level.

It’s a conceptually simple idea, but its effects on reliability and predictability at planetary scale are profound.

Subscribe now

Control Plane Construction

Here is a distilled view of how the control plane constructs the ring:

#define RING_SIZE 65537
#define VIRTUAL_NODES_PER_BACKEND 100

struct backend {
    __u32 ip;
    __u16 port;
    __u32 weight;
};

__u32 ring[RING_SIZE]; // 0 = empty, 1..N = backend indices

for (int b = 0; b < num_backends; b++) {
    int virtual_nodes = backends[b].weight * VIRTUAL_NODES_PER_BACKEND;
    for (int v = 0; v < virtual_nodes; v++) {
        __u64 hash = hash_fn(backends[b].ip, backends[b].port, v);
        __u32 offset = hash % RING_SIZE;
        __u32 skip = (hash % (RING_SIZE - 1)) + 1;

        while (ring[offset] != 0) {
            offset = (offset + skip) % RING_SIZE;
        }
        ring[offset] = b + 1; // backend index + 1 because 0 = empty
    }
}

At first glance, it looks almost naïve: a nested loop, some arithmetic, and a while loop to resolve collisions. But every line has a precise purpose:

1. hash_fn combines the backend’s IP, port, and virtual node index to produce a deterministic pseudo-random value.

2. offset determines the initial slot in the ring for this virtual node.

3. skip ensures that collisions are resolved deterministically: if a slot is already occupied, we advance by skip until an empty slot is found.

4. The final assignment writes the backend index plus one; because zero is reserved to indicate “empty.”

The result is a fully populated, deterministic hash ring. The control plane now has a complete map of which slot belongs to which backend, and crucially, this computation happens entirely outside the kernel, asynchronously from packet forwarding.

Pushing the Ring into the Kernel

Once the ring is computed, the control plane atomically populates a BPF map in the kernel:

for (int i = 0; i < RING_SIZE; i++) {
    __u32 key = i;
    __u32 val = ring[i];
    bpf_map_update_elem(&ring_map, &key, &val, BPF_ANY);
}

From this moment, the XDP program doesn’t touch the ring logic. Each incoming packet:

1. Computes its 5-tuple hash.

2. Calculates the slot index via hash % RING_SIZE.

3. Looks up the slot in the ring_map to find the backend index.

4. Looks up the backend IP and port in the backend_map.

5. Rewrites headers and transmits.

The ring is immutable from the kernel’s perspective during packet processing. Updates, weight adjustments, or backend removals are handled entirely by the control plane, and applied atomically.

This allows Katran to handle failures and reconfigurations without ever stalling packet forwarding.

Why This Matters

The beauty of this approach is that the XDP datapath is entirely stateless and deterministic, yet fully aware of backend weights.

There are no loops in the kernel, no per-flow tracking, no locks or contention. The cost of adding more backends or adjusting weights is absorbed by the control plane; the kernel simply reads the updated maps.

This separation allows line-rate forwarding at planetary scale, where a single ingress could be handling millions of flows per second.

The Maglev ring is the perfect example of precomputation as a scaling strategy: do the complex work once in user-space, push a static, deterministic representation to the kernel, and let the stateless datapath execute at hardware speed.

Every packet becomes a pure function evaluation: input headers → hash → slot → backend → rewritten packet → transmit.

No memory allocations, no state, no concurrency hazards; just physics applied to packets.

Subscribe now

Lookup and Deterministic Forwarding

Once the packet has passed through Ethernet, IP, and TCP validation, the XDP program performs the most crucial step: deterministic forwarding using the precomputed Maglev-style hash ring.

In the kernel, this process is astonishingly simple: and yet it underpins the system’s ability to operate at line rate with planetary-scale fan-in.

__u32 slot = hash % RING_SIZE;
__u32 *backend_idx = bpf_map_lookup_elem(&ring_map, &slot);
struct backend *b = bpf_map_lookup_elem(&backend_map, backend_idx);

Here, a single modulo operation maps the packet’s 5-tuple hash to a slot.

A single map lookup retrieves the backend index, and a second map lookup fetches the backend’s IP and port. That is all.

No loops, no runtime weight calculations, no dynamic per-flow state. O(1) complexity, deterministic results, purely functional evaluation.

Because all kernel accesses are atomic and read-only during packet processing, this architecture scales linearly with CPU cores and NIC queues.

There are no locks, no contention, and no thread coordination. Each core can process its RX queue independently, and the control plane can update ring_map or backend_map asynchronously.

New flows automatically observe updated backends, while old flows continue to use the last-known-good mapping: all without disrupting the kernel datapath.

Subscribe now

Failure as a First-Class Design Input

Statelessness is not just an optimization; it is a failure-handling strategy baked into the architecture. Because the kernel holds no flow tables or connection metadata, failure is trivial to contain.

When a backend server fails, the control plane simply removes its virtual nodes from the hash ring. From the kernel’s perspective, packets that would have landed on that backend now hash to other slots.

There are no stale connection entries to purge, no retries to manage, no cache warm-ups required.

When a Katran node itself fails, upstream Anycast automatically redirects traffic. Each surviving node continues forwarding packets using the last-known-good maps, oblivious to the failure.

There is no need for global coordination, distributed state reconciliation, or multi-step failover. Failure domains remain local, predictable, and instantly recoverable.

This is architectural serenity: the system is designed to survive failures because the kernel never held state that could be lost. Statelessness transforms complexity into predictability.

Visualizing Katran’s Packet Flow

Understanding 5-tuple hashing, virtual node assignment, and BPF map lookups is abstractly simple, but seeing the full flow clarifies the elegance:

[Client Packet]
    |
    v
[NIC RX Queue / RSS]
    |
    v
[XDP Parsing & Validation]
    |
    v
[5-Tuple Hash Computation]
    |
    v
[Ring Map Lookup -> Slot]
    |
    v
[Backend Map Lookup -> Backend IP/Port]
    |
    v
[Packet Rewrite & XDP_TX]
    |
    v
[Backend Server / Direct Server Return]

Control Plane (User Space)
    |
    |-- Computes virtual nodes per backend (weight-based)
    |-- Calculates offsets & skips for Maglev ring
    |-- Updates ring_map in kernel atomically
    |-- Updates backend_map with IPs & ports

Figure 1 illustrates this flow: the kernel operates entirely statelessly and deterministically, while the control plane orchestrates updates asynchronously.

The XDP program is immutable during packet processing, and the system naturally adapts to changes without disruption.

Subscribe now

Direct Server Return (DSR)

Katran very often employs the so-called Direct Server Return (DSR).

Instead of proxying replies through the load balancer, it rewrites only the destination IP and port, leaving the source IP intact. Backends reply directly to clients, bypassing the balancer entirely.

This approach eliminates double traversal of the datapath, removes additional memory and CPU overhead, and allows the system to handle planetary-scale fan-in.

When combined with Receive-Side Scaling (RSS) across NIC queues, each CPU core can process independent flows in parallel, achieving near-linear throughput across thousands of simultaneous connections: all without storing a single flow or connection state in the kernel.

Physics vs. Semantics

It is tempting to ask: “Could Katran replace proxies?” The shortest answer I could give you right now is: hell no.

Proxies operate in the semantic domain, handling TLS termination, header inspection, authentication, authorization, retries, circuit breaking, and protocol translation. They reason about requests, responses, cookies, tokens, and application-layer meaning.

Katran operates entirely in the physics domain. Its responsibility is deterministic packet routing, load distribution, and weight-based selection.

By isolating the concerns, Meta ensures that each layer can scale according to its own constraints: the kernel moves packets at line rate, and proxies reason about requests at application speed.

Collapse these layers, and the system suffers both CPU saturation and state management nightmares.

Katran redraws the boundary between raw throughput and semantic complexity, allowing both layers to excel.

Computation at the Data

Katran is also part of a larger architectural movement: perform computation where the data resides. Modern examples abound:

• eBPF firewalls replacing iptables, operating in-kernel without context switches.

• SmartNICs offloading encryption and filtering.

• Programmable switches performing aggregation in-network.

• io_uring bypassing syscalls for low-latency I/O.

• RDMA eliminating kernel involvement for high-throughput transfers.

The principle is simple but profound: reduce copies, avoid context switches, and specialize each layer for its constraints.

Katran embodies this philosophy at the kernel’s ingress point, turning every packet into a deterministic computation before it ever reaches user-space.

Quiet Minimalism at Hyperscale

Ultimately, Katran’s brilliance is disciplined minimalism. It does not parse HTTP. It does not retry requests. It does not track connections or manage caches.

It simply forwards packets deterministically, guided by the Maglev-style hash ring materialized in kernel maps.

Combined with DSR, RSS, and atomic map updates, Katran achieves planetary-scale throughput. Proxies are freed from the burden of raw packet fan-in and can focus entirely on semantics.

Katran does not render proxies obsolete; it makes them viable at hyperscale. In doing so, it delivers one of the most profound lessons in distributed systems: at extreme scale, less complexity per operation is not a feature: it is survival.

The kernel, stripped to pure functionality, becomes a deterministic engine for traffic physics, leaving higher layers to reason about meaning.

Share

The Hidden Constraint Stateless Systems Cannot Escape

By the end of Part I, we discovered how Katran had collapsed load balancing into something almost “offensively simple”: a pure function executed at the earliest point in the kernel receive path.

A packet arrived via DMA into host memory. XDP intercepted it before socket allocation, before skb creation, before TCP state machines or retransmission queues even existed.

Katran read a fixed set of bytes, computed a hash over the 5-tuple, performed a constant-time lookup inside an eBPF map, rewrote the destination fields in place, fixed the checksums, and transmitted the packet.

No heap allocation.

No locks.

No connection tracking.

No memory proportional to traffic volume.

Routing had been reduced to deterministic evaluation over immutable input.

But this architectural purity comes with a constraint that stateful systems quietly absorb without thinking:

if the system refuses to remember past decisions, it must be able to reproduce them exactly, indefinitely, using only computation and configuration as inputs.

Katran doesn’t operate in a calm, static environment. Backends fail silently, networks partition, nodes drain for rolling upgrades, and autoscaling stretches and contracts continuously.

At Meta scale (tens of thousands of machines) these events happen constantly, while packets flow without pause.

Naive approaches, like modulo hashing, cannot survive this churn: millions of flows would be reassigned simultaneously, TLS sessions break, caches go cold, and tail latency explodes.

Katran survives by remembering nothing. Each packet recomputes its destination independently, deterministically, relying only on the current configuration and immutable header fields.

Continuity cannot come from memory; it must come from mathematics. Formally, the routing decision is a function:

where flow_identity is the 5-tuple and backend_set reflects the current fleet encoded in eBPF maps. Every backend change alters the function. A naive mapping over a changing set would instantly reshuffle all flows, because Katran does not store previous decisions.

There is no per-flow state, no hash table of prior assignments, no flow table to synchronize. Stateful systems externalize continuity into memory; Katran externalizes it into math.

This is where consistent hashing becomes essential. It distributes flows across the ring such that small changes in the backend set affect only a fraction of traffic, preserving mapping continuity without storing anything.

Every packet independently recomputes its backend, yet the global distribution remains stable, predictable, and local.

In other words: continuity is no longer a property of memory; it is a property of the function itself.

Statelessness, determinism, and resilience are inseparable. This is why consistent hashing is not an optimization: it is the backbone of stateless load balancing at hyperscale.

Subscribe now

Why modulo hashing fails under dynamic membership

At first glance, stateless routing appears trivial to implement. The packet already contains a globally unique identifier in the form of its 5-tuple.

Applying a high-quality hash function such as Jenkins hash, MurmurHash, or Toeplitz hash produces a uniformly distributed 32-bit or 64-bit value.

Mapping that value into a backend index can be done using a simple modulo operation:

This approach satisfies nearly all of Katran’s architectural constraints. It is deterministic. It requires no per-flow state. It executes in constant time.

It is also trivial to implement in eBPF. It produces uniform load distribution assuming a well-behaved hash function.

And yet it fails catastrophically the moment the backend set changes.

The problem lies in the divisor.

Modulo arithmetic does not preserve ordering relationships when the modulus changes. When N changes to N+1, the mapping of hash values to indices changes globally.

A hash value that previously mapped to backend 2 may now map to backend 7. Another may map to backend 0. There is no locality preservation property.

To see this concretely, consider a simplified example with N=4 backends. Hash values map into backend indices like this:

Now add a fifth backend, so N=5:

For any given hash value H, the probability that:

is exactly:

This means 80% of all flows are reassigned immediately when a single backend is added. At hyperscale, this is not a theoretical inconvenience. It is an operational disaster.

Every reassigned flow results in packets arriving at backend machines that have never seen that flow before. TCP stacks reject packets with unknown sequence numbers.

TLS sessions fail because cryptographic state does not exist on the new backend. HTTP/2 multiplexed streams break because connection context is lost. Clients initiate retransmissions, exponential backoff, and eventually full connection re-establishment.

This cascades upward.

Connection pools maintained by clients become invalid.
Application-layer caches lose locality.
Backend CPU cache warmth disappears.
NUMA locality is destroyed.
Kernel routing caches become irrelevant.

Traffic patterns that were previously thermally stable suddenly destabilize.

This creates transient overload conditions that amplify latency precisely when the system is undergoing change, which is the worst possible time for instability.

Even more dangerously, this reshuffling occurs regardless of whether the backend change is an addition, removal, or replacement. Removing a single backend causes the same global redistribution effect, forcing nearly all flows to move.

The system becomes topologically fragile. Minor topology changes trigger global traffic churn.

This is fundamentally incompatible with Katran’s stateless model.

Because Katran has no memory, it cannot preserve continuity explicitly. Therefore continuity must be preserved implicitly by the mapping function.

Modulo hashing does not provide this guarantee.

Katran required a routing function whose output changes minimally when the backend set changes. A function whose continuity properties are intrinsic, not emergent.

A function designed not merely for distribution, but for stability under mutation.

This is exactly what consistent hashing provides.

Subscribe now

The missing half of the equation

Up to this point, Katran’s model appears almost suspiciously incomplete. A packet arrives, a deterministic hash selects a backend, the destination IP is rewritten, and the packet is forwarded.

• No connection table is created.

• No session state is remembered.

• No record is kept anywhere.

The load balancer forgets the packet the instant it leaves the NIC. Which raises an immediate and deeply practical question:

how does the response find its way back?

Traditional load balancers solve this by remembering everything. They allocate connection entries, store source and destination tuples, and use that memory later to reverse the transformation.

Return traffic is not discovered: it is looked up. Every response packet must re-enter the load balancer, consult state, and be translated back into the client-visible address space.

Katran does something far more radical. It eliminates the need to remember in the first place.

This works because Katran operates in two distinct forwarding modes, each designed to preserve the illusion of a single virtual service IP while minimizing the amount of work Katran itself must perform.

Both modes rely on the same invariant: the load balancer’s job is not to own the connection, but merely to place the packet correctly at the beginning. Once placed, the network itself can do the rest.

This is the difference between supervising every conversation and simply introducing two parties who can speak directly. Katran prefers introductions.

The art of disappearing

Direct Server Return is the purer expression of Katran’s philosophy. In DSR mode, Katran modifies only the minimum amount of information required to steer the packet to the correct backend.

Specifically, it rewrites the destination MAC address so the frame reaches the selected server at layer 2, while leaving the IP layer logically consistent with the virtual service abstraction.

From the client’s perspective, the packet was sent to the VIP. From the backend’s perspective, the packet appears to be addressed to that same VIP, because the backend is explicitly configured to accept traffic destined for that address via a loopback interface.

This configuration is deliberate sleight of hand: multiple machines simultaneously claim ownership of the same IP, but only Katran determines which one actually receives each packet.

Once the backend processes the request, it responds directly to the client using normal IP routing. The response does not pass back through Katran. It does not need translation, correction, or approval.

The backend already knows the client’s source IP, and the VIP is already a valid source address on the backend.

Katran never sees the response. And that absence is the entire point.

By removing itself from the return path, Katran immediately cuts its packet processing load in half. Every byte of outbound response traffic bypasses the load balancer entirely, freeing CPU cycles, memory bandwidth, PCIe bandwidth, and NIC queue capacity.

Latency improves because an entire network hop disappears. Throughput increases because Katran’s processing budget is now devoted exclusively to ingress traffic.

The load balancer becomes a one-way valve.

This has profound scaling implications. In traditional NAT-based load balancers, throughput is constrained by bidirectional processing capacity. Every request and every response must traverse the same CPU, the same queues, the same kernel structures.

In DSR mode, Katran’s throughput ceiling effectively doubles, not because the hardware changed, but because half the work vanished.

The most efficient packet is the one you never have to touch.

Of course, this requires backend cooperation. Each backend must be configured with the VIP on a loopback interface, must disable certain reverse path filtering protections, and must ensure routing policies allow responses to exit with the VIP as the source address.

These adjustments sound invasive, but at Meta’s scale, where fleets are centrally managed and automatically provisioned, such configuration is routine infrastructure hygiene.

In exchange, Katran achieves something remarkable: it participates only in the part of the connection where its intelligence is actually required.

The rest of the time, it actually “disappears”.

Subscribe now

The compatibility layer

Not every environment can support Direct Server Return. Legacy systems, asymmetric routing environments, and networks with strict topology constraints often require the load balancer to remain in the return path.

For these cases, Katran supports traditional NAT mode, where both inbound and outbound packets are rewritten to preserve the virtual service abstraction.

In NAT mode, Katran performs two symmetric transformations. On ingress, it rewrites the destination IP from the VIP to the backend’s real address. On egress, it performs the inverse transformation, rewriting the source IP from the backend’s address back to the VIP before forwarding the packet to the client.

From the outside, the illusion remains intact. The client believes it is communicating with a single logical endpoint. The backend receives routable packets addressed to itself. Katran acts as the translator between these two realities.

What makes Katran’s NAT implementation unusual is what it does not do.

It does not allocate connection objects. It does not maintain per-flow state. It does not synchronize session tables across nodes. Instead, it relies entirely on deterministic recomputation.

Because the consistent hashing function produces the same backend selection for every packet in a flow, Katran can reapply the same translation logic independently to each packet without remembering anything about previous packets.

Each packet carries sufficient information to rediscover its correct path. This preserves the defining properties of the system:

Routing decisions remain stateless, lock-free, and purely functional. There are no shared data structures that grow with connection volume.

Memory usage does not scale with traffic, and even CPU cost remains constant per packet. Failure recovery requires no state reconstruction because no state exists to reconstruct.

Even when performing NAT, Katran refuses to become stateful. Statelessness is not a feature. It is a constraint the system refuses to violate.

Where intelligence is allowed to exist

At first glance, Katran’s data plane appears almost aggressively unintelligent. It does not monitor backend health. It does not detect overload. It does not adapt dynamically. It does not even know what services exist in any meaningful sense.

It simply reads precomputed values from eBPF maps and executes deterministic transformations on packets.

1. This is not a limitation.

2. It is a deliberate architectural boundary.

3. All intelligence lives elsewhere.

The control plane operates entirely in user space, where complexity is cheap and mistakes are survivable. It continuously observes the backend fleet, performing health checks, capacity measurements, deployment coordination, and traffic engineering decisions.

It computes consistent hashing rings, assigns weights to reflect backend capacity, removes unhealthy nodes, and introduces new ones during scaling events.

Once computed, these decisions are materialized into eBPF maps inside the kernel. This is the only communication channel between intelligence and execution.

The kernel data plane never queries user space. It never performs system calls. It never waits for locks held by other threads. It never allocates dynamic memory. It never blocks on I/O.

It simply reads from maps that already exist in memory and executes instructions that have already been verified.

The relationship between control plane and data plane is asynchronous and one-directional. The control plane writes new configurations when necessary. The data plane continues executing the previous configuration until the new one appears.

There is no synchronization barrier. There is no pause in packet processing. There is no transitional state where routing becomes uncertain.

This separation produces one of Katran’s most important operational properties:

If the control plane crashes, traffic continues uninterrupted.

The kernel retains the last valid configuration indefinitely. Packets continue to be routed correctly. Services remain reachable. The system does not degrade gradually: it simply stops evolving until the control plane returns.

The inverse failure is equally unremarkable.

If a Katran node fails entirely, upstream Anycast routing automatically shifts traffic to other Katran nodes advertising the same VIP. Because no per-connection state exists, there is nothing to migrate.

New packets arrive at different load balancers, which recompute the same consistent hash decisions and forward traffic to the same backends.

There is no recovery phase. Recovery is instantaneous because nothing was lost.

This is resilience not through redundancy, but through the absence of fragile state.

Subscribe now

Watching decisions at line rate

Running routing logic inside the kernel would traditionally be an observability nightmare. Kernel code is opaque, difficult to instrument, and extremely dangerous to modify.

Debugging typically relies on indirect signals: application logs, sampled traces, or inferred metrics reconstructed from incomplete information.

eBPF changes this completely.

Because Katran is implemented as an eBPF program, it can export telemetry directly from the exact instruction path that processes each packet.

Every routing decision can increment counters, update histograms, or emit structured events into shared maps or perf buffers, all without leaving kernel space or introducing meaningful overhead.

This provides observability at the point of truth.

Operators can measure per-service packet rates exactly as they are forwarded. They can observe backend distribution and detect imbalance immediately. They can identify drops caused by invalid packets, resource exhaustion, or configuration transitions.

They can monitor NIC queue utilization and detect early signs of saturation before packet loss begins.

Nothing is inferred.

Nothing is sampled.

Nothing is reconstructed.

The measurements are produced by the same instructions that forward the packet.

Even more importantly, these programs can be updated dynamically. New instrumentation can be deployed without kernel recompilation, without system reboot, and without interrupting traffic.

The kernel effectively becomes a programmable execution environment, capable of evolving while the system remains live.

This represents a fundamental shift in how infrastructure behaves.

The kernel is no longer a static artifact frozen at boot time. It becomes a runtime, executing safe, verified programs that can be replaced as requirements evolve.

Katran is not merely a load balancer.

It is an example of what happens when packet processing stops being a fixed function of the operating system and becomes software again.

Statelessness as resilience

Katran’s architecture repeatedly demonstrates a central theme: statelessness is not merely a performance optimization; it is the source of resilience.

By eliminating memory, per-flow state, and mutable tables from the data plane, Katran transforms a potentially fragile, memory-bound system into a deterministic, local, lock-free computation.

Every section we’ve explored, return traffic handling, control plane separation, observability, and minimal kernel programs, converges on the same insight: the network does not need to remember.

The network needs only to compute, consistently, independently, and correctly.

In hyperscale systems, that subtle architectural shift, from storing what happened to recomputing what must happen, does more than improve speed.

It restores simplicity, predictability, and operational sanity to a layer of infrastructure that was traditionally overburdened with complexity it never required.

Katran does not just move packets faster. It redefines what moving packets efficiently even means. It vanishes in the network, leaving only math, determinism, and physics-aware computation behind.

And in doing so, it proves that sometimes the most radical engineering move is to refuse to remember anything at all.

Share The Software Frontier

The Night Load Balancers stopped feeling like plumbing

The first time I saw a production system degrade because of its load balancer, nothing spectacular happened.

No outages.
No alarms.
No crashes.

Latency just started creeping upward, first p95, then p99, until everything felt sticky. Requests weren’t failing. They were waiting. Queues lengthened. Retries multiplied. Backends looked saturated despite idle CPUs.

What should have been a minor networking blip slowly metastasized into systemic instability.

The system wasn’t compute-bound.
It wasn’t storage-bound.
It wasn’t even network-bandwidth-bound.

It was packet-movement-bound.

At the time, that realization felt wrong. Load balancers were supposed to be solved infrastructure, plumbing you deploy once and forget. You scale them horizontally, tune some buffers, maybe add another tier, and move on.

That night changed how I thought about them. I realized that load balancing isn’t plumbing at all.

It’s distributed systems engineering hiding inside the network layer.

Years later, when I first read about Facebook’s Katran, that same dissonance came rushing back, not because Katran was fast (many systems are fast), but because it treated the problem so differently from everything else I’d seen.

It didn’t try to be a better proxy. It didn’t try to optimize user-space networking. It didn’t try to understand protocols, sessions, or requests.

It tried to disappear.

Katran’s ambition was not to sit in front of traffic, but to move packets before the operating system itself even noticed they existed. Not as an optimization, but as a redefinition of what load balancing actually is.

This piece is an attempt to unpack that redefinition, not as a feature tour, but as a system story:

1. how Meta ended up building a kernel-level, stateless, line-rate load balancer;

2. why proxies stopped scaling; how eBPF and XDP turned Linux into a programmable switch;

3. and what this architecture tells us about where infrastructure is heading.

   Subscribe now

When proxies became the bottleneck

At small scale, load balancers feel like coordination infrastructure.

They terminate TLS, parse HTTP, enforce policies, retry requests, expose metrics, and act as programmable gateways between clients and services.

At medium scale, they become throughput machines. You start tuning epoll loops, socket buffers, thread pools, kernel parameters, memory allocators. But everything still feels tractable.

At hyperscale, Meta scale, something breaks conceptually.

Traffic no longer looks like “clients making requests.”
It looks like “millions of packets per second, continuously, forever.”

In that regime, the bottleneck stops being computation and becomes movement. The system isn’t struggling to compute responses. It’s struggling to move bytes through layers of software fast enough.

A traditional proxy pipeline looks roughly like this:

NIC → kernel networking stack → socket buffers
    → user-space context switch → protocol parsing
    → routing decision → connection pool lookup
    → buffering → write syscall → kernel → NIC

Every arrow carries some real costs:

• Cache misses.

• Branch mispredictions.

• Memory allocation and deallocation.

• Kernel ↔ user context switches.

• Lock contention.

• Scheduling delays.

• Buffer copies.

• Queue management overhead.

At tens of thousands of requests per second, this is noise. At millions of packets per second, this is overhead, but at tens of millions, this becomes physics.

At Meta scale, the load balancer fleet itself grew into one of the company’s largest compute clusters: not because it was doing intellectually difficult work, but because it was burning enormous CPU cycles just forwarding bytes.

Worse, that cost surfaced primarily as tail latency, the kind that cascades invisibly through distributed systems. Slight delays trigger retries. Retries amplify load. Load amplifies queueing. Queueing amplifies latency variance.

What starts as packet-processing overhead becomes application-level instability.

This is where the framing changed.

Meta engineers stopped asking:

How do we build a faster proxy?

And instead started asking:

Why are we proxying at all?

Subscribe now

The conceptual break

Katran begins with a deceptively simple idea:

Load balancing is not about requests.
It’s about packets.

This sounds trivial until you follow it all the way down.

Most systems reason about traffic at the so-called Layer 7. We think in terms of HTTP verbs, paths, cookies, headers, auth tokens, retries, sessions, and request lifecycles. Load balancers become policy engines and application routers.

But stripped to fundamentals, routing is just:

A packet arrives.
It must be forwarded somewhere else.

That decision does not require:

• Parsing payloads.

• Understanding protocols.

• Tracking sessions.

• Maintaining connection state.

• Allocating memory.

• Buffering streams.

It requires in fact:

• Reading a few header fields.

• Mapping them deterministically to a backend.

• Rewriting destination addresses.

Once that’s accepted, the rest of Katran’s architecture follows almost inevitably.

Once you accept that routing doesn’t require user space, a strange realization follows. Why enter user space at all?

If routing doesn’t require sockets, why traverse the TCP stack?
If it doesn’t require state, why store state?
If it doesn’t require buffering, why buffer?

Most traditional networking pipelines do all of these things: not because routing needs them, but because the systems we built around routing do.

Over time, layers accreted: sockets, connection tracking, session tables, queues, retries, buffers. Routing became entangled with everything else, until forwarding a packet meant walking through half the operating system.

Katran asks a very much simpler question.

If routing is just computation, like a deterministic function of packet headers, then why not perform that computation at the earliest possible moment, the instant the packet enters the machine, and then get out of the way?

That’s exactly what Katran does.

It runs in the kernel datapath, before the TCP stack, before sockets, before user space, before state is allocated, before buffers pile up.

A packet arrives. Its tuple is hashed. A backend is chosen. The destination is rewritten. The packet moves on.

No sessions. No tables. No queues. Not even memory.

Just math, applied at line rate, and that’s because Katran doesn’t optimize routing.

It eliminates everything routing never needed in the first place.

The physics of Packet Movement

Before diving deeper, it’s worth stepping back and looking at the constraint Katran is truly built to respect: physics.

At hyperscale, performance isn’t just a matter of elegant algorithms or clever data structures. It’s about the raw realities of moving bits through silicon.

Every design choice carries a cost in CPU cycles, memory bandwidth, and latency. At this scale, even small inefficiencies multiply into catastrophic slowdowns.

Every packet copied between kernel and user space consumes memory bandwidth. Every crossing of privilege boundaries introduces pipeline stalls.

Every memory allocation risks cache thrash or fragmentation. Every context switch adds jitter. Every queue introduces contention, every interrupt triggers scheduling overhead.

NUMA topology, DMA latency, cache line locality, branch predictability; all of it suddenly matters, not in theory, but in the measured tail latencies of millions of concurrent flows.

Traditional proxies accumulate these costs by design. They are built to understand packets: to parse headers, terminate TLS, enforce policies, track sessions.

Each feature is useful at a semantic level, but each comes at a physical price. And at planetary scale, that price dominates everything else.

Katran takes the opposite approach. It doesn’t want to know what the packet means. It doesn’t parse HTTP, it doesn’t validate TLS, it doesn’t track sessions. All it cares about is where the packet should go. That singular focus allows it to escape the usual costs.

By moving routing decisions to the earliest possible point, the kernel’s first receive path, before the networking stack even wakes up, Katran eliminates most of the overhead proxies can’t avoid.

The packet hits the NIC, the tuple is hashed, a backend is chosen, the destination is rewritten, and the packet continues. No user-space copies. No socket buffers. No queues. No connection tables. Almost no memory traffic beyond the computation itself.

In other words, Katran doesn’t just optimize for speed: it co-locates computation with the physics of the system, letting the network move at line rate while avoiding the hidden costs that would crush any traditional proxy at hyperscale.

Subscribe now

XDP and eBPF

None of this would have been remotely feasible a decade ago.

Kernel-level packet processing used to be the domain of fear and ritual. You wrote kernel modules in C, recompiled kernels, rebooted machines, and prayed nothing would panic.

Debugging meant sprinkling printks throughout the code and hoping the kernel didn’t crash before you could see the output.

It was brittle, slow, and incompatible with the dynamic, constantly changing demands of modern production infrastructure.

Then came eBPF, the extended Berkeley Packet Filter, and XDP, the eXpress Data Path. Together, they changed the rules of the game.

eBPF introduced something unprecedented: safe, sandboxed programs that could run directly inside the kernel.

You could load them dynamically, update them on the fly, and remove them at runtime: all without touching the kernel’s core or risking catastrophic panics.

The verifier guaranteed bounded execution, prevented unsafe memory access, and ensured termination. Suddenly, the kernel became programmable, but without the traditional risks that had made kernel development a high-stakes gamble.

XDP tied this capability to the earliest possible packet hook, right after the NIC’s DMA copied the packet into memory, long before the kernel networking stack woke up, before socket buffers were allocated, before protocol parsing began.

At that instant, the packet is nothing more than bytes in memory. And at that instant, you get to run code on it.

This is the environment where Katran lives. It’s not a load balancer in the traditional sense. It’s a programmable L3/L4 forwarding pipeline, implemented in software but operating at the same layer as a hardware switch.

It doesn’t parse requests, terminate TLS, or track sessions. It doesn’t worry about retries, circuits, or headers.

It computes packet destinations at line rate and moves on. Everything else (proxies, policies, application logic) happens after the packet has already been routed correctly and efficiently.

From this perspective, Katran isn’t an optimization. It’s a reframing.

By combining eBPF and XDP, Katran elevates routing to the earliest stage possible, removes the overhead traditional systems carry, and brings software routing into the same performance envelope as hardware switches; all without sacrificing the flexibility, safety, or dynamism that modern production environments demand.

Subscribe now

The Katran Data Plane

Let’s follow a single SYN packet entering a Katran node, step by step, to see exactly what makes this system so different.

Picture it in your mind: every nanosecond counts, every memory access is a potential bottleneck, and every decision must be deterministic.

The packet arrives at the NIC. The network card’s DMA engine deposits it into memory. The kernel is still asleep for the most part, unaware of what just arrived. At this moment, the XDP hook fires, and Katran’s eBPF program takes control.

From here, the packet’s journey is unlike anything a traditional stack would do. Katran doesn’t allocate memory. It doesn’t touch sockets. It doesn’t consult a connection table.

It doesn’t parse HTTP headers or check TLS states. It doesn’t wait for user-space processes to schedule it. It doesn’t buffer or queue.

All it does is look at a few bytes of the header: the classic 5-tuple that uniquely identifies the flow.

struct iphdr *ip = data + ETH_HLEN;
struct tcphdr *tcp = data + ETH_HLEN + sizeof(*ip);

__u32 src_ip = ip->saddr;
__u32 dst_ip = ip->daddr;
__u16 src_port = tcp->source;
__u16 dst_port = tcp->dest;
__u8 proto = ip->protocol;

Those five fields, like source IP, destination IP, source port, destination port, and protocol, become the entire input to Katran’s routing decision. Nothing else matters.

Next comes the whole heart of the computation: hashing and consistent mapping to a backend.

Katran uses the 5-tuple to produce a deterministic key, which indexes directly into a consistent hashing ring stored in an eBPF map.

__u64 key = hash_5tuple(src_ip, dst_ip, src_port, dst_port, proto);
__u32 backend_idx = bpf_map_lookup_elem(&backend_ring, &key);

That index points to another map that contains backend IP addresses and ports:

struct backend backend = backends[backend_idx];

Katran then rewrites the packet headers in place:

ip->daddr = backend.ip;
tcp->dest = backend.port;

Checksums are recomputed using kernel helpers, which are optimized for in-place, zero-copy updates.

And then, this is the magical moment, the packet is sent immediately:

return XDP_TX;

No socket buffers.
No TCP stack.
No connection tables.
No memory allocations.
No context switches.
No scheduling delays.

Just:

Packet arrives → compute backend → rewrite headers → transmit

From the backend’s perspective, the packet looks like it came directly from the client. From the client’s perspective, it came from the VIP. From the kernel’s perspective… the packet essentially never existed outside this tiny, deterministic computation.

The routing decision executes in a fixed, predictable number of instructions. It takes microseconds. There is no jitter from GC pauses, no locks to contend over, no queues to saturate.

The system scales linearly with CPU cores, not with the number of flows or the size of the connection table.

But the performance, impressive as it is, is only part of the story. The deeper shift is architectural: the network stops tracking state. There is no per-flow memory, no NAT tables, no sticky sessions. Routing becomes a pure function:

flow tuple → hash → backend → forward

Everything else (failure recovery, backend updates and scaling) can be handled by updating the small, shared hash ring.

There is no explosion of per-flow state, no coordination across nodes, and no warm-up after failures.

In other words, Katran achieves hyperscale not by faster processors, or clever batching, or hardware acceleration, although it uses those wisely, but by eliminating everything routing doesn’t actually need.

It’s a profound simplification. And once you internalize it, you begin to see why Katran doesn’t just move packets faster: it reshapes the way we think about packet routing entirely.

Subscribe now

Routing as pure computation

Traditional load balancers remember things.

They remember which backend a connection was assigned to, which NAT mapping applies to a flow, which sessions are active, which backends are draining, which retries are still in flight.

Every packet doesn’t just pass through the system: it leaves a trace. A new entry in a table. A new piece of memory that must be managed, synchronized, persisted, migrated, invalidated, and eventually garbage-collected.

At small scale, this feels manageable. At large scale, it becomes suffocating.

State grows with traffic. Traffic grows with success. And in distributed systems, state is not just complexity: it is fragility.

It must survive failures. It must remain consistent across replicas. It must be reconstructed after crashes. Over time, the system stops being a load balancer and becomes a distributed state machine that happens to forward packets.

Katran refuses this entire model. Instead of remembering:

“Flow A maps to backend X.”

Katran recomputes:

“Given tuple A, backend X is the correct destination.”

Every time. From scratch. Deterministically.

Same input → same output.

No memory. No eviction. No synchronization. No garbage collection.
No coordination.

Routing stops being a mutable process and becomes a pure function.

This is not a performance trick; it is a philosophical shift. Once routing is expressed as computation rather than storage, failure semantics collapse into simplicity.

A node can restart and instantly resume forwarding traffic, because nothing was ever stored. Scaling no longer requires migrating flow tables.

Draining no longer requires tracking sessions. There is no warm-up, no replay, no reconstruction; only recomputation.

This is where consistent hashing becomes central.

Instead of tracking where flows went, Katran hashes the flow tuple and maps it deterministically onto a backend via a hash ring. Every packet independently computes its destination. Every node running Katran executes the same function.

As long as they share the same configuration, they produce the same routing decisions: without coordination, without memory, without state.

Routing becomes mathematics.

Not “remember what happened”, but “recompute what must happen”.

And that single shift, from stored decisions to deterministic computation, is the architectural heart of Katran.

Epilogue

Katran isn’t just a faster load balancer. It’s a redefinition of what load balancing means at scale.

Traditional systems treat routing as storage: track flows, maintain tables, buffer packets, reconcile state. At small scale, it works.

At Meta scale, it collapses under its own complexity. Latency creeps up. Queues grow. Retries multiply. The plumbing leaks, invisibly, into application performance.

Katran treats routing as pure computation. A packet arrives, a hash is computed, a backend is chosen, the headers are rewritten, and the packet moves on. No state. No coordination. No memory overhead. No jitter.

The network itself becomes stateless, deterministic, and trivially scalable. Failure recovery, draining, scaling; all of it reduces to updating a simple hash ring. The system doesn’t need to remember. It only needs to compute.

By moving routing into the kernel’s earliest receive path and stripping away everything that isn’t essential, Katran co-locates computation with the physics of packet movement.

It eliminates the overhead proxies carry by design, brings software routing close to hardware performance, and reshapes the architectural landscape for planetary-scale infrastructure.

In the end, Katran is less about “doing load balancing faster” and more about rethinking what load balancing is.

It’s a system that vanishes in the network, leaving only mathematics and determinism behind; a quiet revolution in how packets flow through the world’s largest data centers.

At hyperscale, that subtle shift doesn’t just improve performance. It restores simplicity, reliability, and sanity to a layer of the network that, for too long, had been treated as plumbing when it was always a distributed system.

Katran reminds us that sometimes, the best way to handle complexity isn’t to manage it: it’s to eliminate it entirely.

Share

When halting becomes optional

I remember the afternoon clearly. It was like every other debugging afternoon, except this one felt heavier.

I was chasing a race condition in a distributed system: one of those bugs that appears only when enough users click enough buttons fast enough, vanishes if you trace it, and refuses to reproduce in any deterministic way.

A value would appear in my logs before it was supposed to exist. A future bleeding into the present.

And for a moment, I realized (not dramatically, just quietly, like noticing a stain on your favorite shirt) that the problem wasn’t logic, at all. It was time.

Every failure I’d seen in computation, like for example deadlocks, Heisenbugs, eventual consistency anomalies, was a failure of ordering, not of truth.

And that thought carried me somewhere I had never expected: into the idea that maybe time itself is a kind of computation.

Or maybe computation is a shadow cast by time. Or maybe neither is fundamental, and what we think of as execution is just a trick our brains play on us.

Subscribe now

Waiting for the Future

The Halting Problem, which we’ve alredy covered, is usually presented like a brick wall:

“No algorithm can decide, for every possible program and input, whether it halts.”

It feels eternal. Immutable. Absolute.

But take a closer look. Its “magic” relies on something subtle: time. A program halts if, eventually, a halting state sh​ is reached. If it doesn’t, you can never know in finite time.

In more formal terms, let a program be a sequence of states.

Halting is essentially the predicate:

Non-halting is simply:

Notice the asymmetry: the “yes” case is witnessed finitely, the “no” case is only witnessed infinite time later.

That waiting, the “eventually,” is baked into the problem. Remove time, and halting loses meaning.

Subscribe now

Physics as constraint, not execution

In physics, especially modern physics, “waiting” isn’t guaranteed. Einstein’s equations don’t evolve; they rather constrain:

No “next step”. No “after”. There’s only a four-dimensional block satisfying relational constraints.

Quantum mechanics complicates it further. The Schrödinger equation provides smooth evolution:

…but measurement collapses states discontinuously. And, what about quantum gravity? The hamiltonian version of Wheeler–DeWitt equation annihilates time entirely:

No t. No sequence. No “eventually.” Just a series of solutions that exist or don’t.

Physics does not run; it satisfies.

Execution is our interface, not the universe’s method.

Computation without steps

I first realized this when reading about closed timelike curves (CTCs).

Imagine a world where the past can interact with the future: a universe with loops in causality.

A computer on a CTC doesn’t run sequentially. It must satisfy a self-consistency condition.

Deutsch formalized it in a beautiful way:

Here, ρ is a density matrix representing the state of the system, and Φ is a completely positive map describing interaction with its own past.

Execution disappears. There is no s0​→s1​→s2​. Only a state that works.

Problems that are usually thought to be intractable, suddently collapse to solutions, not by computing faster, but because the problem itself has been reframed: run replaced by exist.

I love this because it feels like cheating, except for the fact that the whole universe isn’t cheating.

We just assumed a rule (sequential steps) that no longer applies.

Subscribe now

Halting as satisfiability

So…. What happens to the Halting Problem, then?

Instead of asking:

“Does the machine eventually reach sh​?”

We ask:

“Does there exist a configuration s* satisfying the transition constraints and the halting condition?”

Symbolically:

This is no longer halting. It’s pure satisfiability. SAT.

Undecidability, once thought eternal, collapses. Not because the universe is super-Turing. But because the universe doesn’t need a clock to decide.

I remember leaning back in my chair and whispering to no one:
“Oh… so this is what time was doing all along.”

The missing clock

Chaos theory feels inevitable. Sensitive dependence on initial conditions:

But if there is no t, there is no exponential divergence. Only complex relational structure.

Chaos becomes not a dynamical phenomenon but a structural one: “I cannot know the whole from partial information”. The unpredictability persists, but it is epistemic, not temporal.

Time was doing invisible work again. Remove it, and the phenomenon mutates.

The structural version

Wolfram tells us that some systems can only be understood by running them. But “running” is a temporal concept.

In a timeless universe, irreducibility morphs. It becomes clear that:

“The only way to know the global structure is to analyze the entire relational network.”

Formally, let a system be defined by a set of constraints C over states S. Irreducibility is:

You cannot shortcut. Not because of time, not at all. But because structure itself resists local reasoning.

Why Time Feels Real

So why does time feel so real? Why does computation work at all? Why do programs execute reliably in the world we inhabit, instead of collapsing into incoherence the moment we press “run”?

At first glance, it’s almost miraculous.

The universe, a jumble of quantum fields and fluctuating geometries, somehow behaves as though it were sequential, as though there were an arrow pointing from “before” to “after.”

But the secret, I realized, lies in three deeply interwoven ingredients:

1. Decoherence — the quantum-to-classical alchemy that stabilizes records. Without decoherence, “memory” cannot exist. A bit in RAM, a log entry, even a thought; all of these are ephemeral superpositions that would collapse if left unobserved. Decoherence carves out islands of classicality from the quantum sea, giving us something we can call a state at all.

2. Entropy gradients — the thermodynamic arrow. Low entropy in the past and higher entropy in the future gives directionality. Irreversibility emerges. A program halts not because it must, but because the universe conspires to make irreversible transitions happen. Without entropy gradients, the notion of “progress” would evaporate. Your loops could oscillate forever without any emergent asymmetry.

3. Causal stability — macroscopic spacetime, the approximate global hyperbolicity that gives us a reliable causal order. Lightcones don’t randomly rotate. Past and future are roughly separable. Partial orders are locally well-behaved. Without this, even classical computation would collapse into ambiguity: a signal could arrive before it’s sent, and your neat sequence of state transitions would have no semantic meaning.

Together, these three factors carve out pockets of reality where causal order, memory, and execution make sense.

Within those pockets, computation works reliably. It feels fundamental, but it’s a phase, not a law.

A convenient feature of the universe’s current “operating mode,” not a guarantee for all of existence.

Subscribe now

Execution is Fragile

And then, like a subtle shock, I realized something terrifyingly mundane: I had been staring at the same principle in my own work all along.

Every time I debugged a distributed system, the failures were rarely about wrong values. They were about ordering:

• Deadlocks — processes waiting on each other forever, trapped by causal dependencies.

• Races — nondeterministic outcomes arising from subtle shifts in ordering.

• Livelocks — systems spinning endlessly without making progress, alive but unproductive.

• Eventual consistency anomalies — the illusion of stability until the system “catches up”…. or doesn’t.

Every one of these problems is a temporal problem. Remove time, or scramble it, and the systems collapse in ways logic alone cannot predict.

Execution is fragile. It relies on scaffolding we rarely notice: stable memory, arrow-of-time, consistent partial order.

Take away any piece, and the machinery of computation, the thing we assume is rock-solid, ceases to make sense.

The universe doesn’t compute. It exists.

Execution, halting, irreducibility: these are projections. They are interfaces for observers embedded in a particular causal phase of reality.

Halting becomes satisfiability. Complexity suddently becomes constraint.

The deep laws of nature don’t “run” programs; they encode global consistency conditions that, to us, appear as sequential evolution.

Beyond Undecidability

So if halting collapses into satisfiability, chaos into structural opacity, irreducibility into relational depth, then what remains? What are the true limits of understanding?

They are no longer temporal. They are structural. They are about existence.

Formally, we can describe them as:

where R(X) is a set of relational or constraint-based rules defining a system.

Notice the shift: the question is no longer “Can this program be computed?” but “Can this global structure exist without contradiction?”

A program that fails to halt in classical computation is harmless: it just spins forever.

But in physics, inconsistency is catastrophic. A relational network with no self-consistent configuration simply cannot exist.

The universe cannot “run” an impossible program. Non-existence is the ultimate computational limit.

Consider a simple example. Imagine a set of fields ϕi with constraints:

Does there exist a set of values {ϕ1​,ϕ2​,ϕ3​,ϕ4​} satisfying all constraints? That is the fundamental question.

Not whether a sequence of updates will reach a solution, but whether the solution exists at all.

And that, I realized, is the fundamental limit of reality: structural consistency. Not undecidability, not non-termination. Just existence itself.

Every time we think we are bumping against the “hard limits” of computation (P vs NP, the Halting Problem, irreducibility) we are really bumping against the limits of the phase of reality we inhabit.

Outside the decoherence, outside the entropy gradient, outside causal stability, those limits may dissolve.

Halting, chaos, and irreducibility are just mere artifacts of our temporally embedded perspective.

Subscribe now

The radical implications

Execution depends on time. Time depends on physical conditions. And undecidability depends on execution.

Step back, and it all becomes a chain of contingencies:

Remove the first link, and the rest vanish.

The universe doesn’t step forward. It satisfies. It exists. It is a structure, not a computation.

And what we perceive as halting, chaos, or irreducibility is simply the shadow of our perspective, projected onto a universe that doesn’t fundamentally need time to exist.

The deepest limits, it turns out, are not computational. They are ontological. Not about “what can be done,” but about “what can exist.”

And in that realization, every programmer, physicist, and philosopher finds a quiet, unnerving thrill: the universe doesn’t need to run to be infinite.

Seeing the universe as constraint

I started Part I with a debugging story. I end Part II the same way.

Systems fail not because they execute incorrectly, but because they try to execute in a regime where only existence matters.

The universe does not run.
It satisfies.

Time is not its clock. It is our interface.

Computation is not its language. It’s our projection.

Undecidability, chaos, irreducibility; these are features of temporally embedded observers. They are not fundamental.

And the question that lingers, the one I cannot shake, is this:

If the universe is a solution, not a process, then what determines which solutions exist at all?

Not what happens. But what is allowed.

I didn’t arrive at this question through theoretical physics. I arrived through debugging.

Late afternoon. Half-broken distributed system. A race condition that only appeared under load, vanished under tracing, and refused to reproduce deterministically. A value read before it had causally stabilized. A future leaking into the present.

Not a logic bug. A temporal one.

And I remember thinking….Not dramatically, just with the quiet unease of someone who has seen this movie too many times:

Every serious failure mode in computation is a failure of ordering, not of truth.

• Deadlocks.

• Non-termination.

• Eventual consistency anomalies.

• Heisenbugs.

They’re not about wrong values, they’re about wrong partial orders.

Which slowly pushed me toward a question that felt, at first, like category error:

Is time something computation happens inside?
Or is there something physics itself must compute?

Because every algorithm presupposes a causal structure.

And physics, increasingly, does not.

That’s the fault line I want to explore.

Not metaphorically.
Not operationally.
But where computation stops being a tool, and starts being a claim about reality.

Subscribe now

The unspoken Axiom of computation

Every formal model of computation begins with a hidden axiom:

There exists a well-founded causal ordering over computational steps.

A Turing machine defines a sequence

A circuit defines a directed acyclic graph with layers. Lambda calculus assumes β-reduction sequences.

Even concurrent and nondeterministic models rely on partial orders, things like Mazurkiewicz traces, event structures, causal cones.

But nowhere in computation theory do we define time.

We assume it and we treat temporal succession as primitive, not derived.

Which is fine, unless we believe computation is physical, because physics does not give you a global ordering for free.

When physics revoked the clock

Classical mechanics offers a global time parameter (t \in \mathbb{R}), and evolution equations of the form:

This fits computation beautifully. Discretize time. Simulate. Iterate.

But relativity removes global simultaneity. Spacetime is actually a Lorentzian manifold ((M,g)) with only local causal structure: lightcones, not layers.

There exists no preferred foliation into spacelike hypersurfaces. Different observers disagree on temporal order for spacelike-separated events.

Still, maybe computation survives by picking a frame.

Quantum mechanics destabilizes that further. Time evolution is unitary:

but measurement is discontinuous, stochastic, and not generated by the Schrödinger equation. There is no closed-form dynamical law for collapse; already a fracture between time and dynamics.

Then quantum gravity removes time entirely. In canonical quantum gravity, the Wheeler–DeWitt equation reads:

with no time derivative. The universe is described by a stationary wavefunctional over spatial geometries and matter fields.

No evolution parameter.
No (t).
No “next”.

Which leaves us staring at something structural:

Computation assumes time as a crucial metric. Fundamental physics does not.

So either computation is not fundamental, or time is not. Either way, something we thought primitive isn’t.

Subscribe now

Can you compute without temporal order?

At first glance, this seems incoherent. Computation is ordered state transition:

Remove well-founded succession, and execution collapses. But physics forces us to consider precisely that regime.

In quantum gravity and emergent spacetime models:

• Geometry emerges from entanglement structure (AdS/CFT, tensor networks).

• Causal relations fluctuate.

• The distinction between “earlier” and “later” may not exist at Planck scale.

Which means there may be physically admissible regions of reality where no global or even local temporal ordering exists.

And in such regions, computation, as defined by state transition systems indexed by N, is simply undefined.

Not because machines fail.

Because the semantic preconditions of “execution” are absent. This pushed me toward an unsettling reframing:

Maybe computation is not primitive.

Maybe causality is.

Not clocks.
Not time parameters.
Not sequences.

Just the partial order relation:

And everything else (time, algorithms, dynamics) is emergent structure layered on top of that relation when it happens to be acyclic, well-founded, and stable.

Which is not guaranteed by physics.

Subscribe now

Maybe time is computation

There’s a competing thesis.

Maybe time doesn’t underlie computation. Maybe time is computation.

This idea recurs across physics:

• Thermodynamics: entropy increase corresponds to irreversible information erasure.

• Quantum information: unitary evolution corresponds to reversible computation.

• Holography: spacetime geometry emerges from entanglement patterns.

• Tensor network models: geometry is literally computational wiring.

From this view, temporal ordering is nothing but dependency ordering between informational degrees of freedom.

Time is not background, it is output.

Computation presupposes time, and time emerges from computation, which creates a loop that suggests neither is truly fundamental, leaving only consistency as the primitive.

Not evolution, not process, not execution, but the satisfaction of constraints over relational structures, exactly as modern physics is written.

The universe is not something that unfolds like a movie but a solution that exists because all conditions fit together.

Causal structure as a computational resource

In standard computational models, causal structure is trivial: a total or partial order on steps.

But in physics, causal structure is dynamical.

In relativity, causal order depends on metric structure:

and (J^+(p)) depends on spacetime geometry.

In quantum mechanics, entangled systems violate classical factorization:

producing nonlocal correlations without signaling.

In quantum gravity, even the causal order relation may not be well-defined.

Which means that causality is not free.

It is constrained by physical law.

And if computation is structured causality (state transitions respecting dependency) then physics is not merely the substrate of computation.

Physics defines what computation means.

Which leads to a sharper version of the Physical Church–Turing Thesis:

Every physically realizable causal structure admits an equivalent computational representation.

And that claim is radically non-obvious.

Because some causal structures are cyclic.
Some are indefinite.
Some are globally inconsistent.

And in such worlds, computation (in the Turing sense) does not merely become inefficient. It becomes ill-posed.

Subscribe now

Closed timelike curves and the collapse of execution

General relativity admits solutions with closed timelike curves (CTCs), where causal order contains cycles:

In such spacetimes, execution semantics fail. There is no acyclic dependency graph. No notion of “before state” and “after state”.

But when computer scientists studied computation in the presence of CTCs, notably Deutsch (1991), and later Aaronson–Watrous, something unexpected happened.

Computational power increased, not incrementally, but qualitatively.

Problems in PSPACE collapse into polynomial time.

Not because you can iterate faster, but because computation no longer proceeds sequentially.

Instead, the system must satisfy a fixed-point constraint:

where

is a completely positive trace-preserving map representing the circuit interacting with its own past state.

The computation is not:

It is:

Execution is replaced by global consistency, which is precisely how fundamental physics operates.

Einstein’s equations:

do not evolve geometry: they constrain the entire spacetime manifold.

Quantum path integrals:

do not generate trajectories; they sum over all histories consistent with boundary conditions.

Physics does not run but satisfies, suggesting that the deepest form of “computation” in nature is not algorithmic but static, constraint-based, and atemporal.

Subscribe now

The day I stopped thinking the universe executes

I used to imagine the universe as a machine.

Initial conditions.
Evolution law.
Future states.

A cosmic program. Then I suddently noticed: that’s not how the equations are written.

They don’t say:

They say instead:

a global constraint over fields on spacetime.

The universe does not step forward; it satisfies equations over four-dimensional structures.

What we call time is not execution but traversal, a worldline slicing a static solution, and what we call dynamics is not computation but projection.

Time is not the medium of physics; it is the interface exposed to embedded observers.

What does “halting” mean without time?

Now things become truly unstable: the Halting Problem assumes a machine starts in state (s0), executes a sequence of transitions, and may or may not reach a halting state (sh).

In a timeless universe, there is no start, no sequence, no eventually: only the question of whether a globally consistent configuration exists.

Halting becomes satisfiability, non-halting becomes inconsistency, and there is no semi-decidability, no asymmetry, no waiting: just the existence or non-existence of fixed points.

This suggests something radical: classical undecidability, like the Halting Problem, may not be a fundamental feature of physical law.

This is not because physics violates Turing limits, but because physics does not instantiate the temporal semantics those limits presuppose.

Undecidability may emerge only from timeful computation, not from reality itself.

The limits of execution

Where does this leave us?

If computation presupposes time, and time is not fundamental, then “algorithm” becomes provisional.

It works inside pockets of reality where causality is well-behaved, decoherence stabilizes information, and entropy defines an arrow.

Outside those pockets, stepwise execution may simply not exist.

The universe may not “run” at all.

It may simply exist: a globally consistent solution to relational constraints, where evolution, dynamics, and causality are emergent features perceived by observers.

Execution is projection. Halting is satisfiability. Complexity is constraint.

Classical computer science tells us what can be computed if time flows like a river.

Physics tells us what is consistent if time is undefined.

Between the two lies a gap:

one about algorithms, the other about existence.

No algorithm, however clever, can escape the possibility that time itself is emergent, and computation exists only because the universe provides the scaffolding to support it.

In Part II, we will explore whether undecidability, chaos, and computational irreducibility are artifacts of timeful computation; or whether they emerge from the very structure of reality itself.

Share

The day I realized data was not the problem

There is a particular kind of frustration that only appears once a data system is, by all reasonable standards, successful.

The pipelines run.
The warehouse scales.
The dashboards load.
The executives nod.

The incident reports stop.
The on-call pager stays quiet.
The architecture diagrams finally look clean.

And yet, something feels deeply wrong.

Not loudly wrong. Nothing is on fire. No red alerts. No missing data. The system does what it is supposed to do.

It produces numbers, charts, trends. It answers questions: at least the ones it was designed to answer.

The discomfort is quieter than that. Harder to name.

I remember the moment clearly. I was reading yet another thoughtful, well-written deep dive on the semantic layer.

It explained metrics, dimensions, abstractions, self-serve analytics.

It talked about centralizing logic, eliminating ambiguity, empowering business users. I nodded along. Everything made sense. The arguments were sound. The tooling was impressive.

This was, by any standard, good work.

Eight hours later, I closed my laptop with an uncomfortable realization I couldn’t quite shake:

If all of this is so well understood,
why does analytics still feel so brittle?

Not broken. Brittle.

The kind of brittle where nothing fails outright, but everything feels slightly unstable. Where confidence depends on context. Where every number comes with a mental footnote. Where trust is provisional.

A single new question could fracture consensus.
A small change in definition could invalidate months of careful work.
Two dashboards could disagree without either being wrong; and without anyone being able to explain, cleanly, why.

Everyone involved was competent.
Everyone followed best practices.
And yet, the system could not absorb change without stress.

That was the moment the problem stopped feeling to be just technical.

This essay is my own attempt to sit with that discomfort and follow it all the way down.

• Not to criticize tools.

• Not to promote frameworks.

• Not to propose yet another layer for its own sake.

But to understand what analytics systems are actually missing,
and why the hardest problems only seem to appear after everything is supposedly working.

Subscribe now

The illusion of maturity

For most of the history of data engineering, scale was the enemy.

1. Could we ingest fast enough?

2. Could we store cheaply enough?

3. Could we query quickly enough?

These were clear, honest problems. They had benchmarks. They had roadmaps. They had satisfying answers, usually involving more cores and a larger bill.

And, eventually, we won.

Cloud storage removed scarcity.
Columnar warehouses removed fear.
Distributed systems removed ceilings.

We learned how to move data around at impressive speeds, how to store absurd volumes of it, and how to scan it so fast that no one even pretends to count rows anymore.

We casually talk about terabytes the way people once talked about megabytes, which should probably concern someone.

By almost any technical measure, the modern data stack worked. And yet, the promised land never quite arrived.

Not because the systems failed, everything was green, but because once scale stopped being the problem, something else quietly took its place.

Meaning.

The remaining problems were no longer about throughput or latency. They were about understanding. About interpretation. About agreement.

The kind of problems that don’t show up in monitoring dashboards and cannot be solved by adding another node.

Because at its core, analytics is not really about numbers.

It is about translation.

From events to facts.
From facts to metrics.
From metrics to decisions.

Each step looks straightforward on a slide deck. Each one feels like it should be automatable. And each one, in practice, smuggles in a handful of assumptions while no one is looking.

• An event becomes a fact once we decide it matters.

• A fact becomes a metric once we decide how to aggregate it.

• A metric becomes a decision once someone decides to trust it; often after asking who built the dashboard.

Every translation introduces interpretation, and it reflects a precise point of view.

And interpretation, left unmanaged, drifts.

Slowly. Politely. With good intentions.

Definitions evolve. Exceptions accumulate. A metric that once meant something precise becomes a rough suggestion.

The system still answers questions: it just answers them with a quiet asterisk.

Scale used to hide this. When data was scarce, ambiguity was tolerable. When queries were expensive, people didn’t ask follow-ups. When dashboards were rare, contradictions stayed invisible.

But once analytics became cheap, fast, and everywhere, meaning became the bottleneck.

And that’s when a data system that looks mature on paper starts to feel oddly fragile in practice: like a very impressive machine held together by shared assumptions and a mutual agreement not to ask that question.

When precision isn’t enough

SQL is one of the most successful languages ever designed.

It scales from hobby projects to trillion-row warehouses, and entire careers are built on fluency. And yet, it answers how, not why.

SELECT SUM(amount)
FROM transactions
WHERE status = 'settled'

Syntactically, this is flawless. The query planner understands it, the optimizer executes it efficiently. Semantically? Empty.

What does “settled” mean? Settled with the customer, accounting, or the bank?

Why are refunds excluded, because they were reversed later, or because someone copied this query from last quarter?

Which FX rate applies; spot, month-end, or the one that makes finance look good?

The database does not know. SQL executes logic; it cannot encode meaning, context, or assumptions.

Every analytical query carries this hidden cognitive tax: is this table canonical? Is this join correct? Is this filter business-approved?

Multiply that across dozens of analysts, hundreds of dashboards, and ad-hoc queries running before meetings, and what looks like efficiency is really entropy: quietly accumulating semantic drift.

The semantic layer emerged as an honest response. Metrics became objects. Dimensions became curated. Business logic was centralized. Joins, filters, and aggregations were standardized.

Dashboards could finally speak a common language. This was progress.

But notice what it does not stabilize: assumptions, temporal validity, domain disagreement. The layer freezes a moment in time while the business continues to evolve.

Products change, markets shift, regulations update: and suddenly a once-“canonical” metric feels negotiable.

Because metrics are not facts. Invoices, payments, FX rates: those are facts.

Metrics are interpretations: stories we tell about those facts, often encoded as SQL transformations, dbt models, or metrics-layer objects. And like all stories, they are context-dependent and time-sensitive.

Meaning, it turns out, does not live in the warehouse. It lives in analyst intuition, tribal knowledge, and half-maintained documentation. The warehouse stores raw events.

The semantic layer stores calculated objects. But understanding, what the numbers actually signify, lives elsewhere.

And until we make that explicit, even the most precise analytics system will remain impressively brittle.

Subscribe now

The Cognitive Data Interface

Ask a large language model to answer a business question, and you get a revelation: it guesses.

• It guesses joins. Should it join orders to payments or transactions? Which key is canonical?

• It guesses metrics. Does “net revenue” include refunds? Discounts? FX conversions?

• It guesses intent. Is the user asking for last quarter’s revenue, or a forecast-adjusted figure?

This is not because LLMs are careless or sloppy. It is because we never encoded meaning explicitly.

All the rules, edge cases, and assumptions that analysts carry in their heads are invisible to machines. SQL queries, dbt models, and metrics layers are only partially helpful: they are brittle shells without context.

Humans quietly compensate for ambiguity. Analysts know that settled = completed minus refunded, even if that logic is scattered across Slack threads, meeting notes, and half-written docs.

Machines do not. AI exposes these cracks, forcing us to formalize what we never bothered to codify.

AI did not create the problem, it exposed it

This is an important distinction. AI did not invent ambiguity. It only surfaced what was already there.

Every analytical system has been silently drifting under layers of human assumptions. Analysts and dashboards absorb ambiguity without complaint; machines cannot.

AI does not “understand” business rules: it follows logic. If the logic is incomplete, inconsistent, or implicit, the AI will guess, and those guesses highlight exactly where meaning was never formalized.

From metrics to knowledge

The semantic layer asks a reasonable question:

How do we define metrics once, so that dashboards, analysts, and AI agree?

But a deeper question is far more consequential:

How does an organization know what it knows?

This is not a data engineering problem in the classical sense. It is epistemic. It is about encoding the logic, context, and history of decisions in a way that both humans and machines can consume reliably.

Without this, every query, model, and dashboard remains an approximation, and AI only makes those approximations painfully visible.

Subscribe now

Definition of the Cognitive Data Interface (CDI)

The Cognitive Data Interface (CDI) is an architectural layer designed to make organizational knowledge explicit, versioned, and machine-actionable. It is built around four core principles:

• Separates facts from interpretations
  Facts are immutable raw events, e.g., (order_id, amount, currency, event_ts); interpretations are calculated metrics, e.g., RecognizedRevenue, which depends on facts plus business logic.

• Versions meaning over time
  Metrics and definitions are not static. Each calculation carries temporal metadata so that historical analyses can reproduce what was believed at the time.

• Encodes domain knowledge explicitly
  Business rules, edge cases, assumptions, and constraints are captured as executable objects rather than scattered in documentation or analyst intuition.

• Serves humans and machines equally
  Dashboards, analysts, and AI agents query the CDI through the same interface, ensuring consistency across all consumers.

The CDI treats meaning as first-class, executable infrastructure, not documentation. It’s the bridge between raw data and trusted insight, humans and AI alike.

Facts vs. Assertions

Consider this concrete example:

Facts (raw events):

order_id | amount | currency | event_ts
--------------------------------------
1001     | 100    | USD      | 2026-01-01
1002     | 200    | EUR      | 2026-01-02

Assertions (interpretations):

RecognizedRevenue :=
  SUM(amount)
  WHERE status = 'completed'
  USING fx_rate_at(event_ts)

• Facts persist: they are immutable observations of what happened.

• Assertions evolve: they encode business logic that may change as definitions, rules, or assumptions change.

This separation allows you to version, audit, and reason about both what happened and what we believed happened, without conflating the two.

Time as a First-Class Semantic Dimension

Every metric, rule, and definition has a lifespan. The CDI tracks two types of time:

• Valid time: when the fact or metric applies in the real world.

• Knowledge time: when the organization codified, recognized, or made a decision based on that fact or rule.

This enables questions like:

“What did we believe was our net revenue for Q4 2025, as of January 1, 2026?”

Even if definitions of “completed” orders, FX rates, or discount rules changed later, the CDI reproduces the answer faithfully.

Dual-temporal modeling ensures metrics are reproducible, context-aware, and auditable, allowing both humans and AI to reason about the past without guessing or hallucinating.

Architecture in Practice

At the heart of the Cognitive Data Interface (CDI) lies the semantic graph, a live, versioned, and queryable representation of organizational knowledge.

Its nodes represent entities, metrics, dimensions, and policies.

Entities might be customer, order, or subscription;

metrics could be RecognizedRevenue or NetRecurringRevenue;

dimensions include region, industry_segment, or customer_size;

and policies encode data access rules or approval constraints.

Edges define relationships such as depends_on for dependencies between metrics or entities, aggregates for roll-ups from facts to metrics, and constrains to enforce policies on metrics or entities.

Versioning ensures that any historical query can reproduce exactly the state of definitions at the time, allowing meaning, not just data, to become first-class infrastructure.

From Intent to SQL

The CDI allows queries to originate from intent rather than pre-written SQL.

For example, when a user asks, “Net recurring revenue growth for mid-market customers last quarter,” the CDI first resolves the metric definitions to determine what qualifies as recurring revenue and which invoice types are included.

It then applies segmentation rules to identify which accounts historically fell into the mid-market bracket, and finally enforces temporal validity, accounting for the FX rates, accounting policies, or definitions in effect last quarter.

Only after these steps does SQL emerge as an execution artifact; it is bytecode, not the source of truth.

Humans and AI alike can rely on the output without ever inspecting the SQL itself.

Subscribe now

Governance without friction

In the CDI, access control is not merely a filter: it is part of meaning.

Who can view a metric shapes how it is interpreted.

A revenue metric visible only to finance but hidden from sales carries different implications than the same metric shared across departments.

Encoding access rules as first-class semantic objects ensures that semantics, security, and intent converge, maintaining both trust and compliance across the organization.

Lineage as Explanation

Lineage is not just a directed acyclic graph.

In the CDI, it becomes a narrative: “This number exists because we aggregated completed invoices from these entities, applied FX rates from these dates, and excluded refunded transactions per the finance policy of Q4 2025.”

Every metric carries its reasoning along with its formula, making lineage an executable, auditable, and machine-readable explanation that preserves context for both humans and AI.

Failure Modes

Not all meaning can or should be encoded. Over-formalization risks rigidity: overly prescriptive rules can stifle decision-making and produce brittle systems.

The CDI must balance formalization with flexibility, allowing analysts to override or annotate interpretations when context demands.

Judgment, intuition, and situational awareness remain human responsibilities.

Organizational Truth

The CDI mirrors the organization. When teams disagree on definitions, or when policies evolve differently across departments, the model will fracture.

That visibility is not a flaw; it is a feature.

Analytics systems should reflect organizational complexity rather than pretending it does not exist, ensuring that inconsistencies become actionable insights rather than silent errors.

Subscribe now

After the Semantic Layer

The semantic layer was truly inevitable.

Centralizing logic, metrics, and definitions was necessary to establish consistency and trust. But it was never the destination.

The next bottleneck is not scale, nor tooling, but meaning itself. Until analytics systems learn to represent what organizations believe: their assumptions, interpretations, and evolving definitions, they will remain fragile.

The Cognitive Data Interface is not a product. It is a recognition: data was never the hard part.

The true challenge lies in capturing, versioning, and operationalizing organizational knowledge, ensuring that humans and machines alike can reason about the past, present, and future with confidence.

Share

The universe whispers in code

The universe has a rhythm that humans barely perceive.

Everywhere I looked, I saw pattern, law, and constraint: not metaphorical, not anthropocentric, but structural and inescapable.

For centuries, humanity has been intoxicated by the same dream: that this lawfulness is totally predictable.

Laplace, in 1814, imagined an intellect that, knowing all positions and velocities of particles, could compute the entire past and future.

Ignorance was a problem of calculation, not principle. Newtonian mechanics, with its continuous trajectories and deterministic equations, seemed to guarantee this possibility.

Yet, even before quantum mechanics, subtle cracks appeared. Henri Poincaré, studying the three-body problem in celestial mechanics, discovered chaotic sensitivity: tiny variations in initial conditions could lead to wildly diverging outcomes.

Determinism remained, but predictability faltered. Later, the discovery of strange attractors and turbulence in fluid dynamics hinted that lawfulness does not guarantee understandable evolution.

The real breakthrough, however, came with computation. The mid-20th century revealed that predictability might have a formal ceiling.

The universe, it seemed, might compute: but it could also refuse to compute answers for us.

This is the second part of a two-part journey exploring the intersection of physics and computation: two fields that have fascinated me for years.

I hope that reading it gives you the same sense of wonder, curiosity, and awe that I felt while writing and researching it, and perhaps even inspires you to look at the universe as a vast, subtle computation in motion.

Subscribe now

Gödel, Turing, and the genesis of limits

In 1931, Kurt Gödel demonstrated to the world that formal systems of arithmetic contain intrinsic limits.

His incompleteness theorems showed that for any system capable of representing natural numbers, there are true statements unprovable within that system.

For the first time in its millennia-long history, mathematics, long seen as a sanctuary of certainty, was revealed to harbor fundamental ignorance.

A few years later, Alan Turing formalized the concept of computation. His Turing machine, basically a read/write head scanning an infinite tape under finite control rules, provided a model of mechanical reasoning.

Turing proved the Halting Problem: no algorithm exists that can determine, for every program and input, whether the program halts or runs forever. Some truths are logically uncomputable, no matter the resources or ingenuity of the observer.

At first, these were abstract results, inhabiting the realm of pure mathematics. But physical systems can implement computation.

Von Neumann’s self-replicating cellular automata, Fredkin and Toffoli’s billiard-ball models, and contemporary quantum spin chains show that the laws of physics can encode universal computation.

Once embedded in reality, undecidability migrates from the abstract to the tangible: physical questions about system evolution inherit computational limits.

Consider a classical billiard-ball system arranged to implement a Turing machine. Each collision represents a logic operation; the trajectory of balls encodes tape symbols.

The system obeys deterministic Newtonian mechanics. Yet, ask whether a particular ball will ever reach a given location, and you face undecidability.

The universe has obeyed the laws faithfully, but no observer can compute the outcome faster than watching the system evolve step by step.

This is the subtle shift: the universe is lawful, deterministic, and continuous, yet it may refuse to reveal some answers.

Undecidability, once a property of formal systems, is now woven into the fabric of reality.

Chaos: lawful but unpredictable

Chaos is seductive because it masquerades as pure randomness.

In the early 20th century, Henri Poincaré’s work on the three-body problem revealed a profound truth: deterministic systems could behave in ways that defied prediction.

A slight perturbation in initial positions (a millimeter, a fraction of a gram) could radically alter long-term trajectories. This is the essence of sensitive dependence on initial conditions, the hallmark of chaos.

Yet chaos is a partial limitation. It is epistemic: our inability to measure initial conditions with infinite precision prevents practical prediction.

If a Laplacean intellect could know the positions and velocities of every particle to infinite accuracy, chaos would vanish, replaced by determinism.

But deeper than chaos is the concept of computational irreducibility, formalized in the late 20th century by Stephen Wolfram. A system is computationally irreducible if no shortcut exists to predicting its state at time t.

The only way to know the outcome is to let the system unfold, step by step. This principle applies not only to simple cellular automata but also to complex natural systems: turbulent fluids, interacting planetary systems, or quantum spin chains.

Consider a turbulent fluid described by the Navier–Stokes equations:

Even when deterministic, predicting the velocity field u(x,t) at a future time is computationally irreducible.

Tiny uncertainties amplify exponentially due to nonlinear terms. Approximations are possible, but exact outcomes remain inaccessible, a physical manifestation of the Halting Problem.

This insight is critical: lawful systems may be inherently opaque.

Determinism is not synonymous with predictability. The universe may compute, but some of its outputs are forever hidden from any observer, not by chance but by computational structure itself.

Subscribe now

Nature as a computational engine

Quantum mechanics takes unpredictability to a new level. In the classical world, determinism reigns, and randomness is epistemic.

In the quantum world, randomness is intrinsic. The evolution of a closed system follows the Schrödinger equation:

where ∣ψ(t)⟩ is the system’s wavefunction and H^ is its Hamiltonian. Unitary evolution is deterministic, yet measurement introduces probabilistic collapse.

Superposition allows a quantum system to “exist” in many states simultaneously, and interference lets amplitudes reinforce or cancel, giving quantum computation its unique power.

Richard Feynman, in the 1980s, realized that classical computers cannot efficiently simulate quantum systems. A system of N qubits has a state space of 2^N dimensions.

Explicit simulation on a classical Turing machine scales exponentially, quickly becoming infeasible. Feynman’s insight birthed the idea of quantum computers: devices exploiting quantum law to compute naturally.

Two canonical examples illustrate quantum advantage:

1. Shor’s Algorithm (1994): Factorization of a large integer N is classically exponential in complexity.

   Shor showed a quantum algorithm can factor in polynomial time, exploiting the quantum Fourier transform to extract periodicity from superposition:

Time complexity: O((logN)^3)

This has profound implications: cryptography, algorithmic efficiency, and the realization that physical law itself can accelerate computation.

2. Grover’s Algorithm (1996): Searching an unstructured database of size N classically requires O(N) queries. Grover demonstrated a quantum search algorithm requiring only O(N​) queries: a quadratic speedup achievable via amplitude amplification and interference.

Yet quantum mechanics does not break the Church–Turing limit. It does not enable hypercomputation; it cannot solve the Halting Problem or decide arbitrary Gödelian statements. It reshapes complexity, not computability.

The universe is a computational engine. Superposition encodes parallel states, interference prunes errors, and entanglement links distant subsystems nonlocally. The physical laws themselves are a kind of hardware optimized for certain classes of computation.

Feynman, Deutsch, and the Church–Turing thesis

David Deutsch, in the 1980s, proposed the Physical Church–Turing Thesis: every function realizable by physical law can, in principle, be computed by a universal Turing machine.

Quantum mechanics challenges this thesis subtly. While quantum computers can accelerate some computations, they respect the fundamental boundary of Turing computability.

Yet they also illuminate a deeper truth: complexity is physically grounded. Some problems are intractable not by lack of ingenuity but because physics enforces resource constraints: time, energy, entanglement, and coherence.

The Lure and Limits of analog computation

In the 1980s and 1990s, computer scientists and mathematicians began to ask: what if the universe could compute beyond discrete machines?

The continuous nature of physics suggested tantalizing possibilities. Consider a pendulum, a flowing fluid, or the precise trajectories of planets: these are analog systems governed by continuous variables.

Could a clever arrangement exploit this continuity to solve problems a Turing machine cannot?