Same GPU, 50% More Output: Why the Operator Matters — Oliver Kennon, SemiAnalysis

Highlights

• A GPU falls to the level of the operator, determining how much useful work is actually delivered.

• Two different serving stacks using B200 GPUs with identical inputs yield a 50 percent difference in generated output tokens per second.

• ClusterMax version 3.0 assesses managed GPU clusters across provisioning, reliability, monitoring, and security.

• Position on the cost curve is relative, meaning a producer can lower its costs but still fall behind if competitors improve faster.

• Rating the operator serves as an essential building block for creating a credible compute futures market.

Timeline

1. 00:00 - 01:05

   Introduction to GPU Economics and Operating Capability

   Hardware availability alone does not dictate performance, as a GPU falls to the level of the operator. In a commodity market, producers align by what they spend to deliver usable work, with lower production costs providing a competitive edge when prices fall.

   • Hardware purchasing gives potential, but operating capability determines delivered useful work.
   • Producers in a commodity market compete to deliver comparable output at the lowest production cost.
2. 01:05 - 05:35

   Inputs versus Outputs and Measured Inference Performance

   Price per successfully completed task at required quality serves as the ultimate North Star metric, though price per token is used as a practical proxy. Data from the public inference calculator demonstrates that choice of software and settings results in drastically different token generation rates using the exact same hardware.

   • GPU hours represent input costs, while intelligence and acceptable tokens represent economic output.
   • Identical B200 GPUs serving the same model produce a 50 percent difference in token output depending on the serving stack and software settings.
3. 05:35 - 07:47

   Cluster Security and Operational Responsibilities

   Fast hardware fails if security is compromised, as seen in instances where monitoring access exposed another customer's logs. Ongoing work is required to test updates, maintain deployments, and manage cluster security, with responsibilities differing between managed services and bare metal.

   • Useful intelligence requires adherence to security and access boundaries.
   • ClusterMax version 3.0 assesses managed GPU clusters on provisioning, reliability, monitoring, and security.
4. 07:47 - 12:35

   Cost Curves and Relative Market Competitiveness

   Better operations lower the cost per acceptable token by spreading rental or equipment expenses over more output. However, cost curve positions are relative to competitors, meaning internal improvements do not guarantee market advantage if rival producers outpace those gains.

   • Full cost per successful task divides total spending by usable customer output.
   • Producers can reduce their own costs yet still become less competitive if rivals improve at a faster rate.
5. 12:35 - 17:34

   Compute Grades, Benchmarks, and Futures Markets

   Rental price benchmarks measure market prices for GPU hours but do not capture internal production costs or output quality. Establishing clear service grades and operator ratings helps manage rental price risk and supports the development of standardized compute futures contracts.

   • Compute grades describe specific services to help buyers identify comparable options.
   • Operator ratings provide the foundational building block needed for a credible compute futures market.