When Running Your Own AI Models Beats Paying for APIs - Steven Latré, imec

Highlights

• A benchmarking study involving thousands of runs evaluated the transition from commercial APIs to self-hosted infrastructure for agentic AI tasks.

• Small open-weight models from Qwen and DeepSeek achieve a resolve rate between 35% and 40% on coding assistant tasks.

• The Kimi K3 model achieves an 86% resolve rate on the same coding tasks while requiring only a 15% increase in GPU rental costs compared to GLM-class models.

• Running a model like Kimi K3 on self-hosted infrastructure supports a maximum of four reliable concurrent users with a latency three times slower than cloud alternatives.

• Lower-tier hardware setups require a GPU utilization rate of 89% to make self-hosting cost-effective, whereas higher systems like an 8x B200 setup require only 15% utilization.

• Selecting specific coding harnesses halves token generation and GPU wall time without reducing application accuracy.

Timeline

1. 00:00 - 05:27

   Evolution of AI Maturity and the Shift to Infrastructure

   Organizations initially adopted any available frontier model before optimizing costs via token pricing and model routing. As token consumption scales, taking control of the AI stack becomes necessary. A recent benchmarking study evaluates thousands of runs to quantify the total cost of ownership, appropriate hardware, and inference engines required when moving from APIs to dedicated infrastructure for agentic tasks.

   • Organizations progress through maturity stages from relying on frontier model APIs and token-based pricing to model routing and infrastructure ownership.
   • Owning the infrastructure introduces a complex design space involving hardware configurations, CPU-to-GPU balance, and KV cache optimization.
   • Long-horizon tasks serve as the primary catalyst for organizations considering a transition from closed APIs to self-hosted open-weight models.
2. 05:27 - 10:10

   Model Performance and Hardware Economics

   Testing across different model parameters and coding harnesses reveals distinct tiers of accuracy. While smaller models handle basic tasks, high-performing models like Kimi K3 achieve high accuracy with manageable cost increases. However, matching these models to hardware requires balancing performance gains against significant variations in GPU rental costs.

   • Coding assistant benchmark tasks categorize models into smaller options achieving 35% to 40% resolve rates and advanced models reaching up to 86% resolve rates.
   • Qwen and DeepSeek deliver comparable application performance, but Qwen requires four times more expensive GPU rentals than DeepSeek.
   • Transitioning between open-weight models creates stepwise cost and hardware demands rather than a linear curve.
3. 10:10 - 14:35

   Concurrency Limits and Total Cost of Ownership

   Moving away from APIs introduces concurrency and memory limitations because model parameters and user KV caches contend for system memory. Total cost of ownership analyses indicate that lower-tier hardware configurations are rarely economical for self-hosting due to high utilization requirements, while high-end hardware architectures achieve a 25% cost saving at realistic utilization levels.

   • Memory constraints and KV cache scaling restrict the number of reliable concurrent users on self-hosted systems.
   • Running large models like Kimi K3 locally supports only four concurrent users with higher latency than cloud baselines.
   • Lower-tier hardware systems require an unrealistic 89% GPU utilization rate to justify self-hosting, whereas high-end systems require only 15% utilization.
4. 14:35 - 18:25

   Harness Impact and Ecosystem Collaboration

   Although multiple coding harnesses achieve identical task success rates, their underlying token generation efficiencies vary drastically. Selecting the right harness reduces the GPU compute bill significantly. Because enterprise environments mix multiple applications and hardware targets, ongoing research efforts and collaborative industry alliances are establishing public benchmarking pipelines to navigate these continuous changes.

   • Different coding harnesses achieve similar application accuracy levels but generate vastly different token volumes.
   • Efficient harness selection cuts token consumption and GPU wall time in half for specific models.
   • An upcoming public dataset and a new industry alliance aim to bridge the gap between software optimization and hardware infrastructure.