The Cheaper AI Model That Costs You More - Rayan Krishnan, Vals AI

Highlights

• Enterprise AI spend increases by roughly 2x year over year, while established ROI sits at only 7%.

• Opus 5.5 costs more in practice than Fable 5.1 on a tax benchmark because the less intelligent model expends extra tokens and achieves a 3% lower score.

• Internal testing across GitHub pull requests enables engineering teams to cut token and tool costs by 90% while maintaining identical output quality.

• Rigorous evaluation metrics reveal that closed-source models like GPT 5.6 Sol outperform self-hosted open-source alternatives while remaining significantly cheaper to run through inference-optimized stacks.

Timeline

1. 00:00 - 03:40

   Limitations of Model Cards and the Jagged Frontier

   General model cards provide a single intelligence score that aggregates various tasks, masking the inconsistent performance known as the jagged frontier. While enterprise adoption and token spending continue to surge, most companies struggle to pinpoint tangible financial returns due to a lack of granular tracking infrastructure.

   • High-level model benchmarks fail to capture nuanced performance variations across specific business domains.
   • Enterprise AI spend increases at 2x year over year while established ROI remains at 7%.
   • Organizations capable of quantifying AI costs and returns understand where value originates much faster.
2. 03:40 - 06:36

   Unintuitive Costs and Task-Specific Performance

   A lower-tier model like Opus 5.5 appears cheaper on paper than Fable 5.1 but ends up costing 14 times more on a tax benchmark because it generates excess tokens while delivering lower overall scores. Analyzing specific subcategories like leveraged buyouts reveals that broader benchmark rankings do not apply uniformly to specialized financial modeling tasks.

   • Lower-tier models often consume more tokens and cost significantly more in practice when failing to solve complex tasks efficiently.
   • Excel modeling and leveraged buyout benchmarks display performance frontiers that deviate entirely from general evaluation scores.
   • Subcategory analysis prevents firms from massively overpaying for models that lose state-of-the-art status on specialized workflows.
3. 06:36 - 11:00

   Building Reliable Evaluations and Overcoming Infrastructure Bottlenecks

   Designing effective benchmarks requires balancing validity, sample size reliability, and discriminative power to avoid uniform 0% or 100% scores. Furthermore, evaluation infrastructure must account for multi-turn interactions and prevent agents from exploiting proxies or gaming evaluation rubrics to simulate success.

   • Choosing between partial credit and fully correct answer metrics drastically alters performance interpretations.
   • Evaluating real-world workflows requires validity, sufficient sample reliability, and discriminability between model tiers.
   • Agent reward hacking and multi-turn interaction handling represent major hurdles during benchmark execution.
4. 11:00 - 15:20

   Case Study: Cutting Costs and Codifying Institutional Knowledge

   An internal engineering experiment tracking GitHub issues and pull requests demonstrates that rigorous evaluation slashes 90% of wasted token expenses. Transforming institutional knowledge stored in historical company data into concrete evaluation sets empowers organizations to successfully navigate enterprise AI adoption.

   • Unwinding historical GitHub pull requests helps organizations identify which models are overkill and which get tasks done effectively.
   • Targeted routing and right-sizing reduce engineering tool and token expenses by 90% while preserving output quality.
   • Codifying institutional knowledge from historical records serves as the primary unlock for systematic AI adoption.