How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada

Highlights

• Air Canada delivers AI solutions across operations, corporate, commercial, and loyalty segments without relying on public benchmarks.

• Data scientists lack domain expertise and have limited access to busy experts, making custom evaluation design extremely difficult.

• A telecommunication Q&A RAG project improved evaluation speed from one test per week to dozens per day by inverting the evaluation process.

• Inverting the test creation involves having an LLM generate questions from specific source pages and using code-based grading to verify agent answers.

• Supplier contract analysis at Air Canada uses five distinct test generation techniques to evaluate agents without an existing benchmark.

• Code-based checks and exact match evaluations avoid the high noise levels associated with using LLMs as judges.

• Human expert feedback feeds directly into generating better tests and prompts rather than just patching individual agent outputs.

Timeline

1. 00:00 - 05:33

   The Evaluation Bottleneck in Enterprise AI

   Enterprise AI development often encounters highly specific tasks where public benchmarks do not exist. Data scientists building specialized agents must rely on domain experts for grading, but those experts face heavy workloads and limited availability. Weekly evaluation meetings involving multiple chatbots frequently produce conflicting human opinions, capping iteration speed at one trial per week. Furthermore, adding complex retrieval techniques often destabilizes previously functional pipelines.

   • Air Canada builds specialized AI agents for operations, corporate, commercial, and loyalty tasks that lack public benchmarks.
   • Data scientists face limited access to busy domain experts who struggle to grade chatbot outputs consistently in weekly meetings.
   • Adding architectural complexity to a telecommunication RAG system broke working features instead of improving overall accuracy.
   • Weekly expert review cycles restricted testing frequency to a single trial per week, severely slowing development velocity.
2. 05:33 - 07:45

   Inverting the Problem for Automated Testing

   Shifting the evaluation paradigm solves the expert bottleneck by starting with source text rather than user prompts. An LLM generates questions based on specific document pages, allowing code to automatically verify whether the agent retrieves the correct source. This inversion elevates testing frequency to dozens of runs per day. Removing unnecessary pipeline complexity allows data scientists to focus purely on foundational retrieval variables like chunking strategies and embedding models.

   • Reversing the evaluation loop uses source pages to prompt an LLM to generate questions automatically.
   • Comparing agent search results against the source pages enables automated, code-based grading instead of subjective human opinions.
   • Automated evaluation increases testing capacity from one trial per week to dozens of evaluations per day.
   • Removing pipeline complexity and focusing on core embedding models, chunking, and BM25 parameters steadily increases retrieval accuracy.
3. 07:45 - 16:40

   Five Techniques for Contract Analysis Testing

   Contract analysis presents a complex multi-step workflow requiring automated evaluation without public benchmarks. Five distinct generation techniques address this challenge, including leveraging past decisions, generating contracts around specific rules, and deliberately stripping clauses from signed documents to test detection rates. Additional robustness checks involve mutating non-semantic elements like supplier addresses or clause ordering to ensure consistent outputs. Reusing pre-existing expert templates allows the generation of numerous test contracts with known compliance statuses.

   • Analyzing supplier contracts at Air Canada involves spotting clause defects, finding supporting amendments, writing text, and classifying results.
   • Generating tests from known rules creates 275 test cases that verify whether agents catch specific contractual breaches.
   • Removing clauses from signed contracts serves as a powerful method to test if agents successfully flag missing requirements.
   • Invariants such as changing supplier names, altering addresses, or reordering clauses test output consistency without altering flag counts.
   • Leveraging existing expert templates allows automated generation of contract tuples that either breach rules or remain compliant.
4. 16:40 - 19:46

   Harnessing Code Over LLM Judges

   Relying on LLM-as-a-judge evaluation frameworks introduces significant noise that proves as difficult to solve as the original problem. Prioritizing code-based checks and exact matching provides fast, reliable evaluation metrics, even if it requires re-engineering the solution design. Ultimately, human feedback channels back into expanding test suites and refining prompt strategies, turning evaluation harnesses into continuous development tools.

   • Exact match and code-based checks execute rapidly and eliminate the heavy noise introduced by LLM-as-a-judge frameworks.
   • Re-engineering solution designs to fit code-evaluable patterns is necessary when original architectures cannot be tested.
   • Expert feedback directs the generation of better test cases and improved prompts rather than direct agent patching.
   • Automated test suites act as developer support tools and starting points rather than absolute go-no-go deployment gates.