How We Took an AI Agent From 41% to 97% Accurate - Milind Sunil Shah, Brainbase Labs

Highlights

• An AI agent built for Domino's improved from 41% accuracy at version one to 97% accuracy at version 11.

• The cost per source dropped from $2 to $1, while runtime decreased from 129 minutes down to 71 minutes.

• The six-step framework consists of kickoff, ontology, benchmarks, first agent, iterating, and handing off.

• Instruction and skill updates alone drove all improvements in accuracy, cost reduction, and processing speed.

• A rule adjustment instructing the agent to scan the entire webpage instead of stopping at linked sections prevented missed discount pricing.

• Instructing the agent to check HTML DOM elements and alt text before zooming into images reduced image reads by two-thirds and cut costs by over 50%.

Timeline

1. 00:00 - 02:23

   Project Background and Initial Performance

   Enterprise back-office automation often begins with tasks that appear simple in isolation but fail due to edge cases. For Domino's, analysts manually spent two hours daily compiling competitor prices, promos, and delivery costs into spreadsheets. Building an automated agent required handling multiple web sources, interpreting varied pricing structures, and flagging uncertain entries for human sign-off.

   • Brainbase deploys and manages internal enterprise agents at scale.
   • A project with Domino's aimed to collect local comparative pricing across competitors and delivery apps automatically.
   • Manual execution by an analyst required opening competitor websites, finding correct stores, comparing prices, and recording results in a spreadsheet.
   • Version one of the agent launched with 41% accuracy, a $2 cost per source, and a 129-minute runtime.
2. 02:23 - 03:34

   Step One: Kickoff

   Clear alignment with the customer prevents misaligned expectations before writing code. Defining what a successful run looks like includes specifying runtime limits, cost limits, required accuracy thresholds, and actions the agent must never perform independently.

   • The kickoff phase establishes agreed definitions of success, baseline metrics, and boundaries between human and agent responsibilities.
   • The Domino's project targeted faster comparative pricing collection without losing judgment for edge cases.
   • Initial requirements mapped out current manual workflows, target accuracy, cost constraints, and operational guardrails.
3. 03:34 - 05:43

   Step Two: Ontology

   Without explicit definitions, AI agents rely on guesses that introduce errors. Providing a targets table, a rules file, and a price observations table gives the agent a structured memory environment. Defining precise criteria for bundle items, delivery methods, and promotion math ensures consistent data entry into the final spreadsheet.

   • Ontology creation involves defining every term explicitly so the agent avoids making assumptions.
   • A structured PostgreSQL database primitive stores item targets, rules, and price observations.
   • Explicit rules direct the agent to record final checkout prices rather than struck-through prices and to document data gaps instead of guessing.
4. 05:43 - 07:43

   Step Three: Benchmarks and Evals

   Evaluating agent outputs requires multiple validation layers. Code checks verify spreadsheet completion and math accuracy, LLM judges review execution traces for correct item selection, and agent judges audit error flags. Storing historical browser snapshots enables consistent offline testing against past data.

   • Offline evaluations utilize historical spreadsheets and archived web snapshots to test agent performance.
   • Evaluation methods combine deterministic code checks, LLM-as-a-judge trace reviews, and agent-as-a-judge flag verifications.
   • Browser screenshots captured from version one onward establish a reliable testing environment.
5. 07:43 - 08:48

   Step Four: First Agent

   Starting with a minimal slice of the overall problem allows teams to measure runtime, cost, and failure modes immediately. Expanding the scope happens only after meeting an established baseline score.

   • Version one prioritizes measurement over high performance by launching with a narrow scope.
   • Initial tests run on a single app, a single item, and a single city to establish a baseline score.
   • Initial metrics for the Domino's agent showed 41% accuracy and a cost exceeding $2 per source.
6. 08:48 - 11:55

   Step Five: Iteration

   Analyzing failure traces reveals expensive or frequent errors that guide rule updates. For example, adding a rule to inspect entire web pages rather than stopping at linked sections prevented missed discounts. Similarly, optimizing browser navigation loops by prioritizing HTML DOM data over repeated image zooms eliminated infinite processing loops.

   • Iteration involves scoring, fixing, and shipping adjustments to instructions, rules, and ontologies one change at a time.
   • Version 11 achieved 97% accuracy and a reduced cost of $1 per source.
   • Modifying browser behavior instructions to check HTML DOM structures and alt text before reading images reduced image reads by two-thirds and cut costs by 50%.
7. 11:55 - 013:13

   Step Six: Handoff

   Empowering the end user to manage exceptions and convert corrections into new benchmark cases ensures long-term system stability. Domino's integrated the agent into regular operational schedules while expanding deployment across multiple markets.

   • Handoff transfers the operational loop rather than just the agent code.
   • Scheduled runs execute daily or via email triggers, with human analysts reviewing only flagged exceptions.
   • Corrective actions feed directly back into new benchmark cases and rule definitions.