How to Write Your First AI Eval - SallyAnn DeLucia, Arize AI

Highlights

• Vibe checks fail to scale because prompt adjustments without evals cannot detect regressions or gate releases in CI/CD pipelines.

• Traditional software unit tests require one deterministic output, while agent evals must handle non-deterministic execution paths across complex multi-step traces.

• LLM-as-a-judge setups perform best when evaluating exactly one concrete quality dimension per evaluator prompt.

• The internal agent Alex started with basic QA correctness checks and expanded over time to run 32 distinct evaluations in production.

• Filtering spans by parent ID equals null isolates the root interaction between user request and final output for evaluation datasets.

• Metrics flag that an agent is broken, while evaluation rubrics and explanations diagnose the specific failure mode.

Timeline

1. 00:00 - 04:13

   Limitations of Vibe Checks and Foundations of AI Evaluation

   Shipping prompts based on subjective feeling leaves teams unable to prove whether an agent update fixed a problem or broke adjacent workflows. Unlike traditional software with visible code logic, agent decisions emerge dynamically during execution. Observability traces provide the foundational data required to evaluate multi-step tasks such as day-by-day itineraries and budget constraints.

   • Informal vibe checks create silent regressions when prompt adjustments accumulate without objective baseline comparisons.
   • Agent evaluations differ from deterministic unit tests because identical inputs yield multiple valid outputs across unreadable runtime paths.
   • Traces serve as runtime logs for AI agents, while evals function as the actual integration tests.
2. 04:13 - 07:29

   Taxonomy of Evaluation Architectures

   Evaluations span deterministic code tests, gold-standard human annotations, model-based judges, and live production feedback. Core evaluation judges should operate across both development sandboxes and production environments to maintain continuity without wasted engineering effort. Explanations generated alongside labels verify whether an evaluation model followed the intended reasoning steps.

   • LLM-as-a-judge pairs categorical labels with natural language explanations to justify scoring decisions.
   • Decision model APIs output raw probabilities for fast binary judgments without explanatory rationales.
   • Code-based checks handle deterministic constraints like regex patterns, JSON schema validation, ROUGE, and BLEU scores.
   • Agent-as-a-judge workflows deploy external autonomous agents to evaluate complex, multi-turn task success criteria.
3. 07:29 - 19:03

   Trace Inspection and Multi-Agent Architecture Anatomy

   Tracing exposes each granular unit of work, showing whether an agent routed properly, picked the correct external tool, or synthesized context accurately. In a travel planning agent, one model span selects an external retrieval tool for weather and local sights, while a subsequent model span summarizes raw payloads for downstream components. Tracing libraries like OpenInference instrument agent frameworks neutrally without vendor lock-in.

   • A trace captures the chronological graph of agent decisions, tool invocations, and nested model calls required to fulfill a prompt.
   • Root spans represent top-level user inputs and final outputs where parent ID equals null.
   • Multi-agent patterns divide labor among specialized agents for distinct sub-tasks like tool selection, data fetching, and synthesis.
4. 19:03 - 25:19

   Formulating Concrete Single-Dimension Criteria

   Vague instructions cause evaluation models to grade inconsistently across datasets. Writing specific criteria mirrors briefing a human reviewer on exact acceptance criteria, such as verifying that travel itineraries stay strictly within an 800-dollar budget. Complex applications scale evaluation suites incrementally over time rather than attempting all-in-one rubric prompts.

   • Broad quality labels like helpful or accurate fail without concrete criteria tied to user constraints.
   • Restricting each LLM judge to evaluate one specific dimension prevents degradation in judgment quality.
   • Prompt criteria require clear bullet points detailing three explicit conditions for passing and failing.
5. 25:19 - 33:03

   Building, Filtering, and Testing Judges in the Playground

   Curating focused evaluation datasets involves filtering traces by span type, such as isolating root spans to check end-to-end plan fidelity or tool spans to check payload accuracy. Variable abstraction prevents broken evaluators when data keys shift between local test datasets and live traces. Running small models like GPT-4o-mini inside an interactive playground highlights prompt flaws quickly through generated rationales.

   • Span filters isolate target evaluation nodes like root spans or specific tool names for dataset curation.
   • Evaluator prompts establish role context, input variables, exact pass-fail rubrics, and target label sets.
   • Variable mapping decouples evaluator prompt schemas from changing production trace attributes.
   • Prompt playgrounds enable iterative tuning of evaluator instructions against sample trace runs before deployment.
6. 33:03 - 35:18

   Production Deployment, Alignment, and Lifecycle Strategy

   Deploying evaluators online creates continuous operational signal, enabling automated agents to flag regressions and suggest targeted prompt fixes. Quantitative metrics indicate when an application deteriorates, but qualitative evaluation rationales reveal why the underlying agent failed. Teams maintain evaluation accuracy by comparing judge outputs against human labels before expanding scope.

   • Human alignment validates LLM judges by cross-referencing model scores against annotated golden datasets.
   • Online evals run continuously in the background to score incoming production traces automatically.
   • Cross-functional evaluation development unites product, engineering, support, and domain specialists to define success.
   • Effective evaluation suites begin with small 20-trace baselines assessing one dimension with readable explanations.