Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

하이라이트

• Traditional system metrics remain green even when an agent completely hallucinates tool execution and fails to process an action.

• Customer-facing financial architectures require client-side masking functions to redact PII like credit card numbers before persisting traces into observability storage.

• Domain experts annotate targeted 10% to 20% production trace samples, after which an LLM clusters their raw comments into reusable failure categories.

• Combining LLM-as-a-judge scoring with deterministic regex code evaluators eliminates evaluation drift and provides a repeatable baseline.

• Langfuse prompt version promotions trigger automated GitHub Actions CI jobs that run experiments against benchmark datasets to enforce pass/fail deployment gates.

• Handling 1,400 deflection interactions over seven days cost under $7 in model tokens, demonstrating measurable human capital displacement.

타임라인

1. 00:00 - 05:40

   Silent Agent Failures and the AI Engineering Loop

   A user support agent can falsely claim to issue a refund without ever executing the underlying refund tool, creating major business liability that surfaces weeks later. Langfuse addresses these non-deterministic failure modes by ingesting over 90 billion observations monthly across enterprise deployments to provide inspectable visibility into multi-step agent decisions. Establishing systematic tracing across every generation, tool call, and token metric provides the raw baseline needed to detect user friction and operational regressions.

   • Standard software monitoring tools miss AI agent failures because infrastructure latency and HTTP status codes remain green despite fabricated outputs.
   • The continuous AI engineering loop consists of five operational phases: tracing, monitoring, building datasets, running experiments, and evaluating outcomes.
2. 05:40 - 12:12

   Ground Truth Generation in Regulated Industries

   Trace logs capture raw root inputs, generation latency, tool invocation sequences, and cumulative costs. Regulated systems cannot afford unverified behavior, making curated benchmark datasets containing expected responses, keyword constraints, and boundary test cases mandatory for CI/CD pipelines. Teams systematically migrate from manual human workflows to automated systems by comparing live shadow-run agent responses directly against verified human decisions.

   • Low-stakes internal bots can deploy directly to production with post-hoc monitoring, whereas regulated domains require verified offline ground truth datasets before launch.
   • Regulated teams establish ground truth using three parallel strategies: expert curation, shadow runs alongside manual human workflows, and phased human-in-the-loop automation.
3. 12:12 - 20:52

   Live Trace Redaction and Guardrail Enforcement

   In banking assistants, persona impersonation tests whether guardrails adapt appropriately across retail accounts versus high-net-worth clients. A live test reveals plain-text credit card inputs automatically redacted inside the Langfuse execution tree before database persistence occurs. The agent processes multi-modal speech-to-text contact center transcripts and flags unauthorized interactions—such as cryptocurrency purchase inquiries—as investment advice risks.

   • Automated client-side redaction masks sensitive credentials and credit card numbers before persisting payloads into observability platforms.
   • Input guardrails detect out-of-scope customer intents like requests for investment advice and tag traces with categorical risk scores.
4. 20:52 - 30:07

   Dataset Curation and SME Annotation Queues

   Datasets import either from historical CSV test suites or by promoting high-value production execution traces into test cases. Instead of forcing domain experts to manually define taxonomy structures, teams present reviewers with rapid true/false toggles and open text fields. Pattern-matching LLMs then ingest the completed annotations to surface structured failure categories like missing citations, unresolved disputes, or informal phrasing.

   • Benchmark datasets serve as unit tests for AI agents, asserting intents, behaviors, and boundary handling rather than exact string matches.
   • Annotation queues streamline expert review by capturing raw commentary on 10% to 20% trace samples in under an hour.
   • LLMs aggregate qualitative domain expert notes to automatically classify global failure mode categories.
5. 30:07 - 39:09

   Deterministic and Model-Based Evaluators in CI/CD

   Relying purely on LLM evaluators introduces variance, while pure code checks fail to capture nuanced output quality. Automated webhooks detect prompt version promotions in the Langfuse UI or API, launching automated CI test runs across specific metrics such as source recall, language matching, and non-solicited upsells. The pipeline enforces numeric minimum thresholds, establishing an audited paper trail before code merges to main.

   • Production evaluation requires a balanced combination of 100% consistent deterministic regex functions and semantic LLM-as-a-judge scoring models.
   • System prompt changes in the registry trigger GitHub Actions CI workflows to benchmark configurations against test datasets before merging.
6. 39:09 - 46:00

   Production Dashboards and Targeted Error Analysis

   A live production dashboard reveals 1,400 customer inquiries deflected from human tier-one support over seven days at an infrastructure cost under $7. Rather than manually inspecting thousands of traces, operators use categorical filters to isolate the 89 instances flagged for investment advice or zero-resolution customer dispute cases. Combining explicit thumbs-up/thumbs-down ratings with structured risk categories directs developer attention to high-liability failures.

   • Executive dashboards convert trace aggregations into business value by comparing model token costs against human contact center hourly labor rates.
   • Categorical error-mode tracking isolates dominant operational regressions, including out-of-scope inputs, informal register, and unresolved disputes.
7. 46:00 - 53:22

   Implementation Strategy and Architectural Scalability

   Autonomous agents excel at writing regex evaluators, instrumenting codebases, and scaling experiment variations, but human engineers must define domain boundaries and dataset ground truth. The platform supports both self-hosted AWS EC2 deployments and managed SaaS models, backed by ClickHouse after relational databases failed to handle scale requirements. Aligning automated judges with human baseline raters involves biasing initial runs toward higher sensitivity so teams catch edge cases before tuning accuracy over time.

   • Delegating agent self-repair to open-ended coding agent prompts produces low-quality code, requiring human taste to guide evaluation design.
   • To process billions of daily trace events without latency degradation, the platform backend relies on ClickHouse storage.
   • LLM-as-a-judge scorers calibrate against human ground truth samples by tolerating higher initial false-positive rates to ensure no critical failure goes undetected.