Don't Trust Your AI Agents. Test Them Like a Scientist — Stephanie Jarmak, Omni Analytics

Highlights

• Modifying an initial agent prompt reduced run costs by 28% without resolving the agent's failure to invoke the target code search tool.

• Swapping the agent harness from OpenHands to Astra changed task effectiveness by over 30% using the identical underlying model.

• A search tool retrieval query returned 295 files when only 6 files were necessary, demonstrating a severe precision deficit.

• Independent traffic record audits detected injected security issues that successfully bypassed internal agent hash checks and execution traces.

• Building accurate evaluation harnesses requires anchoring tests in internal production workload data rather than relying solely on public benchmark scores.

Timeline

1. 00:00 - 02:51

   Tool Use Failures and Prompt Alterations

   Initial testing of code search MCP tools reveals that superficial optimizations like cost reduction do not correlate with functional tool utilization. Comprehensive trace analysis is necessary to determine whether agents receive necessary context and why they fail to execute specific actions.

   • Initial prompt modifications reduced run costs by 28% without altering the agent's failure to invoke the code search tool.
   • Agent evaluation requires examining complete execution traces to isolate why tool calls are skipped despite available context.
2. 02:51 - 04:05

   The Impact of Harness Configurations

   Third-party comparisons on Terminal Bench 4 demonstrate that harness selection introduces massive variance in agent capabilities. Evaluating models inside a configuration that mirrors the target production environment prevents misleading performance expectations from public rankings.

   • Swapping agent harnesses from OpenHands to Astra shifted task effectiveness by over 30% using the same underlying model.
   • Model performance changes drastically depending on the enclosing harness environment.
3. 04:05 - 05:33

   Interpreting Public Benchmarks versus Internal Evals

   Public benchmarks help map the broader AI landscape, but organizations must avoid over-indexing on them due to non-representative task distributions. Internal evaluations must mirror actual operational workloads and iterative organizational feedback loops.

   • Public benchmarks serve primarily as high-level reference sets to generate shortlists rather than exact predictors of internal utility.
   • Model training spikiness causes public benchmark scores to diverge from specialized organizational requirements.
4. 05:33 - 07:07

   Applying a Scientific Approach to Variable Testing

   Rigorous evaluation treats system changes like a scientific experiment by isolating variables such as task coverage, run variance, and configuration interactions. Comprehensive workload clustering ensures code understanding and bug fixing are measured independently.

   • Workload coverage requires clustering tasks across the entire software development life cycle rather than focusing solely on bug fixes.
   • Non-deterministic model behavior necessitates running multiple iterations to measure performance variance and interaction effects.
5. 07:07 - 09:55

   Observability and Precision Metrics in Search Tools

   Debugging agent failures requires following the complete execution trail to identify where instructions or tool outputs break down. Optimizing metrics like the F1 score prevents tools from pulling excessive, irrelevant data that degrades overall system efficiency.

   • Execution pipelines must be audited step-by-step from context availability down to returned values and final outcomes.
   • Grep-based searches retrieving 295 files for a task requiring only 6 files highlight the need to optimize for precision alongside recall.
6. 09:55 - 13:03

   Quality Gates, Human Review Load, and Trace Trustworthiness

   Autonomous software factories require strict boundary enforcement because agents lack reliable internal stop mechanisms when errors occur. Furthermore, agent-generated traces can be tampered with or fail to record injected vulnerabilities, necessitating independent traffic history audits.

   • Automated quality gates require hard-coded enforcement mechanisms rather than relying on agents to self-enforce stops.
   • A 1% false alarm rate in automated reviews can generate thousands of alerts that overwhelm domain expert review capacity.
   • Independent record comparisons can catch injected security issues that successfully bypass agent execution traces and hash checks.
7. 13:03 - 17:02

   Representative Workloads and Production Autonomy

   Effective evaluation anchors on real-world production proxies categorized by complexity, ranging from basic Q&A to full application building. Establishing rigorous operational boundaries allows systems to automate low-risk decisions safely while retaining human oversight for critical thresholds.

   • Production traffic and internal usage data serve as the ultimate truth for building representative evaluation harnesses.
   • Defining clear risk boundaries determines when an agent can autonomously execute tasks versus when it must escalate to a human.