How to Test Your Agents on Real Tasks With Harbor - Xander Song, Arize AI

하이라이트

• Harbor runs tasks inside isolated Docker sandboxes and scores agent actions via verifier scripts that produce rewards such as binary zero or one pass/fail values.

• Oracles provide deterministic reference scripts that validate test environments, seed data, and verifier logic before incurring model token costs.

• Mounting verifiers into Docker containers only after an agent finishes execution prevents the model from inspecting scoring scripts or cheating during runtime.

• Sandbox providers like Daytona enable horizontal scaling across up to 200 concurrent workers to parallelize independent agent trials and calculate aggregate pass@k metrics.

• Agent Trajectory Interchange Format (ATIF) captures JSON-based intermediate steps and tool invocations, which can convert into OpenInference semantic telemetry within Phoenix.

• Evaluating aggregate query routing confirmed that adding a direct read-only SQL tool alongside a focused prompt tweak cut unnecessary multi-turn pagination cycles.

• Testing agent frameworks extends beyond core foundation models to the surrounding system layer comprising CLI tools, skills, prompt instructions, and MCP servers.

타임라인

1. 00:00 - 07:49

   Core Architecture and Verifier-Driven Testing

   Agent evaluation requires moving away from qualitative impressions toward repeatable evidence. Harbor standardizes this process by combining instruction markdown files, custom container environments, and independent verifier scripts into repeatable jobs. Because verifiers measure post-execution state changes like modified files or database updates, agent implementations can undergo refactoring while holding scoring definitions constant. Adding trajectory tracking via the Agent Trajectory Interchange Format records internal agent steps for debugging without coupling success criteria to specific execution paths.

   • Harbor executes agent instructions defined in markdown files against isolated Docker containers preloaded with task-specific data.
   • Verifier scripts examine environmental side effects rather than agent internals to compute pass/fail rewards.
   • Deterministic oracle scripts execute known solutions to validate environment setup and verifiers before spending token budgets.
2. 07:49 - 14:19

   Decoupled Verification in a Database Migration Scenario

   A broken database migration provides a concrete benchmark task. The container supplies an up-migration script seeded to fail, alongside verifiers that check exit codes, schemas, and table indexes after the agent attempts a fix. Delaying the mounting of test files until execution finishes ensures the agent cannot read grading scripts to bypass real problem solving. This decoupled interface allows engineers to update agent prompts, swap model providers, or refactor skill libraries while keeping evaluation criteria fixed.

   • A broken schema migration scenario demonstrates end-to-end task setup using actual SQL files and failing DDL code.
   • Mounting verifiers into the container after the agent finishes prevents the model from inspecting tests or gaming the evaluation.
   • Treating agents as black boxes mirrors integration testing principles and decouples evaluation logic from internal agent architectures.
3. 14:19 - 23:21

   Parallel Scaling and Observability Integration

   Because language models exhibit non-deterministic behavior, measuring performance requires repeated trials across identical starting states. Remote container platforms like Daytona execute large datasets concurrently across 200 workers, yielding robust aggregate metrics like pass@k. Integrating this execution pipeline with Phoenix converts raw execution trajectories into OpenInference telemetry. This data mapping provides visualization dashboards for tracking reward shifts across prompt versions and inspecting multi-turn execution traces.

   • Independent trials running across platforms like Daytona scale horizontally across 200 concurrent workers to compute pass@k metrics.
   • Phoenix maps Harbor data sets, jobs, and binary rewards directly into observability dashboards and evaluation traces.
   • OpenInference translates ATIF trajectory logs into OpenTelemetry standards to support step-level inspection of tool calls and model reasoning.
4. 23:21 - 33:23

   Evaluating SQL Tools to Eliminate API Pagination

   Supporting multiple access surfaces, including CLIs, MCP servers, and background harnesses, requires evaluating the end-to-end tooling system rather than raw model weights. In one scenario, agents attempting analytical queries paginated through REST and GraphQL endpoints because no aggregate endpoints existed. Providing a read-only SQL tool resolved the infrastructure gap, but required testing to confirm the agent would invoke SQL over existing APIs. Using Claude to generate synthetic tasks alongside oracle queries established a deterministic test suite on real SQLite trace databases.

   • Packaging the single-node Phoenix server into a container provides a complete testing harness containing realistic SQLite data and APIs.
   • Supporting CLI, MCP, and server-side interfaces requires evaluating the surrounding tooling ecosystem rather than isolated base models.
   • Synthetic benchmark tasks paired with SQL oracles measure whether agents select direct query paths over inefficient pagination.
5. 33:23 - 44:12

   Experiment Comparison and Optimization Trade-Offs

   Modifying system instructions to route aggregation questions through SQL decreased average tool counts per task without degrading answer correctness. Comparing pre-change and post-change jobs within Phoenix confirmed that the agent executed direct SQL queries instead of repeatedly calling a GraphQL-backed bash emulator. While canonical verifiers evaluate only external filesystem side effects, inspecting trajectory logs helps engineers diagnose tool selection frequency and optimize operational latency.

   • Updating system instructions to prefer SQL for aggregation cut overall tool calls and reduced multi-turn token latency.
   • Comparing experiment runs side by side in Phoenix confirmed that rewards held steady while tool invocation efficiency improved.
   • Inspecting internal trajectory logs serves optimization workflows even when pure correctness verifiers remain decoupled.
6. 44:12 - 55:52

   Workflow Skills, Agent Harnesses, and Practical Constraints

   Skill regression tests safeguard multi-step agent workflows across different foundation models, though overlapping trigger descriptions can cause unintended routing failures. Verifying skill activation can involve targeted trajectory inspection to ensure specific tools fire as intended. While single-node backends easily package into Docker images for Harbor runs, distributed dependencies require modularizing API boundaries or spinning up multi-container clusters. Securing representative task data remains the most demanding prerequisite for building realistic evaluation datasets.

   • Testing skill workflows mirrors end-to-end user journeys to identify prompt collisions where overlapping definitions prevent execution.
   • Single-node architectures simplify containerization, whereas distributed microservices require isolating agent interfaces or spinning up multiple images.
   • Data preparation represents the primary operational bottleneck when constructing realistic, high-signal benchmark tasks.