Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS

Highlights

• A custom verification CLI gives agents the exact deterministic feedback loops needed to self-heal and confirm real persistence rather than trusting superficial UI changes.

• Engineers managing 10 to 30 concurrent agent workflows scale throughput to over 900 PRs a month when agents verify their own state of completion.

• Prompting agents with specific end outcomes rather than rigid step-by-step instructions allows frontier models like Sol 6.1 and Opus 5.5 to catch edge cases creatively.

• A single screenshot is insufficient proof of task completion unless accompanied by verifiable end-to-end trace logs, network responses, and runtime artifacts.

• System 1 models such as Jev and Clef provide rapid, flexible evaluation to verify application assertions without brittle, exact string matching.

• Preserving execution receipts across all agent runs creates the training datasets and evals required to power recursive self-improvement loops.

Timeline

1. 00:00:00 - 00:07:31

   The Failure Modes of Unverified AI Coding

   High agent velocity creates a severe signal-to-noise deficit for human reviewers. The Midnight Snack Club demo shows how an agent can create a form that renders an email on the screen, creating an illusion of completion. Reloading the page clears the entry because the data was never written to the backend database. Superficial visual success masks underlying logic failures when verification does not inspect state persistence.

   • Unchecked agents can cause catastrophic incidents, such as deleting production databases in nine seconds.
   • Code that builds and passes basic UI checks often fails to persist data upon a page reload.
   • Managing dozens of concurrent agent sessions makes manual line-by-line PR reviews impossible.
2. 00:07:31 - 00:16:35

   Verification Loops and Centralized Evaluation Architecture

   True agent velocity comes from moving away from multi-day, monolithic agent sessions toward small, decomposed tasks with crisp verification boundaries. Tools like CTX centralize authority by communicating through MCP to check GitHub APIs and merge requirements before certifying completion. Riker monitors automated PR review feedback and test runs so human engineers only handle higher-level system architecture.

   • Agent scaling depends entirely on an agent's ability to verify its own work.
   • Decomposing tasks with exact completion criteria enables developers to merge up to 900 PRs per month.
   • Centralized verification services like CTX and Riker enforce merge checks and monitor PRs through API queries rather than agent prose.
3. 00:16:35 - 00:26:01

   Building Bespoke CLI Verifiers for Deterministic Feedback

   Vague natural language prompt instructions like 'verify changes in the browser' introduce high error rates. Wrapping bash checks, headless browser drivers, and health probes into a single command provides deterministic test output. The Snack CLI template demonstrates commands such as doctor, join, reload, and storage to let the agent independently verify that state changes exist in local storage.

   • LLMs produce high-variance text, whereas dedicated CLI scripts enforce deterministic checks.
   • A bespoke verification CLI such as Snack equips agents with operational commands to start, inspect, and test an application.
   • Terminal agents require external observation tooling like Playwright snapshots and DevTools inspection to validate runtime state.
4. 00:26:01 - 00:37:53

   Outcome-Driven Development and Preserving Receipts

   Prescribing every intermediate step restricts frontier models like Sol 6.1 and Opus 5.5 from finding optimal code structures or handling subtle edge cases. Supplying explicit desired outcomes and rich failure logs turns errors into self-correction loops. Verification artifacts must prove that actions actually happened rather than taking a screenshot of a hardcoded UI state.

   • Detailed error messages provide the essential signal for an agent to self-heal without human intervention.
   • Outcome-driven development defines the target completion state while letting frontier models devise the implementation path.
   • A passing screenshot is invalid proof unless trace logs confirm the agent executed the correct end-to-end user path.
5. 00:37:53 - 00:48:42

   System 1 Judges and Data Foundations for Recursive Self-Improvement

   Hardcoded assertions break when an application shows 'your seat is confirmed' instead of 'waitlisted', despite the action succeeding semantically. Sub-second System 1 models provide fast semantic evaluation to resolve this mismatch. Collecting these structured receipts across hundreds of daily runs generates the data foundation required to train and tune self-evolving agent harnesses.

   • System 1 models like Jev and Clef evaluate semantic correctness without requiring exact string matching.
   • Historical execution receipts supply the benchmark datasets needed for recursive self-improvement loops.
   • Failing evaluation runs map application edge cases and prevent behavioral regressions.
6. 00:48:42 - 01:00:55

   Scaling Verification from Local Runtimes to Production

   Relying strictly on remote CI creates unproductive context switching when running 10 to 15 concurrent agents. Placing clear verification instructions inside an agents.md file ensures every local runner executes checks before pushing code. Post-deployment safety patterns deploy secondary watcher agents that monitor live telemetry and feature flags, automatically reverting deployments if production errors spike.

   • Local verification loops in files like agents.md prevent developers from getting blocked by remote CI latency.
   • Automated code review scores from platforms like Greptile combined with green CI pipelines establish shipping confidence.
   • Autonomous deployment pipelines use dedicated monitoring agents to track Datadog metrics and trigger instant rollbacks.