ShipSF Workshops - V, from Scratch

Highlights

• A single routing contract file defines when the super agent routes incoming requests to a specific remote specialist agent.

• Replacing high reasoning with a minimal, bare-bones model severely degraded request routing quality, forcing an immediate rollback.

• Production traces automatically purge input and output content whenever a query executes in private channels or direct messages.

• User downvotes trigger an LLM-as-a-judge system to extract context, distinguish routing errors from answer quality failures, and draft test cases for review.

• Switching the core reasoning model behind the Eve framework requires changing only a single configuration line through AI Gateway.

Timeline

1. 00:00 - 04:19

   Origins of the Super Agent Architecture

   Prototyping domain agents across engineering, finance, and support created an alphabet soup of isolated bots such as D0, V0, and P0. Users lacked visibility into which bot owned specific questions or datasets. Building V established a centralized entry point that evaluates authority and relevance, letting contributors turn local workflows into shared capabilities.

   • Specialist agents like D0 originally fragmented internal discovery across multiple Slack channels and naming conventions.
   • V functions as a unified front door that routes user requests to specialized tools and synthesizes context across Slack, Notion, and the web.
   • The web interface acts as an internal marketplace where teams publish reusable procedures, scheduled workflows, and domain skills.
2. 04:20 - 09:17

   Specialist Ownership and the Remote Agent Protocol

   D0 demonstrates how a specialist provides both raw computation and context, such as identifying jumps across 450 million downloads over an eight-week window. Instead of merging every team's instructions into one central prompt, remote agents operate as standalone micro-services. This division isolates code reviews and domain adjustments within individual teams while maintaining a clear routing contract with V.

   • D0 provides plain-language analysis of warehouse metrics while supplying analytical interpretations alongside raw data tables.
   • The Eve remote agent protocol lets domain teams manage their own tools, prompt logic, permissions, and release cadences independently.
   • Precise boundary descriptions prevent routing conflicts when multiple specialists handle adjacent domains like general data usage versus specific system metrics.
3. 09:18 - 15:45

   Executing Multi-Agent Workflows Across Eve and MCP

   Handling multi-stage requests requires tracking asynchronous execution states and managing dependencies across distinct execution surfaces. In the demonstration, V runs on Claude Opus 5.5 fast to orchestrate an analysis from D0, transforms the tabular payload into an interactive chart using v0 via MCP, and publishes a snapshot report to Notion. The architecture chains sequential tasks together while maintaining references to the original warehouse sources.

   • The Eve framework provides execution primitives for sessions, durable task pauses, authorization steps, and sandboxed code execution.
   • End-to-end task fulfillment coordinates sequential handoffs between D0 for data retrieval, v0 via MCP for live charts, and direct Notion tool integrations.
   • Model swapping operates through a single line of configuration in AI Gateway.
4. 15:46 - 24:03

   Handoff Integrity, Packaging, and End-to-End Evals

   Information loss between steps poses a major operational hazard, such as metric caveats disappearing and causing raw download tallies to be mislabeled as unique user growth. Evaluating system health requires validating the final assembled output against missing data points, broken links, or authorization failures rather than just monitoring API status codes. Migrating capabilities from isolated remote agents into unified monorepo extensions minimizes unnecessary handoffs.

   • Successful intermediate tool calls can still produce incorrect outcomes if assumptions, units, and scope parameters drop during handoffs.
   • Bundling capabilities into Eve extensions reduces coordination overhead, latency, and context transfer compared to running standalone reasoning loops.
   • Production downvotes feed an LLM judge that drafts targeted regression tests for human review queues.
5. 24:04 - 33:00

   Identity Forwarding, Delivery Boundaries, and Trace Policies

   Authenticating a caller as a trusted service does not grant blanket authorization to read sensitive data. Eve decouples caller authentication from forwarded user identity, enforcing credential checks at the source and preventing sensitive summaries from reaching public channels. Tracing systems also forward data-scrubbing restrictions downstream so that specialist agents omit private variables from debug logs.

   • Source authorization and output delivery represent separate boundaries that prevent private data from surfacing in broad Slack channels.
   • Vercel Connect issues ephemeral Slack prompts for missing OAuth credentials and resumes blocked tasks automatically upon authorization.
   • Trace content policies suppress input and output logging in private Slack environments to prevent sensitive prompts from leaking to internal dashboards.
6. 33:01 - 43:01

   Audience Q&A: Framework Boundaries, Routing, and Freshness

   Discussion addresses enterprise implementation boundaries, confirming that sensitive database access stays strictly within D0 while cross-functional utilities live in shared extensions. To protect data privacy in team channels, sensitive reports fall back to private direct messages or ephemeral threads. When handling user feedback, team members resolve ambiguous ratings by inspecting linked Slack threads and adding clarification evals to the core testing pipeline.

   • Eve functions as the foundational runtime for V, though non-Eve external agents can interface via standard protocol wrappers.
   • Capabilities that directly touch the internal data warehouse remain strictly isolated inside D0 for security.
   • V disambiguates underspecified user queries by asking clarifying follow-up questions instead of guessing intent across competing specialists.