No, That's Not a Software Factory — Ryan Cooke, WorkOS

Highlights

• Measuring software factory success by PR volume or code output disguises whether automated systems actually drive business outcomes.

• WorkOS built two distinct systems for its factory: TARS to integrate coding agents into Slack, Linear, and GitHub, and Horizon as the infrastructure orchestration layer.

• An internal Model Context Protocol (MCP) gateway connected to Snowflake enables AI agents to query semantic tables on product utilization and customer conversations.

• Webhooks from project management tools like Linear allow the agent system to pick up dependent tickets and evaluate missing project scope autonomously.

• Engineers at WorkOS kick off entire product engineering workflows from Slack using brief descriptions that trigger AI agents to draft initial specifications and Linear resources.

Timeline

1. 00:00 - 03:11

   Shift from Output Metrics to Outcome Metrics

   Standard software factory setups run AI coding agents inside cloud sandboxes to generate pull requests from prompts. Relying on output metrics creates a false signal of efficiency because thousands of generated pull requests do not necessarily yield finished products. Evaluating a factory based on feature delivery velocity ensures automation investments translate into tangible engineering output.

   • High volumes of AI-generated pull requests do not guarantee improved engineering velocity.
   • Output metrics like line counts or total pull requests obscure the true efficacy of automated coding setups.
   • Software factory success relies on accelerating feature delivery rather than generating raw code volume.
2. 03:11 - 05:07

   Architecture of the WorkOS Software Factory

   Initial tests with Cloudflare sandboxes and model routers yielded productivity identical to developers running AI coding tools locally. Driving exponential velocity gains required embedding actual engineering workflows into the system. WorkOS split its factory into TARS for developer interface integrations and Horizon for infrastructure orchestration, connected through a central Model Context Protocol gateway.

   • Basic cloud sandboxes with open-code routers perform no better than local AI coding tools.
   • TARS handles developer interactions across Slack, Linear, and GitHub via webhooks.
   • Horizon functions as the infrastructure orchestration layer sitting in front of an MCP gateway.
3. 05:07 - 08:52

   Automating Product Engineering Processes

   Instead of relying on human operators to prompt every step, TARS listens to project management webhooks to track progress and pick up dependent tickets automatically. A specialized PM agent ingests brief prompt descriptions to draft comprehensive PRD documents—referred to as Hilltop documents—incorporating market feedback and design references. Human engineers step in to review, cut excess scope, and refine generated tasks rather than creating assets from scratch.

   • Webhook triggers allow AI agents to execute dependent Linear tickets sequentially without human intervention.
   • A dedicated PM agent generates initial draft PRD documents directly from short project briefs.
   • Continuous project re-evaluation helps identify missing scope items as execution proceeds.
4. 08:52 - 11:59

   Interactive Workflows and Developer Interfaces

   Initiating a project requires only a brief Slack message, after which TARS instantiates tracking resources, initial task breakdowns, and decision logs. The structured context created in Linear and Notion feeds seamlessly into various coding environments, including third-party agents like Devin or local instances of Anthropic Claude. Centralized Slack channels ensure security teams and collaborators can weigh in early during automated project initiation.

   • Short Slack commands trigger TARS to populate Linear projects, decision logs, and Notion drafts.
   • Structured documentation enables third-party agents like Devin or local model harnesses to execute tasks.
   • Automated project channels give cross-functional teams like security immediate visibility into new initiatives.
5. 11:59 - 14:06

   Unlocking Leverage with an Internal MCP Gateway

   WorkOS built an MCP gateway, called the context engine, to supply agents with explicit rules on navigating internal tools and data storage. Connecting the server directly to Snowflake data warehouse tables lets agents craft queries covering customer behavior and usage metrics. Beyond coding automation, employees across the organization leverage this MCP server directly within Slack to complete operational and data analysis tasks.

   • An internal MCP gateway provides structured context and tool usage instructions to coding agents.
   • Connecting Snowflake to the MCP server enables automated querying of semantic product and customer data.
   • Non-engineering internal teams use the central MCP gateway to perform direct data analysis from Slack.
6. 14:06 - 18:39

   Expanding Automation and Infrastructure Control

   Automation extends across the software lifecycle to triage incoming Slack bug reports, assist customer support through code analysis, and build out custom sandbox infrastructure. Migrating away from external sandbox-as-a-service providers gives WorkOS full control over session memory and context distribution. Analyzing execution logs allows AI agents to identify missing capabilities, retire obsolete instructions, and continuously refine the factory's codebase.

   • TARS handles automated bug triaging, support ticket resolution, and pull request generation.
   • Owning sandbox infrastructure provides granular control over session data and workload distribution.
   • Session analysis enables automated self-improvement by detecting outdated agent skills and engineering skill gaps.