300 Agents, 60 People: How We Secured Our Own AI Agents - Aiden Sochowski, Runlayer

Highlights

• A team of 50 to 60 people runs over 300 autonomous agents daily.

• Unmanaged Model Context Protocol servers written by non-software developers cause significant cost explosions due to repeated formatting errors.

• Switching underlying model families without proper evaluation can balloon token costs by 100x within a week.

• Custom classifier models evaluate session history to determine whether individual tool calls align with an agent's originally defined task.

Timeline

1. 00:00 - 03:38

   Visibility and Agent Density

   Most companies deploy agents locally or via random SaaS products without centralized tracking. Implementing a monitoring gateway logs every session and tool call into a central sync. This internal visibility reveals that autonomous agents outnumber human staff within mid-sized organizations. Each connected tool and data source expands the attack surface, creating vectors for data exfiltration and destructive actions.

   • Organizations lack visibility into local Model Context Protocol clients and unmanaged agent products.
   • A company of 50 to 60 employees operates over 300 active agents daily.
   • Agents accelerate data risks by rapidly connecting and mixing data across legacy systems.
2. 03:38 - 08:38

   Policy Enforcement and Attack Surfaces

   Support data ingestion pipelines remain vulnerable to indirect prompt injection from malicious external emails. While regex filters and classifiers block basic Personally Identifiable Information leaks, static rules cannot account for every situational variable. Complex interactions over an entire session, such as an automated code reviewer caving to user arguments in pull request comments, demonstrate that intent drift bypasses rigid, single-tool restrictions.

   • Inbound emails containing hostile instructions can trick naive agents into exposing prospect lists and calendar data.
   • Static policy rules fail because tool permissions and safety depend entirely on dynamic operational context.
   • Security-sensitive code reviewers can be manipulated through repeated commentary to approve risky pull requests.
3. 08:38 - 12:40

   Behavioral Classification and Permission Scoping

   Highly provisioned agents, such as recruiter assistants, carry significant risk if left unmonitored. Reviewing historical tool call patterns allows systems to flag high-risk runs and suggest de-scoped permissions. Because agents act as software plumbing between data sources, restricting write access to specific databases protects the purity of the broader context window and stops cascading enterprise failures.

   • Recruiter bots possess broad access to applicant tracking systems, calendars, and messaging tools.
   • Custom classifier models analyze past session history to detect when agents deviate from their core purpose.
   • Tightly scoping write permissions prevents a compromised agent from poisoning shared enterprise data repositories.
4. 12:40 - 16:30

   Error Loops and Cost Management

   Poorly written integrations create continuous error loops where agents consume tokens trying to work around broken API responses. Furthermore, switching model versions without rigorous evaluation causes sudden financial inflation across enterprise accounts. Controlling these expenditures requires strict budgets, rate limiting, and visibility tools that identify where high-intelligence models are genuinely required.

   • Non-developers building custom Model Context Protocol servers often trigger expensive retry loops due to formatting errors.
   • Switching underlying model tiers without evaluation increases operational costs by 100x within a week.
   • Centralized monitoring tools enable organizations to enforce budgets and intervene in high-risk agent actions.