We Kept Rebuilding Our AI Agent. Here's What Survived - Fuad Ali, Arize AI

Highlights

• The in-product AI agent Alex uses custom harness code rather than a commercial agent framework to maintain full control over the loop, plans, stopping rules, and memory.

• A single markdown file capped at 8,000 characters per user successfully replaced complex architectures like vector retrieval and temporal knowledge graphs for long-term memory.

• Moving a to-do list tool into the harness code eliminated wasted LLM iterations where the agent spent most of its time checking off list items.

• Truncating large JSON tool results within the response and allowing the agent to fetch specific data using JQ code prevents context window exhaustion.

• Shifting from ten specialized sub-agents to a single agent that loads skills and tools on-demand simplified the architecture and improved performance.

Timeline

1. 00:00 - 06:19

   Evolution and Architecture of Alex

   The development team spent two years refining Alex from a simple router over skills into a robust production agent. By keeping the execution loop, planning mechanism, and stop rules within custom harness code rather than prepackaged frameworks, the engineering team retained total control over agent behavior across different model providers.

   • Alex is the in-product AI agent inside the Arise AX platform that handles trace analysis, synthetic data creation, and multi-step task planning.
   • Early prototypes in late 2023 suffered from a 50 percent hallucination rate using early era models.
   • The architecture evolved from a basic skills router into a structured system powered by custom harness code without external agent frameworks.
2. 06:19 - 09:43

   Core Design Decisions: Planning and Termination

   Initial attempts to let the agent manage to-do lists failed because the model became trapped in repetitive check-off cycles. The team transitioned the plan into the harness code, rendering current objectives alongside data inputs. Additionally, inconsistent termination behavior across different model providers necessitated an explicit finish gate enforced by the harness.

   • Allowing models to manage their own to-do lists resulted in wasted LLM calls spent updating lists rather than performing actual work.
   • Moving the plan into the harness code ensured the model received updated state context with each new call.
   • An explicit finish call enforced by the harness prevents infinite loops when models fail to terminate automatically.
3. 09:43 - 13:33

   Managing Context Window and Long-Term Memory

   Large tool outputs, such as detailed agent traces, quickly overwhelm context windows if loaded directly. The system addresses this by truncating large results and letting the agent query specific data via code-like tools. For long-term memory, complex solutions like knowledge graphs were abandoned in favor of a straightforward, human-auditable markdown file managed by background scripts.

   • Treating massive tool results like external files instead of dumping raw JSON into the context window prevents context blowups.
   • A single markdown file capped at 8,000 characters per user outperformed vector stores and temporal knowledge graphs for long-term memory.
   • Background jobs update the user memory file after a session concludes, keeping the data auditable and safe.
4. 13:33 - 15:26

   Simplifying Specialists and Selecting Models

   Maintaining ten separate specialist sub-agents created unnecessary complexity as model capabilities improved. The team streamlined the platform into a single agent capable of dynamically loading required skills. Rather than chasing every new frontier model, engineering efforts focus on optimizing harness code and utilizing smaller single-call models for isolated tasks.

   • The system consolidated ten page-specific specialist sub-agents into a single unified agent that loads necessary skills on demand.
   • The newest frontier model is not always the default choice, favoring fast and reliable models paired with strong harness logic.
   • Narrow tasks like filter generation bypass the agent loop entirely in favor of single model calls.
5. 15:26 - 19:57

   Testing, Evaluation, and Permanent Best Practices

   Because agent bugs rarely appear as traditional software errors, the team relies on automated tools like Signal to flag anomalous behaviors such as redundant retry loops. Production sessions continuously expand the regression test suite. Six core practices, including executing math inside code rather than raw text and maintaining golden datasets in CI, survived every major architectural iteration.

   • Live production sessions automatically feed into regression datasets to serve as living test fixtures.
   • Agent bugs manifest as qualitative issues like repetitive tool loops rather than explicit syntax errors.
   • Core principles that survived all architectural rebuilds include trace monitoring, math execution via code, and CI regression datasets.