Your Agent Fix Just Broke Something Else - Sofia Jakovcevic, Arize AI

Highlights

• A simple one-line prompt change to add confirmation steps to a support agent generated a 17-point improvement in action safety alongside a 17-point degradation in task completion.

• Traces serve as the new source of truth because agent logic executes at runtime rather than existing solely within static code.

• A five-step regression testing loop involves selecting production datasets, building evaluators, running baseline experiments, testing modifications, and automating checks within CI pipelines.

• Coding agents connect directly to observability platforms using agent skills to bridge local code generation with live production telemetry.

Timeline

1. 00:00 - 05:00

   The Hidden Costs of Prompt Modifications

   AI systems operate as black boxes during execution, making runtime behavior difficult to predict without proper observability tools. A security modification requiring user confirmation before agent actions improved safety scores by 17 points but decreased task completion by an identical 17 points because users abandoned incomplete workflows. Numeric evaluation suites prevent engineering teams from shipping changes based on intuition.

   • Observability uncovers systemic agent behavior that static code reviews miss.
   • Simple prompt adjustments frequently trigger unintended performance trade-offs across different operational metrics.
   • Evaluating single dimensions like safety without tracking task completion yields a net-zero overall performance impact.
2. 05:00 - 06:27

   Tracing Runtime Execution Versus Static Code

   Traditional software logic remains explicit prior to production deployment, whereas agentic systems generate logic dynamically at runtime. Modifying a single markdown system prompt alters how agents interact with external tools and other sub-agents. Observability platforms bridge the gap between written code and actual runtime behavior by storing structured traces.

   • AI agents fail during runtime rather than at the traditional code level.
   • Local code diffs fail to capture thousands of potential runtime trajectories involving tool calls and sub-agent communication.
   • Structured log traces accurately represent live system operations.
3. 06:27 - 11:51

   Building the Five-Step Regression Loop

   Establishing a robust regression testing process requires pulling real production traces rather than relying on synthetic data. Evaluators automate the scoring of non-deterministic metrics like hallucinations and step efficiency using binary or multi-point classification models. Baseline experiments compare baseline metrics against modified prompts, model changes, or tool updates before code reaches production.

   • Production datasets incorporate real queries, known failures, edge cases, and successful workflows to protect existing functionality.
   • Evaluators leverage language models as judges to convert subjective metrics into quantifiable numeric scores.
   • Automated regression experiments run inside CI pipelines to block pull requests that fail established performance thresholds.
4. 11:51 - 16:15

   Executing Experiments and Connecting Coding Agents

   Platforms like Arize allow engineers to execute comparative experiments inside user interfaces or programmatically via software development kits. Remote agent connections evaluate how multi-agent architectures communicate across different configurations. Connecting coding agents with external observability backends allows developers to unite local code generation with live production telemetry.

   • Prompt playgrounds and agent playgrounds test single-agent prompts or complex multi-agent remote systems side by side.
   • Agent skills connect coding agents directly with external databases and observability platforms.
   • IDE coding workflows integrate runtime observability data to evaluate code changes before submission.
5. 16:15 - 19:15

   Closing the Loop and Self-Improving Agents

   Continuous improvement relies on feeding failed production traces back into the observability layer to strengthen future agent performance. Human oversight during PR reviews limits scaling velocity, shifting the focus toward agent-driven self-improvement. Equipping coding agents with runtime trace data enables them to understand production outcomes and autonomously implement corrections.

   • Every production failure transitions into a new test case within the continuous regression testing suite.
   • Manual dashboard reviews create operational bottlenecks that slow down agent iteration speeds.
   • Future observability focuses on agents using runtime telemetry to investigate, evaluate, and improve themselves autonomously.