Building the AI Agent Is Easy. Proving It's Safe Isn't - Jonathan Osborne, NatWest

Highlights

• NatWest serves approximately 19 million retail customers through its digital assistant, Cora, which handles around 100,000 conversations daily.

• The engineering team implements a multi-layered routing funnel that filters out about 20 percent of common events via contextual filters before invoking any machine learning models.

• Human validation across 111 payment conversations reveals a significant discrepancy, showing a 97 percent pass rate for happy path scenarios but only 43 percent for topic changes.

• Analysis of 14,000 complaints agent cases reveals that LLM-as-a-judge models inherently exhibit a strong bias toward overly generous scoring with minimal spread.

• Shadow deployment enables the bank to capture real production data and evaluate model responses with zero risk to actual customers.

Timeline

1. 00:00 - 03:41

   Scaling AI in a Regulated Retail Bank

   Building an AI agent is straightforward with modern coding assistants, but proving its safety in a high-stakes financial setting remains challenging. A failure in customer interactions impacts real money and invites strict regulatory scrutiny. Traditional scripted decision trees fail when customers alter topics mid-conversation, such as switching from mortgage inquiries to account closures.

   • NatWest operates as the second largest retail bank in the UK with 19 million retail customers.
   • Cora, the primary digital assistant, manages roughly 100,000 conversations per day using primarily scripted answers.
   • The organization transitions approximately one-third of customer journeys to AI-generated answers backed by RAG and knowledge bases.
2. 03:41 - 06:38

   Failure Modes and the Routing Funnel Architecture

   Traditional software engineering assertions and regression tests fail against non-deterministic AI outputs. To mitigate risk, the routing architecture enforces access control, guarantees complete auditability, maintains resilience during model outages, and stores configurations in version control for rapid market adaptation.

   • Agentic systems introduce four primary failure modes: confident hallucinations, prompt injection, unhelpful loops, and non-deterministic behavior.
   • The routing funnel applies contextual filters to remove common button clicks, accounting for about 20 percent of traffic.
   • Deterministic pre-classification rules direct utterances without machine learning before deploying neural network intent classifiers and policy rules.
3. 06:38 - 10:35

   Evaluating Performance with Human Validators and LLM Judges

   Relying on a single quality score masks critical failures in edge cases. When automating evaluation using an LLM as a judge, models tend to be overly generous and show near-zero correlation with human error distributions unless properly calibrated and decoupled from the primary agent model family.

   • Performance scoring splits into a vector across three distinct categories: safety, task resolution, and customer experience.
   • Human validation across 37 payment journeys shows an 83 percent overall pass rate, dropping to 43 percent for topic changes.
   • Human validators achieve 95 percent agreement on objective metrics like bias but only 64 percent agreement on subjective intent understanding.
4. 10:35 - 15:06

   Production Infrastructure, CI/CD, and Deployment Strategies

   Continuous evaluation operates within an air-gapped environment with strict zero-tolerance policies for personal identifiable information. Traces are captured via OpenInference standards and stored in Comet. Input feature drift monitoring acts as an early canary indicator for model degradation before f1 scores drop.

   • Live conversations are sampled at a default rate of 5 percent alongside 100 percent of failed or escalated sessions.
   • CI/CD pipelines enforce hard regression gates that block prompt or context updates from merging without baseline comparisons.
   • Shadow deployments process live customer utterances to test classifications against the existing scripted solution with zero customer risk.
5. 15:06 - 17:52

   Feature Rollouts and Enterprise Governance

   Continuous evaluation and rigorous validation frameworks transform deployment from a high-risk big-bang release into controlled, measurable steps. Enterprise AI success depends less on the initial code creation and more on building a reusable, robust evaluation spine.

   • Brand-by-brand percentage rollouts use automated and manual kpis to govern progressive deployment.
   • Global and dimension-specific feature flags allow engineers to route traffic back to scripted solutions in seconds.
   • The internal complaints agent requires mandatory human feedback before handlers can rerun investigations, building a high-quality dataset of 14,000 labels.