What a $900M Bank Error Teaches Us About AI Agents - John Feminella, Two Sigma
Key Takeaway
Deploying autonomous AI agents safely in enterprise environments requires a composable set of controls—including registries, risk-based reviews, tool guardrails, shared evaluations, and automated runtime classifiers—that enable speed without sacrificing governance.
Highlights
A Citibank processing software error in August 2020 mistakenly sent the entire outstanding loan principal of approximately $900 million instead of $8 million in interest to over 300 lenders.
An agent registry makes agents legible by recording key facts such as declared tools, target endpoints, product association, and ownership.
Allowing agents to declare their purpose and required tools upfront provides a granular alternative to blanket blocks or unrestricted access.
Independent conservative classifiers handle routine execution decisions, sending requests to humans only when ambiguity or significant risk arises.
Timeline
When Controls Fail: The $900 Million Lesson
Financial institutions operate under strict regulations and complex controls, making failures valuable case studies for understanding risk. In August 2020, Citibank processed a Revlon loan payment using software named FlexCube. Due to unclicked checkboxes and overridden warning prompts, the system distributed the entire outstanding loan principal to 300-plus lenders instead of routing it to an internal wash account. This incident demonstrates how human operators can mistakenly bypass safety interventions when routine actions mask underlying errors.
- A catastrophic software error at Citibank transferred $900 million of loan principal to lenders instead of scheduled interest.
- Enterprise software controls failed when human operators overrode warning notifications because the transaction matched expected interest payment behavior.
- Lenders refused to return the funds because receiving full principal repayment on troubled debt was highly favorable.
Balancing Safety and Autonomous Innovation
Managing AI agents parallels managing human workers who are also non-deterministic and prone to mistakes. Completely locking down agents or relying solely on un-enforceable policy text fails to provide practical utility. Effective architectures establish a balance where safety and innovation coexist through structured boundaries and system controls.
- Non-deterministic AI agents mirror human operations, requiring managed boundaries and visibility rather than total isolation.
- Sandboxing limits agents too severely, while complete freedom creates unacceptable enterprise risks.
The First Three Controls: Registry, Risk Triage, and Guardrails
To govern diverse AI agent implementations across teams, organizations deploy an agent registry that tracks what each agent does and who owns it. Because a simple ticket summarizer carries vastly different risks than a production deployment agent, risk-based categorization ensures reviewers are not overwhelmed. Furthermore, runtime environments enforce granular security policies based on the specific tools each agent explicitly declares during initialization.
- An agent registry records ownership, tool access, and endpoints to eliminate shadow IT and ensure governance visibility.
- Differentiating low-risk tasks from high-risk production actions prevents review queues from becoming unmanageable bottlenecks.
- Agents declare their required tools upfront, enabling managed environments to enforce precise host and endpoint access policies.
Evaluations and Runtime Classifiers
Just as software requires unit tests, AI models require shared evaluation systems to monitor runtime dependencies and output quality. To prevent human review fatigue and context loss during parallel agent execution, independent classifiers evaluate routine calls automatically. Humans only review high-impact or ambiguous requests, ensuring safe operations without stalling velocity.
- Shared evaluation frameworks act like unit tests, verifying output quality when models or prompts change.
- Human-in-the-loop review fatigue leads to dangerous rubber-stamping if every minor action requires manual approval.
- Independent conservative classifiers handle routine calls automatically, routing only ambiguous or dangerous actions to human reviewers.
Description
In 2020, Citibank accidentally sent about $900 million to Revlon’s lenders instead of an $8 million interest payment, even though the procedure, the warning pop-up and three approvers all did what they were supposed to. John Feminella, who works on AI architecture and engineering at Two Sigma, uses that story to show how to give AI agents real autonomy without locking everything down. He walks through five composable controls: an agent registry, risk-based review, tool access that follows an agent’s declared purpose, shared firm-wide evals, and classifiers that send humans only the approvals that matter. Views are his own, not Two Sigma’s. Chapters: 0:00 Intro 0:43 When controls fail 1:03 The $900M Revlon payment mistake 4:58 What this has to do with AI agents 5:22 Humans aren’t deterministic either 6:07 Approaches that don’t work 7:32 Control 1: An agent registry 9:47 Control 2: Risk-based review 11:47 Control 3: Tools follow purpose 14:42 Control 4: Shared evals 16:17 Control 5: Ending approval fatigue 18:07 Putting it all together 18:32 Why cars have brakes John Feminella https://www.twosigma.com Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!