1,368 AI Failures and What They Teach About Evals — Vishal Srivastava & Tanmay Sah, New York Life

Highlights

• AI failures in foundational models have increased exponentially with a nearly vertical jump over the last four years.

• A structured framework consisting of 13 sub-dimensions across development, deployment, and business context covers almost 98 percent of AI failures.

• On average, six failure sub-dimensions are present simultaneously in each investigated AI incident.

• Deployment controls are more critically associated with harm than technical model failures or business context alone.

• Approximately 79 percent of people tested wrong in Wason's experiment due to confirmation bias, a pattern inherited by AI systems trained on human data.

Timeline

1. 00:00 - 06:56

   Mapping AI Failures and Dimensions

   Analyzing data from the public AI incident database reveals that enterprise use cases outpace manual review capacity, requiring a structured investigative approach. Technical strength encompasses data quality, task design, predictive validity, fairness, failure modes, and explainability. Deployment controls manage decision authority, human mediation, exposure control, and monitoring controls. The final business context dimension accounts for financial impact, legal consequences, and user scale.

   • AI failures across foundational models show an exponentially increasing, vertical jump over four years.
   • AI incidents span three core dimensions: development stage, deployment phase, and business context.
   • Thirteen specific sub-dimensions across technical strength, deployment controls, and business impact account for 98 percent of AI failures.
2. 06:56 - 10:41

   Prioritizing Sub-Dimensions and Materiality

   Plotting sub-dimension frequency against association to harm demonstrates that not all failure modes carry equal weight. Deployment controls and monitoring mechanisms sit on the upper left of the impact plot, showing strong ties to actual harm despite appearing less frequently than structural design factors. Establishing a materiality scale allows organizations to direct review resources toward high-risk AI deployments rather than treating all use cases uniformly.

   • Individual AI incidents involve an average of six failure sub-dimensions occurring simultaneously.
   • Deployment controls such as human mediation and harm detection associate more heavily with harm than scale or system design.
   • A materiality scale rank-orders AI use cases by potential harm to prioritize resource allocation.
3. 10:41 - 17:03

   Building System-Level Evaluations and Falsification

   Traditional machine learning validation on held-out data falls short when applied to autonomous agentic systems that execute action rollouts and tool calls. Drawing from Peter Wason's cognitive experiments on confirmation bias, evaluation frameworks must actively try to break system claims rather than confirm them. This first-principles approach uses an iterative build, break, fix loop to map failure regions and restrict AI usage to high-confidence operational boundaries.

   • Risk decomposition requires evaluating how often a model fails, whether failures reach a decision, remain uncaught, and cause severe consequences.
   • Agentic systems expand testing boundaries to include retrieval quality, tool failures, human review effectiveness, and recovery mechanisms.
   • Evaluation systems must be failure-driven and falsification-based to counteract human and model confirmation bias.
4. 17:03 - 19:51

   Real-World Application and Framework Conclusion

   Applying the build-break-fix framework to a customer service scenario demonstrates how defining strict policy requirements and challenging agent responses exposes flaws early. End-to-end evaluation must integrate technical failure metrics, deployment controls, and business context into a unified stress-testing protocol. Comprehensive risk-based governance ensures AI systems operate only within validated, highly confident regions prior to production deployment.

   • An airline chatbot failure where incorrect post-travel discount advice led to tribunal liability illustrates how narrow evaluations miss real-world risks.
   • Pre-deployment stress testing and policy check integration catch critical failures before customer exposure.
   • Effective evaluations evaluate the full system based on risk and materiality rather than isolated technical components.