Don't Approve the AI Model. Approve the App - Doneyli De Jesus, Langfuse

Highlights

• Regulated financial institutions certify applications rather than individual AI models.

• A baseline model scoring 97 percent in grounding failed significantly on financial math and numerical accuracy.

• Integrating a deterministic Python calculation tool elevated numerical accuracy from 60 percent to over 90 percent.

• Evaluating applications across multiple dimensions prevents failures in binary areas like regulatory compliance.

• Establishing a structured evaluation pipeline reduced risk and compliance assessment times from months to minutes.

Timeline

1. 00:00 - 05:00

   Context and Real-World Requirements in Regulated Industries

   Public AI benchmarks fail to reflect the internal data and specific tasks utilized by major financial institutions. Developing tools such as an earnings call analyst requires integrating internal firm context that models lack out of the box. Security and privacy mandates require keeping sensitive workloads within localized data centers and air-gapped environments.

   • Financial institutions prioritize internal data sensitivity and proprietary use cases over public benchmark scores.
   • Infrastructure requirements mandate on-premises data centers, gated access, and air-gapped deployments.
   • Production deployment depends on validating compliance requirements and user adoption metrics.
2. 05:00 - 11:35

   Application Testing Framework and Golden Datasets

   Systematic evaluation requires establishing a golden dataset comprising inputs, expected outputs, and metadata. Running test cases generates traceable evidence that allows teams to compute scores across distinct dimensions. While a model demonstrated 97 percent grounding accuracy for internal documents, it struggled with financial math because large language models operate as next-token predictors rather than native calculators.

   • Golden datasets act as standard software engineering test cases for operationalizing AI evaluation.
   • Execution traces provide explicit evidence of model performance against selected evaluation criteria.
   • Models can achieve high performance in grounding while failing basic calculations due to inherent limitations with numbers.
3. 11:35 - 18:00

   Tool Integration and Multi-Dimensional Scoring

   Providing a dedicated calculation tool allowed the system to bypass native math errors and elevate numerical accuracy from 60 percent to over 95 percent. Effective quality gates require balancing code-based checks with qualitative assessments of the path taken by the agent. Because regulatory compliance operates on binary thresholds, failing a single compliance metric blocks production promotion regardless of aggregate scores.

   • Equipping agents with deterministic tools like Python code execution resolves calculation flaws.
   • Comprehensive evaluations combine deterministic scoring, path verification, and language model judges.
   • Domain experts must define distinct passing thresholds for regulatory compliance and risk dimensions.
4. 18:00 - 19:12

   Human-in-the-Loop Feedback and Core Principles

   Structuring an annotation queue enables human experts to examine failure modes and refine evaluation criteria over time. This iterative process cuts risk evaluation time from weeks and months down to minutes. The overarching strategy relies on proprietary benchmarks, application-level harnesses that permit seamless model swapping, and active collaboration with compliance stakeholders.

   • Curation queues route compliance failures directly to domain experts for review and feedback.
   • Reviewed edge cases feed back into the golden dataset to strengthen future evaluation cycles.
   • Organizations must build custom benchmarks, gate applications instead of models, and involve risk teams early.