Do Not Let Models Invent Numbers - Vinoo Ganesh, Kepler

Highlights

• Language models operate as probability machines, making them unreliable for mission-critical numerical tasks without strict architectural constraints.

• Native RAG tools and vector databases fail in financial analytics by producing thousands of unverified or incorrect citations.

• Kepler uses an architecture where language models plan and cite numbers, while deterministic code handles all arithmetic and calculations.

• The platform processes over 30 million SEC documents and registers citable numerical anchors to ensure full reproducibility.

• Any citation that cannot be verified against trusted sources is aggressively stripped out by a post-processing filter to prevent hallucinations.

Timeline

1. 01:46 - 04:27

   The core challenge of numerical accuracy in language models

   Financial analysts operate under strict accuracy standards where unvalidated numbers are unacceptable. Although large language models excel at creative tasks, their probabilistic nature causes severe reliability issues when applied to finance. Attaching sources after the fact often fails because semantic search retrieves semi-relevant documents or pulls numbers out of context, such as confusing page numbers, currencies, or historical fiscal years with current data.

   • Financial analysis requires numbers that can be explicitly traced and validated.
   • Language models are probability machines that create challenges in mission-critical numerical situations.
   • Traditional source attachment fails because models perform semantic search rather than reading documents, leading to wrong sources or incorrect numerical contexts.
2. 04:27 - 08:31

   The Kepler architecture: Retrieval, code, and post-processing

   The platform solves numerical inaccuracy by implementing a rigid division of labor. Retrieval systems pull verified sources, pre-written deterministic code handles calculations, and language models manage the overarching workflow. By preprocessing massive corpuses of SEC filings and establishing a citation registry, the system ensures every displayed number maps back to an auditable source document.

   • Kepler processes over 30 million SEC documents to register citable numerical anchors.
   • Deterministic code performs all math while language models handle high-level planning.
   • Unverifiable citations are completely refused by the platform to prevent hallucinated outputs.
3. 08:31 - 13:33

   Hybrid search infrastructure and tenant isolation

   Operating effectively requires integrating public filings with internal corporate documents stored across various enterprise platforms. The search infrastructure avoids embedding bottlenecks by searching wide with approximate nearest neighbors and BM25, followed by cross-encoder re-ranking. Security is enforced at the document level, ensuring that internal research and proprietary data remain strictly isolated between different teams.

   • Hybrid search combines approximate nearest neighbors and BM25 across public filings and internal enterprise data.
   • Cross encoders re-rank retrieved results to de-conflict discrepancies between different document types.
   • Every search hit undergoes strict authorization checks in Postgres to maintain tenant isolation and data security.
4. 13:33 - 20:54

   Evaluating native RAG failure rates and enforcing deterministic guarantees

   Standard vector database and RAG implementations fail rigorous financial evaluations, generating thousands of incorrect numbers that make them dangerous for auditing. To eliminate this risk, the platform enforces a strict invariant where models can never write numbers they cannot cite. While this prevents the model from acting as a speculative thought partner, it creates a reliable ground truth engine capable of automating tedious financial data extraction.

   • Standard evaluation tests reveal that native RAG setups generate thousands of straight-up wrong numbers.
   • Post-processing filters strip out unverified citations to enforce structural incapability of producing hallucinated metrics.
   • The architecture shifts the industry toward quantum-mental workflows where AI handles tedious data extraction instead of creative ideation.