A Bank Trained Its Own Foundation Model to Catch Fraud — Freddy Lecue, Wells Fargo
Key Takeaway
Training a custom 1-billion-parameter transformer foundation model from scratch on billions of customer transactions outperforms traditional XGBoost models by boosting fraud detection rates and reducing false positives while maintaining sub-250-millisecond inference latencies.
Highlights
Wells Fargo built a custom 1-billion-parameter foundation model trained from scratch on billions of customer financial transactions.
The transformer architecture processes five core dimensions—who, what, how, when, and where—across hundreds of raw tabular features.
The custom transaction model surpasses traditional XGBoost performance by increasing fraud detection rates while reducing false positives.
Inference processing runs well under the 250-millisecond threshold required for transaction decline decisions.
The single backbone model streamlines multiple downstream tasks, including fraud detection, credit line increases, anti-money laundering, and marketing.
Timeline
Motivation and Shift to Transaction Foundation Models
Traditional large language models process text tokens, but financial institutions primarily handle structured tabular data from credit cards, debit cards, and checks. Rather than relying solely on siloed XGBoost models, Wells Fargo developed a transformer-based foundation model to capture patterns across all customer transactions. This unified backbone supports downstream tasks like credit scoring, fraud detection, and marketing.
- Financial organizations possess massive volumes of structured tabular transaction data across multiple banking channels.
- A single transformer backbone replaces siloed models by learning customer transaction representations at scale.
- The custom model outperforms existing production XGBoost solutions in both latency and fraud detection accuracy.
Data Scale, Dimensions, and Tokenization
The training process ingests billions of transactions monthly, involving hundreds of transactions per client. Instead of relying on heavily engineered features, the team focuses on raw feature units across five core dimensions representing who, what, how, when, and where. Converting tabular variables—such as transaction amounts, time deltas, and postal codes—into effective tokens requires careful vocabulary design to avoid overly massive or messy token spaces during training.
- The training dataset spans billions of monthly transactions across millions of accounts.
- Input features are structured around five primary dimensions: participant, product, channel, timestamp, and location.
- Complex tokenization maps heterogeneous numerical and categorical variables into standard embedding lookups.
Model Architecture and Pre-training Strategy
Guided by scaling laws adapted from open solutions, the team built a 1-billion-parameter decoder-only model optimized for predicting subsequent transactions. Pre-training runs from scratch on a dedicated GPU cluster over several weeks using causal and masked modeling techniques. Engineering practices like mixed precision, macro-batch sizing, and activation checkpointing ensure efficient handling of large-scale sequence histories.
- The initial pre-trained model contains 1 billion parameters, with an 11 billion parameter version currently in development.
- A decoder-only architecture pre-trains from scratch on a GPU cluster using causal language modeling objectives adapted for transactions.
- Training utilizes mixed precision, distributed data parallel, and flash attention to optimize memory efficiency and convergence speed.
Downstream Performance, Results, and Business Impact
Applying the pre-trained backbone to fraud classification allows the model to evaluate current transactions against complete historical sequences through attention mechanisms. This temporal awareness catches more fraudulent activity while lowering false positive rates compared to standard production XGBoost classifiers. Meeting strict latency requirements under 250 milliseconds enables real-time decision-making, translating directly into millions of dollars in operational savings and preserved customer transactions.
- Fine-tuning the foundation model for fraud classification significantly improves detection rates over XGBoost.
- The architecture successfully maintains inference processing times well below the required 250-millisecond window.
- Reduced false positive rates prevent unnecessary transaction declines, saving the firm millions of dollars in operational costs.
Description
Freddy Lecue explains how Wells Fargo pre-trained its own transformer from scratch on billions of customer transactions instead of words. The model learns to predict the next transaction, and that one backbone can then be fine-tuned for fraud detection, credit line decisions, anti-money laundering and marketing. He covers the hardest part, turning tabular data like amounts, timing, merchants and zip codes into tokens, plus the model's size and training setup. On fraud, it caught more fraudulent transactions with fewer false alarms than the production XGBoost model, while staying fast enough for real-time card approvals. Chapters: - 00:00 Why a foundation model for transactions - 02:14 Why not just XGBoost? - 02:47 Billions of transactions - 03:22 Language models vs. transaction models - 04:45 Pre-training from scratch - 05:58 Who, what, how, when and where - 07:59 Turning tabular data into tokens - 09:42 Architecture and scaling - 12:10 Training setup - 13:44 Fraud detection results - 16:47 Recap Freddy Lecue, Wells Fargo: https://www.wellsfargo.com Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!