Backtest Your Agent Against Yesterday's Web — Ryan Ahern, Exa

Highlights

• Exa provides agentic web search used by major coding agents and applications like Cursor.

• Exa Snapshot covers an index of approximately 100 billion documents tracking trillions of URLs across the web.

• Content versioning and version search allow users to query webpages and execute searches as of a specific date in the past.

• Live search results constantly update article publish dates for SEO purposes, which leaks future knowledge into past evaluations.

• Workshop participants tested binary features over a 30-minute exercise to predict company acquisitions using historical snapshots.

• Overfitting remains a significant risk in predictive models when backtesting without a dedicated holdout set.

Timeline

1. 00:00 - 04:12

   Introduction to Agentic Web Search and Exa

   AI agents require external search tools like Exa to answer questions outside their internal knowledge windows. Exa balances quality, cost, and latency across various architectural requirements, ranging from 300-millisecond queries to long-running batch searches for thousands of entities.

   • AI systems search the internet more frequently and efficiently than human users.
   • Exa provides search infrastructure for major coding agents and financial applications.
   • Hedge funds, asset managers, and venture capitalists leverage Exa to source alpha, forecast risk, and find startups.
2. 04:12 - 11:41

   Exa Snapshot and Historical Version Control

   Article publish dates frequently change due to genuine updates or SEO optimizations, making historical analysis unreliable without version control. Snapshot prevents future data leakage during backtesting by constraining search visibility to information available prior to a target event, such as an corporate acquisition or financial disclosure.

   • Exa Snapshot enables users to view web pages and execute searches as of a specific date in the past.
   • The index tracks approximately 100 billion documents and trillions of URLs.
   • Live search indices leak future knowledge by displaying updated information and modified publication dates.
3. 11:41 - 17:05

   Backtesting Patterns and Model Integration

   Developers deploy backtesting through three primary patterns: gathering static historical website data, adding agentic search features to traditional machine learning models, or building fully agentic prediction engines. Freezing external repositories and data sources right before specific events ensures that performance improvements stem from model weight updates rather than future information leakage.

   • Traditional backtesting models gather static historical features without live search execution.
   • Agentic features convert unstructured web data into structured signals for investment models.
   • Fully agentic prediction systems combine internal knowledge and external web searches at runtime.
4. 17:05 - 26:34

   Hands-On Acquisition Prediction Exercise

   Participants used Exa Snapshot to test binary prediction features against historical company data across 1-day, 7-day, and 30-day lookback windows. Reviewing top submissions revealed that without a holdout set, participants could easily overfit their search queries to specific companies in the evaluation dataset.

   • Workshop participants competed in a 30-minute exercise to identify features predicting company acquisitions.
   • Scoring awarded one point for correct positive predictions 1, 7, or 30 days prior to acquisition, and deducted one point for false positives.
   • Top-performing participants demonstrated the danger of model overfitting when testing without a holdout set.