How to Build an AI Code Review Agent - Daksh Gupta, Greptile

Highlights

• About 1% of all pull requests on GitHub run through Greptile.

• Running code reviews with coding agents costs between $3 and $25 per pull request.

• Claude models produce one-off errors at three times the rate of GPT models.

• Model inversion between code generation and review yields a 10% higher bug catch rate.

• Only one out of every 1,000 pull requests contains a catastrophic bug.

Timeline

1. 00:00 - 04:21

   Introduction to AI Code Review and Initial Architecture

   Code validation bottlenecks emerge as engineers and non-engineers increasingly generate code using AI tools. While a basic webhook and harness setup can return review feedback within five minutes, it quickly exposes systemic flaws. Unfiltered agents flood pull requests with pedantic stylistic complaints that coding agents blindly attempt to fix, resulting in overly defensive codebases.

   • Greptile reviews and tests pull requests across thousands of companies to catch bugs and logical errors from AI-generated code.
   • Building a basic review agent requires a sandbox environment, an open-source harness, and a GitHub or GitLab webhook to receive pull request data.
   • Initial agent setups encounter major issues including missed bugs, missing codebase context, high token costs, and excessive nitpicky comments.
2. 04:21 - 09:02

   Prompt Optimization and Historical Data Sets

   Prompt modifications cause dramatic fluctuations in comment volume, creating uncertainty about whether critical bugs are being missed. Because catastrophic bugs occur rarely, standard reviews cannot easily prove the absence of errors. Maintaining a real-time updating dataset of past engineering incidents provides a reliable benchmark to evaluate agent performance over time.

   • Cataloging historical outages allows teams to tune prompts to detect specific high-propensity bug types while explicitly excluding nitpicks.
   • Catastrophic bugs appear in roughly one out of every 1,000 pull requests, making it difficult to measure the absence of bugs.
   • Testing agent versions against a historical dataset of past incident-causing pull requests improves review reliability.
3. 09:02 - 14:26

   Model Inversion and Codebase Context

   Matching the review model to the generation model creates blind spots because both models fail in identical patterns. Inverting models—such as reviewing Claude-generated code with a GPT model—captures errors that would otherwise slip through. Furthermore, reducing reliance on imprecise grep commands by maintaining centralized code knowledge bases and living wikis prevents unnecessary token consumption during context retrieval.

   • Claude models generate one-off errors three times more frequently than GPT models, while GPT models produce more authentication bypasses.
   • Pairing a different model for code review than the one used for generation increases bug catch rates by 10%.
   • Standard grep operations consume vast amounts of inference tokens and lack the precision needed for deep architectural understanding.
4. 14:26 - 18:19

   Autonomous Testing and Inference Cost Allocation

   Visual UI bugs often bypass static code analysis, requiring virtual machine sandboxes to spin up application stacks and test user flows visually. Because running frontier models across entire review pipelines drives costs up to $20 per pull request, optimizing inference allocation is essential. Delegating routine retrieval to decision models and smaller background agents cuts operational costs significantly.

   • Configurable virtual machine sandboxes enable end-to-end computer use testing for visual and UI regressions.
   • Allocating smaller, cheaper models to exploratory file reads and lightweight tasks preserves expensive frontier models for critical review steps.
   • Complex review pipelines split tasks across roughly 20 different sub-agents to optimize performance per dollar.