Building uReview, Uber’s Multi-Agent Code Review Engine — Will Bond & Ameya Ketkar, Uber
AAI Engineer
컴퓨터/소프트웨어경영/리더십AI/미래기술
스크립트
00:00:00All right. Hello everyone. My name is Will and I'm here to talk to you about automated code review.
00:00:20My teammate Amaya and I work at Uber and we're gonna be walking through uReview, a system that
00:00:27Uber has built to help increase the velocity of our software engineering teams. For a little
00:00:34bit of context about what software engineering org at Uber looks like, we have thousands of
00:00:39software engineers who work across hundreds of teams located across 12 different sites
00:00:46and they work in primarily one of six language-specific monorepos. As many of you have probably noticed
00:00:53over the past 24 months, the volume of PRs, the size of PRs has been growing. One of the ways that
00:00:59that's been exposed to us has been through the metric that we track of the first time to review. Back in
00:01:052024, we were seeing that engineers would get their first review within three hours. Now in 2026 that has
00:01:12grown to nine hours in addition to all of the volume changes. So in short, code review is now the bottleneck
00:01:19that we are running into. Specifically around automated code review, there are various options available in the
00:01:27industry, but Uber spent the time to invest in building an in-house solution due to some of the constraints
00:01:33that we have. One of those is we currently use Fabricator and have for a long time and are in the process of
00:01:40migrating to GitHub. Most of the solutions do not provide support for Fabricator. In addition, if you were at the
00:01:47previous talk, you saw Uday and Adam talking about the agentic SDLC. A big part of what we want to do is bring a
00:01:54consistent code review experience to the inner loop so that our agents are getting the same code review, the same rules,
00:02:00everything applied as our humans do. With hundreds of teams across the company, we can't have centralized
00:02:08management of our code reviews, our customizations, our rules, and even the knowledge that goes into those
00:02:14code reviews, we need to distribute that. So we have a need for plugging into an existing team ownership
00:02:21system, rather than trying to replicate that externally. Finally, with the volume of code reviews that we
00:02:28perform, we need the ability to take factors like the risk profile and the complexity of a code change
00:02:36change, and factor that in when deciding how we're going to run a code review. Not all code gets the exact
00:02:42same review. And then finally, consistency. We need to make sure that we have security and compliance
00:02:47reviews run across everything. We can't rely on teams hoping to run the skill, the code review skill
00:02:54that happens. We need reliability there. With all that said, I wanted to give you an overview of the
00:02:59architecture of what you review looks like. We'll talk about a couple of the big pieces. And then we're going to dive into a
00:03:05few focus areas. At the top, you'll notice that we have our code review surface areas, GitHub,
00:03:12fabricator, and the agent loop. These all feed into you review service. This takes in requests for reviews,
00:03:21it brings in feedback from users, and it routes it. We have a number of different generators. Now, these generators
00:03:28are tuned for different performance and cost avenues. There are -- we also have the ability to plug into third
00:03:36party code review systems so that we can compare ourselves to what's available more broadly. Finally, with all these
00:03:44different generators, we might be duplicating comments, and we can actually create quite a high volume of
00:03:51comments. If you've ever used AI to run a code review, you've probably seen that. So, we run through a number
00:03:57of steps in the post-processing where we both rate, categorize, filter, and deduplicate comments so that our
00:04:04engineers get only the highest confidence comments that are actionable for them to work on. You'll also notice
00:04:11that, along the bottom, we talk a little bit about feedback in our evaluation. But with this context of
00:04:16the overall system, I'm now going to hand it off to Amaya to dive into our first focus area.
00:04:24Hello. Hello, everyone. So, I will be talking about how we evolve Ureview with observability and
00:04:31evaluation. So, Ureview had a very humble beginning. Basically, it was a single prompt that you should do logic
00:04:38checks per file, a simple agent which used to do thorough review. And we had a dispatcher to decide
00:04:44whether to go which generator to choose. Even what we used to collect as observability was very surface
00:04:51level. We used to collect cost. We used to run an NPS survey, have Google forms being filled, Slack
00:04:56support. And with all of this, we saw that our quality to cost ratio was like all over the place. Like,
00:05:03our goal is to be in the second quadrant. That is the top left quadrant. But you can see we were all over
00:05:10the place. Then what we did is that we started collecting more data. So, we started collecting the
00:05:18sentiments of the replies that were made to the Ureview, that the Ureview, you know, the Ureview agent got from the
00:05:27developers. So, we categorized them into positive, negative. We classified them into various categories.
00:05:34And we found a bunch, a lot of classes of bugs and issues that we could actually solve. And with that,
00:05:40we improved the system. And we were able to move a large number of PRs to a high quality to cost ratio.
00:05:48But we still felt that this was not enough. We need to know more of how the review is done. So,
00:05:54we started tracking things like addressal rate. So, basically, when a Ureview comment is made,
00:05:59does the developer go and actually address the comment? We started tracking that. And then we also
00:06:07started doing more like a runtime profile, which is like the agent trajectory, which told us why the
00:06:14agent is doing what it did. We get to know what tool calls it made. We get to know what thinking process
00:06:21it had. And then with that insight, we were able to actually tune our runtime, tune our performance,
00:06:27such that the agent could very quickly give us high quality results at a low cost.
00:06:35One of the biggest learnings in this process was like the model doesn't know that it's wrong.
00:06:39It always confidently says 100% sure that, yeah, this is the review for your code, go ahead. But we saw that,
00:06:46no, it actually needs a lot of guidance from the teams because each team has its own style guide,
00:06:52its own patterns or anti patterns that they want to look for. So, that all should be like baked into
00:06:59the agent. And we also realized that we need to have guardrails for the agent. So, we need to tell
00:07:05the agent what not to waste turns doing. Like code review is something that has to happen in like a
00:07:11specific time span. And then if it starts spending time doing things that it should not be doing,
00:07:17leads to a bad quality code review. Second focus area for you review has been
00:07:24We went very deep on team customizations because as Will presented that we have hundreds of teams and
00:07:32everyone has like their own way or their own thing for code review. So, our review stack is pretty
00:07:38straightforward. We have single file reviewers and multi-file reviewers. We basically do a general
00:07:44purpose, hey, find me all logic bugs per file kind of a review. And then we also do a deep review because
00:07:52we have like six mono repos. So, all these mono repos have their own anti pattern style guides and all
00:07:57baked into this agent review, which does a nice multi-file review. But then we extended it further
00:08:04basically to AI linters. These are basically few short AI prompt or like a few short system where
00:08:14developers can basically kind of deterministically get more context and then run rules with that context and
00:08:21like a file and find some systematic and mechanical issues. And finally, the most powerful thing is the
00:08:29custom agent where the teams could basically define their own custom agent, link it to like a knowledge
00:08:35base, link it to their past PRs, have like a skill to do the review and so on. But all of this was not
00:08:43simple because we had to actually piggyback on our ownership model, which is at Uber, so that we can like very
00:08:50logically roll out to all the teams. We had to basically do a, what do you say, co-locate the
00:08:58customizations next to where the developers write their code so that they can like quickly
00:09:03keep updating these customizations. We had to implement a smart deterministic routing so that we could route which
00:09:11team gets what kind of review with which model, what kind of generators and so on. And finally, the hard thing was like we had to actually surface all of the
00:09:20all of this observability that I talked before, like the agent trajectory, address and rate, sentiment analysis back to the teams
00:09:27so that the teams could actually understand that, oh, I wrote this rule, but maybe not a lot of developers are liking it in my team, so let me go and update it. And then we had to give bubble up that kind of observability to all the people who are contributing to the platform.
00:09:42So one thing that we learned is that actually writing the skill was very easy, like teams just very quickly wrote a skill by asking Claude to write one, go over my previous PR reviews and write me a skill, but the hard part was how to run these skills at scale with consistent quality and low cost.
00:10:04And that required a lot of iterations not only from the review team side, but also like for each team who was trying to write these rules.
00:10:11In results, we basically see that, you know, you review does like around 25,000 comments a week, and we get 10% of them actually get some feedback and only 4% of the PRs actually get some negative feedback.
00:10:27We also saw that the overall address rate was around 67%, and almost three quarters of the high severity issues were usually addressed by the developers, which shows that you review actually adds some value to the entire development lifecycle.
00:10:45And then with all the observability and evils that I showed that I went through, we saw that against like a very naive implementation, our costs were down by 60% and our quality and our accuracy was up by around 70%.
00:11:00For our last focus area, I'll give the mic back to Will, and he will go over the inner versus outer loop.
00:11:07Awesome. So, now that we've talked about some of the details of actually implementing high-quality reviews, it kind of brings us to the last area, which is where we start talking about where things are going, right?
00:11:25With moving to the agentic SDLC, we're moving software into a model where engineers are interacting with the code less.
00:11:35We're oftentimes not as involved in authoring the code.
00:11:38Currently, we still have humans approving the code, but we see a short path in the near future to a percentage of our code landing automatically, having automatic approvals, right?
00:11:52There's various parts of the industry are already moving there.
00:11:55There's a lot of work.
00:11:56Part of the way along the process was figuring out by having our single code review platform, what did we need to tune for the various audiences that are actually getting these code reviews?
00:12:09You know, the interface, that's one area that's sort of intuitive there.
00:12:13One thing that might be less intuitive is around accuracy.
00:12:17With the inner loop, our accuracy needs actually need to go up or else we can result in dealing with cavitation of an agent where it fixes something, goes back, gets another code review, and has to kind of like fix backwards because the quality of the comment was low.
00:12:33One of the other interesting things is agents are more than happy to go through and fix 100 nits on a pull request where your engineers really get frustrated in situations like that.
00:12:46But probably the most interesting aspect of this transition is the feedback.
00:12:52As you can see, quite a bit of what went into getting high quality code reviews at Uber was bringing the human feedback into the system and using that to figure out how to tune our prompts, how to tune our agents.
00:13:07And so as we move to a model where humans are less in the loop, where software engineering is moving to an agentic model, we're effectively going to a place where we're starting to talk about, are we going to kill the outer loop?
00:13:23Is the human engineer not going to be involved in the code review?
00:13:28Some people are already here.
00:13:30some people are already there.
00:13:31Now, with the feedback taken into consideration, you start wondering, all right, what could this result in, right?
00:13:38I'll let your imagination go there in terms of quality degradation, slop, and so forth.
00:13:43But rather than killing the outer loop, I think that we believe, and the industry has just started to really kind of coalesce on this idea that we're really expanding the outer loop.
00:13:54Rather than removing humans from the code review process, we are moving their responsibilities up a layer.
00:14:02Rather than them dealing with the details of the implementation, the agent is great at writing the software.
00:14:08The agent is getting much, much better at reviewing the software as a human would.
00:14:13But now, as software engineers, we still are going to have an outer loop, it's just going to look a little different.
00:14:18Instead of you worrying about the optimization of the performance and the API compatibility, you're going to be thinking more about architecture in your code reviews.
00:14:27You're going to have time to focus on the domain expertise that you have and product thinking.
00:14:33So, we believe that as we adopt this automated code review, this is going to be the result of how our engineers are interacting with the system and guiding it.
00:14:43And that's it. Thank you so much for coming.