What we can learn from Cursor's SQLite Rust experiment
MMaximilian Schwarzmüller
Computing/SoftwareInternet Technology
Transcript
00:00:00SQLite was rewritten in Rust.
00:00:02And I know we just had the bun rewrite in Rust
00:00:04and you may wonder why is everybody rewriting
00:00:06everything in Rust, but this is not about Rust.
00:00:09This is not even about SQLite.
00:00:11I am aware that there is the Terso database,
00:00:15which already is a modernized re-implementation
00:00:18of SQLite and Rust.
00:00:19That is the one you want to use
00:00:21if you want to use a production ready
00:00:23SQLite Rust-based database.
00:00:26Instead, this experiment, mini SQLite,
00:00:29which is linked below, which you can check out,
00:00:32is not about SQLite or Rust.
00:00:34It's instead an experiment by the Cursor team,
00:00:37which is all about agent swarms and engineering
00:00:40a system of AI agents and finding out what works
00:00:44and what doesn't work that can build something
00:00:47like SQLite just from its docs,
00:00:51because that is what that experiment is about.
00:00:53There is a super detailed, very interesting blog post
00:00:56and we'll dive into that.
00:00:57There are a lot of interesting learnings in there about
00:00:59which we got to talk, which you also find linked below,
00:01:02where they explain how they ran this experiment and the starting point
00:01:07and the idea behind that experiment was to take the SQLite documentation,
00:01:12which is in the end 835 pages, if you were to put it all into one document,
00:01:18which is, of course, primarily written for humans.
00:01:23I mean, it's not the most approachable documentation I have ever seen,
00:01:27but obviously it's meant for humans because it's way older than all the AI agent stuff.
00:01:32Nonetheless, it also kind of acts as a super detailed spec,
00:01:37a super detailed specification because it describes in detail how to use SQLite
00:01:43and what the intended behavior or what the intended features are.
00:01:48And the cursor team took that documentation
00:01:52and then they took a public test set, the SQLogic test,
00:01:57which is a bunch of tests that test the behavior of SQLite
00:02:02or that test queries specifically.
00:02:05And they used that to check whether the implementation
00:02:09their agents built based on that documentation
00:02:13actually works with that official or with that huge test suite.
00:02:19Now, a couple of important words of caution up front.
00:02:23This test suite here is all about testing queries and query behavior.
00:02:29It does not test all the features, all the capabilities SQLite has.
00:02:35It does not test performance in general.
00:02:38It does not test concurrency.
00:02:40There is a lot of stuff in SQLite, which this does not test.
00:02:44And this mini SQLite result of that experiment, one of the results,
00:02:49they actually rebuilt SQLite multiple times with different agent combinations.
00:02:53And we'll dive into that is there for really just something to explore.
00:02:57It's not production ready.
00:02:59It's not what you want to use.
00:03:01It's just the output of an experiment where the goal was to use the documentation
00:03:06and then rebuild SQLite and make that rebuild version pass all those tests.
00:03:13And the cursor team used various model combinations here.
00:03:17Why combinations?
00:03:18Because as we'll learn, they used an approach
00:03:21where multiple agents worked together, planner, worker and reviewer agents.
00:03:28And they rebuilt that SQLite database with different combinations here and then also measured for similar quality.
00:03:37So all these combinations achieved the same kind of quality, the same amount of tests passing, but they measured which combinations cost how much.
00:03:45So for example, GPT 5.5 used for everything for planner agents and for worker agents led to an implementation cost of SQLite in Rust based on a documentation of around $10,000.
00:03:59On the other hand, combining Opus 4.8 with Composer 2.5 and Composer 2.5 is that super fast, very cheap, very efficient, but not super intelligent model by cursor, combining these two led to the same kind of quality, the same amount of passing tests as these other combinations for only a fraction of the cost for only $1,300.
00:04:24And the idea here was to use Opus 4.8, which is the more capable model, of course, for planning, for designing the tasks, which are then handed off to those worker agents, which are using Composer 2.5.
00:04:39And that already is one important takeaway from this article, which is not brand new, of course.
00:04:44You can do that yourself, too, if you're building software.
00:04:47It's a good idea to split your work, depending on the complexity of the work, of course, into different tasks that are executed by different agents, using sub-agents, if you will, where some agents focus on the planning,
00:05:03depending on designing a focused task of the overall task, so as an individual job, you could say, and then having number agents that implement that task.
00:05:14Because it turns out that for just spitting out good code, you don't necessarily need frontier intelligence if the context is good.
00:05:24So if the task is clearly defined, if all the useful information is in that task description, and then, of course, other factors can matter, too.
00:05:33For example, it can matter how the surrounding code base looks like, how examples look like you provide to an agent.
00:05:39That all influences the output, but worker agents that just write code can be numbered if the task is well-specified and the context is good.
00:05:49And that's what most of the experiment was about.
00:05:52How to design a system that can tackle a task of this scale.
00:05:57Because, of course, as I mentioned, for your projects as well, it is worth a consideration to have planner plus worker agents.
00:06:07Now, obviously not for all tasks.
00:06:10If you have a quick bug fix, if you have a very simple task, it's absolutely fine to just tell your agent codex, clot code, whatever it is.
00:06:19Hey, I have this problem.
00:06:20I want you to do this.
00:06:21Give it some extra context and let it do its job.
00:06:24It may, depending on the coding agent harness, spin up sub agents.
00:06:28Nonetheless, clot code may do that.
00:06:31Other harnesses like PI may not do that if you don't give it the right extensions.
00:06:36But even without sub agents, many tasks can be tackled by just one agent and you'll be fine.
00:06:43But for more complex projects, more complex tasks, having that split can be useful, including reviewer agents.
00:06:51That's something I personally also like to do.
00:06:54Again, depending on the complexity of the task.
00:06:56But having that split is something that works really well.
00:06:59And that's, of course, not groundbreakingly new.
00:07:02What is new is that for something like that SQLite rewrite here, you have multiple parallel processes of planner, worker and reviewer agents, multiple workers, but also multiple planners and reviewers.
00:07:18And they clash all the time.
00:07:19That is what the cursor team in the end found out here.
00:07:22Now, in this blog post, which is very, very interesting.
00:07:26They mentioned that earlier this year, they already ran an experiment with an agent harness where they built a web browser from scratch.
00:07:33And now they used that same harness or that same system of agents, I should say, to do that SQLite rewrite.
00:07:40But they also built a new system simply based on learnings they now have as an experiment.
00:07:47And then in this experiment and in this blog post, they compare these different approaches and dive into all the challenges they encountered whilst trying to set up and run that system that rebuilds SQLite.
00:07:59And one of the first challenges they faced when working with a system where multiple, many hundreds or thousands of workers work simultaneously is that traditional version control, Git, doesn't do it anymore.
00:08:13As they write in an earlier post about the swarm, we know that the tools like Git and Cargo rely on course logs for concurrency control, meaning that the same piece of data is being locked so that it can't have multiple writers simultaneously.
00:08:30This is fine for one developer, but unworkable for the volume of work produced by hundreds of concurrent agents.
00:08:36The browser swarm from earlier this year peaked at roughly 1000 commits per hour.
00:08:41So that's the swarm that rebuilt that browser.
00:08:45The new system, which they designed for this experiment, peaks at around 1000 commits per second.
00:08:52So the old swarm, which they ran earlier this year, had 1000 commits per hour, which is a bit more than most humans have, of course, obviously.
00:09:02The new system, however, had around 1000 commits per second, which is mind blowingly much and clearly, clearly not what Git was built for.
00:09:14Obviously.
00:09:15To facilitate this rate of activity, we built a new version control system from scratch.
00:09:21Throughput was not the only reason to own this layer.
00:09:25Every change in the system passes through the version control system.
00:09:29So it's where collisions first become visible.
00:09:31And several of the coordination mechanisms in the next section are implemented directly inside of it.
00:09:37And that's really interesting.
00:09:39They built a new version control system for the AI agent age.
00:09:43Because the old one, Git, which we all use, of course, there's nothing wrong with it.
00:09:48Just to be very clear.
00:09:49We're talking about an experiment here at a scale and on a task that many of us will never tackle, at least not soon.
00:09:57But still, the old system, Git, is not built for having hundreds of agents, hundreds of entities working on the same code simultaneously.
00:10:07So they built a new version control system, which can handle insanely high concurrency, but which can also help with resolving conflicts via agents.
00:10:19Because obviously it's in the version control system where conflicts become visible if two changes affect the same piece of code in a file.
00:10:28So that's the first important thing here.
00:10:30They built a brand new version control system for this experiment to be able to efficiently run it or perform this experiment.
00:10:37Now, naturally, as they mentioned here, they encountered many problems at that scale and rate of change at 1000 commits per second.
00:10:48For example, and this is really interesting, all these issues they encountered and how they solved it, because it gives us a glimpse at how software engineering may look like in the future, at least in certain scenarios.
00:11:01The split brain design problem.
00:11:03Two planners, unaware of each other, implement the same concept in different ways in different parts of the code base.
00:11:10So duplication, same concept in different ways in different parts of the code base.
00:11:16You typically would want to extract that and reuse that logic, right?
00:11:21We fixed this through prompting.
00:11:24So no fancy new system built here, instead through prompting.
00:11:28Planners make the signed decisions, the planner agents themselves, rather than delegating them.
00:11:34And we require them to ensure that no two delegated subtrees decide the same question.
00:11:39So it's a setup question here.
00:11:40It's all about ensuring that as you split that system into planners, workers and so on, you ensure that your different planners, because it's not just parallel workers, it's all the parallel planners, have clearly defined tasks that are highly unlikely to clash and overlap.
00:12:02So that, of course, starts with the design from humans.
00:12:08So how you set up the task, how you prompt, right?
00:12:11We fixed this through prompting.
00:12:13So that, of course, then kind of goes down that tree of agents with all those parallel nodes and all these parallel leaves, where you want to ensure that as you have agents split up tasks into subtasks, those agents are prompted to cut tasks that have a low chance of overlapping.
00:12:34So that is ultimately a planning challenge for humans, which is all about setting up the system in the right way from the human side.
00:12:43So that's how they solved or how they tackled this problem.
00:12:47Another problem they faced was contention between planners.
00:12:51A harder form of contention is when two planners know about each other and fight through back and forth changes over the same files.
00:12:59The problem is two pictures of reality and merge tooling can't fix a disagreement.
00:13:04Instead, we have agents record decisions in shared design docs.
00:13:08Code that depends on a decision carries a compiled check reference back to its doc.
00:13:13When planners unknowingly contradict each other, a reconciler merges the docs and the references propagate the resolution downstream.
00:13:21So in the end, related to the previous point, when splitting work across planners still, of course, naturally in software development, you can't entirely avoid overlaps or shared domains, shared logic, shared areas that need to be touched by planners and ultimately workers in a code base.
00:13:43So that is when planner agents started to fight over an implementation and they fixed that by bringing in a reconciler agent, I assume, that merges the docs that were crafted by those planners.
00:13:59So that were crafted to then hand out to the workers.
00:14:02They had a reconciler step in between that merges those docs of fighting planner agents so that they would talk a consistent language and agree on an implementation to also help with ensuring that the same thing would be reimplemented in different ways in different parts of the code base.
00:14:21So these two work together, as I understand it.
00:14:24Now, naturally, they also encountered merge conflict.
00:14:29So planning and ensuring that there is no overlapping there or as little overlapping as possible and that planners speak the same language is the first important step.
00:14:38But still, multiple workers, even multiple workers working on one and the same plan, are, of course, very, very likely to touch the same files to contradict with each other.
00:14:50So there are a lot of other workers that are not just going to make sure that they're not going to work on the same files.
00:14:55So that's why they're not going to work on the same files.
00:14:57So that's why they're not going to work on the same files.
00:14:58In order to resolve a collision, they would have to stop, absorb the other agent's context and merge around it.
00:15:03Naturally, if two agents or two humans, for that matter, work on the same file, in order to resolve that conflict, both have to stop, no matter if it's agents or humans, or normally they would have to stop, so that you find a decision,
00:15:19an implementation that resolves the conflict.
00:15:23Worker agents, however, are bad at this and in practice either override the other change or abandon their own.
00:15:29And maybe you noticed this as well.
00:15:31I certainly have.
00:15:32If you work in a code base together with one or more AI agents and you make a change.
00:15:38Okay, I know it's scary, but you can still write code.
00:15:40So let's say you make a change.
00:15:42You change something in code.
00:15:44The agent will always just undo it and override it.
00:15:48It does not respect those changes.
00:15:50It has its agenda.
00:15:52And if it's decided that it must edit that given file, it will do that.
00:15:57And it does not care if you made any change to it in the meantime.
00:16:01It's a bit different if you committed to that change because these agents are post trained, fine tuned to not easily undo your commits and stuff like that.
00:16:13But if it's an uncommitted change, the worker just doesn't care.
00:16:17The agent just doesn't care.
00:16:19And that's exactly what they encountered here too.
00:16:21To fix this, we created a system where a neutral third party agent intervenes on merge conflicts and resolves them on behalf of all parties.
00:16:29Its only goal is to be impartial and efficient, similar to the way merge queues work in engineering teams.
00:16:35And I think that is also interesting.
00:16:38It's again, a form of reconciliation.
00:16:41It's again, as I understand it, about stopping those agents, just like in the old world where you had to stop and take a step back and find a
00:16:49resolution for a conflict.
00:16:51But what this clearly shows, and that's also not a new learning, is that fresh context with the right context given, though, is super, super important.
00:17:03So no matter if you're tackling a large scale task like cursor here, which we all don't do, or if you're just working on a smaller scale project,
00:17:12the huge advantage of having a split across planner and worker and reviewer agents is primarily, or very often at least, that you work with fresh context windows.
00:17:24This does not mean that it's empty context windows.
00:17:26It just means that you have fresh agent sessions populated with just the right context for a given task.
00:17:33For example, a worker is pretty bad at reviewing its own work because it has all that context from implementing that stuff in its context window.
00:17:41So it's biased, if you want to call it like this.
00:17:44That's why a reviewer agent should start in a fresh context window, should get the information, what the worker worked on, what the plan was, which files were touched, but nothing else.
00:17:55So that it can honestly review that work.
00:17:59That's why fresh context windows populated with the right context are so important.
00:18:04And that's exactly the same thing here where they resolved merge conflicts through a new agent with just the right context without having a bias that could then resolve a conflict.
00:18:17And then I assume the system was set up such that those worker agents received information that this is the conflict resolution and they must not override it or new worker agents were started.
00:18:30That's not entirely clear to me here.
00:18:32Another issue they encountered were mega files.
00:18:35Some files are particularly popular places for agents to work.
00:18:39Each agent might add only a small amount of code and no single agent is responsible for keeping the file small.
00:18:45These mega files choke everything.
00:18:49They're expensive to transport, diff, merge, and become the site of constant collisions.
00:18:54Again, that is also something which on a far smaller scale you may have encountered as well.
00:19:00I certainly have.
00:19:01One of the things that we have.
00:19:02Especially for testing.
00:19:03My experience is that agents love to just add more and more tests in the same file.
00:19:08And it's of course not just testing, but that's one area where I can frequently see it.
00:19:13And especially if you have multiple agents working and each agent has its agenda.
00:19:18They don't care because they're not humans.
00:19:21How could they care about anything?
00:19:23They're just executing tasks, right?
00:19:24They don't care about the size of a file or the general architecture of a system.
00:19:30If you just have a bunch of agents executing their tasks, your code base will drift into chaos at some point.
00:19:37Because agents don't care.
00:19:39They care about executing their task.
00:19:42And these mega files, of course, are one clear indicator of that problem.
00:19:48That they become a thing the more agents work for a longer period of time in your project.
00:19:55There is no agent there that's responsible for splitting that file or for keeping your code base well architected.
00:20:02That's just not their agenda.
00:20:04So, to fix this, we gave worker agents a way to flag bloated files.
00:20:09Once flagged, we blocked new commits and an outset agent decomposes the overgrown file into smaller modules.
00:20:16So, again, a fresh agent coming in.
00:20:19It's a pattern we see here.
00:20:21For all these problems, it was about identifying a problem and then using fresh agents with the right task given,
00:20:28with the right context given, to resolve that problem so that then the other agents can continue their work.
00:20:35And that's the same here for the mega files.
00:20:38Ossification, another problem they faced.
00:20:40Agents have learned from working in existing code bases with humans in the loop,
00:20:44not to touch core code even when it needs to change.
00:20:48So, this is not what I meant before.
00:20:50When you make a change in a file the agent is working on and it just froze that away.
00:20:55It's instead about in general.
00:20:57An agent has a clear task based on a plan, based on a prompt you gave it.
00:21:03And that does, of course, involve changing certain files.
00:21:07Now, one thing we already know or see every day when working with agents is that depending on the model,
00:21:16some models are super hesitant to let go of existing code.
00:21:21They robber add 10 fallbacks, 10 if checks and more and more legacy code to a code base instead of deleting it and cleaning it up.
00:21:31You have to explicitly prompt to make sure that an agent really deletes a function or gets rid of some code file.
00:21:39They don't really do that on themselves because of fine tuning, because clearly these model providers don't want to build models,
00:21:47which then roam freely and break all kinds of production code.
00:21:50But when not working in a brownfield project, when not working in an existing code base that's maybe running in production,
00:21:57then this tendency to not really touch code and to keep all the code around forever can be super problematic and annoying.
00:22:05And it can also lead to other side effects like they encountered here where agents would just not improve code written by other agents,
00:22:16but just build on top of it over and over again, leading to a bloated code base ultimately, of course.
00:22:23To fix this, we license intentional breakage.
00:22:27An agent that judges a core change worthwhile can make a focused patch outside its scope and leave a comment explaining why it did it.
00:22:36And that's, again, on a smaller scale what you can do in your projects to what I'm doing.
00:22:41You want to explicitly license and tell your agents, hey, we're building this.
00:22:46We're in early development.
00:22:48This is not live yet.
00:22:49I want breaking changes.
00:22:51So clean the code up aggressively.
00:22:54Refactors are welcome.
00:22:56Stuff like that.
00:22:58You want to encourage agents and these AI models and overwrite their fine tuning instructions.
00:23:04So to say they're built in knowledge and get rid of that to make sure they can indeed evolve a code base instead of just adding more and more code to it.
00:23:14So again, something we can see on a smaller scale here, of course, seen on a large scale.
00:23:19Now for reviewing, they used an approach called review lenses.
00:23:24So we have the planner and worker agents, but of course that work must be reviewed to then create follow up work and restart the loop until a certain bug is fixed until the code base is in a better shape.
00:23:38In a system that is both long running and multi-agent errors accumulate and the swarm needs a way to correct itself before small mistakes become foundational.
00:23:47Again, makes sense.
00:23:48We've all seen this on a smaller scale too.
00:23:50We experimented with many kinds of review lenses, such as giving a review agent the workers full transcript or only its output or nothing but the code base.
00:24:00We also tried reviewers running on different models with different training and a different personality.
00:24:05No single lens catches everything, but the correlated lenses stack the way self-driving systems reach above human reliability without any single perfect component.
00:24:16The compute spent on review is high return since review is much cheaper than the work it audits.
00:24:21We suspect this stack review system was a major contributor to the sustained quality of the runs.
00:24:28So key takeaway here.
00:24:30It's impossible to have one or more reviewer agents each review the entire code base.
00:24:39It's way too much.
00:24:41Instead, they experimented with different approaches like giving it the full transcript or only the output or nothing but the code base.
00:24:47And what they find out in the end is that what helped them is to have different reviewers with different personas with different focus lenses where they would focus on different aspects.
00:25:00Give them the code base, as I understand, and maybe some information about what the worker did.
00:25:06And then it was the combination, the output of multiple reviewers combined, essentially, that led to an overall review result that then could be picked up by a planner again to turn it into a plan and have workers fix the code.
00:25:23And again, on a smaller scale, I think that is something you can or we can apply to.
00:25:28Now, obviously, we're again, we're not building stuff like that.
00:25:34But what works really well in my experience, too, is to have multiple reviewer agents with different tasks where one of them may focus on, hey, is this ideomatic rust?
00:25:46Another one may focus on performance and security issues if Fable 5 lets you.
00:25:51Another reviewer may focus on naming patterns if that's something you want to focus on and so on.
00:25:58So you have different lenses and then you give those reviewers just the right context.
00:26:03Again, something like, hey, we had workers work on that feature.
00:26:07Maybe give them the plan of the of the worker and maybe some information about the rough steps the worker did, but nothing more.
00:26:15And then you have all this output from the different reviewers and you can combine that again, maybe with another reviewer.
00:26:22What I also like is have a reviewer review the review results, which simply has the idea that depending on the model again, they tend to like to find stuff.
00:26:36No matter which code you hand them, it could be a one liner.
00:26:40I sometimes feel like and they would find five issues in it.
00:26:43So have a reviewer categorize those review findings and drop the ones that aren't really issues can work really well in my experience.
00:26:55And it's this again, as they write combination of reviewers, this stack of reviewers that can help produce good results, which then can be picked up and implemented again.
00:27:06Now, again, always depends on the scale of your task of the software you're building.
00:27:11Obviously, for many, many pieces of software, this is all way too complex, but it's a nice glimpse into how software engineering could look like, how such systems could look like in the future, which I personally find very, very interesting.
00:27:27Now, one last thing they also did is they let agents shape the environment.
00:27:32Here, the idea was that they let agents write a field guide, in the end, a document or a collection of documents where they did not give the agents any instructions other than that this field guide should act as context for the overall task, where agents could therefore build memory, shared memory, you could say, which is all about learnings or key issues that maybe were identified and things like that.
00:28:00So that they had this extra memory system, if you want to call it like this, in place for the agents to keep notes and document decisions.
00:28:10And overall, I find, just like the BUN rewrite on Rust, I find this experiment very, very interesting.
00:28:16It can also be scary.
00:28:17I totally get this.
00:28:18And I think we should not infer that this is how all software should be built from now on.
00:28:24I mean, for one, this is not even a production-ready piece of software.
00:28:28And getting it production-ready would certainly take a considerable amount of time.
00:28:33This is not to be underestimated.
00:28:35It's not like you can build something like this in a couple of hours and then making it production-ready is only a few hours more.
00:28:43The first 80% can be way faster to achieve than the last 20%.
00:28:48We all know that.
00:28:49So that's one important takeaway.
00:28:51It's also important, of course, to realize that this specific task of rewriting SQLite, yeah, it may just have received those docs and then this test suite here was used.
00:29:04But obviously, for one, this is an incredibly detailed spec here, something you don't have for new projects.
00:29:13If you're building a new piece of software, you don't have a specification as detailed as the documentation for a software that's over 20 years old.
00:29:24And of course, even if only the documentation was given to those agents, the SQLite source code and also maybe other source code like other re-implementations like in Rust here by Terso may and is very likely to be part of the training data of most or all of these models that were used.
00:29:46So it's not like it was brand new for these models.
00:29:51It's not the same as a brand new piece of software being built where iteration is also an important part of building it.
00:29:58You will have a very hard time and I would even say it's impossible to build a new piece of software no matter what it is from scratch without it changing all the time.
00:30:11Because you can't write a perfect spec up front and then be done with it.
00:30:17You always discover new stuff or things you want to change whilst you're building something, no matter on which scale.
00:30:26And therefore, of course, this is not representative of how software will or should be built in general.
00:30:34It is a very interesting experiment, though.
00:30:37It is a very interesting experiment and it has key learnings that matter for all of us.
00:30:43Key learnings that, of course, aren't brand new, like splitting your work, having fresh context windows with just the right context.
00:30:50Interesting insights like that maybe new version control systems will emerge and will be required in the future.
00:30:57And, of course, that multi-agent orchestrations become a thing.
00:31:01Now, that all, of course, also proves that humans designing these systems, these agent systems and humans, of course, for new software, especially also making decisions on how that software should be architected.
00:31:21Writing spec files, even if they're not as detailed as this and coming up with software architectures and then building agent systems that can implement them and then reviewing that, which is important.
00:31:33This is where we're all heading and which, even though it's all changing and definitely not how we built software six years ago, is something that gets me excited.
00:31:45I think it's really interesting that we're moving into that systemic thinking area, both for building the agentic systems as well as designing the software architecture itself.
00:31:59And then we have both work together.
00:32:01I find experiments like this very, very interesting.
00:32:04The learnings here are very, very interesting.
00:32:07And some of these learnings on a smaller, more simplified scale may matter to everyday software and development projects as well.
00:32:17But as always, let me know what your thoughts are and what you think about experiments like this.