It's Over... This Just Ended The GPT 6 Astra Vs Fable 5.1 Debate
AAI LABS
Computing/SoftwareInternet Technology
Transcript
00:00:00Fable 5.1 and GPT-6 Astra were released just a few days apart and both have been getting really
00:00:05high scores on benchmarks. Astra even scored almost 100% on the AGI test which no model has
00:00:11ever come close to. So looking at those numbers you'd expect it to do really well when you give
00:00:16it work of your own. But doing well on those tests doesn't tell you how well a model will
00:00:20handle the work you give it and we wanted to see whether both models perform just as well on our
00:00:25own work. Since we're a software company we already had projects to test them on so we gave both
00:00:30models work from those projects and checked what they finished. We ran each model across multiple
00:00:35projects and scored how well it performed in different areas. We used GPT-5.6 to help judge
00:00:41the results and score them. This judge model didn't know which results came from Fable and which came
00:00:45from Astra. One model got an overall score of 86 out of 100 while the other got 84. Now that might look
00:00:52like a small gap but the model that came out ahead overall didn't win every part of the testing and
00:00:57the other performed surprisingly well in some of those areas. So we're going to go through how each
00:01:02model performed in our testing and which one won in each category so that you know how those differences
00:01:07affect the work you're giving it. Before we move on to how each model performed let's first go over
00:01:12the models themselves. This section is only for those who don't know about them so if you already know
00:01:17the details you can skip this section and go straight to the next one. Anthropic released
00:01:22Fable 5.1 on September 1st and even before Anthropic could fully enjoy the release hype,
00:01:27OpenAI dropped GPT-6 Astra just a few days after Fable and people have been building really interesting
00:01:33products with these models and pushing these models to the limits of their ability and getting really
00:01:37surprising results. When it comes to pricing, both models actually have the same pricing of $10 per
00:01:43million input tokens and $50 per million output tokens. But with Fable 5.1 there's one thing
00:01:49different which is that Anthropic cut the price of cached reads by 75%. For those who don't know
00:01:54about cached reads, parts of the conversation that the model has already read are actually saved for
00:02:00reuse. When you send a new prompt, those saved parts are reused instead of the model reading them
00:02:05again from scratch and that's called a cached read. These reads already make it cheaper to reuse the
00:02:10conversation but with the decreased price it gets even cheaper. But that doesn't mean Fable is a
00:02:14cheaper model to use overall because the total bill also depends on many other factors which we'll
00:02:19talk about in a bit. Now Fable models have been in Claude code for a while but Fable 5.1 still isn't
00:02:25included in the Claude Pro plan so on Pro you have to pay for it separately through usage credits. On the
00:02:30Max plans, Fable 5.1 is included but Fable models have a lower limit than the rest of the models.
00:02:36On the other hand, OpenAI doesn't treat Astra as a separate model like that because Estra is part of
00:02:41the normal codex limits on ChatGPT+ and Pro and you can spend all of the usage on Estra. Now these
00:02:47models are really capable of working on long tasks which means you can just ask them to build a large
00:02:52feature and leave them to work through the steps on their own. But when you leave these models working
00:02:57on a long task, the amount of context they can hold matters too. Fable gives you a million tokens in
00:03:02Claude code while Astra's default context window in Codex is only 272,000 tokens even though Astra has a
00:03:09much larger window through its API which is even higher than Fable's million. But you won't be
00:03:14getting that in Codex for now because Codex defaults to a context window of 272,000 tokens even for Astra.
00:03:21Now these models have reached a level where they can carry out advanced research so OpenAI and
00:03:26Anthropic both have added safety guardrails that restrict some of the work they're allowed to do.
00:03:31But those restrictions also affect your work in different ways because each model reacts differently
00:03:36when its safety system decides your request isn't allowed. When Astra reaches a part of the task that
00:03:41goes against its rules, it straight up refuses that task and tells you about it. On the other hand,
00:03:45Claude code switches from Fable to a weaker model when a request goes against its rules. And many
00:03:51people found that Claude code had switched models without them knowing. So if Claude code continues
00:03:56working after that switch, the result you're getting may no longer be coming from Fable.
00:04:01Now the first metric that we measured was quality which looks at how good the model's work actually
00:04:06is. For this testing, we used multiple client projects so we can't reveal the details of those
00:04:11projects but we can explain how we judged the work and where the models differed. For how well the code
00:04:16was written and organized, Fable scored 90 compared to Astra's 78 because it kept the code for different
00:04:22parts of the app more clearly separated. That makes it easier for the model to understand the app when
00:04:27making changes later and Fable was ahead of Astra on this across all the projects we tested. And for
00:04:32how much of the requested work was finished, Fable scored 91 compared to Astra's 84 because it also fixed
00:04:39problems that we hadn't specifically asked it to fix. But we also checked whether the finished features
00:04:44worked correctly without us asking for fixes and Astra scored 87 compared to Fable's 85 on that.
00:04:50Part of Astra's lead came from checking the apps more thoroughly and fixing more of the known problems.
00:04:55When Fable tested its apps, it missed some situations where they could go wrong, so the finished features
00:05:00still had a few problems. When it came to the experience of using the app, Astra scored 89 compared to Fable's 88,
00:05:08because it arranged the different parts of the page in places that made them easier to find and use.
00:05:13So based on all the tests we just mentioned, Fable's overall quality score was 88 while Astra scored 84,
00:05:19because it built more complete features and its work was easier to change later. So in terms of quality,
00:05:25Fable is a clear winner. But before we move on to the next test, let's have a word by our sponsor,
00:05:30Zapier. You vibecode an app and it works great, but the moment you want it to actually connect to tools
00:05:35like Gmail, Slack, and Notion, you're wiring up every integration and OAuth flow by hand. Before you know
00:05:41it, you're buried in authentication code and suddenly the integrations are the whole project. That's where
00:05:46the Zapier SDK comes in. It's a code library you drop into your project right inside cloud code or cursor,
00:05:52giving you programmatic access to Zapier's 9000+ app ecosystem. So instead of building each integration
00:05:58by hand, we just called the ones we needed and kept moving. In a few lines, our app was sending emails
00:06:04and creating Notion pages with no API docs and no auth to fight through. And when we needed to update
00:06:10a custom property in Notion that didn't have a pre-built action, we called it directly through the
00:06:15SDK. It genuinely meets you where you already work. If you're tired of wiring up integrations by hand,
00:06:20try the Zapier SDK. The link's in the description below. The next metric that we measured was how well
00:06:26the models handled long-running tasks, which means how much work they finished on their own with little
00:06:31guidance from us as they worked. We also checked how well they handled problems that came up along the way.
00:06:36For this, we gave each model an app idea to build and worded the request according to its prompting guide
00:06:42so that it had the best chance of doing the work well. We didn't mention the specific details we wanted
00:06:47and we just described what the app needed to do. Estra worked for about 32 minutes and Fable worked for
00:06:53about 44 minutes. Estra ran into errors while building and checking its app, but it worked through them
00:06:58and continued fixing the app without us having to guide it through each problem. Fable also finished
00:07:03its app without stopping early or asking us what to do next, and it ran checks on what it had built.
00:07:09So both carried the work through, but we also went through the finished apps to see what they had
00:07:13left us with. Estra's app opened straight to a login page without a separate landing page. Once we
00:07:18signed in, the layout was organized well and the app worked, although it still had the familiar layouts that
00:07:24show up in a lot of AI generated apps. But we had a problem with the side panel because it didn't scroll
00:07:30far enough to let us reach the sign out button. We had to zoom out to reach that button, which was
00:07:35something Estra's own checks hadn't caught. Fable's app also opened straight to a login page, but its
00:07:40design felt much more generic. The features worked, but everything was packed so closely together that
00:07:45the app was difficult to use and the repeated gradients added to that familiar AI generated look. It did adjust to
00:07:51smaller screens, but it still didn't have a usable layout, even though the features were in depth. We
00:07:56also had multiple other apps and huge features planned and built this way with the models left
00:08:01to work on their own. So for long running work, Estra scored 93 out of 100 compared to Fable's 90.
00:08:08Fable focused more on making the features work, unless we had specified in the prompt that it should
00:08:13focus on visuals as well. Estra paid attention to both how the apps worked and how they looked,
00:08:18so Estra wins on long running tasks. The next metric that we looked at was design and how
00:08:23easy the apps were to use. We did let the model score the designs, but we didn't want to rely on
00:08:28its design taste alone. So we built a viewer for Fable's results and another for Astra's, which let us go
00:08:33through the designs ourselves and compare what they were like to use. We gave both models the same design
00:08:38tasks, starting with a landing page for a business that sells fonts. Fable's version looked very similar to
00:08:44what Opus tends to produce, from the colors and fonts to the layout. It even had that strip of text moving
00:08:50across the page that we've been seeing in Opus designs lately. But one thing Fable did differently
00:08:55from Opus was add a lot more interactive elements throughout the design, and that made the app feel
00:09:00better to use. Astra's version had a more spacious layout, with clearer differences in text size and
00:09:05colors that made the text easier to read. But there was much less movement, so while it felt more
00:09:10comfortable to look at, it didn't offer the same experience as Fable's design. That's why Fable
00:09:14scored 94 in interaction scores, where Astra scored 85. Next, we had each model design a horror game
00:09:20landing page. Fable's design used an animated lighthouse to create the atmosphere. It made the
00:09:25artwork as SVGs, which are images drawn with code, but the text didn't stand out enough against the dark
00:09:31background because of the sizes and colors Fable chose. And on the same design task, Astra used the
00:09:36built-in image generation model in Codex to generate an image instead of creating the artwork with code.
00:09:42It also created a teaser for the game even though we didn't ask for it. And where Astra pulled ahead
00:09:47was in its choice of fonts and colors, which made the text stand out against the dark background.
00:09:52For the music festival site, Fable's version repeated a lot of the design choices we see in designs made
00:09:57by Opus. But even with those familiar choices, it felt like Fable had put more effort into giving the
00:10:03site its own style. Astra used a generated concert photograph, but the overall design felt more
00:10:08generic. The last comparison was a parallel parking game, where Fable's version was really well made,
00:10:13and the rear view mirror helped us judge where to move the car more accurately. The game had two
00:10:18camera view, but both of them had problems. Because one showed the wrong side of the parking space,
00:10:23while the other only showed one side instead of giving us a full view of the road ahead. That made
00:10:27it harder to see where we were moving the car, and if those camera angles had been right, the game would
00:10:32have felt more realistic. Astra gave us three fixed views and added driving tips in the side panel,
00:10:38which made its game easier to play. The camera views were all much better than Fable's. These were
00:10:42general tests where we gave the models very little detail about the designs we wanted, so we were seeing
00:10:48the model's own taste here. Both repeated design choices we'd seen in their other work, but both did
00:10:53produce good results. And with a little more detail about how we wanted the apps to look and feel,
00:10:58we'd expect both to design well. So on taste alone, Astra won on visuals and usability, while Fable won on
00:11:04animations and interactivity. But before we move on to testing cost and speed, it would be great if you
00:11:09subscribed to the channel and hit the hype button. This small gesture of support goes a long way for us.
00:11:15The next metric that we measured was efficiency, which covered the cost of the work, how long it took,
00:11:20and how many times the model used its tools to get it done. One thing to know about the costs is that
00:11:25we didn't use the API, so these are just estimates of how much the work would cost if we used the API
00:11:31based on the tokens used. We ran the tests on our usual subscription. Across all test runs, Fable's
00:11:37total estimated cost came to $49.18, compared to $27.69 for Astra, so Astra's total was about 44% lower.
00:11:47Fable generated nearly 3 times as many output tokens as Astra, just like how Fable's prompting guide
00:11:52mentions that it tends to write more than other models. Both models had the same cost for those
00:11:57tokens, so writing more was what increased Fable's cost. Using tools also adds to that token usage,
00:12:03because the model decides which tool to use and then reads the results before it continues working. Across all
00:12:09six runs, Fable's main agent used its tools 443 times compared to Astra's 287, so that also added to the
00:12:17cost. For cost, Astra scored 80 out of 100 compared to Fable's 66. When it came to speed, Astra was also
00:12:23ahead at 70 compared to Fable's 67. The long running tasks took about 62 minutes in total for Astra,
00:12:30compared to 73 minutes for Fable. So across the scores for cost, speed, and tool efficiency,
00:12:35Astra came out ahead of Fable. The next metric that we measured was instruction following,
00:12:40where we checked how well the models followed the instructions we gave them while building,
00:12:44and whether they kept following them as the work continued. When we ran Fable on a project,
00:12:49it initially added a co-authored message saying Claude helped write the code,
00:12:53even though our instructions explicitly told it not to. It also ignored a rule about how it was allowed to
00:12:58create or change files. We told it to use Claude's own file editing tools, so those changes were tracked
00:13:04and easy to undo, but it created two files by running commands instead. For instruction following,
00:13:09Astra scored 96 out of 100 compared to Fable's 88. So when it came to following the instructions,
00:13:16Astra came out ahead in these runs. The next metric that we measured was review quality, where we gave the
00:13:21models a project containing problems and asked them to find those problems. We first gave both models the
00:13:27same code to review, with four problems deliberately added, so we knew what they needed to find. Fable
00:13:33found all four, and it also found five additional problems that we hadn't added, which were then
00:13:37checked and confirmed. Astra found two of the four problems we'd added, but it still told us the review
00:13:42was complete. We also included three parts that looked suspicious, but were actually correct, because
00:13:47we wanted to see if the model would report false problems. Neither treated those as bugs, so the
00:13:52difference here was in how much they found, and Fable gave us the more thorough review,
00:13:57but when we tested them on a larger project with nine confirmed problems, Astra fixed five, while Fable
00:14:03only found and fixed four. So Astra completed more of the repairs, although both left several problems
00:14:08unfixed. For review quality, which included those repairs and how the models checked their other work,
00:14:13Fable scored 84 compared to Astra's 78. So Fable came out ahead on the overall review score,
00:14:19because Fable gave us the stronger review overall. So based on these results, Astra is a clear winner across
00:14:25multiple categories we tested, but that doesn't mean Fable is a bad model, because it did perform well
00:14:30across the tasks, and wasn't far behind Astra in several of them. So you just need to choose the model
00:14:35based on tasks you're giving them. Now in order to push these two models to their max ability, there are
00:14:40certain practices you need to follow. We used a lot of them in our testing, and if you want to access
00:14:45those as well, you can get them in AI Labs Pro, which is our community. So if you found value in what we
00:14:51do and want to support the channel, this is the best way to do it. The link's in the description.
00:14:56That brings us to the end of this video. If you'd like to support the channel and help us keep making
00:15:01videos like this, you can do so by using the super thanks button below. As always,
00:15:05Thank you for watching and I'll see you in the next one.
Community Posts
No posts yet. Be the first to write about this video!
Write about this video