Kimi K3 Is Fable Level... (they should be worried)
BBetter Stack
Computing/SoftwareSmall Business/StartupsInternet Technology
Transcript
00:00:00Kimi K3 is here and honestly this one is kind of shocking. It didn't just catch up with Opus and
00:00:04GPT 5.5, it caught up with Fable and GPT 5.6. The open models are not as far behind as we first
00:00:10thought. Let's take a look. Let's start with our announcement. Kimi K3 is a 2.8 trillion
00:00:19parameter model built on Kimi Delta Retention and Attention Residuals with native vision
00:00:24capabilities and a 1 million token context window. It's the world's first open 3 trillion class model
00:00:29designed for frontier intelligence across long horizon coding, knowledge work and reasoning.
00:00:34On all of the benchmarks that they show in this blog, Kimi K3 is either trailing behind Fable
00:00:38and GPT 5.6 Soul or even beating them on some of the other areas. Like this is such a crazy jump
00:00:43that I don't think many people predicted. Obviously though this is on their own benchmark so let's take
00:00:47a look at artificial intelligence and our third party ones and I'm happy to say the results pretty
00:00:52much match up. On this benchmark we can see that around 80% of you watching this aren't subscribed
00:00:56so go ahead and click that subscribe button if you'd like to stay up to date with all things software
00:01:00development and AI. On to the real benchmarks though starting out with the coding index it scores a 76.2 on
00:01:06this that is 0.3 points behind Fable 5 and is slightly more than GPT 5.5 more than Opus 4.8 and
00:01:12more than GPT 5.6 Luna and it only slightly loses to GPT 5.6 Soul and Terra. Like look at this leap they
00:01:19have made from their last model. If we then take a look at the agentic index it's the same story.
00:01:23It's third here losing only to Fable and Soul but that is seriously still impressive and you're used
00:01:29to this by now but it's the exact same on the intelligence index as well. So even the third
00:01:32party benchmarks confirm that this model is awesome. Sadly we don't yet have the deep SWE results as of
00:01:38recording this but if we take a look at our own benchmarks you can see that it came third again.
00:01:42Like this model just seems too good to be true. The only downsides that I can find with this model is
00:01:46that it is a little bit expensive. Unless you have a supercomputer in your house I don't think you're
00:01:50going to be running this one locally so if you go through the API it's actually priced at $3 for a
00:01:54million input tokens and $15 for a million output tokens. This is quite a big step up from Kimi K 2.6
00:02:00and it's around the same price as Sonic 5 when that discount goes away but it is cheaper than Fable 5,
00:02:06Opus and GPT 5.6. API pricing also isn't everything so if we take a look at it in practice with the cost
00:02:12per task here it's actually similar to GPT 5.6 Soul and it's about half the price of Opus 4.8 and of
00:02:17course it's cheaper than Fable and Sonic 5. So it is competitive with those Frontier closed models but
00:02:22for those of you that love open models because they're a little bit cheaper this model probably
00:02:26won't be for you. GLM 5.2 is the second best option here and it's half the price. It's also not the
00:02:31fastest model ever but to me it seems fairly competitive for this level of intelligence.
00:02:36Overall it's honestly pretty hard to have anything negative to say about this model so let's just jump
00:02:40in and see some of my local testing. The first test that I put these models through was creating me a
00:02:44complete Formula 1 racing game in 3.js that's just all in one index.html and this one is Kimi K3 going
00:02:50through Open Router. I also tested it in Kimi Code that CLI just to see if the harness does change
00:02:56anything. You can see down here that it thought for 14 minutes before it started and overall it took it
00:03:0035 minutes to make this 3.js web app and it took $1.24 in API tokens. And the results I got back are
00:03:06pretty impressive. Obviously this has no access to any external assets so it's doing its best at creating
00:03:11these models. But the track does work. We have our AI racers and they seem to be able to go around the
00:03:16track. You can see the barriers actually work so I can't clip through them. Our position is updating.
00:03:21We have our map updating. We have our current time updating as well as the laps as well. I have done
00:03:25a full lap and confirm that that all works. So it's a pretty good result for what Kimi K3 gave us.
00:03:30As I said I also tested out the different harnesses. So this is Kimi K3 but in Kimi Code. And this one
00:03:35I would say looks a little bit nicer. It's done a better job with the assets although that is just a
00:03:39style guide. There's a bit of lag there with those models but the handling is a little bit better.
00:03:43The controls are still inverted and actually seems to be a pattern in a lot of these 3.js applications
00:03:48for some reason. But again all of this handles really well and it has features like going off
00:03:52track is going to slow me down. And yeah it's done a pretty impressive job as well. But now let's
00:03:57compare this to the Frontier models. We have Fable 5 here in Claude Code that was on max and it took
00:04:0144 minutes to make my 3.js application. Then we also have GPT 5.6 Sol on max as well. And this
00:04:07one took 18 minutes. This is the result that Fable gave me. So we have Fable GP here. And I would say
00:04:12that it looks slightly nicer maybe. But again all of this is just a stylistic choice. And the controls
00:04:17aren't actually inverted in this one. There's also a bit of camera wobble which I do think is a nice
00:04:22style. And again everything seems to be working in this one like all of the others. I've tested doing
00:04:26a full lap of this as hard as it is. We do also have some collisions here as you can see. But yeah
00:04:31everything seems to be working and it's pretty comparable to what Kimi K3 gave us. It's a similar story
00:04:35with GPT 5.6 Sol. As you can see we have a working game here. Although I will note that the road here
00:04:40is grass. It didn't actually render that in. And everything does seem to be working. The controls
00:04:44are inverted on this one as well. And we also have a few backward assets. I don't know why these models
00:04:48seem to like doing that. The only one that didn't was Fable. And also the track here does get a bit
00:04:53jumbled up. So this is actually the only one that didn't make me a working track. To add in one final
00:04:57comparison. I also checked what GLM 5.2 would give me as it's the next best open model. And this one
00:05:02thought for 5 minutes and 58 seconds. Although it didn't finish it in one prompt. There was actually
00:05:07a few bugs. So I had to go back with another prompt. Even then when I did finally get the game running
00:05:11you can see that there's a lot of bugs in this one. Firstly we don't start the right way on the track.
00:05:15And there isn't really a track to begin with. And also it just looks a lot worse than the other ones.
00:05:20So there is a bit of a step up when we go up to Kimi K3. Moving on to test 2 though. I asked for a
00:05:25personal finance management dashboard. This is going to build the full stack application with a front
00:05:29end and a back end. The only thing I said I didn't want is auth. And I also asked it to seed some data.
00:05:33I did this one again in open router with K3. And then also in Kimi code. So this one is in open
00:05:38router. And you can see in total this one took 56 minutes to make me the application. And also cost
00:05:43me $1.30. This is the result they gave me though. And honestly I can't really complain about this.
00:05:48This is exactly what I asked for. It is a personal finance management dashboard. I'd say it looks
00:05:53really nice as well. And we have all of the features like the extra pages here. And I have
00:05:57checked the adding funds and sending money. It all works. So it's done a very good job. Now I'm also
00:06:01always curious to see what tools it used to do the task. Because I don't actually give it any
00:06:05instructions on what stack to use in the prompt. You can see for the front end here it went with
00:06:09react and react dom. It didn't actually use any router or anything like that. It actually built its
00:06:12own view system. So I probably would have preferred that you use something like react router,
00:06:16tan stack or next.js. And then on the back end it also built out its own server. I believe this
00:06:21one is using express.js as we can see down here. But then it also built its database just using a
00:06:26JSON file. Didn't use any SQLite or anything like that or drizzle which I probably would have liked to
00:06:31see if we were to scale this app in the future. But again obviously me knowing that I could have
00:06:35given that in the prompt. But I always like to see what these will pick as their default. This is a
00:06:38result I actually got using K3 via Kimi code. And you can see this one is actually a bit simpler.
00:06:43It's only one page and it doesn't have our sidebar. Again all of the features are working and it has
00:06:47a pretty similar look to what we got out of open router. Looking at the code though I
00:06:51do actually prefer what Kimi code gave us here with K3. It's got a pretty similar front end where it
00:06:55just uses react and no routing library. But the server did actually use a database. You can see
00:06:59it's built and expressed here. We do actually have a finance database and it's actually using node
00:07:04SQLite. Now if you compare this to the frontier models again this one is Fable 5 in Claude code.
00:07:08You can see it worked for 56 minutes on that max effort. And then I also have GPT 5.6 soul max
00:07:13where it thought for 31 minutes to make the application. Here's what Fable 5 gave us and
00:07:17honestly it's a pretty similar design. I mean there's not much more you can actually do with a
00:07:21personal finance dashboard. And all of the features do work in here. It's actually gone with a bit of a
00:07:25different style for the fonts which I've seen Fable and Opus do quite a lot lately. They seem to be
00:07:29retraining it to move away from some of those previous AI looks that they had and they're building in a new
00:07:34AI look. And this seems to be the one they're picking. I'd say these charts are actually a little
00:07:38more useless than what we got out of Kimi K3. But overall it looks pretty similar and it works
00:07:43basically the same. Even the code is pretty similar. You can see it chose React for the front
00:07:47end and it didn't actually use a router either. It wrote its own. And then also in the database it
00:07:51did successfully use a database properly. So we're using Node SQL in this one as well. And for the
00:07:57actual server it's just running Express 2. Moving on to GPT 5.6 soul though. This one is a bit of a
00:08:02weird one. It's definitely my favorite design out of the lot. And again all of the features are
00:08:06working. Let me know if you agree that this is the best design. But the weird part is in the code.
00:08:11This is definitely the most complete application code that we have seen in these examples. It
00:08:15actually built out a full NextJS application and it used Drizzle just as I would. And it used NextJS
00:08:20as well for the front end as well as the server. But the part I find a little bit weird is that it
00:08:24chose to host it on Cloudflare which isn't odd by itself but it also used for Next. And it's not
00:08:29necessarily that I wouldn't recommend that. I just think it's a little bit untested. And it's pretty
00:08:33new which shows that they're actually pushing GPT 5.6 soul to use this. And I can only speculate as to
00:08:38why but it seems it might be because that's how that ChatGPT sites actually work. You can
00:08:42see here actually decided to host this for me on that ChatGPT site feature even though I didn't ask
00:08:47it to. Personally I don't really like that very much when I'm coding an application. I don't want
00:08:50it to host it for me. I'd prefer to handle all of that myself. But I could have asked that in the
00:08:54prompt. I kind of would prefer that I have to ask it to host it instead of the other way around.
00:08:58The final result I have here is GLM 5.2 and this one's done a pretty similar thing to what Kimi K3
00:09:04did in Kimi Code where it's just built this single page. But again all of the features work and I do
00:09:08think this design is pretty nice. GLM 5.2 actually took 15 minutes to do this and you can see it cost
00:09:13me a dollar and 11 cents. As for the actual code GLM 5.2 chose to go down the Next.js route as well
00:09:19which I do actually like. But it did build out its own store instead of using a database. So all of this
00:09:24it's just in memory in JavaScript. So from my quick test there I think Kimi K3 does stand up with Fable
00:09:29and GPT 5.6 Sol and live up to the hype that the benchmarks are giving it. And honestly it's getting
00:09:33quite hard to show you just how powerful these models are in a video demonstration. You tend to
00:09:38have to use them over a course of a week in a production code base to actually see the quality
00:09:42of the code. If you have any ideas for tests that I can actually show off in a video that would
00:09:46actually put these models through a hard test let me know in the comments down below. I do also want
00:09:50to show off some of the examples that they had in their tech blog. You can see Kimi
00:09:53K3 built this game here but did have another tool to actually generate these assets so the cowboy and
00:09:58the horse weren't generated by K3. They were gone elsewhere. You can see he's done a very good job
00:10:03of this in 3GS. And the other thing that's crazy is this chip design one. You can see as an early
00:10:08proof of concept Kimi K3 designed a chip to serve a nano model built on its own architecture. In a single
00:10:1348 hour autonomous run K3 built, optimized and verified the chip using open source tools. So hopefully
00:10:19you've seen that this model is absolutely awesome and it's definitely going to be a bit of a hit to
00:10:23these frontier closed labs. Or at least it will be when they actually drop the weights. They haven't
00:10:27dropped them yet but they do plan on doing so. So I hope we see them very soon. What do you think
00:10:32about this? Did you expect a Chinese lab to catch up this quickly? And does this model appeal to you?
00:10:36Let me know in the comments down below. While you're there, subscribe. And as always, see you in the next one.