I Made Qwen 3.8 and Claude Opus 5 Build the Same Game
BBetter Stack
Computing/SoftwareVideo & Computer Games
Transcript
00:00:00A 2.4 trillion parameter open-weight model just scored 58 on artificial analysis.
00:00:06That's the exact same score as Quinn's proprietary Max model.
00:00:10Claude Opus 5 is still ahead at 63, so we just assume this is simple.
00:00:14Claude is smarter, use Claude.
00:00:16But I gave both models the same real task.
00:00:19Same prompt, same constraints.
00:00:22And once you actually use them and see how close they are, Claude might not seem like the first choice.
00:00:30Now, key point to make here.
00:00:34I'm using the hosted version here, so I'm still paying for inference.
00:00:37The difference is that, unlike Claude, Quinn also gives you access to the weights if you want to deploy it yourself.
00:00:45But at 2.4 trillion parameters, running the full model locally is not realistic for most of us, at least not for me.
00:00:52This is Quinn 3.8 2.4TA95B.
00:00:57Yeah, okay, it's a mouthful.
00:00:59It has roughly 2.4 trillion total parameters, but unlike its mixture of expert model, only around 95 billion are active at any given time.
00:01:08It's the open-weight counterpart to Quinn 3.8 Max.
00:01:13A95B is the open-weight base model, and 3.8 Max is the enhanced production-ready version of that same model.
00:01:21But remember, they have the same score.
00:01:23Except Max has some extra capabilities layered onto it.
00:01:27Vision, flexible thinking, tools, and better defaults.
00:01:30But both of these score 58 on artificial analysis, with Claude Opus 5 Max scoring 63.
00:01:36So, on Raw Intelligence, yeah, Claude wins.
00:01:40But we don't always buy the benchmark points.
00:01:42We care about more things.
00:01:43We care about things like, how much does it cost?
00:01:47How fast is it going to respond?
00:01:49Can it handle a huge code base?
00:01:52Can it, I actually control how I deploy it?
00:01:54And that's where this comparison is going to get interesting.
00:01:57Quinn gives you roughly 1 million token context window.
00:02:01Hosted pricing is around $2 per million input tokens, and $6 per million output tokens.
00:02:07And you get access to the model's weights.
00:02:09That's the big win.
00:02:10But the catch here is, it's slower.
00:02:13Artificial analysis puts it around 47 to 48 tokens per second.
00:02:18And it really likes to talk.
00:02:19So, if you ask it for a small fix, there's a decent chance it's going to give you the fix,
00:02:23plus a small life story that we didn't even ask for.
00:02:26That sounds annoying, and sometimes it is.
00:02:29But for bigger coding tasks, that same behavior could actually become useful.
00:02:33So, let's test it.
00:02:34If you enjoy coding tools that speed up your workflow, be sure to subscribe.
00:02:37We have videos coming out all the time.
00:02:39Now, I'm using the hosted version here.
00:02:42You can call it through OpenRouter, DeepInfra, Fireworks, or another provider that actually supports it.
00:02:47The nice part is that the API is OpenAI compatible.
00:02:51So, the setup, we already know how the setup goes.
00:02:54And just to clear up one thing here.
00:02:56Yes, this is an open-weight model.
00:02:58So, no.
00:02:59You are probably not running the full thing on the PC under your desk.
00:03:03A serious local deployment needs multi-node data center class hardware.
00:03:07This is huge.
00:03:08So, for most of us today, hosted inference is the realistic way to use it.
00:03:13Now, for the actual test.
00:03:15I didn't want to give both models a cheap prompt like build a to-do app, right?
00:03:19We can build a to-do app.
00:03:20That's fine.
00:03:21That's easy to just fake to see which model is going to perform best.
00:03:25A coding model can generate something that looks impressive and still completely falls apart when you actually try to play it.
00:03:31Play it.
00:03:32Remember that.
00:03:32I want to build a game.
00:03:33So, I gave both models the exact same task.
00:03:36Build a complete playable 3D endless runner inspired by Temple Run on a single HTML file.
00:03:43Use 3.js for a CDN.
00:03:46The player runs forward automatically, can switch between three lanes, jump, and slide.
00:03:52The path needs to generate procedurally with turns, gaps, and obstacles.
00:03:57Score is based on distinctive survival, and it needs game over and restart.
00:04:02A smooth follow camera, clean code structure, and most importantly, needs to actually be fun, maybe a little fun, and playable for at least 30 to 60 seconds.
00:04:11Same prompt, same rules, no extra help.
00:04:15Quen is on the left.
00:04:17Opus 5 is on the right.
00:04:19And almost immediately, they take two different approaches.
00:04:23Quen slows down at first.
00:04:25It starts thinking through the game loop, player state, collision detection, procedural generation, and how all these systems need to work together.
00:04:33Claude, as we know it, it moves faster.
00:04:35The 3.js scene goes up, players appear, and the 3D world starts taking shape.
00:04:41And just watching the code come in, Claude, feels like it's already winning.
00:04:45There's more code.
00:04:46We use Claude all the time.
00:04:47This must be winning.
00:04:49It's faster progress.
00:04:50But again, at the same time, that could be a problem.
00:04:53Because I didn't ask them to generate a lot of JavaScript.
00:04:56I asked them to build a playable game.
00:04:59So the real test starts when we hit run.
00:05:02Let's see what happens, right?
00:05:03Can I actually switch between all three lanes smoothly?
00:05:07Does jumping work?
00:05:09Does sliding work?
00:05:09Have you subscribed to the Better Stack channel?
00:05:12If not, hit that subscribe button.
00:05:14And the other question is going to be, can the game generate turns, gaps, and obstacles without creating impossible situations?
00:05:20These are the questions that we want answered.
00:05:23Can we die?
00:05:24Can we restart?
00:05:25Can I get a clean new game?
00:05:26So once both models finish, I run them side by side.
00:05:29Let's give it a run.
00:05:32Now you tell me which one is more interesting here, right?
00:05:35Which one is Quinn?
00:05:37You know that.
00:05:37Which one is Claude?
00:05:38Which one even looks better?
00:05:39Because now we're not comparing how convincing the code looks.
00:05:42We're comparing what the code actually does.
00:05:44One game feels smoother.
00:05:46One handles the path better.
00:05:48One does a better job with movement, collisions, and game state.
00:05:52And underneath all that, one gives you code that would be much easier to actually keep working on.
00:05:58What is it?
00:05:58You tell me again.
00:05:59Remember, this is the same prompt, just two different results.
00:06:03And that's where this comparison starts to really show.
00:06:06This version makes the test much stronger because every payoff maps directly back to something you would explicitly required in the prompt.
00:06:13Now, let's pull that result beside the benchmark numbers.
00:06:17Quinn 3.8 Max in the Open 2.4T model, both score 58.
00:06:23They both do it.
00:06:24Claude Opus 5 Max scores 63.
00:06:26So if the question is, which one has a higher intelligence score?
00:06:30I mean, come on.
00:06:30I just told you that, right?
00:06:32There is no debate there.
00:06:34Claude wins.
00:06:35But that's not the same thing as asking, which one should I actually build with?
00:06:39Because Quinn gives you something Claude doesn't give us.
00:06:42Open weights.
00:06:44And hosted.
00:06:45You're looking at roughly $2 per million input, $6 per million output tokens.
00:06:49That matters if we're building agents.
00:06:52An agent might read a giant repo, call the model again and again, generate tasks, inspect errors, and keep going.
00:06:58At that point, a model that's slightly weaker on a benchmark but much cheaper to run might actually make more sense.
00:07:05So the benchmark winner is obvious.
00:07:08The practical winner depends on what we're doing with it.
00:07:10And for me, this whole comparison comes down to three things.
00:07:14Speed, verbosity, access.
00:07:16First, speed.
00:07:17Quinn runs around 47 to 48 tokens per second in artificial analysis testing.
00:07:21For a coding agent running in the background, maybe that's fine.
00:07:24If you're sitting in your editor waiting for an answer, it's not great.
00:07:28And that gets worse because the second problem, verbosity.
00:07:31Quinn produces a lot of output.
00:07:34Sometimes it's useful.
00:07:35If I give an autonomous agent a complicated migration, I don't really care whether it gives me a short answer.
00:07:42I care whether it understands the migration and doesn't break production.
00:07:46Now, what do I actually like about this?
00:07:47First, getting weights for a model this capable?
00:07:50That's a big deal.
00:07:51Second, Quinn feels built for bigger, longer coding jobs.
00:07:54Not write a function.
00:07:56More like understand this repo, figure out how these pieces connect, make a plan, change multiple files.
00:08:03Claude can do it, Quinn is good at it, but more code base, what's going to be running longer?
00:08:08That's a much more interesting use case.
00:08:10Then, of course, there is cost.
00:08:12I just touched on cost.
00:08:13If your agent makes a few calls, pricing doesn't really matter.
00:08:17If you're making hundreds of calls, pricing might become more part of your actual decision.
00:08:22A cheaper model that gets you most of the way there can be more valuable than a slightly smarter model.
00:08:27You can't actually afford to run constantly.
00:08:30Now, all this control is useful, but it also leads directly to Quinn's biggest weakness.
00:08:35Quinn can overthink things.
00:08:37And when I say overthink things, I mean a lot.
00:08:39For complicated agent work, that can be a strength.
00:08:41For normal development questions, that is not a strength.
00:08:45That might just ramble on, take more time.
00:08:47And remember, that extra thinking is coming from a model that's already relatively slow.
00:08:53Now, yes, we get the weights, but this model, at 2.4 trillion parameters, most of us are still going to use it through hosted infrastructure.
00:09:01So, in practice, we gain openness, we gain control, without suddenly turning this into a casual local model.
00:09:08But if you're building a real product, you still need to pay attention to all the little details.
00:09:13So, there is real freedom here.
00:09:15It's just not zero-cost freedom.
00:09:16The giant 2.4T model gets all the attention, but for a lot of us devs, the smaller Quinn 3.8 models may end up being more useful.
00:09:24The 2.4T model is the showcase.
00:09:27It's what you use hosted when you want maximum capability.
00:09:31But something like the 27B version, it's much closer to the kind of model devs can realistically experiment with locally.
00:09:38And that creates different questions.
00:09:40The interesting question isn't only now how close can Quinn get to Claude with 2.4 trillion parameters.
00:09:47It's more along the lines of how much of that capability can Quinn keep when the model gets small enough for devs to actually run themselves.
00:09:55If you only look at the benchmark, Claude wins, 63 versus 58.
00:09:59If you only look at speed and polish, I'd still rather use Claude for a lot of everyday coding work.
00:10:04But if you care about cost, long-term agentic tasks, huge context, Quinn becomes much harder to ignore.
00:10:11If you enjoy coding tips and tricks like this, be sure to subscribe to the BetterStack channel.
00:10:15We'll see you in another video.
Community Posts
No posts yet. Be the first to write about this video!
Write about this video