I Tested Both New 30B Models... One Of Them is TERRIBLE! (Muse Glimmer & Lightning 3.5)
BBetter Stack
Computing/SoftwareInternet Technology
Transcript
00:00:00you know i'm rarely this harsh about ai model outputs but this time i just have to say it
00:00:08this is probably the worst model i've ever tested so meta and nvidia both dropped new models this
00:00:15week one day apart from each other and suspiciously enough both of them are 30 billion parameters large
00:00:21both of them are designed to run comfortably on consumer hardware but there are stark differences
00:00:27in the way they operate i tested both of them on a different set of tasks and honestly one of them
00:00:32is complete trash the other one is actually quite usable which one is which well you'll just have to
00:00:39stay tuned to find that out so in this video we'll take a look at both of these models see how they
00:00:44work and put them in a challenge match against each other to see which one is ultimately the
00:00:50better model it's gonna be a lot of fun so let's dive into it
00:00:58so first let's look at both of the models first up we have muse glimmer which was published on monday
00:01:04which is a distilled version of their flagship model muse spark and it's built specifically so it can run
00:01:10on consumer gpus like the rtx 5090 and they even have quantized versions that can run on just a regular m4 or
00:01:18m5 mac now rumor has it the reason why they released this model was because they realized that muse spark
00:01:25is not that popular because everyone is still using either clod or codex so they decided to go the open
00:01:31source route instead i'm not sure that rumor is true but if it is i wouldn't be surprised but anyway
00:01:38muse glimmer also ships with d flash support which basically enables the model to do speculative decoding which
00:01:45essentially works by having a smaller drafter model that guesses a whole chunk of 16 words at once and
00:01:52then the main model just checks that guess and fixes whatever it got wrong this gives you a three times
00:01:58bigger speed boost without sacrificing any of the quality so that's a nice add-on but does it actually
00:02:04work well we'll see that in the tests now let's talk about what nvidia put out on tuesday this is nemotron
00:02:12lightning 3.5 and it's also a 30 billion parameter model but it's built completely differently it's
00:02:18what's called a mixture of experts model and what that actually means is that instead of running the
00:02:24whole model every single time you ask it something it's got a router that picks a handful of smaller
00:02:30expert sub networks to handle each word as it goes so even though the model is 13 billion parameters on paper
00:02:38only about 3 billion of those are actually doing work at any given moment and that's the whole
00:02:44trick behind why it's so fast and cheap to run despite being this big model on paper now nvidia also
00:02:50really wants you running this thing inside their own routing setup which is called nemo switchyard it's
00:02:56basically a system that automatically hands off each step of the task to whichever model is best suited for it
00:03:03so it might bounce between open models proprietary ones even nvidia's own models depending on what
00:03:10the task needs so realistically the best case scenario for lightning is probably running it inside of this
00:03:16setup but for this video i'm going to run it standalone so we can compare it fairly against muse glimmer so
00:03:22how do they both stack up on paper against each other so they're both 30 billion parameters they both support
00:03:30tool calling which is great news because it means we can actually run some fun experiments with them but
00:03:35nemo tron lightning has a bit of a disadvantage it can't see images at all you can't hand it a picture
00:03:41and ask it what's going on muse glimmer on the other hand has actual vision system built in so it can look
00:03:48at an image and reason about it there's also a real gap in context window size muse glimmer can hold about
00:03:54128 000 tokens at once but nemotron lightning can hold a full million so on paper lightning should be the
00:04:03model you reach for when you've got like a massive document you need to dig through while glimmer is
00:04:09more built for shorter denser back and forths all right now let's do some proper testing on both of these
00:04:15models so i have a small home lab workstation here with an rtx 50 90 and 64 gigabytes of ddr5 ram
00:04:23and i've loaded up both of these models on an olama server and i'm going to be querying
00:04:29this server through open code and basically this is what my open code json setup looks like
00:04:34so we got two models right here and i've also got two mcp servers loaded and one of them is an adobe
00:04:41premiere plugin built to control the editing software externally and another one is an mcp server for
00:04:47epidemic sound to fetch some music and this will come in handy in one of the tests that we're going
00:04:52to be doing later okay but first we're going to start with a different test i'm going to ask both of
00:04:57the models to build a simple web page showing how an esp32 c3 microcontroller connects to a breadboard
00:05:06with a switch and a diode on it i've got a pretty long prompt for this one so i'll drop it in the github
00:05:12gist if you want to follow along before doing these tests i gave this prompt to opus 5 to see what a
00:05:18state-of-the-art frontier model would give us and i'm pretty satisfied with the result basically it's
00:05:24just a visual guide how this electronic setup works with some kind of interactivity to see how the
00:05:30current flows and everything now i'm not expecting that any of the models will do as good of a job
00:05:36but i expect them to at least try to set up a simplified version of this so first let's run this
00:05:42test with nemotron lightning and as we can see the model immediately fired a bunch of curl requests to
00:05:47fetch some pages and it was working through the task for about five minutes and by the end of it we got this
00:05:54a blank page with some buttons there are no visual elements the buttons don't work so at this point
00:06:00it seems like lightning did not do this task properly but i wanted to give it the benefit of
00:06:05the doubt so i prompted it again saying that there are some issues on the page to see if the second
00:06:10round would fix it and the second round took lightning six minutes to finish and by the end of it we got
00:06:16exactly the same broken result nothing is displayed the buttons are not working so at this point
00:06:21unfortunately i have to declare that lightning just completely failed this task all right let's
00:06:26switch to muse glimmer and see how it did with the same task with the same prompt so muse glimmer
00:06:32fired up a bunch of web fetch tool calls to fetch some of the pages and the entire round was completed in
00:06:38roughly two minutes and 50 seconds and by the end of it we got this web page which honestly was a lot
00:06:44better than what lightning did at least the page had buttons that were functional there was some kind of
00:06:49representation of the current and it did actually make a pin layout of the controller and there are
00:06:55even some tool tips that explain what each area is and what it does but unfortunately it still did not
00:07:02create a proper wiring diagram so i wouldn't say i'm satisfied with this result but this is a big step up
00:07:08from what lightning did with the same task so i'm gonna give it to muse glimmer in this case at least it tried and
00:07:15honestly i reckon with a few additional rounds of bug fixes we could get it to a working version but
00:07:21now let's move on to the second test and i'm pretty excited about this one because specifically for this
00:07:26test i created an adobe premiere plugin that connects to the editor via the uxp adobe bridge so this means
00:07:35that any ai model that supports tool calls can theoretically navigate premiere autonomously and do
00:07:41certain actions on the editor i tested it out with opus 5 and it works very well and if you want to
00:07:47check out this plugin i also added a link to the github repo in the description below all right so since
00:07:52both of these models support tool calling they should have no problems working inside of premiere and since
00:07:58nemotron lightning is custom tailored for more quick turnaround tasks like something that requires
00:08:05several rapid tool calls i expect lightning to be able to execute this task better than muse glimmer
00:08:11but let's see if that is actually the case so for the task itself i have premiere opened in a project
00:08:16here and i have a recent google keynote presentation recording and it is already transcribed and just for
00:08:22the fun of it i want the model to parse through the transcript and find all the occurrences where
00:08:28someone says the word ai and i wanted to create a fast montage of short clips of people just saying ai
00:08:35all the time and then use epidemic sounds mcp server to add appropriate background music to it so
00:08:41a pretty straightforward simple task for an ai model so let's see how lightning performs so it starts out
00:08:47pretty strongly we can see that lightning recognizes the tool calls and fires them very rapidly
00:08:52and it was able to fetch the transcript and also find all the occurrences of people saying ai and
00:08:58afterwards it also successfully managed to cut the individual pieces and lay them out in a separate
00:09:04sequence so that is all great but then when it comes to adding the background music with epidemic sound
00:09:09this is where things started to get a bit tricky it found the soundtrack and it laid it on the sequence
00:09:15but on the first audio track which basically layered it on top of the speaker audio which basically means
00:09:22that now there's no actual audio and we're just left with the background music so i called this out in a
00:09:28follow-up prompt to see if it would fix the mistake and indeed it recognized the mistake and moved the
00:09:33background music to a separate audio track and then declared the task done when in fact we still don't
00:09:39have the speaker audio because it was deleted earlier so then i prompted it to bring back the speaker audio
00:09:45and at this point it went completely off the rails because while trying to salvage the audio it started
00:09:50deleting the original tracks and it completely messed up the timeline so at this point i was just too
00:09:56frustrated to continue so i have to declare again that lightning unfortunately failed at another task
00:10:03which is a real bummer because i thought this is the sort of task lightning would be best suited for but
00:10:08anyway let's now switch to muse glimmer and see how it performed on the same task so once again we started
00:10:14with the same prompt and i noticed that muse glimmer was a bit slower in tool calling but it did
00:10:19everything properly it laid out the sequence correctly and found the background music and applied it to the
00:10:25separate audio track and it even trimmed the duration of it to match the video track and also added a fade out
00:10:31effect to the music as was required in the prompt so this already shows that muse glimmer is much more
00:10:37intelligent when it comes to identifying these sorts of details and small issues so i was pretty happy
00:10:42with the first pass and i can already confidently declare that muse glimmer is the winner of this task
00:10:48but there's one more thing i wanted to ask it to do i wanted it to change the background music because
00:10:53i wasn't quite satisfied with the first choice so then i prompted it again to find another background
00:10:59track from epidemic sound and it did so elegantly and successfully replaced the old music track with the
00:11:05new one so well done muse glimmer you know i'm actually considering using this one on a daily basis
00:11:11if i ever max out on my clawed token usage so at this point even with a task that is kind of custom
00:11:18tailored to the sorts of tasks that lightning should be superior on it still failed so i want to do this
00:11:24final task where lightning should be able to perform better just because it has a 1 million token context
00:11:30window so for this third task i wanted to actually put that context window gap to real use so i grabbed
00:11:36something real off of scc's website and it is nvidia's own fy 2026 10k filing it's a big document with
00:11:46over 100 pages of dense financial and legal limbo and there's a little bit of a fun angle here too
00:11:52because we're essentially asking nvidia's own model to go read nvidia's own paperwork about themselves so
00:11:59the question was pretty simple i asked the model to find what percentage of revenue came from nvidia's
00:12:04two largest customers which business segment that revenue was attributed to and then cross check
00:12:10whether that same number shows up again later in the notes to the financial statements so first let's see
00:12:16how lightning did on this task honestly i expected this to be an easy one especially for lightning because
00:12:23it is tailored to read through a large chunk of documents with its big context window but look
00:12:28what happened its fetch request to the sec came back with an error and instead of just retrying it fell
00:12:35back to the general search and ran with whatever came up which turned out to be wrong and it told me two
00:12:41customers made up 23 percent of revenue combined split roughly 13 and 10 percent attributed to the data
00:12:48center segment none of that is correct because the actual filing says 22 and 14 36 combined so not only
00:12:57did it fail at this task but it also failed to properly utilize its built-in tool calls because it
00:13:03couldn't even fetch a web page properly but worst of all it hallucinated the numbers that simply were not true
00:13:13you know i'm rarely this harsh about ai model outputs but this time i just have to say it
00:13:19this is probably the worst model i've ever tested i mean i even made it simple for nemotron's
00:13:25lightning i tried to find a task that would be custom tailored to it and it still failed miserably
00:13:32god but anyway we still have to check how muse glimmer did with the same task so when prompted
00:13:38muse glimmer with the same task it took about three minutes to finish it and it made a bunch of web
00:13:44requests some of them failed but nonetheless it still managed to work its way around it and retrieve the
00:13:50correct information which in this case is 22 and 14 and 36 combined so once again well done muse glimmer
00:13:59another test passed so overall where do we stand with this well i would give muse glimmer a two out of
00:14:06three score because it did accomplish two of the tasks with an acceptable result but when it comes to
00:14:12nemotron's lightning dude i am really disappointed i wouldn't have expected something like this from
00:14:18nvidia honestly i mean do they even test their own model can we actually trust their benchmarks is
00:14:26anything real anymore these are good existential questions that we will not answer in this video
00:14:31but subscribe to our channel and maybe we'll do something about it so anyway folks those are the two
00:14:36models in a nutshell so what i can take away from these tests is that i was pleasantly surprised by muse
00:14:42glimmer and i would actually consider using this model on a day-to-day basis for certain types of tasks
00:14:48but when it comes to nemotron's lightning i've said what i've said and i don't want to comment on it anymore
00:14:55but those are just my thoughts and observations what do you folks think let us know in the comment section
00:15:00down below this time instead of doing the regular outro i'm gonna leave you with something funny at the end of this video
00:15:06so let me present to you with the ai montage that muse glimmer performed on one of the tasks enjoy and i will see you in the next videos
00:15:14next videos
00:15:22is
00:15:34a i