This Open Source Model Fixes AI's Biggest Flaw (ThinkingCap)

EnglishالعربيةDeutschEspañolFrançaisहिन्दीBahasa Indonesia日本語한국어PortuguêsРусский中文
BBetter Stack
Computing/SoftwareSmall Business/Startups

Transcript

00:00:00This is the same question asked to the same base model.
00:00:03Watch the token counters here.
00:00:05One of them is spending a lot of time reasoning about the question,
00:00:09while the other one has already completed the task.
00:00:12And if we check both outputs, the answer to the question is basically the same.
00:00:16But the model on the right wasted far more tokens to get to the same conclusion.
00:00:21And that is pretty cool, because the only difference between these two models
00:00:24is that one on the left is running Thinking Cap,
00:00:27a custom fine-tuned version of the same model,
00:00:31which aims to ensure you get the same quality outputs
00:00:34while sacrificing less tokens and getting to your answer two times faster.
00:00:39So in today's video, we'll take a look at Thinking Cap,
00:00:42see how it works, and then we're going to test them out ourselves
00:00:45to see if this really is a novel approach to fine-tuning AI models.
00:00:50It's going to be a lot of fun, so let's dive into it.
00:00:57Before we talk about Thinking Cap, let's rewind the clocks a bit to see how we actually got here.
00:01:03The first ever broadly available model, GPT-3, was launched back in 2020,
00:01:09and by the late 2022,
00:01:11ChatGPT put GPT 3.5 in front of the general public for the first time.
00:01:17That's when most people's relationship with AI actually started.
00:01:21AT4 followed in 2023, which was another jump-in capability,
00:01:25but it still had fundamentally the same idea.
00:01:28Predict the next token and answer immediately.
00:01:31But then, in September 2024, OpenAI released O1,
00:01:37and this was the first model which used a thinking mode before answering.
00:01:42So instead of firing off a response immediately,
00:01:45it spent extra time and tokens reasoning through the problem first.
00:01:49That one architectural shift kicked off the reasoning model era.
00:01:54DeepSeq R1 followed a few months later
00:01:56and brought the same idea to the open source world.
00:02:00And pretty quickly,
00:02:01reasoning became the thing that every major AI model maker was shipping.
00:02:05But now, a little over a year later,
00:02:08we're starting to see the opposite trend show up.
00:02:11Models fine-tuned specifically to think less and not more.
00:02:15In a way, we're going back to basics,
00:02:17just with everything we learned from the reasoning era baked in.
00:02:21But here's the thing.
00:02:22That shift towards more thinking solved one problem and quietly created another.
00:02:28Nobody actually taught these models when to stop thinking.
00:02:31If you ask a reasoning model something genuinely simple,
00:02:34it will still burn through thousands of tokens,
00:02:37re-deriving things it already figured out three sentences ago,
00:02:41restating the same point in slightly different words,
00:02:44or in worst case scenario, it's going to start looping.
00:02:48And I mean that almost literally.
00:02:50Let me show you one of the most ridiculous examples I've run into.
00:02:54This is step 3.5 flash.
00:02:56And when I was testing this reasoning model,
00:02:58I just sent it a single word, hello.
00:03:01And look at what it does with that.
00:03:03It spends multiple paragraphs debating whether it's required to introduce itself by its exact model name,
00:03:09whether always use the exact name when introducing yourself means it must introduce itself every time or only when it chooses to.
00:03:19Then it weighs whether mentioning today's date is relevant.
00:03:22And eventually, after all that, it lands on a response any of us would have typed in about two seconds.
00:03:28Hello, I'm step 3.5 flash.
00:03:31How can I assist you today?
00:03:33So that's not really reasoning.
00:03:35That's a model that's trained to think, but it was never trained to know when thinking is done.
00:03:41Now, BottleCap AI is a European startup focused specifically on making inference more efficient.
00:03:47And what they did with ThinkingCap is honestly really clever.
00:03:51So they took the model Quen 3.6, the 27 billion parameter version,
00:03:55and they weren't trying to make it smarter and they weren't teaching it new skills or changing its personality or touching its safety behavior either.
00:04:04The only thing they wanted to change was how much compute it burns getting to an answer it was already capable of giving.
00:04:11And that sounds simple, but it's actually harder than it seems.
00:04:15So the lazy way to make a model faster is to just cut its reasoning short.
00:04:20Force it to stop after X amount of tokens, whether it's actually done or not.
00:04:25That will absolutely reduce token counts.
00:04:27But it will also make the model wrong more often because sometimes it genuinely needs those extra steps.
00:04:34So the real engineering challenge here isn't making it shorter.
00:04:38It's making it shorter without secretly making it dumber.
00:04:42And here's how they actually did it.
00:04:44Starting from the Quen 3.6, 27 billion parameter model base checkpoint,
00:04:48they trained it on a curated set of problems spanning multiple domains and difficulty levels.
00:04:55And instead of rewarding the model for getting the right answer,
00:04:58they rewarded it for getting the right answer efficiently.
00:05:02Because if a model only gets rewarded for correctness, being verbose doesn't cost anything.
00:05:08There's no incentive to stop thinking early.
00:05:10But by explicitly rewarding efficiency alongside correctness,
00:05:14the model learns to recognize when it already has enough to commit to an answer.
00:05:19So in result, they got a model that behaves almost identically to the original.
00:05:23Now let's actually look at the numbers because this is where it gets interesting.
00:05:27Across 12 out-of-domain benchmarks, meaning problems that weren't part of the training data,
00:05:33thinking cap cut its thinking tokens by an average of 45.8%.
00:05:37And the accuracy barely changed.
00:05:40It moved by less than one percentage point on average.
00:05:43And on the benchmarks that were part of the training domain,
00:05:46the token reduction was even bigger, almost 58%.
00:05:49And on the GSM 8K specifically, accuracy actually went up from 93.3% to 96.5%.
00:05:58And they also tracked something called a looping rate.
00:06:00How often a model gets stuck rephrasing the same reasoning over and over without converging,
00:06:06like we just saw in that step 3.5 flash example.
00:06:10That dropped too on most benchmarks,
00:06:12which tells us that a lot of the extra reasoning these models were doing was never actually productive.
00:06:18It was just noise that happened to look like effort.
00:06:21Okay, numbers on a page is one thing, but let's actually test it ourselves.
00:06:26So I'm running this test on a machine with an RTX 5090 with 64 gigabytes of RAM.
00:06:32And I'm going to ask both of the models the same question.
00:06:35What is the smallest positive integer n such that n factorial has exactly 100 trailing zeros?
00:06:42And to get to the answer of this question, it has to use the Legendres formula,
00:06:47in which case for this particular question would yield the answer of 405.
00:06:51There's only one correct answer.
00:06:53There's no in-between.
00:06:55So if we get 405, then we know that the model has answered correctly.
00:07:00Okay, so first let's run the standard 27 billion parameter QUEN 3.6 quantized model
00:07:05and see how it performs.
00:07:07And as you can see right off the bat, the token speed is not that bad.
00:07:12It's averaging about 60 tokens per second.
00:07:14But look how much thinking this model is doing.
00:07:17It's absolutely ridiculous.
00:07:20We're well past a minute now, and it's still churning out tokens just on the reasoning part.
00:07:26I had to speed up the preview here because the total time was running over two minutes.
00:07:30But nonetheless, at the end of it, we did get the correct result, which is 405, which is good.
00:07:37But look at the total elapsed time.
00:07:39It's 140 seconds.
00:07:41And look at the token expenditure.
00:07:43More than 7,000 tokens were spent just on reasoning alone, and merely just 900 tokens were actually spent on printing out the answer.
00:07:52So this example clearly demonstrates how over-the-top ridiculously QUEN is reasoning about every request.
00:07:59Now let's switch to the thinking cap version of the same model.
00:08:02And just on the record, we're using the same 27 billion parameter model with the same quantization.
00:08:08And as you can see here, we're 20 seconds in, and we're already done with reasoning.
00:08:13And just 9 seconds later, we get the answer, which in this case is also correct, 405.
00:08:19And look what a stark difference this is.
00:08:21Not only did we only spend less than 2,000 tokens in total, of which only 1,100 was allocated for reasoning,
00:08:30but we also got a slightly faster tokens per second speed of 62 tokens per second.
00:08:36So in every metric, this model just blows the base QUEN 3.6 model out of the water.
00:08:42It's faster, it's more efficient, it costs less tokens, and it gives you the same correct answer.
00:08:48So this is a pretty impressive improvement.
00:08:51And here's a detail from their write-up that I have to mention, because it's genuinely a funny accident.
00:08:57Their original goal was to only shrink the thinking trace, so that's the reasoning part,
00:09:02while keeping the final answer exactly the same length as before.
00:09:06But they had a bug.
00:09:07The shortening accidentally applied to the final answer too, not just the thinking.
00:09:12So they fixed the bug, but apparently, internally, everyone actually preferred the buggy version.
00:09:18So their theory is that humans get tired writing long answers, so we naturally compress what we say.
00:09:25And these models never had that kind of fatigue built in until now.
00:09:29So they ended up shipping the broken, concise version anyway, and saving the technically correct one for a future release.
00:09:37So there you have it, folks.
00:09:38That is Thinking Cap in a nutshell.
00:09:40I guess the big takeaway from all of this is that we've always believed that the more the models think, the more intelligent they are.
00:09:48Whereas Thinking Cap just proved that is not necessarily the case.
00:09:53Once you actually train against it, you can get the same quality output for a fraction of the cost, with basically nothing given up.
00:10:01If you want to try the model on your own, it's freely available on Hugging Face under Apache 2.0 license.
00:10:08And I want to hear from you, folks.
00:10:10What do you think about this new approach?
00:10:12Do you see any pros or cons to this technique?
00:10:15Let us know in the comment section down below.
00:10:17And, folks, if you like these types of technical breakdowns, please let me know by smashing that like button underneath the video.
00:10:23And also, don't forget to subscribe to our channel.
00:10:26This has been Andres from BetterStack, and I will see you in the next videos.

Key Takeaway

Thinking Cap fine-tunes the 27 billion parameter Qwen 3.6 model by rewarding efficiency alongside correctness, cutting thinking tokens by nearly half without sacrificing output accuracy.

Highlights

  • Thinking Cap is a custom fine-tuned version of the 27 billion parameter Qwen 3.6 model that reduces thinking tokens by an average of 45.8% across 12 out-of-domain benchmarks while maintaining nearly identical accuracy.

  • OpenAI introduced the reasoning model era with O1 in September 2024, shifting the industry standard toward spending extra time and tokens reasoning through problems before answering.

  • Reasoning models frequently loop or burn through thousands of tokens on simple inputs like the word 'hello' because they lack training on when to stop thinking.

  • During training, Thinking Cap rewards efficiency alongside correctness, teaching the model to recognize when it has enough information to commit to an answer.

  • On the GSM 8K benchmark, Thinking Cap's accuracy increased from 93.3% to 96.5% while reducing token expenditure.

  • Testing on a math problem requiring Legendre's formula showed the base Qwen 3.6 model taking 140 seconds and over 7,000 reasoning tokens, whereas Thinking Cap completed the task in under 30 seconds using less than 2,000 total tokens.

  • Thinking Cap is freely available on Hugging Face under the Apache 2.0 license.

Timeline

Introduction to Thinking Cap

  • Thinking Cap is a custom fine-tuned version of an existing base model.
  • The model achieves the same output quality while using significantly fewer tokens.
  • Inference speed reaches the final answer two times faster than the unmodified model.

Two models process the exact same question simultaneously, but one spends excessive time reasoning while the other completes the task rapidly. Both arrive at the same conclusion, yet the standard model wastes tokens. Thinking Cap resolves this inefficiency by reducing token usage without compromising response quality.

The Evolution of AI Reasoning Models

  • GPT-3 launched in 2020, followed by GPT-3.5 via ChatGPT in late 2022 to establish standard next-token prediction.
  • OpenAI released O1 in September 2024, initiating the reasoning model era by incorporating a thinking mode prior to answering.
  • DeepSeek R1 brought the same reasoning architecture to the open-source community shortly thereafter.

Early foundational models predicted the next token and answered immediately. The release of O1 shifted the paradigm by forcing models to spend extra time and compute reasoning through problems internally. This architectural shift quickly became an industry-wide standard among major model creators.

The Flaw of Unconstrained Reasoning

  • Reasoning models lack training on when to stop thinking.
  • Simple inputs, such as the single word 'hello', cause models to waste thousands of tokens debating formatting and metadata.
  • Excessive reasoning often manifests as looping or re-deriving already established points.

The shift toward extensive reasoning solved complex problem-solving capabilities but introduced a new flaw of unconstrained computation. When given trivial prompts, models burn through massive token counts debating irrelevant details instead of providing direct answers. This behavior stems from training models to think without teaching them when thinking is complete.

Engineering Efficient Inference with Thinking Cap

  • BottleCap AI developed ThinkingCap using the 27 billion parameter Qwen 3.6 model.
  • The training methodology rewards the model for reaching correct answers efficiently rather than just for correctness.
  • Simply cutting token generation short risks lowering accuracy, making efficient reward structuring the primary engineering challenge.

European startup BottleCap AI targeted inference efficiency without altering the underlying intelligence, personality, or safety behavior of the model. Standard methods of shortening outputs often degrade accuracy because models are cut off mid-thought. By training the network to optimize for both correctness and efficiency, the model learns to commit to an answer earlier.

Benchmark Performance and Accuracy Metrics

  • Across 12 out-of-domain benchmarks, Thinking Cap reduces thinking tokens by an average of 45.8%.
  • Accuracy shifts by less than one percentage point across these external benchmarks.
  • GSM 8K accuracy improves from 93.3% to 96.5% alongside a token reduction of nearly 58%.

Evaluations across diverse datasets prove that reduced thinking time does not equate to diminished performance. In-domain benchmarks show even sharper token reductions. Furthermore, tracking looping rates reveals that much of the reasoning performed by standard models consists of unproductive noise.

Hardware Benchmarking and Comparative Testing

  • Tests executed on a machine equipped with an RTX 5090 and 64 gigabytes of RAM compare the base and fine-tuned models.
  • A mathematical question requiring Legendre's formula and yielding the integer 405 serves as the evaluation test.
  • The standard Qwen 3.6 model takes 140 seconds and over 7,000 reasoning tokens, whereas Thinking Cap finishes in under 30 seconds using fewer than 2,000 total tokens.

Live testing validates the efficiency gains. While the standard Qwen model spends over two minutes churning through tokens for a single mathematical calculation, Thinking Cap reaches the exact same correct answer in a fraction of the time and token expenditure while maintaining high generation speeds.

The Accidental Concise Output and Final Conclusions

  • An initial bug caused the shortening mechanism to apply to final answers as well as thinking traces.
  • Developers retained the concise output style because humans naturally compress language when fatigued.
  • Thinking Cap is available freely on Hugging Face under the Apache 2.0 license.

A coding bug inadvertently compressed the final answers alongside the reasoning traces, but internal testing favored the shorter responses due to their human-like conciseness. Ultimately, the project demonstrates that increased token expenditure does not correlate with higher intelligence. The model is publicly accessible for developer experimentation.

Community Posts

View all posts

How to Deploy a 27B Model on an Internal Server and Control Token Costs

makedreamAug 11, 20269220
Write about this video