Inkling: This Open-Weight Model Wants To Be Fine Tuned

Englishالعربية Deutsch Español Français हिन्दी Bahasa Indonesia 日本語 한국어 Português Русский 中文

Computing/SoftwareSmall Business/StartupsInternet Technology

Transcript

00:00:00Thinking Machines, the 12 billion startup from ex-OpenAI CTO Mira Moretti, just dropped its

00:00:05first ever model. There's 975 billion parameters, its Apache 2.0 license, and the weights are all

00:00:12on Hugging Face for anyone to download. And although it's not competing with frontier labs

00:00:17like Anthropic, its actual goal is to be fine-tuned specifically with its own platform,

00:00:22Tinker. So if you need a model that's trained on a highly specific task,

00:00:25then this may be exactly what you're looking for. Today we'll cover why you'd want to use

00:00:29this model specifically and all of the reasons why you'd want to fine-tune. And there's a super

00:00:35interesting way that this model processes audio and visual, which I've never seen before. So stick

00:00:40around and we're going to cover that as well in this video. And we cover AI topics constantly on

00:00:44this channel. So if you want to stay up to date, then subscribe to BetterStack.

00:00:52Thinking Machines are very upfront about the fact that their new model is not as strong as basically

00:00:57any general model on the market. And they say this in their launch post. So they're just managing

00:01:01expectations and you can see why in the benchmarks. On instruction following, it does beat GLM 5.2,

00:01:0779.8 to 73.3. But on terminal bench, where agents go off and actually do work, it gets beaten badly

00:01:14with 63.8 to 82.7. So it's good at following orders, but much weaker at working things out on its own.

00:01:21However, it is a great starting point if your intention is to fine tune. Under the hood, it's a mixture of

00:01:27experts. So there's 975 billion parameters in total, but each layer holds 256 unique experts and each token

00:01:35only gets sent to six of them, which means only about 41 billion parameters are doing any work at any

00:01:41single time. It's also multi-model. So images, text and audio all go in. And the way it does this is much

00:01:47different to any model you would have seen before. Normally, when a model handles audio, for example,

00:01:52there's a separate speech model in front of the language model that transcribes the text and then

00:01:56passes that transcript to the language model. Images can work in the same way. A vision encoder looks at

00:02:02the picture and then creates a description for the image and then passes that to the language model.

00:02:07And of course, doing this means that there's going to be some form of data loss along the way.

00:02:11Inklin doesn't do any of that. The audio goes straight in as a spectrogram, so the raw frequency bands of the

00:02:16sound are chopped up into buckets. Images also go in as 40 by 40 pixel patches. Both get dropped straight into the

00:02:23same space where the text tokens live and the model chews through all of them together. So nothing's getting

00:02:28squashed down into text. So theoretically, you should get much less data loss and compression when processing

00:02:35things like audio and visual specifically with the Inklin model. And in fact, compared to most

00:02:40generalist models which don't support raw audio at all, Inklin clearly stands out. Now, whether you

00:02:46should even fine tune at all comes down to your specific goals. The rule of thumb is generally that

00:02:50fine tuning teaches a model how to answer, not what to know. For example, tone of voice or rigid

00:02:57structured output where thousands of examples would be needed in context to guide the model's output

00:03:02because passing thousands of examples via a prompt would just be impractical. Also cost and latency

00:03:08distillation where performing a narrow task with a general model would be considerably more expensive

00:03:13than fine tuning a smaller model. Basically, if there's any situation where you need thousands of

00:03:18examples to get decent output, then that could benefit from fine tuning. But if you need the model to know

00:03:23things it doesn't know, then don't fine tune. There's actually a well cited paper which puts retrieval up

00:03:28against fine tuning and retrieval one consistently. And if you want better reasoning, then fine tuning can

00:03:34actually make it worse because it degrades the model's chain of thought. And interestingly, smaller models

00:03:40suffer the most. But specifically for narrow jobs, the results from fine tuning can be great. If we take

00:03:46Harvey who build AI tools for law firms, they trained a small model to pull citations out of legal documents.

00:03:52They scored its F1, which is a metric used to evaluate the performance of a classification model. And fine

00:03:58tuning took them from 0.56 to 0.68. And head to head against GPT 4.0, the smaller model won or tied 93%

00:04:07of the time. And it even ran faster. Thinking machines even show a demo forcing Inkling to never

00:04:12use the letter E, where you can literally prompt it to fine tune itself via Tinker. It then goes off and

00:04:18creates the training data set and can run the full fine tuning process completely automatically. And

00:04:24the result of this is a model that has severe epsilon phobia and somehow the responses still make sense.

00:04:29But you're not retraining the entire model here. You're actually using something called LoRa,

00:04:32which stands for low rank adaptation. Instead of training the entire model, LoRa freezes the original

00:04:38weights and injects lightweight smaller matrices to capture new specific data. This provides advantages of

00:04:44cost and speed for post training compared to fine tuning an entire model, which likely would just be

00:04:49impractical. Now Tinker already supports a bunch of open models like Quen, Kimi, Deepsea, Nvidia's

00:04:55NemoTron and OpenAI's GPT OSS. And Inkling is one of the pricier ones, so why would you even bother with it?

00:05:02Well, there's two reasons and we've covered them a little bit already. The first is audio. Out of every

00:05:06model on the platform, Inkling is the only one that takes audio at all. Plenty of them do images,

00:05:11Kimi and most of the Quen models handle vision fine, but not one of them actually handles raw

00:05:16sound. And they've also shipped three cookbook recipes to go with it, like speech recognition

00:05:20and speaking style classification. So the model can tell you how something was said, not just what was

00:05:26said. And there's also a medical example trained on dictated prescriptions, teaching it drug names that

00:05:31most models would otherwise mangle. The second is thinking effort. Inkling lets you set how hard it

00:05:36thinks as a number from zero to one. Most models give you a switch of two to three settings. GPT OSS

00:05:42gives you low, medium and high, for example, but Inkling gives you an entire slider. So the big

00:05:47question is, should you actually use it? Well, if you want a model to run as it comes, then this probably

00:05:52isn't it. And they'll be explicit about that themselves. But if you've got a narrow high volume

00:05:56job, some real label data and the way to grade the output, fine tuning a small open model is one of the

00:06:02most underrated things you can do right now. And with platforms like Tinker, it's so much easier.

00:06:07And whether Inkling is the one you're trying really comes down to what you're feeding it.

00:06:11If there's audio going in, Inkling is a strong choice and nothing else on Tinker even takes raw

00:06:16audio right now. And if you're looking for the best open generalist model right now, we've just done a

00:06:21video on Kimi K3 you can watch here, and that has had some incredible results. But otherwise,

00:06:26thank you for watching. I've been Warren from BetterStack and I'll catch you in the next one.

Key Takeaway

Inkling provides a raw audio and visual processing architecture designed for specialized fine-tuning via Tinker rather than general-purpose out-of-the-box tasks.

Highlights

Thinking Machines released Inkling, an open-weight model with 975 billion total parameters under an Apache 2.0 license.
Inkling processes raw audio as frequency spectrograms and images as 40 by 40 pixel patches directly in the token space without intermediate text conversion.
On instruction following, Inkling scores 79.8, beating GLM 5.2 at 73.3, but scores 63.8 on terminal bench compared to 82.7 for agents.
Law firm AI tool developer Harvey trained a small model on legal citations, raising its F1 score from 0.56 to 0.68 and winning or tying GPT-4.0 in 93 percent of head-to-head tests.
Tinker supports automated self-fine-tuning using LoRa, freezing original weights while injecting lightweight smaller matrices.

Timeline

Model Overview and Architecture

Thinking Machines released its first open-weight model, Inkling, with 975 billion parameters under an Apache 2.0 license.
Inkling scores 79.8 on instruction following compared to GLM 5.2 at 73.3, but scores 63.8 on terminal bench against 82.7 for agent workflows.
The architecture utilizes a mixture of experts with 256 unique experts per layer, routing each token to six experts so that 41 billion parameters operate concurrently.

Inkling targets fine-tuning applications via the Tinker platform rather than competing directly with frontier general-purpose models. The creators explicitly manage expectations regarding standalone general capability, noting strong instruction following paired with weaker autonomous agent execution.

Native Multimodal Processing

Raw audio enters the model as frequency spectrograms split into frequency bands without prior speech-to-text transcription.
Images enter the model as 40 by 40 pixel patches directly into the shared token space alongside text tokens.
Eliminating intermediate text transcription layers prevents data compression and loss during multimodal input ingestion.

Unlike standard architectures that rely on separate encoders or speech models to convert audio and images into text before language model ingestion, Inkling processes raw audio and image patches concurrently within the same token space. This native ingestion minimizes information degradation.

Fine-Tuning Strategy and Performance

Fine-tuning teaches models output format, tone, and structured style rather than adding new factual knowledge.
Retrieval systems consistently outperform fine-tuning for factual knowledge integration, and fine-tuning can degrade model chain-of-thought reasoning.
Harvey trained a legal citation model that achieved an F1 score increase from 0.56 to 0.68 and matched or beat GPT-4.0 in 93 percent of head-to-head tests.

Fine-tuning suits narrow, high-volume tasks requiring thousands of structured output examples that exceed prompt context limits. When applied to specific legal document extraction, smaller fine-tuned models outperform generalist flagships while operating at higher execution speeds.

Automated Fine-Tuning and Tinker Platform

Tinker enables automated dataset creation and fine-tuning using LoRa by freezing original weights and injecting lightweight adaptation matrices.
Inkling stands out on Tinker as the sole platform model supporting raw audio input, accompanied by speech recognition and medical prescription recipes.
Inkling offers a continuous slider from zero to one to configure thinking effort rather than discrete low, medium, or high switches.

Users can prompt Tinker to automatically fine-tune Inkling for custom constraints, such as eliminating specific letters from generated responses. While generalist models excel without modification, specialized audio-driven tasks and granular thinking control position Inkling as a distinct option on the Tinker platform.

Community Posts

Write about this video