Run Claude Code with Ollama for 99% Cheaper AI

Englishالعربية Deutsch Español Français हिन्दी Bahasa Indonesia 日本語 한국어 Português Русский 中文

Computing/SoftwareInternet Technology

Transcript

00:00:00olama is a headless alum server with over 176 000 stars on github that lets you run open models

00:00:06through any app or agent that includes clawed code codex and even open claw you can point your

00:00:12tools at olama which even manages local models or route your traffic to hosted models in clawed

00:00:18code for example you can just change the base url environment variable to point to your olama

00:00:23server now everything works exactly the same as before but you've unlocked access to hundreds of

00:00:28models instead of four models like glm deep seek and even local models like gemma and quen are all

00:00:34available this is a huge unlock because you can carry on using all the tools you like in exactly

00:00:38the same way but now you've unlocked access to every model you can think of today we'll test out olama

00:00:44we'll try running open models through clawed code and see if it's worth using over other competitors

00:00:50like vllm and lm studio when we launch the olama cli directly it gives us options to launch other cli

00:01:01tools like clawed code or open code then being able to select our preferred model now i'm inside open

00:01:07code but i'm running gemma 4 via olama and now i can type something like are you subscribed to better

00:01:12stack and if the answer is no then why not subscribe below this video we don't even need to go via the

00:01:17olama cli at all with clawed code for example you can just change the environment variables to point to

00:01:23olama changing the base url and off tokens and now you can just run clawed code directly pointing at

00:01:29whichever model you like the result is now i'm inside clawed code i can use it exactly as i did before but

00:01:34now i'm running it with an open source model you can even run models directly with olama like running

00:01:40gemma 4 which is a multi-model i can ask it questions like to identify what is inside an image and you

00:01:46can even enter olama's harness directly to chat to models like gemma so you don't need any third

00:01:51party tool whatsoever okay so we're in the terminal here and i'll just type in olama to enter the cli

00:01:57and you can see here we've got a bunch of options of different agents or harnesses that we could enter

00:02:01we could just chat to it directly but in this case i'm just going to enter clawed code and you can see

00:02:05also that the default model which i've already downloaded is gemma 4 so now we're inside clawed code and

00:02:11you can see we're on gemma 4 with high effort it says ipi usage billing but because this is actually

00:02:17a local model it won't cost us anything at all so we can type in a message like hello so once a model

00:02:23replies you can still see that it still thinks that it's clawed code except it's still running the gemma

00:02:284 model so we get response hello i'm clawed code your assistant for all things software engineering so

00:02:34this means you can now just carry on using clawed code exactly as you did before but you've tricked

00:02:39it in a way because instead of using anthropics models you're actually using a local and open

00:02:44source model now there are tons of other models available you can see here at olama.com search

00:02:49we've got the options of many many models including things like quen 3.6

00:02:55we've got minimax i'm sure we've got yeah the deep seek family is here as well so basically any popular

00:03:01model that you can think of is going to be available and to run this we can just type in a command like

00:03:05olama run quen 3.6 and then that will if the model is not downloaded start to download that model so

00:03:11you'd have to wait a while for that to download but once it's available you can then run it on your

00:03:15machine now let's just check out that image detection script so i can say olama run gemma 4

00:03:20and then inside quotes we say what's inside this image and then just point it to an image that just

00:03:25happens to be in my downloads folder and then straight away very quickly actually we get a response and

00:03:29it's analyzed this and it says that there's a cat in the image it's a tabby cat the pose and action

00:03:35the cat is lying down all of this is true and this is the image that it did the detection against now

00:03:41your available memory and system performance really matters when running local models if you've got

00:03:45less than 24 gigabytes of vram you'll just get 4k of context 28 to 48 gig of vram gets you 32k of context

00:03:52and more than 48 gig of vram gets you 256k of context recently olama also made huge updates when

00:03:59running on mac os back in march olama ported over to mlx for apple silicon which is apple's machine

00:04:06learning framework this allows models to take full advantage of your unified memory and allows olama to

00:04:11leverage gpu neural accelerators to accelerate both times first token and generation speed basically it's

00:04:18faster aside from your local machine olama can also run inside docker so you can run your own services

00:04:24which rely on local models the docs include a full guide on using olama inside a container either using

00:04:30cpu only or with nvidia and amd gpus then you can launch the model inside docker and even curl request

00:04:37directly to it the api reference also includes several other endpoints you can call now comparing olama to

00:04:44other tools vlm seems far more suited to running models as services rather than a local cli tool

00:04:49and is significantly faster for server deployments due to a whole host of performance improvements

00:04:54like state-of-the-art serving throughput efficient memory management and quantization so if you're

00:05:00running a hosted service then vllm may be the better choice lm studio is much closer to olama in terms

00:05:06of local workflows and also comes with a beautiful gui however i would mention that olama does have its own

00:05:12official gui if that's something you're interested in there are of course tons of other tools in this

00:05:17space like everything llm which we've covered an entire video on here otherwise i hope you found this

00:05:22useful i've been warren from better stack thank you for watching and i'll see you next time

Key Takeaway

Ollama enables the use of open-source models within existing AI developer tools by redirecting base URL environment variables, effectively removing costs associated with proprietary API usage.

Highlights

Ollama functions as a headless server to run open models through existing agents like Claude Code or OpenCode by modifying environment variables.
Changing the base URL environment variable allows developers to route traffic from tools designed for proprietary models to local models like Gemma, Qwen, or DeepSeek.
Ollama porting to MLX for Apple Silicon leverages unified memory and neural accelerators to increase token generation speed on macOS.
Local model context windows scale with VRAM: under 24GB allows 4K context, 28–48GB allows 32K, and over 48GB allows 256K.
Ollama supports containerized deployment via Docker for both CPU-only environments and systems with NVIDIA or AMD GPUs.

Timeline

Integration with Developer Tools

Ollama acts as a local server for running open models through existing third-party agents.
Redirecting the base URL environment variable in tools like Claude Code routes traffic to a local Ollama instance.
Users gain access to a broad library of models including Gemma, Qwen, and DeepSeek while maintaining existing tool workflows.

Ollama serves as a headless interface for managing models locally or routing them to hosted services. By simply updating the base URL and authentication tokens in configuration files, tools originally designed for Anthropic or other proprietary APIs operate with local, open-source alternatives. This configuration swap allows developers to keep familiar interfaces while switching the underlying engine to run locally on their own hardware.

Local Execution and Capabilities

Ollama provides a direct command-line interface to interact with models without third-party agents.
Local execution of multimodal models enables features like image content identification directly on local machines.
Downloaded models such as Gemma 4 or Qwen 3.6 function within existing code assistants without API billing costs.

The command-line interface allows users to pull and run models directly using commands like 'ollama run'. Once downloaded, these models can be prompted to perform tasks such as image analysis, returning results locally. Because these tasks execute on the user's local machine, they bypass external API billing structures, allowing for software engineering assistance at no cost.

Hardware Requirements and Performance

VRAM availability directly determines the size of the context window, with 48GB+ supporting up to 256K context.
macOS performance benefits from MLX integration, utilizing Apple Silicon neural accelerators for faster generation.
Docker support enables deployment of local models in containerized services via CPU or GPU acceleration.
VLLM is recommended for high-throughput hosted services, while Ollama serves as a local CLI tool.

System performance depends heavily on available hardware, specifically VRAM and unified memory on macOS. Recent optimizations using Apple's MLX framework improve generation speeds by directly accessing neural accelerators. While Ollama is optimized for local workflows, users requiring state-of-the-art throughput for hosted services may prefer VLLM for its advanced memory management and serving capabilities.

Community Posts

Write about this video