스크립트
00:00:00olama is a headless alum server with over 176 000 stars on github that lets you run open models
00:00:06through any app or agent that includes clawed code codex and even open claw you can point your
00:00:12tools at olama which even manages local models or route your traffic to hosted models in clawed
00:00:18code for example you can just change the base url environment variable to point to your olama
00:00:23server now everything works exactly the same as before but you've unlocked access to hundreds of
00:00:28models instead of four models like glm deep seek and even local models like gemma and quen are all
00:00:34available this is a huge unlock because you can carry on using all the tools you like in exactly
00:00:38the same way but now you've unlocked access to every model you can think of today we'll test out olama
00:00:44we'll try running open models through clawed code and see if it's worth using over other competitors
00:00:50like vllm and lm studio when we launch the olama cli directly it gives us options to launch other cli
00:01:01tools like clawed code or open code then being able to select our preferred model now i'm inside open
00:01:07code but i'm running gemma 4 via olama and now i can type something like are you subscribed to better
00:01:12stack and if the answer is no then why not subscribe below this video we don't even need to go via the
00:01:17olama cli at all with clawed code for example you can just change the environment variables to point to
00:01:23olama changing the base url and off tokens and now you can just run clawed code directly pointing at
00:01:29whichever model you like the result is now i'm inside clawed code i can use it exactly as i did before but
00:01:34now i'm running it with an open source model you can even run models directly with olama like running
00:01:40gemma 4 which is a multi-model i can ask it questions like to identify what is inside an image and you
00:01:46can even enter olama's harness directly to chat to models like gemma so you don't need any third
00:01:51party tool whatsoever okay so we're in the terminal here and i'll just type in olama to enter the cli
00:01:57and you can see here we've got a bunch of options of different agents or harnesses that we could enter
00:02:01we could just chat to it directly but in this case i'm just going to enter clawed code and you can see
00:02:05also that the default model which i've already downloaded is gemma 4 so now we're inside clawed code and
00:02:11you can see we're on gemma 4 with high effort it says ipi usage billing but because this is actually
00:02:17a local model it won't cost us anything at all so we can type in a message like hello so once a model
00:02:23replies you can still see that it still thinks that it's clawed code except it's still running the gemma
00:02:284 model so we get response hello i'm clawed code your assistant for all things software engineering so
00:02:34this means you can now just carry on using clawed code exactly as you did before but you've tricked
00:02:39it in a way because instead of using anthropics models you're actually using a local and open
00:02:44source model now there are tons of other models available you can see here at olama.com search
00:02:49we've got the options of many many models including things like quen 3.6
00:02:55we've got minimax i'm sure we've got yeah the deep seek family is here as well so basically any popular
00:03:01model that you can think of is going to be available and to run this we can just type in a command like
00:03:05olama run quen 3.6 and then that will if the model is not downloaded start to download that model so
00:03:11you'd have to wait a while for that to download but once it's available you can then run it on your
00:03:15machine now let's just check out that image detection script so i can say olama run gemma 4
00:03:20and then inside quotes we say what's inside this image and then just point it to an image that just
00:03:25happens to be in my downloads folder and then straight away very quickly actually we get a response and
00:03:29it's analyzed this and it says that there's a cat in the image it's a tabby cat the pose and action
00:03:35the cat is lying down all of this is true and this is the image that it did the detection against now
00:03:41your available memory and system performance really matters when running local models if you've got
00:03:45less than 24 gigabytes of vram you'll just get 4k of context 28 to 48 gig of vram gets you 32k of context
00:03:52and more than 48 gig of vram gets you 256k of context recently olama also made huge updates when
00:03:59running on mac os back in march olama ported over to mlx for apple silicon which is apple's machine
00:04:06learning framework this allows models to take full advantage of your unified memory and allows olama to
00:04:11leverage gpu neural accelerators to accelerate both times first token and generation speed basically it's
00:04:18faster aside from your local machine olama can also run inside docker so you can run your own services
00:04:24which rely on local models the docs include a full guide on using olama inside a container either using
00:04:30cpu only or with nvidia and amd gpus then you can launch the model inside docker and even curl request
00:04:37directly to it the api reference also includes several other endpoints you can call now comparing olama to
00:04:44other tools vlm seems far more suited to running models as services rather than a local cli tool
00:04:49and is significantly faster for server deployments due to a whole host of performance improvements
00:04:54like state-of-the-art serving throughput efficient memory management and quantization so if you're
00:05:00running a hosted service then vllm may be the better choice lm studio is much closer to olama in terms
00:05:06of local workflows and also comes with a beautiful gui however i would mention that olama does have its own
00:05:12official gui if that's something you're interested in there are of course tons of other tools in this
00:05:17space like everything llm which we've covered an entire video on here otherwise i hope you found this
00:05:22useful i've been warren from better stack thank you for watching and i'll see you next time