This Open Source Repo Solves Claude Code's Biggest Problem
CChase AI
Computing/SoftwareSmall Business/StartupsInternet Technology
Transcript
00:00:00Can a single skill make CloudCode wildly more efficient?
00:00:03Can it make CloudCode faster, cheaper, and write less code
00:00:06while still giving us the same sort of high-level results we're used to?
00:00:10Well, that is exactly what Ponytail is claiming to be able to do,
00:00:13and it's caused it to hit 40,000 stars only seven days after its release.
00:00:18Now, Ponytail is not the first tool that we have seen claim to do something like this.
00:00:22We've talked about Caveman in the past, and all of these tools tend to have the same idea.
00:00:26The idea is that CloudCode is naturally verbose,
00:00:29and if we tell it, hey, stop talking so much,
00:00:32we can get a much more concise answer that is ultimately just as correct
00:00:36or, like we remember with Caveman, might even be more correct.
00:00:40Ponytail is simply the latest version of it,
00:00:42but it's a version that's claiming numbers that are better than anything we've seen in the past.
00:00:45And we can see those numbers right here.
00:00:47We can see lines of code versus tokens versus cost and versus time.
00:00:52And across the board, gray being sort of the baseline with none of these tools
00:00:55and green being Ponytail,
00:00:58Ponytail pretty much leads the pack everywhere or gets pretty close.
00:01:03Now, the numbers you've seen here are aggregates.
00:01:05This is the average taken across a number of different tests,
00:01:08and this is also done using Haiku 4.5.
00:01:11And don't worry, later we're going to A, validate these tests,
00:01:14take a look at a real model because none of us are using Haiku 4.5 really.
00:01:18We're using Opus 4.8, so let's see what those numbers look like.
00:01:20And when it comes to lines of code, it's about 50% less lines.
00:01:24And we're looking in terms of tokens, cost, and time,
00:01:27about 20% to 30% improvements versus the baseline.
00:01:31And that's no small amount,
00:01:32especially when we extrapolate this to something like Fable,
00:01:36which is wildly expensive.
00:01:37So if I could tell you, hey, if you're using something like Fable,
00:01:40it's going to be faster and cheaper.
00:01:42Well, we would love that, wouldn't we?
00:01:43Now, before I go into how this works
00:01:45and showing you what the benchmark scores look like,
00:01:47when I tested it, a quick word from today's sponsor, me.
00:01:50So inside of Chase AI+, I have my Clawed Code Masterclass,
00:01:53which is the number one way to go from zero to AI dev,
00:01:56especially if you don't come from a technical background.
00:01:59I update this every single week,
00:02:01and it also includes masterclasses on codecs
00:02:03and how to build your own agentic OS.
00:02:06You can find a link to it in the pinned comment.
00:02:08And again, I update this every single week,
00:02:10and we focus on real use cases.
00:02:12So if you want to begin to master Clawed Code,
00:02:15this is the place for you.
00:02:16So how does Ponytail work?
00:02:17Well, it goes through the six-step process
00:02:19before it writes code.
00:02:20So the first question is,
00:02:22does this even need to exist?
00:02:24If the answer is no,
00:02:26well, then we just don't write code for it at all.
00:02:28Relatively obvious.
00:02:29After that, we ask, does the standard library do it?
00:02:33If the answer is yes,
00:02:34we're going to use the standard library.
00:02:36The big thing that you're going to see with the benchmarks
00:02:38is there are instances where Clawed Code
00:02:41will recreate features from scratch that already exist,
00:02:45either within some sort of library or as a platform feature.
00:02:49So Clawed Code has the problem where,
00:02:51hey, the wheel's already been built.
00:02:52We have the wheel here in this program.
00:02:53And it's like, you know what?
00:02:55I'm going to build a wheel from scratch.
00:02:56And because of that,
00:02:57that's how you get lots of code
00:02:59when you don't necessarily need it.
00:03:01That's something you see over and over again
00:03:03in these benchmarks.
00:03:04And to step away for a second,
00:03:05these six steps all are pretty much asking Clawed Code,
00:03:09like, hey, does this feature already exist natively?
00:03:12Do we need to create something custom?
00:03:15Because Clawed likes to create custom things,
00:03:17even if it doesn't have to.
00:03:18So if the standard library doesn't do it,
00:03:20then it's saying, hey, is this a native platform feature?
00:03:22Is this an installed dependency?
00:03:24Can this be one line?
00:03:26Do we need to be verbose?
00:03:27And if it gets through all that,
00:03:28and it's essentially like, no, no, no, no, no,
00:03:30then we're saying, whatever you write,
00:03:33just do the minimum that works.
00:03:35Don't go over the top.
00:03:36Don't create it if we don't need it.
00:03:37And if we do need it, do the bare minimum.
00:03:40So the idea here is to make Clawed Code lazy,
00:03:42but not negligent.
00:03:44Anything that has to do with trust boundary validations,
00:03:47data loss, handling, security, and accessibility
00:03:48are never on the chopping block.
00:03:50So it's kind of smart about what it applies this process to.
00:03:53Now in terms of install, relatively straightforward.
00:03:55You're just gonna copy this command right here.
00:03:57And I'll put a link down in the description
00:03:58for this repo, obviously,
00:04:00and this is going to install it for you.
00:04:01And you can also use this for codecs,
00:04:03or really any AI agent out there.
00:04:05There's a few commands when it comes to Ponytail.
00:04:07Namely, light, full, ultra, and off.
00:04:10Again, very reminiscent to Caveman,
00:04:12like the levels of Caveman we're going for.
00:04:14We can have it review our code.
00:04:16We can have it audit a repo.
00:04:18And then we also have the debt, gain, and help skills.
00:04:20Again, you can really drill down to these
00:04:22if you want to inside the GitHub repo.
00:04:24But none of this really matters
00:04:24if the benchmarks don't hold up.
00:04:26And the nice thing about this repo
00:04:28is they give us the benchmarks.
00:04:29We can run this for ourselves.
00:04:31And guess what?
00:04:32That's exactly what I did.
00:04:34You can do this yourself too.
00:04:36There is a full write-up
00:04:37on how they got the benchmarks
00:04:39right here on the README.
00:04:40And it also gives you the ability to reproduce these.
00:04:43And so what I'm going to show you
00:04:44is the numbers I got
00:04:45when I reproduced all of these benchmarks.
00:04:48And I reproduced them not only with Haiku 4.5,
00:04:51which is what you see in the repo,
00:04:52but also did it with Opus 4.8.
00:04:54Because again, none of us are using Haiku.
00:04:56I don't really care about Haiku.
00:04:58I care about Opus.
00:05:00And the results were honestly pretty interesting.
00:05:02So here's the tests, and here's the scores.
00:05:04You see their published numbers.
00:05:07You see our run with Haiku.
00:05:09And then over here on the far right
00:05:10is our run with Opus.
00:05:12At the bottom, you have the aggregate.
00:05:14So the 54%, this is again, looking at lines of code.
00:05:17It is 54% less lines of code, according to Ponytail.
00:05:21When we ran it, it was 56% on Haiku.
00:05:24So essentially the exact same.
00:05:27And on Opus, it was 71%.
00:05:29So we saw even greater gains or more efficient code using Ponytail when using Opus.
00:05:36Why is that?
00:05:36Because these more powerful models kind of like to talk, right?
00:05:40They like to be verbose.
00:05:41Again, kind of a callback to Caveman.
00:05:43You'll remember one of the studies that is talked about in there
00:05:45is this whole idea that very verbose models like to talk a lot
00:05:50to the point that sometimes they talk themselves out of the right answer.
00:05:53So kind of interesting and actually like it's sort of a boost to this thing.
00:05:57And it's interesting.
00:05:58And they talk about why they used Haiku in the testing and it was for costs.
00:06:02I really think they should have done this whole thing with Opus
00:06:04because when we ran it, Opus actually makes it look better.
00:06:09You know, and this is the model people are using.
00:06:11So if anything, they sort of undersold its efficiency in regards to lines of code.
00:06:15And this also applies to costs.
00:06:17When we looked at Haiku 4.5, what was the aggregate on our tests?
00:06:21We saw about a 25% reduction in the cost versus Opus 4.8, a 53% reduction,
00:06:28which is wild.
00:06:3053% less it's costing us.
00:06:32Imagine this was Fable.
00:06:33And you can see all the tests and the numbers across the board.
00:06:35And the lowest one was 13%.
00:06:38And in some cases, it was high as 73% for a multi-step wizard.
00:06:42Now you might be like, do we even need Opus for some of these?
00:06:45Fair point.
00:06:45But just understand what's being sort of illustrated here.
00:06:48What would it cost it $1.39 normally using standard Opus without the skill instead cost us $0.38
00:06:55using Ponytail.
00:06:57And if we look at Haiku, these smaller models in some instances actually ended up costing more using Ponytail.
00:07:04So this whole idea of cutting down the lines of code and making it more effective is way better when we're talking about more powerful models.
00:07:11In some cases, we have an opposite effect with the smaller models because they were already going to be efficient because they're kind of just like dumb and quick.
00:07:18You can see here in the count items benchmark, it was 21% more expensive to use Ponytail with Haiku.
00:07:27Now we're talking about a difference of two cents, but still point remains.
00:07:31The stronger the model, the more effective this architecture is.
00:07:34And I would love to see what this looks like using Fable.
00:07:37Again, 53% is no joke.
00:07:39And what about speed?
00:07:40Again, we're seeing the same thing play out with Haiku.
00:07:43How much faster was it?
00:07:44About 31% more, 31% quicker to use Haiku with Ponytail versus not.
00:07:51With Opus, 71% faster.
00:07:5571% faster.
00:07:56And again, what do we see with Haiku?
00:07:58There are instances, three in fact, where it was slower using Ponytail.
00:08:03You know, in some cases, 22% slower versus every single benchmark across the board on Opus up to 88%.
00:08:10In some instances, it was always faster, right?
00:08:13Again, we see multi-step wizard 78%, date picker 88%.
00:08:17And in the worst scenario, it was a 27% difference.
00:08:22So we look at these numbers with Ponytail and we're like, ah, take it with a grain of salt, even though I can do the benchmarks, like what really is 20%?
00:08:31And then you're like, oh, it's Haiku.
00:08:33So this is kind of BS.
00:08:34Then we test it on Opus and it's wildly different.
00:08:36It's wildly more effective.
00:08:37And I think the obvious question becomes like, well, what about the benchmarks themselves?
00:08:41Like how effective are these benchmarks?
00:08:42Are they realistic?
00:08:44First of all, go to the repo, test these out for yourself or run your own benchmarks that you think fit the bill for what you deem legitimate.
00:08:52Either way, I think when we're talking about, I think the 19 different benchmarks it ran, we're starting to see the same thing across the board.
00:08:59When we look at a more powerful model like Opus, I mean, honestly, I kind of like ignore these for Haiku.
00:09:04I don't care about Haiku.
00:09:06It's cheaper.
00:09:07It's faster.
00:09:08And therefore, it's more efficient.
00:09:11And again, since we're talking about what is essentially just a skill, what's the downside for trying this out?
00:09:16These numbers look really good.
00:09:17I highly suggest you go to this repo, download and start using it yourself.
00:09:21In the worst case scenario, let's say for your particular project, it's so complicated that telling it to be, you know, less verbose actually sort of backfires.
00:09:30Well, I mean, I think it's kind of like a no harm, no foul scenario, right?
00:09:34So that's the worst case.
00:09:37The best case is you're saving like 50% on your Opus usage and it's 70% faster.
00:09:43So really interesting stuff.
00:09:45I'm definitely going to be using this in my day-to-day.
00:09:47I've been using Caveman for like a month or two now for all the time, just automatically loaded.
00:09:52And I'm going to be switching over to Ponytail and see how I like it.
00:09:55I think the more stuff that comes out like this, the better.
00:09:58All you hear about these days is token cost, token cost, token cost.
00:10:03So anything that can lower that for us is going to be well received.
00:10:07So that's where I'm going to end this video.
00:10:08As always, make sure to check out ChaseAI Plus if you want to get your hands on my Cloud Code Masterclass.
00:10:13Let me know what you think in the comments and I'll see you around.