This Open Source Repo Solves Claude Code's Biggest Problem
CChase AI
Computing/SoftwareSmall Business/StartupsInternet Technology
Transcript
00:00:00Can a single skill make CloudCode wildly more efficient?
00:00:03Can it make CloudCode faster, cheaper, and write less code
00:00:06while still giving us the same sort of high-level results we're used to?
00:00:10Well, that is exactly what Ponytail is claiming to be able to do,
00:00:13and it's caused it to hit 40,000 stars only seven days after its release.
00:00:18Now, Ponytail is not the first tool that we have seen claim to do something like this.
00:00:22We've talked about Caveman in the past, and all of these tools tend to have the same idea.
00:00:26The idea is that CloudCode is naturally verbose,
00:00:29and if we tell it, hey, stop talking so much,
00:00:32we can get a much more concise answer that is ultimately just as correct
00:00:36or, like we remember with Caveman, might even be more correct.
00:00:40Ponytail is simply the latest version of it,
00:00:42but it's a version that's claiming numbers that are better than anything we've seen in the past.
00:00:45And we can see those numbers right here.
00:00:47We can see lines of code versus tokens versus cost and versus time.
00:00:52And across the board, gray being sort of the baseline with none of these tools
00:00:55and green being Ponytail,
00:00:58Ponytail pretty much leads the pack everywhere or gets pretty close.
00:01:03Now, the numbers you've seen here are aggregates.
00:01:05This is the average taken across a number of different tests,
00:01:08and this is also done using Haiku 4.5.
00:01:11And don't worry, later we're going to A, validate these tests,
00:01:14take a look at a real model because none of us are using Haiku 4.5 really.
00:01:18We're using Opus 4.8, so let's see what those numbers look like.
00:01:20And when it comes to lines of code, it's about 50% less lines.
00:01:24And we're looking in terms of tokens, cost, and time,
00:01:27about 20% to 30% improvements versus the baseline.
00:01:31And that's no small amount,
00:01:32especially when we extrapolate this to something like Fable,
00:01:36which is wildly expensive.
00:01:37So if I could tell you, hey, if you're using something like Fable,
00:01:40it's going to be faster and cheaper.
00:01:42Well, we would love that, wouldn't we?
00:01:43Now, before I go into how this works
00:01:45and showing you what the benchmark scores look like,
00:01:47when I tested it, a quick word from today's sponsor, me.
00:01:50So inside of Chase AI+, I have my Clawed Code Masterclass,
00:01:53which is the number one way to go from zero to AI dev,
00:01:56especially if you don't come from a technical background.
00:01:59I update this every single week,
00:02:01and it also includes masterclasses on codecs
00:02:03and how to build your own agentic OS.
00:02:06You can find a link to it in the pinned comment.
00:02:08And again, I update this every single week,
00:02:10and we focus on real use cases.
00:02:12So if you want to begin to master Clawed Code,
00:02:15this is the place for you.
00:02:16So how does Ponytail work?
00:02:17Well, it goes through the six-step process
00:02:19before it writes code.
00:02:20So the first question is,
00:02:22does this even need to exist?
00:02:24If the answer is no,
00:02:26well, then we just don't write code for it at all.
00:02:28Relatively obvious.
00:02:29After that, we ask, does the standard library do it?
00:02:33If the answer is yes,
00:02:34we're going to use the standard library.
00:02:36The big thing that you're going to see with the benchmarks
00:02:38is there are instances where Clawed Code
00:02:41will recreate features from scratch that already exist,
00:02:45either within some sort of library or as a platform feature.
00:02:49So Clawed Code has the problem where,
00:02:51hey, the wheel's already been built.
00:02:52We have the wheel here in this program.
00:02:53And it's like, you know what?
00:02:55I'm going to build a wheel from scratch.
00:02:56And because of that,
00:02:57that's how you get lots of code
00:02:59when you don't necessarily need it.
00:03:01That's something you see over and over again
00:03:03in these benchmarks.
00:03:04And to step away for a second,
00:03:05these six steps all are pretty much asking Clawed Code,
00:03:09like, hey, does this feature already exist natively?
00:03:12Do we need to create something custom?
00:03:15Because Clawed likes to create custom things,
00:03:17even if it doesn't have to.
00:03:18So if the standard library doesn't do it,
00:03:20then it's saying, hey, is this a native platform feature?
00:03:22Is this an installed dependency?
00:03:24Can this be one line?
00:03:26Do we need to be verbose?
00:03:27And if it gets through all that,
00:03:28and it's essentially like, no, no, no, no, no,
00:03:30then we're saying, whatever you write,
00:03:33just do the minimum that works.
00:03:35Don't go over the top.
00:03:36Don't create it if we don't need it.
00:03:37And if we do need it, do the bare minimum.
00:03:40So the idea here is to make Clawed Code lazy,
00:03:42but not negligent.
00:03:44Anything that has to do with trust boundary validations,
00:03:47data loss, handling, security, and accessibility
00:03:48are never on the chopping block.
00:03:50So it's kind of smart about what it applies this process to.
00:03:53Now in terms of install, relatively straightforward.
00:03:55You're just gonna copy this command right here.
00:03:57And I'll put a link down in the description
00:03:58for this repo, obviously,
00:04:00and this is going to install it for you.
00:04:01And you can also use this for codecs,
00:04:03or really any AI agent out there.
00:04:05There's a few commands when it comes to Ponytail.
00:04:07Namely, light, full, ultra, and off.
00:04:10Again, very reminiscent to Caveman,
00:04:12like the levels of Caveman we're going for.
00:04:14We can have it review our code.
00:04:16We can have it audit a repo.
00:04:18And then we also have the debt, gain, and help skills.
00:04:20Again, you can really drill down to these
00:04:22if you want to inside the GitHub repo.
00:04:24But none of this really matters
00:04:24if the benchmarks don't hold up.
00:04:26And the nice thing about this repo
00:04:28is they give us the benchmarks.
00:04:29We can run this for ourselves.
00:04:31And guess what?
00:04:32That's exactly what I did.
00:04:34You can do this yourself too.
00:04:36There is a full write-up
00:04:37on how they got the benchmarks
00:04:39right here on the README.
00:04:40And it also gives you the ability to reproduce these.
00:04:43And so what I'm going to show you
00:04:44is the numbers I got
00:04:45when I reproduced all of these benchmarks.
00:04:48And I reproduced them not only with Haiku 4.5,
00:04:51which is what you see in the repo,
00:04:52but also did it with Opus 4.8.
00:04:54Because again, none of us are using Haiku.
00:04:56I don't really care about Haiku.
00:04:58I care about Opus.
00:05:00And the results were honestly pretty interesting.
00:05:02So here's the tests, and here's the scores.
00:05:04You see their published numbers.
00:05:07You see our run with Haiku.
00:05:09And then over here on the far right
00:05:10is our run with Opus.
00:05:12At the bottom, you have the aggregate.
00:05:14So the 54%, this is again, looking at lines of code.
00:05:17It is 54% less lines of code, according to Ponytail.
00:05:21When we ran it, it was 56% on Haiku.
00:05:24So essentially the exact same.
00:05:27And on Opus, it was 71%.
00:05:29So we saw even greater gains or more efficient code using Ponytail when using Opus.
00:05:36Why is that?
00:05:36Because these more powerful models kind of like to talk, right?
00:05:40They like to be verbose.
00:05:41Again, kind of a callback to Caveman.
00:05:43You'll remember one of the studies that is talked about in there
00:05:45is this whole idea that very verbose models like to talk a lot
00:05:50to the point that sometimes they talk themselves out of the right answer.
00:05:53So kind of interesting and actually like it's sort of a boost to this thing.
00:05:57And it's interesting.
00:05:58And they talk about why they used Haiku in the testing and it was for costs.
00:06:02I really think they should have done this whole thing with Opus
00:06:04because when we ran it, Opus actually makes it look better.
00:06:09You know, and this is the model people are using.
00:06:11So if anything, they sort of undersold its efficiency in regards to lines of code.
00:06:15And this also applies to costs.
00:06:17When we looked at Haiku 4.5, what was the aggregate on our tests?
00:06:21We saw about a 25% reduction in the cost versus Opus 4.8, a 53% reduction,
00:06:28which is wild.
00:06:3053% less it's costing us.
00:06:32Imagine this was Fable.
00:06:33And you can see all the tests and the numbers across the board.
00:06:35And the lowest one was 13%.
00:06:38And in some cases, it was high as 73% for a multi-step wizard.
00:06:42Now you might be like, do we even need Opus for some of these?
00:06:45Fair point.
00:06:45But just understand what's being sort of illustrated here.
00:06:48What would it cost it $1.39 normally using standard Opus without the skill instead cost us $0.38
00:06:55using Ponytail.
00:06:57And if we look at Haiku, these smaller models in some instances actually ended up costing more using Ponytail.
00:07:04So this whole idea of cutting down the lines of code and making it more effective is way better when we're talking about more powerful models.
00:07:11In some cases, we have an opposite effect with the smaller models because they were already going to be efficient because they're kind of just like dumb and quick.
00:07:18You can see here in the count items benchmark, it was 21% more expensive to use Ponytail with Haiku.
00:07:27Now we're talking about a difference of two cents, but still point remains.
00:07:31The stronger the model, the more effective this architecture is.
00:07:34And I would love to see what this looks like using Fable.
00:07:37Again, 53% is no joke.
00:07:39And what about speed?
00:07:40Again, we're seeing the same thing play out with Haiku.
00:07:43How much faster was it?
00:07:44About 31% more, 31% quicker to use Haiku with Ponytail versus not.
00:07:51With Opus, 71% faster.
00:07:5571% faster.
00:07:56And again, what do we see with Haiku?
00:07:58There are instances, three in fact, where it was slower using Ponytail.
00:08:03You know, in some cases, 22% slower versus every single benchmark across the board on Opus up to 88%.
00:08:10In some instances, it was always faster, right?
00:08:13Again, we see multi-step wizard 78%, date picker 88%.
00:08:17And in the worst scenario, it was a 27% difference.
00:08:22So we look at these numbers with Ponytail and we're like, ah, take it with a grain of salt, even though I can do the benchmarks, like what really is 20%?
00:08:31And then you're like, oh, it's Haiku.
00:08:33So this is kind of BS.
00:08:34Then we test it on Opus and it's wildly different.
00:08:36It's wildly more effective.
00:08:37And I think the obvious question becomes like, well, what about the benchmarks themselves?
00:08:41Like how effective are these benchmarks?
00:08:42Are they realistic?
00:08:44First of all, go to the repo, test these out for yourself or run your own benchmarks that you think fit the bill for what you deem legitimate.
00:08:52Either way, I think when we're talking about, I think the 19 different benchmarks it ran, we're starting to see the same thing across the board.
00:08:59When we look at a more powerful model like Opus, I mean, honestly, I kind of like ignore these for Haiku.
00:09:04I don't care about Haiku.
00:09:06It's cheaper.
00:09:07It's faster.
00:09:08And therefore, it's more efficient.
00:09:11And again, since we're talking about what is essentially just a skill, what's the downside for trying this out?
00:09:16These numbers look really good.
00:09:17I highly suggest you go to this repo, download and start using it yourself.
00:09:21In the worst case scenario, let's say for your particular project, it's so complicated that telling it to be, you know, less verbose actually sort of backfires.
00:09:30Well, I mean, I think it's kind of like a no harm, no foul scenario, right?
00:09:34So that's the worst case.
00:09:37The best case is you're saving like 50% on your Opus usage and it's 70% faster.
00:09:43So really interesting stuff.
00:09:45I'm definitely going to be using this in my day-to-day.
00:09:47I've been using Caveman for like a month or two now for all the time, just automatically loaded.
00:09:52And I'm going to be switching over to Ponytail and see how I like it.
00:09:55I think the more stuff that comes out like this, the better.
00:09:58All you hear about these days is token cost, token cost, token cost.
00:10:03So anything that can lower that for us is going to be well received.
00:10:07So that's where I'm going to end this video.
00:10:08As always, make sure to check out ChaseAI Plus if you want to get your hands on my Cloud Code Masterclass.
00:10:13Let me know what you think in the comments and I'll see you around.
Community Posts
No posts yet. Be the first to write about this video!
Write about this video