Transcript
00:00:00The biggest competitor to Pandas is about to drop version 2.0.
00:00:04The Polaris update is almost here, and the biggest change is probably not the one you'd expect.
00:00:09By default, every query you've already written moves onto a different engine.
00:00:14You don't change a line.
00:00:15Polaris also stops promising that your rows come back in the order you gave them.
00:00:19It sounds like regression, though it isn't, is the price of making the first change possible.
00:00:24As Polaris gains more traction with 2.0, it's giving Pandas a run for its money.
00:00:30Let's take a look at the biggest update yet.
00:00:37Alright, now, from the start of time, Pandas has always been the go-to for most data work.
00:00:43It's simple, it's used by the masses, not only works great for data analytics, but it's also key in machine learning as well.
00:00:49Then a while back came Polaris.
00:00:51It's always been faster than Pandas.
00:00:53We'll talk about that.
00:00:54Now, with this major 2.0 update, it may just be getting closer to more adoption.
00:01:00Since you got the quick backstory, Polaris has quietly had two engines for a while.
00:01:05The one most people have been using is the in-memory engine.
00:01:08You can think of that as, like, a warehouse.
00:01:10It loads your entire data set onto the floor, the warehouse, then runs each step of the query across the whole floor.
00:01:17And that is extremely fast.
00:01:19As long as everything fits in RAM.
00:01:21The streaming engine works differently.
00:01:23Instead of one giant floor, it's more or less like an assembly line now.
00:01:27Polaris chops the data into small pieces the maintainers call Marcel's.
00:01:32Those pieces are sized to fit in your CPU cache.
00:01:36Then they're pushed through the query plan.
00:01:39Now, different parts of the query can keep moving instead of waiting on the whole data set.
00:01:43That's where the speed comes from.
00:01:44And eventually, it's also the path toward working with data larger than your memory.
00:01:49But there's a catch.
00:01:50If you have eight workers on an assembly line, they don't necessarily finish in the same order they started.
00:01:56And neither does Polaris.
00:01:58So, in 2.0, joins, group buys, unpivots, the docs literally say, etc.
00:02:04No longer guarantee row order.
00:02:06Unless you explicitly ask for it.
00:02:08If you enjoy coding tools to speed up your workflow, be sure to subscribe.
00:02:11We have videos coming out all the time.
00:02:13Right, so here's what this actually looks like.
00:02:15I'm going to UV pip install pre-polar.
00:02:18And remember that pre-flag.
00:02:20We're going to circle back to that in a second.
00:02:22Okay.
00:02:22I've got a lazy frame with the keys 210.
00:02:27I left join it to another frame, then collect.
00:02:32And look at the result.
00:02:34Same code as Polaris 1.
00:02:35Order is no longer guaranteed.
00:02:37Same code, it's just acting different.
00:02:39Now I add maintain order equals left to the join.
00:02:43I can execute it again.
00:02:44I'll run it here again.
00:02:45And the original order is back.
00:02:47So, this isn't Polaris randomly shuffling around our data.
00:02:51It's Polaris saying, if order matters, you tell it.
00:02:54You have to actually tell it.
00:02:56And there's another nice detail here.
00:02:58Run the explain function on a query.
00:03:01It's not hiding the behavior.
00:03:03You just have to know to look for it.
00:03:05That's the big behavioral change.
00:03:06But 2.0 is strange because this isn't really a feature release.
00:03:10It's actually just a cleanup.
00:03:12And there's a decent amount being cleaned up here.
00:03:15Originally pandas Polaris, I have read CSV, which is now just scan CSV, collect underneath that.
00:03:22Lazy frame profile is gone.
00:03:25Melt now becomes unpivot.
00:03:28Join nils becomes nils equal.
00:03:30Casting an integer directly to categorical is removed.
00:03:35You can use cat2 now to help with this.
00:03:38Casting a string directly to a date has been removed.
00:03:41That's string to date.
00:03:43Add a signed and unsigned 64-bit integer together, and Polaris now gives you int 128 instead of turning it into a float and silently losing more precision.
00:03:53And there's even more coming.
00:03:55You can add concat.
00:03:57That now refuses mismatched heights instead of trying to guess what you meant.
00:04:02This is actually a smart choice on Polaris' part.
00:04:05Every removed thing throws an attribute removed error.
00:04:09That tells you what replaced it.
00:04:12Call melt, and the error literally tells you to use unpivot with index and on.
00:04:17All these errors more or less become the guide on how to actually use it.
00:04:20Run the code, break something, fix exactly that break based on the error, and just keep going.
00:04:26And that gets to the reason Polaris has become a daily driver for so many in the first place.
00:04:31Pandas pushes a lot of problems to runtime.
00:04:35Polaris tries to catch them early through all this.
00:04:37You can call collect schema, and you can find out the types are wrong before Polaris reads a single row.
00:04:44It's just anticipating what's about to come.
00:04:47And it's becoming more and more efficient.
00:04:48So, where does Polaris actually fit?
00:04:50Well, DuckDB is SQL first.
00:04:53Polaris is an expression API built for Python.
00:04:56There's also Dask and Spark.
00:04:58Those are distributed.
00:04:59Polaris is focused on one machine.
00:05:01The open source library itself is open source.
00:05:04The Polaris cloud is not.
00:05:05And that key thing changes things once we get to performance.
00:05:09Now, this announcement says the new streaming engine is easily five times faster.
00:05:13Now, I've played around with Polaris over the years, and yeah, it is faster than Pandas, but I've never gotten any crazy speeds like five times faster.
00:05:21Try it out on some larger data sets.
00:05:23You will definitely see a speed difference.
00:05:25It'll vary each run and with each data set, but you're going to feel the difference and actually see it in runtime.
00:05:30And this is probably the most important distinction in the whole release.
00:05:34The part that would let the engine spill to disk, true out-of-core execution, hasn't landed yet.
00:05:40So, today, streaming means chunked and piped line.
00:05:43It does not yet mean your data set can magically be bigger than RAM.
00:05:46The 5x number is Polaris' own expectation.
00:05:50There's no benchmark table in the post anywhere.
00:05:53And honestly, most people seem to be fine with that.
00:05:55A lot of users seemed happy with a useful yet boring release.
00:05:59There's no giant feature dump.
00:06:01You don't have to learn anything new.
00:06:02Those errors, as you're coding, they're going to tell you what to replace it with.
00:06:07Just breaking, clean up, and semver being used the way it's supposed to be used.
00:06:12But two complaints keep coming up repeatedly.
00:06:14The first is row order.
00:06:15If maintain order defaults to false, you can create some nasty bugs in the code.
00:06:19Because nothing crashes.
00:06:21Your numbers can still be completed correct.
00:06:23They're just attached to rows in a different order.
00:06:26That's much harder to notice than an exception.
00:06:29The second complaint was the word streaming here.
00:06:32People argued that it's confusing for an engine that still isn't truly out-of-core.
00:06:38Then there are actual release candidate bugs.
00:06:41Since the RC landed, there's been a P-high issue where
00:06:44group by dynamic throws a date time out-of-range error on the streaming engine.
00:06:49Call the limit method.
00:06:51Well, limit isn't early exiting after join.
00:06:54And string to date time can return nil where it used to just raise errors.
00:06:58Which is why I told you to remember the install command pre.
00:07:02Pre 2.0.
00:07:04This is still a release candidate.
00:07:06And right now, it behaves like one.
00:07:08Which is fine.
00:07:09Right?
00:07:09It's not the full version yet.
00:07:11One more thing.
00:07:12The Rust crate is still version 0.55.
00:07:15So, if you're using Polars from Rust, there's no Polars 2.0 for you yet.
00:07:19But I'm going to assume that most of us are probably just going to be using Python here.
00:07:24Now, should you upgrade?
00:07:25Well, for a new project, yeah, I'd do it.
00:07:27Right?
00:07:27We are adapting to new technologies, new updates.
00:07:30If your pipeline already uses Sync Parquet or Sync CSV, you've effectively been using
00:07:34streaming this whole time anyways.
00:07:36But there are a few use cases where I just wait.
00:07:39Right?
00:07:39If your code depends on that row order, just don't upgrade and hope.
00:07:43Go grep for group buys that don't have a sort after them.
00:07:47If you're using group buy dynamic, I'd just wait for all this to settle down.
00:07:51And I mean, this is cool to see where this most likely is going.
00:07:54This probably will be 2.0, but it's not the official release yet.
00:07:57You could get warmed up with it, but it's not quite there.
00:08:00The thing I keep coming back to is how unusual this release feels.
00:08:03Polars 2.0 ships zero new features.
00:08:07And it still changes what your existing code does.
00:08:10The new engine is faster on my machine, just not five times faster.
00:08:14So 2.0 isn't exciting because it adds a bunch of new stuff.
00:08:18It's exciting because Polars is using a major version bump to really just clean up the foundation
00:08:24underneath everything that we're already using.
00:08:26I'm Josh from BetterStack.
00:08:28If you enjoy coding tips and tricks like this, be sure to subscribe.
00:08:30We'll see you in another video.
Community Posts
No posts yet. Be the first to write about this video!
Write about this video