DuckDB is becoming unstoppable...
BBetter Stack
Computing/SoftwareSmall Business/StartupsInternet Technology
Transcript
00:00:00For years, the moment our data outgrew a spreadsheet, the advice never changed.
00:00:05Go spin up a cloud data warehouse, Snowflake or BigQuery.
00:00:09But there's this tiny free database that runs entirely on your laptop,
00:00:13and while everyone was staring at the cloud, it quietly grew up.
00:00:16It now has real encryption, Git-style upserts, and its first ever long-term support release.
00:00:22This is DuckDB, and I want to show you exactly how it stacks up in 2026,
00:00:26including the one big thing that people keep getting wrong about it.
00:00:35So, DuckDB.
00:00:36Just think, SQLite, but for analytics.
00:00:40You know, SQLite.
00:00:41One file, no server, embedded right inside your app.
00:00:45DuckDB is that exact same idea, except instead of being built for transactions,
00:00:50it's built for crunching numbers.
00:00:52It's built to scan and add up millions of rows really fast.
00:00:56And one of the best parts is it reads Parquet, CSV, and JSON files directly.
00:01:02There's no import step, no load the data first.
00:01:05You just point SQL at the file.
00:01:08Let me show you what that feels like.
00:01:09If you enjoy coding tools that speed up your workflow, be sure to subscribe.
00:01:13We have videos coming out all the time.
00:01:15Now, here in my terminal, I type DuckDB, and I'm sitting in a SQL prompt instantly.
00:01:20Now, watch this.
00:01:21I select from a Parquet file that lives on a URL out on the internet.
00:01:27I didn't download it.
00:01:28I didn't define anything or start a server.
00:01:30DuckDB reached out, streamed a remote file, and ran real SQL on it in a single line.
00:01:36That right there, in five seconds, is a reason people love this.
00:01:40But here's the thing people are getting wrong.
00:01:43There are two releases in play here, and the internet keeps blurring them together.
00:01:48The big, exciting features everyone's talking about, full AES-256 encryption of your database,
00:01:54a merge-into command for Git-style upserts, and the ability to write Apache Iceberg tables,
00:01:59none of those are in 1.5.
00:02:02They all shipped in DuckDB 1.4 in September, which was also DuckDB's very first long-term
00:02:08support release.
00:02:09Then, 1.5, this past March, we got the updated version of that.
00:02:14A nice new command line client with colors and a pager, a new variant type for messy semi-structured
00:02:20data, and geometry baked right into the core.
00:02:24Instead of just one quick query, let me show you what the updated DuckDB actually feels like
00:02:30in practice, the 1.4 features, plus 1.5 on top of that.
00:02:34Now, I still have the parquet query we saw earlier right here.
00:02:38Now, here is the new feature in 1.5, the variant type.
00:02:43I create a quick table and insert mixed types, integers, strings, arrays, and objects, all in
00:02:50the same column.
00:02:51There's no schema, no JSON parsing, it just works.
00:02:54And it stores typed binary data, which compresses and queries better than plain JSON.
00:02:59That is the new variant feature.
00:03:02Next up is the feature that actually changed what you can build with DuckDB, merge into.
00:03:08This is from 1.4.
00:03:09Don't confuse that.
00:03:10One clean SQL statement.
00:03:12There's no app logic.
00:03:13This is the kind of thing that used to require a Spark or custom Python.
00:03:17Then the encryption piece as well, since here we are.
00:03:21Now, I can query it normally, but if I try to open the same file in a new session without the
00:03:27key, it fails exactly as it should.
00:03:30AES-256 at the page level, you bring the key.
00:03:33DuckDB doesn't store it or manage it.
00:03:36And that's before we even get into iceberg writing or geometry support, which a lot of
00:03:41you guys might just use spatial instead of geometry.
00:03:43This is why 1.4 was the big update.
00:03:45It went from a query engine to something you can actually trust with real data on a single
00:03:50machine.
00:03:51Secure files, reliable upserts, and the modern lake house formats.
00:03:551.5 just made the experience a lot better with improved CLI and the new variant type.
00:04:00Okay, so how is this different from the tools already on your machine?
00:04:04Let's take SQLite.
00:04:05Same, one file, no server feeling.
00:04:08But SQLite is a row store built for transactions.
00:04:12And DuckDB is a column store built for analysis.
00:04:15Then we could take Pandas.
00:04:17DuckDB hands you a real SQL optimizer and multi-threaded joins.
00:04:21So on a big group buy, it's often a little bit faster.
00:04:24Then if we took Snowflake or BigQuery, those are cloud warehouses built for entire teams and
00:04:30petabytes of data.
00:04:32DuckDB is one machine running inside your own process for free.
00:04:35Now, the number one complaint over and over and over again is memory.
00:04:40Point DuckDB at a billion rows and it can run clean out of memory and fall over.
00:04:45It might be a bit too flaky for production too, if that's what you're dealing with.
00:04:49The second thing is, this is not a transactional database.
00:04:53It's a single writer.
00:04:54One process writes at a time.
00:04:57So you don't wire it up on your app's back end or your session store.
00:05:01That's still Postgres's job or SQLite for small MVPs.
00:05:05And that whole encryption part, it's real, it's thorough, but it's bring your own key.
00:05:10DuckDB does not store the key.
00:05:12It does not rotate the key.
00:05:14It doesn't watch anything.
00:05:15You lose the key.
00:05:16Your data is gone.
00:05:17And the iceberg writing lives in an extension that's still pretty new.
00:05:22If you're doing analytics, crunching parquet, and CSV running ELT transforms, exploring data
00:05:28in a notebook, anything with a few megabytes up to a single machine scale, DuckDB is one
00:05:33of the best tools you can possibly install.
00:05:35And it's completely free, MIT licensed with no paywall.
00:05:39But if you need a transactional backend for an app, or you're routinely firing billions
00:05:44of rows, and you don't want to hand-tune memory, this is the wrong tool, and that's fine.
00:05:50Right database for the right job.
00:05:52If you enjoy coding tips and tricks like this, be sure to subscribe to the BetterStack Chick.
00:05:56We'll see you in another video.