I Tried the Tool Trying to Kill Apache Airflow (Kestra)

EnglishالعربيةDeutschEspañolFrançaisहिन्दीBahasa Indonesia日本語한국어PortuguêsРусский中文
BBetter Stack
컴퓨터/소프트웨어창업/스타트업AI/미래기술

스크립트

00:00:00This is Kestra, a startup that recently raised $25 million on a single promise,
00:00:06kill Apache Airflow. And their entire trick is almost insulting in how simple it actually sounds.
00:00:12You stop writing your data pipelines in Python, and you start writing them in YAML.
00:00:17Stay with me because that one thing is bigger than it looks.
00:00:21Let me show you how all this works in the next couple minutes.
00:00:29And now picture a chain of jobs that have to run in order. Pull some data, clean it up,
00:00:35load it into a database, then hit an API to kick off the next thing. You could wire that up with
00:00:41Chrome. But the moment one step fails, it's no bueno. No retry, no log, no clue what actually broke.
00:00:48That's the exact mess orchestration tools were built to actually fix. And Airflow honestly has
00:00:54been king of them for years. The problem with Airflow is that every single pipeline is a
00:00:58Python program you have to write, maintain, and debug. The whole system is heavy and just getting
00:01:04it running on your own machine, most of us wouldn't actually enjoy anyways. If you enjoy coding tools
00:01:09to speed up your workflow, be sure to subscribe. We have videos coming out all the time.
00:01:14Now, Kestra's whole thing is that workflow shouldn't be a program at all. It should just be
00:01:19a configuration. Let me show you what that actually means. I'm in the browser editor here,
00:01:24and I'll write a tiny flow in YAML. A couple of steps. One runs a Python script. One runs a shell
00:01:32command. I can hit execute and watch the screen. The diagram lights up with one bot at a time live
00:01:41as each step runs. I can jump to a timeline view to see how long each piece took and click straight into
00:01:47a log for any step that I want. That entire pipeline just ran, and I never wrote a single line of
00:01:53orchestration code. So here's how this thing actually runs. Every workflow is what they call
00:02:00a flow, and really it's just YAML. A list of tasks and a trigger that starts them. The tasks are language
00:02:08agnostic, and that's big here. One flow can run Python, the node, then bash, then a SQL query,
00:02:15and then fire up a container all in one row. Airflow really wants you in Python. N8N really wants
00:02:22you in JavaScript. Kestra actually doesn't even care what each step is written in, and the triggers are
00:02:27built right into the whole thing. Run it on a schedule like Chrome, set it off when a webhook fires,
00:02:34or a file lands in a bucket, or call it from the API. One last thing I like, the code and the visual
00:02:39builder stay locked together. So editing one actually automatically updates the other one. Now, why should
00:02:46you care versus the tools already we're using, aka Airflow, right? Now, against Airflow, your pipeline
00:02:53turns into a clean config that someone who doesn't know Python could actually read and approve at pull
00:02:59requests. And the reviewers say the engine chews through parallel work better than Airflow's
00:03:05scheduler does. Now, that's great, but against Zapier and Make, there's no SaaS and no getting build per
00:03:12task. It's built for devs and real infrastructure, and you host it yourself. And against plain old Crone,
00:03:19you get retries, timeouts, a real dependency map, and an actual UI right out of the box. Now,
00:03:26Kestra says it ran 2 billion workflows in 2025. That's 20 times the year before. And it's got names
00:03:32of customers like Apple, JP Morgan, Toyota, and Bloomberg. Now, those growth numbers come straight
00:03:38from the company, not from an outsider audit. So take it all with a grain of salt. But more and more
00:03:43people are betting that this declarative config first way of doing orchestration is where the whole field
00:03:49is heading. Okay, now a few catches here. First, this is a Java app. And the JVM, it's hungry. You want
00:03:58around 4 gigs of RAM and a couple of cores just to run the server in peace. Second, YAML is a dream for
00:04:06clean linear pipelines. But the second you need heavy dynamic branching logic, you know, it starts to fight
00:04:11us. And honestly, Python based tools handle that a lot better. And lastly, Kestra is open core. So
00:04:18the engine itself is truly open source, but single sign on rule based access control and audit logs,
00:04:24those live behind some kind of paywall. So the free version gives you one shared login. That's it. It's
00:04:31perfectly fine if you're doing this alone. But a problem the day you need a real multi user control
00:04:36system without wanting to pay. So should you use it? Well, that last part might be the dictator to all
00:04:42of that. If you want your orchestration to be readable config instead of Python, this is honestly
00:04:47cool. This is a cool tool. And it runs natively on Apple Silicon. Spinning it up on max is a single
00:04:53Docker run command and the dashboard comes up on localhost. Give it a try. Or if you have already
00:04:59tried it, drop your thoughts below. If you enjoy coding tips and tricks like this, be sure to subscribe.
00:05:03We have videos coming out all the time.

핵심 요약

Kestra enables data orchestration by shifting pipeline management from complex Python code to readable, language-agnostic YAML configurations that automatically generate visual dependency maps.

하이라이트

  • Kestra replaces Python-based pipeline orchestration with declarative YAML configuration.

  • The tool executes language-agnostic tasks including Python, Node, bash, SQL, and containers within a single flow.

  • Pipeline code and visual builders stay synchronized, where editing one automatically updates the other.

  • Kestra reported 2 billion workflow executions in 2025, a 20-fold increase from the previous year.

  • The platform requires approximately 4GB of RAM and multiple CPU cores to run the Java-based server.

  • The open-core model restricts features like single sign-on and role-based access control to paid plans.

타임라인

Limitations of Traditional Orchestration

  • Traditional orchestration often forces reliance on complex Python programs for simple pipeline tasks.
  • Simple automation scripts like cron lack critical features like retries, logging, and dependency tracking.
  • Airflow requires maintaining and debugging significant amounts of boilerplate Python code for basic job chains.

Orchestration tools address failures in linear job chains that simple cron tasks cannot handle. While Apache Airflow is a dominant solution, it imposes a heavy burden by requiring every pipeline step to be a custom-coded Python program. This complexity creates barriers for teams looking to manage simple data workflows efficiently.

Kestra Workflow Architecture

  • Workflows consist of YAML configurations rather than procedural code.
  • Tasks are language-agnostic, allowing execution of Python, SQL, shell commands, and containers in one sequence.
  • The system provides live execution tracking with automatic visual diagrams and log access for every step.
  • Triggers include schedules, webhooks, file storage events, and API calls.

Kestra treats orchestration as a configuration problem. By using YAML, it decouples the workflow logic from the language used in individual tasks. The system maintains a real-time link between the configuration and the visual builder, providing an interactive dashboard that updates as tasks execute.

Strategic Advantages and Limitations

  • YAML-based pipelines are more readable and easier to review via pull requests compared to Python scripts.
  • The engine handles parallel work processing more efficiently than standard Airflow schedulers.
  • Java-based architecture requires substantial hardware resources, specifically 4GB of RAM.
  • Dynamic branching logic remains easier to implement in Python-based tools than in YAML configurations.
  • The free version limits multi-user access controls and auditing features.

Kestra offers significant benefits over self-hosted cron and Python-based orchestrators, though it introduces trade-offs. The declarative approach simplifies collaboration but can be restrictive for complex dynamic branching requirements. Users must also account for the resource-heavy nature of the JVM and the gated features in the open-core business model.

커뮤니티 글

아직 글이 없습니다. 이 영상에 대한 첫 번째 글을 작성해 보세요!

이 영상에 대해 글쓰기