Discussion about this post

User's avatar
Ketan's avatar

I remember having this feeling a few years ago. What I realized is that airflow has taught us a few bad habits and also brought ahead an interesting paradigm of the vertical workflow engine.

I agree airflow is old, legacy and ideally folks should not use it, reality is there is a lot of pipelines already built with it - sadly. I think as a community we have to start moving away from it for more complicated problems.

Disclaimer: I created Flyte.org and heavily believe in decentralized development of DAGs and centralized management of infrastructure

Expand full comment
Aman Gupta's avatar

Stephen, this was a fantastic read! I wholeheartedly agree with your candid reflections on Airflow's limitations and the need for platforms that support faster, simpler, and more decentralized data workflows. Your insights on the evolving role of data engineers truly resonate with the current shift towards enabling broader access and agility in data operations.

I also wanted to highlight a solution from our side at dlt, which you might find interesting given your quest for more flexible and less cumbersome tools. In our blog post ['https://dlthub.com/blog/first-data-warehouse'], we delve into the benefits of using the dlt library for building custom pipelines. Unlike the traditional heavy, centralized orchestrators like Airflow, dlt offers a more lightweight, Pythonic approach to data ingestion and transformation, promoting database agnosticism and facilitating easier migrations. This aligns well with the decentralized and autonomous data platform philosophy you advocate for.

Looking forward to more of your insightful posts!

Best,

Aman Gupta,

DLT Team

Expand full comment
10 more comments...

No posts