Fantastic writing as always! This Event Bus reminds me of the homegrown "orchestration" we had in one of my previous workplaces: it was just Google Pub/Sub consuming logs and triggering functions in response to them. Setting up was a pain and observability essentially inexistent, but when it did work, it was super smooth and quick to propagate changes across the whole graph.
Oh it was much worse, there's a reason why I elected for spinning up orchestrators everywhere else after that. Dagster's declarative scheduling gave me the same benefit of seeing updates propagate across the graph, with the difference that it comes with much better monitoring and observability, and a lot less engineering overhead for someone who's not intimately familiar with the system's inner workings.
Again a masterpiece 🌟
Fantastic writing as always! This Event Bus reminds me of the homegrown "orchestration" we had in one of my previous workplaces: it was just Google Pub/Sub consuming logs and triggering functions in response to them. Setting up was a pain and observability essentially inexistent, but when it did work, it was super smooth and quick to propagate changes across the whole graph.
So the question is -- where did you find it stacks up against something like Dagster? Was it less framework overhead and easier to just sling code?
Oh it was much worse, there's a reason why I elected for spinning up orchestrators everywhere else after that. Dagster's declarative scheduling gave me the same benefit of seeing updates propagate across the graph, with the difference that it comes with much better monitoring and observability, and a lot less engineering overhead for someone who's not intimately familiar with the system's inner workings.
I enjoy your writing style.. Thank you.
How abt take the topic of Data Observability one of these weeks !?;)
what is data sprawl (defined as used here) and why is it bad?