This symposium asks the wrong question

When Stephen asked me to participate in this symposium, my first reaction was “Are you sure?” After all, I am the founder of a company whose sole purpose is to build a system that many would call an “orchestrator.” I'm not exactly impartial.

The question assumes there is an agreed-upon definition for the orchestrator. If that definition is “the system whose sole responsibility is scheduling and ordering of tasks in production and nothing else,” then my answer is: it’s alive, but no one likes it, and it probably deserves to die.

If that’s my answer, then why am I doing what I am doing with my life? Worry not, this symposium has not caused some sort of existential crisis. Instead, I think “Is the orchestrator dead or alive?” asks the wrong question. Instead, we ought to ask about orchestration, not the orchestrator.

Orchestration is an essential capability and isn’t going anywhere. The right question is: what is the future of orchestration?

Orchestration versus orchestrator

Modern organizations build data products and assets to power analytics, ML, and their production applications. To do this, data practitioners use a variety of technologies to create data pipelines. From the first pipeline to the full data platform, you need orchestration.

It’s worth defining orchestration:

Orchestration is the coordination and management of multiple computer systems, applications and/or services, stringing together multiple tasks in order to execute a larger workflow or process. — Databricks

Data will, for the foreseeable future, be stored and computed in many storage systems and runtimes. All the data within an organization is not going to live in a single cloud data warehouse, executing on a single compute substrate. Organizational dynamics, economic realities, and technical constraints will not allow it.

But that definition of orchestration is too narrow in the data domain. Even with a single system, there is a dependency graph of data assets that constitute a data pipeline, as all data must come from somewhere and go somewhere. Computations to produce that data must be ordered and scheduled. 

If you are a data practitioner, you are orchestrating whether you call it that or not. If you are writing a data notebook that processes a file dropped in S3 and manually running it once a day, you’re orchestrating. If you are using dbt on a single warehouse, you are orchestrating: somewhere in the bowels of that codebase, there is a topological sort determining the order of model execution and launching compute into the data warehouse. If you have set up cron jobs in Fivetran, Snowflake Tasks, and Hightouch to flow data through your platform, you are orchestrating. 

Overlapping cron jobs in the modern data stack SaaS tools has become a popular way to avoid orchestration tools.

However, it’s just distributed orchestration, and that leads to operational silos. Good luck debugging an upstream data pipeline that breaks the ML team in a totally separate stack. It results in an operationally fragile data platform that leaves everyone in a constant state of confusion about what ran, what's supposed to run, and whether things ran in the right order. 

Control plane, not an orchestrator

We at Dagster conceptualize ourselves as a data management tool. All production data assets in an organization are represented, in software, within Dagster. As they are software artifacts, change management is done with software engineering processes. Keeping those data assets up to date is an essential responsibility of Dagster, and so orchestration is a core capability.

This is a team- and tool-spanning layer, not a silo. Keeping a graph of assets up to date for stakeholders is the core function of any data, analytics, or ML engineering team. And those teams are stakeholders with respect to each other. Sharing a control plane, while bringing their own transformation and domain-specific tooling, is proper and natural.

This control plane also has an active metadata layer. Dagster streams information about the assets into an immutable, structured event log. Users can plug in their own metadata as well. This serves as a ledger for the data platform usable for many purposes: versioning, quality, and others. Directly within the system, users can schedule based on activity in this ledger. This can take the form of policy or explicit scheduling.

By its nature, a system of record of production assets combined with metadata naturally should and will incorporate lineage, data quality, cataloging, observability, and governance. With well-defined APIs, this will integrate an entire ecosystem of tools that provide specialized, higher-level functionality in all of those domains.

Here comes the iPhone/iOS analogy

What I am describing here is a rebundling dynamic. Try as I might, I cannot help but reach for the analogy to the iPhone – it is too apt.

Sending emails on the go, digital photography, contact management, texting, and voice communication are essential capabilities. That does not mean that Blackberries, mass-market digital cameras, PDAs, and flip phones deserved to survive as standalone devices.

Consolidating those capabilities into a single device was a watershed moment in personal computing, on the order of the broad adoption of the original PC.

And it wasn’t just the iPhone, it was iOS. All of those capabilities needed organization, coherence, and rules, or else the user would live in an untrusted, chaotic world. iOS did that. The grid of applications on your home screen is the manifestation of that ordered heterogeneity. The user can organize and catalog myriad capabilities within a single, trusted, coherent experience.

In our vision, the asset graph is data’s in-product manifestation of that ordered heterogeneity. Assets computed by any runtime and stored in any system conform to a common protocol. The graph they reside in is not a post hoc observation of your assets, but a system of record that is alive.

So just like the iPhone is still a “phone”, we might still end up calling the new type of orchestrator an orchestrator. But it will in no way resemble the orchestrators of the past. And any system that claims to be an orchestrator without these capabilities will be viewed as woefully deficient.

Conclusion

While the standalone orchestrator might be going out of fashion, orchestration is alive and well. 

The “orchestrator” will fade in the same way the Blackberries, digital cameras, palm pilots, and flip phones did. But capabilities and tools are very different things. The capability that is orchestration is an essential, undeniable one. It must and will live on. The only question is where. And the answer to that question will be one of the most consequential in data infrastructure in the next decade.

Hell is other services.

If you’re building a system today, you live in hell, and you know it. Your eyes water at the scorched wasteland of internal applications. Your face burns as you stumble through a miasma of cloud infrastructure. Your skin sloughs off when you sift through some startup’s Swagger sewage.

There is no god in hell. I miss him, that emergent phenomenon of server racks, impenetrable protocols, and Cheeto-fingered priests. He used to dwell among us. We placed him in arks in office basements, in rooms 20 cubits high by 14 cubits deep. You could beseech God for assistance, and he would reply: “You are a sinner, full of error. Repent and resubmit.”

But he’s gone now, stuffed into data centers, and squeezed through tubes as we write letters through his stewards. Any chance at a personal relationship is gone, forever.

It’s a virus, this services thing. It dehumanizes everything and everyone it touches. Your face is an endpoint, your organs a module, your appendages a series of plugins to interface with supported integrations.

I have my own theories on how we got here: a secret AWS lab, a slipped hand, a cracked beaker, a drop of serum, an exposed toe. Regardless: everything is now unbundled, a hive mind of services backed by the same four companies, connected by cables to a room inside the earth’s molten core, where demons plug and unplug them intermittently to make humans dance between Twitter, Slack, and stale status pages.

We are survivors of a plague. This wasteland is our new home.

It’s changed us. If there’s one thing survivors don’t count on, it’s tomorrow. We need our infrastructure to rise and fall on demand, a virtual FEMA camp. We don’t get attached: yesterday’s architecture is today’s scrap heap, salvaged for only the most essential parts.

Survivors depend on the services they hate. They ambush them in alleys, plunge a needle into their intracranial space, and withdraw as much value as they can. Stitching services together into a soulless gesture at our cultural past is the only way to survive, in these end times.

The orchestrator, though, is a special kind of service.

Humans fashioned the orchestrator in their own image: it is an extension of the human mind, like another set of hands, like an infinite set of hands. The orchestrator, true to its name, is about turning the ideas of one into the actions of many. Orchestration is how psychological energy becomes kinetic energy.

The orchestrator’s essential role is this: run the thing. Run it now, run it later, run it after that, run it again if it fails, run it tomorrow, run it like this, run it over here, run it, run it now, run it again, again, again.

Yet as things have fragmented, the data orchestrator has found itself becoming not more important, but less. Like civilization itself, the data orchestrator was unbundled — the catalog-as-orchestrator, the warehouse-as-orchestrator, the sql-formatter-as-orchestrator. Instead of asking the survivors to run the things, the things are running themselves.

Data survivors, therefore, are even more adrift than others. One response has been to call for more control over the hellscape. Services are out of control. Data sprawl is unmanageable. We need better ways to coordinate. We need stoplights. Air traffic controllers. Bus schedules. Zoning ordinances. Crossing guards. Traffic lines. Segues.

These calls make sense — if you live in a civilization. But survivors do not sit on town councils. They do not go to the BMV and get driver’s licenses. They do not slow down for children at play.

Getting all these services, all this activity, to make sense, is a daydream of a time lost. It will not happen — not here, not today. But that doesn’t mean all hope is lost.

What if we built an orchestrator, not for architects, but for survivors?

Chaos. We would embrace chaos.

We would build tooling that is, above all, versatile and fast. It could be triggered from anywhere at any moment. It would have no opinions. It would require no thinking, no planning, no setup. It would be dangerous.

Hellscape tooling is dangerous. We have plenty of examples.

Adopting Segment is like handing out an assault rifle to every man, woman, and child in your cult. Without warning, you can go from having no user behavior events to having millions, billions of events flowing into Segment and thence into seven other systems. The startup cost is next to zero: you click some buttons, turn off the safety, and press the trigger whenever a user touches your product.

Using Segment will result in pain, agony, and analysts writhing on the floor, their legs shorn off. Segment does not protect you. Segment is not your friend. 

But Segment works, because Segment is fast. Datadog, similarly, is fast, and it also penetrates your systems, nestling into their squishy insides to feed and grow. Datadog is valuable with just a couple metrics, but every additional sacrifice provides more new correlations, more value within the same interface.

Datadog will balloon into an unmanaged mess of millions of metrics. It doesn’t matter, though, not really — the first concern is to survive, and to survive, you need to hit your target. You don’t say, “I wish Datadog had fewer metrics,” you say, “I wish Ripley never learned Kubernetes.”

On the orchestration front itself, we have examples: GitHub Actions, for example, is not a control plane — the only useful part of its UI is a button that generates a status snippet you can embed in the README. It has no local development story. If you are creating a workflow from scratch, you will experience nothing but pain.

Github Actions thrives because it eschews any meta-narrative around what it should be used for, who should use it. It’s just Actions. It runs things. And with a marketplace of over 18,000 actions, chances are that you won’t even know how awful the developer experience is. It’s just ready for you, ready for action, ready to run code.

Here’s the through line for hellscape tooling: they cater to the individual, not the organization. They spread out across survivor clans and create long-distance dependencies between people who don’t know each other. And they are fast, requiring no thinking or training, just an initial thirty-second trial and error period to find the most ergonomic way to fire the thing.

The system emerges from activity; it’s a bottoms-up phenomenon. But this also means that the system’s survival is tied to its penetration into the culture. Its goal is not to evolve a single perfect organism, but to spawn an infectious alien growth that occasionally metastasizes into massive new fauna.

Simplicity and speed. This is where current data orchestrators fail. They make you think, not just about their opinionated frameworks — directedness, acyclicity, node/edge graph distinctions — but about whether they are even the right way to do a task.

Data ingestion — one process an orchestrator must own — illustrates this perfectly. Managed tools are easier for the simple case; managed resources such as Snowpipe are more effective in the advanced case.

The Snowpipe case is informative because it highlights the architectural limits of current orchestrators. A Snowpipe executes SQL whenever a new file lands in storage. It uses a message queue to trigger the compute, which updates the target table within seconds.

The Snowpipe doesn’t have good ergonomics. The monitoring sucks, it requires substantial infra work, and troubleshooting it is awful. But it is, above all, fast and simple. If a Snowpipe works for a use case, it’s probably the best solution — bypass the orchestrator.

There is a broader post-apocalyptic trend toward serverless pipelines. The DAG itself has been stuffed into a little box that’s run on ephemeral compute: event-driven, cross-platform, single-step executions, with fast code execution.

This leaves the orchestrator in a dissatisfying middle ground — neither dominant enough to bring every workflow to heel, nor flexible enough to spread virulently across teams.  But what if the Snowpipe model was the core of the platform? What if, instead of DAGs, we littered the world with even more tiny services mediated by a message bus — pouring fuel on the fire, rather than trying to put it out?

It would be a push towards a distributed, business-level task manager. The chief adaptation would be to move to an entirely activity-driven model, and a consequence would be that the orchestrator would become flat. Not tall buildings that form permanent infrastructure, but a flotilla of boats drifting on a sea of activity. Nimble, mobile, discardable.

How many applications in the unbundled data stack could you power with a message bus, a Lambda function, a database, and access to a shared resource configuration?

All of them?

The orchestrator that’s useful is the one plugged into everything. That’s the interesting play from the data catalog — in a reactive environment, awareness is as important as skill. The orchestrator is losing today because it’s not fast enough to onboard the use cases the engineers want, and it’s not simple enough to justify displacing or running managed services through it. It needs to do both.

I’ve sketched out aspects of this chimeric product before — it’s part Segment, part Zapier, part Airflow, part Lambda function, and part Datadog. It doesn’t have to manage the compute itself — who cares whose EC2 instance it is — but it needs to have its tentacles in everything.

Here are the essential elements:

Event Bus: Ingest every potentially relevant triggering event. Every single object change in S3, every Snowflake user query, every test failure, every new hire, every first kiss, every lame icebreaker. It all goes in, or at least essential derivatives do. This is your potential energy, the reptile sensory brain, the primordial soup from which all resulting process emerges.

Environment: A bespoke orchestrator has over a cloud provider its ability to adapt to a particular context. The orchestrator speeds up development by making the environment easy to manipulate. It should take two minutes — literally two minutes — to write a task that queries Snowflake and sends a Slack message with a summary of the results every hour.

History & Context: The orchestrator embraces chaos, but only of a certain kind: ordered chaos, not wasteful chaos. If you tell it to fire at anything in its periphery, it will do that — then stop. This requires some meta-awareness, retries, task ids, and the like.

Artifact Catalog: Passing data between tasks is one of the key elements that drive DAGs. But artifacts are simply a subset of the environment. They are, essentially, an address and an object type. That’s not to say they’re simple — but as dbt has shown, if they are properly cataloged, passing them between operations is simply a matter of having the right context and name available.

Results and Monitoring: The orchestrator feeds itself, an Ourobouros of data processes. It’s directed, but it’s not deterministic, not necessarily. As each process completes, it adds results to the activity stream, which triggers downstream processes. But that’s not all: high-level systemic monitoring must be a primary concern to support all the chaos that’s being generated.

In the event-driven world, the graph exists, but it emerges from the architecture, rather than being the primary concern. You don’t control it, you create it, much in the same way you don’t truly control the activity patterns of users or the correlations of metrics in your infrastructure. You build funnels and piece together the processes after the fact.

And that’s the wedge the orchestrator can latch onto: the more chaotic the landscape, the more useful intelligence is. Compute is the content; the orchestrator is the feed.

Chaos or control?

It boils down to activity or graphs. 

A desire to control and unify leads you to dbt: a comprehensive description of the ideal state, an approximation of the übergraph. This is a convergence problem. You have an idea of the world, and the challenge is to make it a reality.

There’s a place for this in our world, even today: graphs are thunderdomes, the last remnant of real human culture, a sealed-off sphere where contests can be fought and arguments can be settled.

But thunderdomes are tiny, and the wasteland is enormous. There are so many teams out there who want to run so many things. Many of these are terrible ideas that might result in catastrophe.

The orchestrator should make catastrophe easy.

Our descent into services hell is the perfect opportunity for the orchestrator to become the authority on the environment, to connect — and disconnect — services, to arm survivors with the intelligence they need to react to new threats and to monitor the sliver of reality they care about.

The orchestrator does not have to settle arguments or broker consensus. It doesn’t have to choose between truth and lies, monoliths or meshes, good or bad quality— why not do all of it? Its natural role is as an arms dealer, distributing compute, secrets, data.

Existing tools are moving towards a less controlled model, but not fast enough. Astronomer’s push-button deployments, Dagster’s declarative scheduling, Prefect’s UI-driven flow registry, Datahub’s stream-based metadata ingestion, Atlan’s Zapier-like metadata actions.

How much of this boils down to an event stream, trigger logic, and a Lambda function? Make it lighter! Make it faster! Make it simpler!

But don’t make it weak, leaving engineers in the cold. Give survivors the tools they need to do ludicrous things. They don’t want to wait at bus stops, they want to jerry-rig ATVs and sky-hook onto passing choppers and wave chainsaws at zombies.

Man’s descent into chaos positions the orchestrator to become one of the few services we trust, a bulwark against the neverending hordes of point solutions. The orchestrator should applaud every new category, every new cloud provider, every bit of fragmentation, because each one adds the cacophony that can only be managed by a service that welcomes it.

There is no longer any direction, any acyclicity, any distinction between nodes and edges. Not anymore, not for us who live in these dark times. We only have the present, the finger on the trigger, the running of things, an infinite number of things. 

If you’ve already got a workflow orchestrator, it’s very tempting to treat data materialization – the act of turning abstract data models, typically defined in SQL, into relational tables in your data warehouse / lake / lakehouse / swamp / outhouse / etc. – as an orchestration problem. After all, the problem sounds like a straightforward set of related, imperative tasks: make a job DAG where every task represents a model and edges represent data dependencies between models, long-poll data sources until they become “fresh”, freshen models in data-dependent order, repeat frequently enough to meet your recency SLOs.

Easy, right? 

Unfortunately, models are rarely that well-behaved. Models change all the time, data dependencies between models can be complex, and SLOs change as business demands shift. Backfills, data deletion requests, and run-of-the-mill bugs in query logic happen all the time. The more complex and dynamic your data becomes, the more shoehorning is required to fit all that dynamism into the workflow orchestrator’s imperative, scheduled, task-oriented abstraction. All too often, the result is a giant shambling mess of interdependent job DAGs that are tedious to operate, difficult to reason about, and dangerous to change.

I’ve become increasingly convinced that we’ve been trying to shove a square peg into a round hole by treating data materialization as a workflow orchestration problem. Data materialization isn’t an orchestration problem, it’s a convergence problem, and we need a new system to handle it. 

Why Convergence?

I’m far from the first person to have longed for a convergence-based solution to data materialization. Benn Stancil wrote a post on his blog in August of last year called “Down with the DAG”, where he laments the current state of workflow orchestration in the (ugh) Modern Data Stack. The entire post is worth a read, but this excerpt sums up his main point:

I don’t actually want to think about when to run jobs, how to define DAGs, or to manually orchestrate anything. I just want my data to be fresh—where I can declare what fresh means—and to know when it’s not.

Earlier in this symposium, Vinnie echoed the same sentiment:

Let’s all just stop writing ETL altogether. Let’s declare the outputs of our pipelines and let our systems figure out the rest. If humans don’t wanna deal with it, let the machines do it, at least while we can still command them.

In a perfect world, we’d tell the system what models we have, when we need them, and maybe what queries we’re running on them, and let some kind of data auto-materialization system do the grunt work of turning those models into tables in our warehouse. Instead, we’re left telling a workflow orchestrator both what we want and how to get it, and having to do a lot of undifferentiated toil when either of those things change.

In the abstract, the job of this hypothetical data auto-materialization system is to continually compare the state of the warehouse against the desired state of the models and their SLOs and make modifications to the warehouse until the models are materialized and the SLOs remain satisfied. We want the reality of the data in our warehouse to continuously converge toward our model definitions as those definitions change.

The software world is littered with solutions to convergence problems. Infrastructure-as-Code tools like Terraform and CloudFormation are convergence-based tools that read a representation of the desired end state, compare that desired state to the current state of the infrastructure, and construct a plan to converge the current state with the desired one. Kubernetes is also a convergence-based tool: its controller-manager compares the set of declarative resource specifications with the current state of the cluster and makes changes to the cluster (adding and removing cluster nodes, starting and stopping containers, etc.) until all the desired resources are running.

In a convergence-based view of data materialization, a “materialization controller” could periodically evaluate the state of all models, inspect their materialized counterparts, and issue queries to the warehouse to partially or completely (re-)materialize them as necessary. If a few of a model’s partitions need to be recomputed, the user could simply drop the impacted partitions and allow the controller to notice their absence and re-populate them. The controller could choose how to materialize a model by comparing the costs of each materialization method and adjusting the materialization method in the background as workloads or model sizes change. It could also handle prioritizing materialization queries based on the warehouse’s current load and the models’ SLOs, automatically shifting costly materializations that can tolerate some delay to off-hours. All the incidental complexity that was once the domain of human operators disappears, leaving data teams to focus more on the data and less on the nitty-gritty technical minutia of getting it there.

Applying convergence to data materialization appears to be an idea whose time has come. Dagster is already taking a crack at this idea with their notion of software-defined assets. What I’m calling “convergence” here, they’re calling “reconciliation”, but we’re both essentially hitting the same high points. I haven’t played with Dagster’s implementation enough to form an opinion on it yet, but what I’ve seen so far looks promising (though defining assets as Python functions make the declarative purist in me a little itchy).

If the benefits of this approach are so apparent and work is already underway, you might wonder why we’re not already swimming in competing materialization controller implementations. I think it’s because building one is going to be really hard.

The Trouble with Convergence

The industry has enough collective production experience and battle scars with convergence-based systems to know that they’re not as easy to build or operate as they might first appear. In particular, we need to be wary of three big problems with building and operating convergence-based systems: explainability, over-eagerness, and drift.

The first big problem with convergence-based systems centers around explainability. Everything looks like magic in a convergence-based system when things are going well, but if something goes wrong it can be difficult to understand why. If you’ve ever gotten Kubernetes stuck trying to launch a pod, you’ll know what I’m talking about. Detailed machine- and human-readable logs would go a long way toward making the system easier to troubleshoot and could also serve as a rich piece of lineage information that could be used elsewhere. Those logs would need to be built into the system as a first-order architectural concern to be truly useful, though. Bolting logging onto the side once the system is already built won’t cut it.

The second big problem with convergence-based systems is that it’s easy for an over-eager controller to spend a lot of money and/or resources either doing too much or doing it too fast. The controller will need some guardrails to prevent it from trying to rebuild the universe from scratch in reaction to a misconfiguration or a hastily done refactor. These guardrails aren’t just necessary for emergencies or controller meltdowns, however. Even in normal operation, operators will need to be able to tell the controller to either stop or slow down sometimes, if only to keep the system from live-locking itself when a lot of models change at the same time.

The third and arguably thorniest problem with convergence-based systems is drift. If the controller were the only thing changing the warehouse, it could keep a cache of the warehouse’s state and use that cache to decide what actions to take. With that local cache as its source of truth, it could iterate faster and lighten the query load on the warehouse itself.  Unfortunately, the controller is rarely the only thing manipulating the warehouse, and its state cache can get stale very quickly. Even if you could prevent anything but the controller from modifying the warehouse, you probably wouldn’t want to; you need some way to break the proverbial glass and bypass the controller in an emergency.  

There are a few ways you might be able to keep the controller’s cache warm in the presence of drift. The controller could hook into some kind of notification system so that it’s aware of changes relatively quickly. Data warehouses are good at exposing what’s being done to them via query logs, and we can attach post-commit hooks to version control systems, so both systems are ready sources of notifications. Even with that notification system in place, there will still be some latency between a change happening and the controller becoming aware of that change. To account for that latency,  you’ll need to assume some amount of staleness and limit the controller to operations that are “safe” if the controller’s view of the world is out-of-date. Restricting the controller to either idempotent or reversible actions seems like a good way of doing this, but that feels like it’ll be one of the trickier parts to get right.

I, For One, Welcome our Convergent Overlords

While there are a lot of challenges facing the implementers of a materialization controller, I’m cautiously optimistic that it can be done. We’re in such an early stage that we can and should be bringing all the lessons that we’ve learned from prior convergence-based systems into the development of this one. I’m hopeful that we’re gaining clarity on not just what’s breaking with workflow orchestrators, but why it’s breaking, and that we’ll use that clarity and a healthy dose of past experience from adjacent parts of software engineering to build a new generation of even better tools.

How much time have you spent managing tasks on an orchestrator? Hundreds of hours? Thousands? Whichever it was, those were some low-value activities, and you probably felt that way too.

The root problem is that the orchestrator tries to control the data operations, rather than letting the operations stem from the data.

Why do we have orchestrators?

tl;dr: legacy.

As Louise wrote, the data orchestrator is “a software solution or platform responsible for automating and managing the flow of data across different systems, applications, and storage locations.” I would supplement it with the idea that, in the end, an orchestrator is a stateful solution to manage the execution of scripts.

Looking back, the need for data orchestration stemmed from the complexity of triggering logic, where simple script scheduling failed to account for dependencies. One script depended on another, and there needed to be a stateful software piece in the middle to not only trigger the entire process but to ensure individual steps were started only after all their upstream dependencies had successfully completed. 

As a standalone brick, the orchestrator serves multiple purposes, the main ones being:

• Keep state: keeping tabs on what has happened and what hasn’t is likely the core feature of the orchestrator.

• Trigger stuff: the actual mechanism of executing the script, often using a set of parameters set at the task or job level.

• Enforce dependency constraints (DAG): sort out which script should be executed before which and make sure there is no loop.

• Surface logs: provide a comprehensive view of task statuses that serves as a troubleshooting entry point.

What’s the issue?

Hint: it’s the orchestration.

I could ramble on about the maintenance time spent managing deep dependency graphs (high number of successive tasks), the latency they introduce by forcing workflows to adjust to the slowest task, the cost they bear in an already resource-hungry practice, their poor interoperability with modern tools (or utter incompatibility like with stream processing), the burden they put on data consumers who are forced to depend on engineering work for the tiniest updates…

But, in the interest of keeping it efficient, the orchestrator’s problem boils down to two things: they’re just another brick in the stack, and, most importantly, new data technologies don’t need them.

What are the post-orchestrator people doing?

Just-in-time data ops.

Displacing the orchestrator means removing synchronization overhead, and the chief way to do this is to try and make the time of execution matter less. Today, this means implementing either Asynchronous Processing or High-Frequency batches.

High-frequency batches

Let’s cover high-frequency batches first because I see it as an aberration.

High-frequency batching is the idea of running processing steps at very high frequency (per batching standards) – for instance, every 5 min or so. In doing so, one can almost treat every independent task as asynchronous. This helps make individual tasks run independently, without the need for a stateful orchestrator at the helm, but it forces you down the darker path of one of the hardest trade-offs: agility for cost.

Anyone who has tried running high-frequency batches at scale has seen two challenges. First,  latency adds up fast - it doesn’t take a deep dependency graph before high-frequency tasks stop correlating with high-availability datasets. And second, pricing hurts. Most data solutions aren’t designed for this usage for a simple reason: running a LEFT JOIN every 5 min means redundantly scanning a ton of data. 

Asynchronous processing

Asynchronous processing, or continuous systems – think incremental engines like Stream Processors, Real-Time Databases, and to a certain extent HTAP Databases – do not require an orchestrator since they are continuous in nature and data is self-updating: a change is instantly reflected in consumer systems and downstream views/tables.

For anyone who isn’t too familiar with streaming yet, that’s the difference between Pull and Push data operations: the former requires you to constantly trigger the actions you would like performed (like in the ETL / ELT model for instance) while the latter computes and propagates data points passively and incrementally. No orchestration! Each task is almost like a microservice, a data service of sorts.

Unfortunately, Benoit already beat me to quoting the “Why not Airflow” documentation, so I’ll spare you that much. But as mentioned earlier, one of the core difficulties of orchestrators is interoperability: you simply can’t orchestrate everything. It introduces complexity and latency into your data operations and is generally a very efficient way to obliterate many weekly hours into an activity that ultimately shouldn’t exist. 

Imagine if everything you run in prod had to be orchestrated, that would be the death of agile. Really, why is this still a thing in the data world? (Rhetorical question, please refer to the first paragraphs).

What’s the new toolbox then?

It’s already here, sort of.

If you’re spending time fixing an Airflow DAG, sorry for saying this but that’s unlikely to be your greatest contribution to your company. Fortunately, the alternatives are out there, so it’s already down to your organization’s techno-political leadership to make that choice. With regards to spinning up data services, all the asynchronous processing options mentioned earlier are already live and running at scale in an enterprise near you.

What’s really missing is some form of reusable Control Plane with basic enforcements: ensure that the Acyclic and Directed aspects of dependency rules are met, and surface operational commands and statuses at a central level for convenience. This is more of a generic DataOps tool that inherits some of the Orchestrator responsibilities, especially on metadata and command exploitation, albeit without meddling in the task lifecycle – hence no longer a mandatory piece of the stack. That’s where the “active metadata” trend could be an interesting play by acting as a passive observer of the pieces involved, rather than intruding on the data operations by having an active role.

At Popsink we have no extra brick for orchestration, and it saves us time and money. We did end up building our own control plane for convenience, partly because some of the open standards today are incompatible with the idea of not having an orchestrator (like OpenLineage which is built on the foundation that tasks have a “start” and an “end”).

Yet it works great: jobs are now asynchronous constructs with passive consumers that do not need to pull the data when predefined conditions are met. To use a metaphor I like: we’re working with pipes instead of buckets. There’s a lot less lifting so you can sit back and watch the data flow.

Final words

You made it!

In many organizations, data products are still the output of a long line of data manufacturing that requires a plethora of orchestration. But it doesn’t have to be this way, the solutions are already out there, like the Streaming Workflow Builders Hubert mentioned or what we’re up to at Popsink.

With the ongoing shift to continuous operations, it’s worth exploring what the post-orchestrator world feels like, if not for your sanity then for your wallet. I also grew pretty fond of the term “data services” as it sits well with the idea of on-demand data subscription as opposed to off-the-shelf “data products”.

It’s going to take a bit more time to reach full feature parity and for good reusable standards to materialize, especially as current active metadata and lineage systems are deeply rooted in orchestrators, but the capacities are already here.

It’s time to move on with your life.

In a previous post, I talked about how data lineage can be difficult for stream processing workflows. The asynchronous tasks in streaming make it hard to conceptualize existing data lineage solutions. It’s because streaming tasks don’t have a start or an end. Lineage tools are mainly built for batch processing which expects synchronous tasks which do have a defined start and end. This same issue exists for workflow orchestrators.

Workflows are everywhere. If you’ve ever viewed a Spark DAG or a Flink plan, you’ll know that those are workflows. If you’ve ever explained a SQL statement in a popular database, you’ll also see a workflow. When implementing a SAGA pattern for your microservices, that is also a workflow. These workflows contain business logic which is always being updated for optimization to save cost and improve user experience. Workflows need to be agile. We need tools that enable this agility to quickly react to improved changes to the business.

Event-driven applications and (a)synchronous tasks

To better understand asynchronous applications, we’ll need to understand Event Driven Architecture (EDA). EDA is an acronym that categorizes solutions that are triggered by events. An example of an event is a file appearing in a directory or a message appearing in a queue. EDAs are always running and listening for an event to occur. Then they act on that event by either running a task or processing data.

EDA applications do not require the producer of the event to know about the consumer of the event. Producer and consumer are not communicating synchronously like request-response RESTful APIs. Therefore EDA applications are asynchronous.

Event-driven orchestration vs stream processing

Orchestration involves using a tool like Airflow or Dagster to execute tasks in a workflow. These tools make it easy to build complex workflows, schedule, and monitor them. Workflows in these tools are called DAGs (directed acyclic graphs) and are executed on a schedule. Event-driven orchestration follows EDA semantics and can be done with these same tools. Instead of running on a schedule, they are triggered based on an event.

Stream processing is also an EDA. Stream processing involves listening for events that have occurred in a data store or messaging system and processing those events while the data is in motion. 

The main difference is that the events in an event-driven orchestration don't hold business data while the events in a stream processor do hold business data. Examples of business data would be account records, user records, product records, etc.

Hubert’s Substack

"Streaming Data Mesh" OReilly - Supporting Monolithic Data Engineers

By Hubert Dulay

Can orchestrators do stream processing?

Both Airflow and Dagster have the ability to subscribe to an event to trigger a workflow. In Airflow and Dagster they are called sensors. These components allow the orchestrator to listen for events wherever they may occur. Some examples of this would be listening for a file to appear, waiting for an HTTP request, or even consuming from a Kafka topic.

Stream processing can also read from Kafka. So can event-driven orchestration support stream processing? In order to answer this question, we need to understand the differences between batch processing and stream processing.

Older ETL/ELT processes required writing their data into a database because they didn’t have a way to perform complex transformations like joins and aggregations. When you persist your data in a database, you're forced into batch processing semantics which is why most older ETL/ELT processes needed to use scheduling tools like CRON. Airflow and Dagster are similar to CRON with additional enhancements.

With this understanding of orchestrators, I’ll try to use another event-driven orchestrator called Prefect.

Prefect

Prefect is an open-source orchestration platform that looks promising. It natively has a way to consume events. In the code below, line 42 is an annotation that decorates a function. When this Python script is run, it will be submitted to a running Prefect server for execution. Line 9 contains the configuration to connect to Kafka. Line 21 contains the code that consumes data from Kafka and then processes the data by just printing it out to the console.

At first glance, this code looks like it may be the solution we need, but can it replace the popular stream processing platforms available today? Let’s see what we actually need for stream processing.

Stream processing requirements

Stream processing is an alternative way of transforming data that keeps your data in motion. This maintains real-time semantics but there are a few requirements orchestrators need to be able to handle stream processing.

• Need to be able to hold state.

• Need to distribute tasks to execute in parallel.

• Need to shuffle data between tasks.

State enables stream processors to perform aggregations like counting records. Otherwise, it will not be able to remember the current count. These orchestrators do not have this ability.

Streams also need to evenly distribute their tasks across multiple workers so that they can process large amounts of data in parallel. Processing TBs of data through a machine will take a very long time. In a high throughput use case, the Kafka topic may have many partitions to allow for horizontal scaling. These orchestrators don’t take advantage of these partitions. They don’t have a way to distribute the data across multiple workers to execute parallel tasks. They deploy only one task runner which will need to subscribe to all partitions and process them synchronously.

Shuffling data is the way distributed databases move around data when performing a join. Shuffling and state enable stream processors their ability to perform complex transformations. Since orchestrators are neither distributed nor hold state, they cannot perform complex transformations.

So no, orchestrators cannot replace today’s popular stream processors. So what is event-driven orchestration supposed to do? The answer is really notifications. Event-driven orchestrations are really notification consumers. They only trigger tasks and do not define the task itself. The tasks that orchestrators invoke could be a batch process that can handle large volumes of data but that data wouldn’t come from Kafka. Batch data comes from data stores that hold data at rest.

Drivers are like DAGs

In Spark and Flink, you define your workflow logic in a driver just as you would in an Airflow DAG. Drivers are written in either Java, Scala, or Python and follow a functional programming (FP) paradigm. FP is beyond the scope of this post but to summarize, FP treats functions as first-class citizens. You can serialize a function passing them to other workers in a distributed system to run in parallel. These drivers define the logic in the workflow without actually running the logic locally. Instead, the functions get executed remotely to distribute the workload and scale out horizontally.

These drivers aren’t orchestrators like you would think of Airflow, Dagster, and Prefect. The main difference is DAG orchestrators define task workflows but do not implement the logic in the task. Drivers do implement the workflow logic.

Streaming workflow builders

There are nice streaming workflow builders in the market today but they are not orchestrators. Confluent’s stream designer and StreamSets are examples. These builders can build a streaming data pipeline from source to sink.

These builders have the ability to put tasks together like puzzle pieces using schemas. Schemas define the shape of the data. Stream builders use schemas to put tasks together with the same schema. If the schemas don’t match between the tasks, then the builder cannot put them together.

Is Closing The Gap Possible?

I really liked using Prefect as a way to trigger a workflow. It has a beautiful dashboard for monitoring your event-driven workflows. Plus, it’s open source which is even better. I’d say it’s a “prefect” orchestration tool. But it’s event-driven, not stream processing. If you need to do real-time processing of data, today you’ll need to leverage the stream processing tools available today like Apache Flink, Apache Spark, ksqlDB, etc.

But can orchestrators do stream processing?

Not really. Orchestrators assemble discrete tasks in a workflow. To the orchestrator, tasks can be arranged in any order like square blocks. Tasks in stream processing are arranged like puzzle pieces and must follow the input and output of each subsequent task. Think of stream processors as a large SQL statement with many CTEs that feed into one another sequentially. Then at the end of the statement, there is one data set that is stored somewhere. That is simply not what workflow orchestrators do.

The goal of engineers is to be as lazy as possible. If your system doesn't allow you to sit and wait for completion, you're missing something.

That was probably the day I knew I wanted to be an engineer.

The IT teacher was writing some wizardry on a very desaturated terminal projected on a big white wall. He said this sentence while I was on the point to get a "wow effect" on one of my first Python code snippets. It’s now carved in my memory.

Though a bit sarcastic, I really think he highlighted something that day.

What's our job - data engineers - if not trying to reach end-to-end automation?

I once wrote that we don't need orchestrators. We need orchestrators, but not ones that end up in spaghetti, not ones that need Italian data engineer chefs to maintain its codebase.

The orchestrator is often our best friend when speaking about automation, but I think we still lack something. We are to a certain extent just a replacement for old BI tools... We should drive greater value.

Data engineers are not used to their full potential. Writing the same Python code over and over, dealing with the same data issues, migrating systems to new ones, etc. This is not really an engineering job, is it?

The outcomes we should look for are automation at its full potential, laziness at its climax, and optimized engineering costs. Our main duty should be to design architecture, to build systems that allow us to be lazy, not manage all this stuff we have built.

From An Engineer Sight

A periodic about data, engineering and design

By Benoit Pimpaud

How to clean up our city?

As Antoine de Saint-Exupery said, "a designer knows he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away."

We definitely missed something with the modern data stack. Unbundling and bundling cycles, some would say. Or we just got blinded by marketing and hoped that our problem could be solved by a myriad of new tools.

The orchestrator’s neighborhood is not a pretty sight. It’s like a continuation of gray buildings trying to grow wherever they can while one wonderful house keeps standing here in the middle of the street. There are shadows on this house. (There are actually several houses.)

What we might just want is to bring some sunshine back to the city. To remove some of those ugly buildings and keep the ones that really support us. And why not at the same time raze some big houses nobody really wants to buy anymore?

Yes, the Airflow house is great. I loved Airflow’s garden, like any front-end developer who loved jQuery lighting at some point. But now the house is too big to keep all our furniture and decorations tidy.

 “Airflow was not built for infinitely-running event-based workflows”.

Playing with words here, but even if that myriad of tools didn’t replace Airflow, they still highlighted something: we need a new central control plane to deal with our event-based reality and its infinitely running data flows.

Fortunately, we are starting to catch up. 

Yes, we have and use better tools: dbt, new declarative orchestrators, Terraform, duckDB, CI/CD, etc... The declarative paradigm is nudging every part of the data stack.

But as disruptive as those new tools are, we still need vision, automation, and architecture design. We need the mayor to wake up and stop wondering if building a new pool will allow him to be re-elected.

Tools and codebases are often the elements of the debate. And while they support our concrete daily stuff, it's rare to take the same amount of time to think about the underlying system architecture they question and the legacy they will create.

One key to uncovering some vision and paving the way toward persistent roads is to ask ourselves probing questions :

• Should my orchestrator be in the central space of my data stack? Does it fit with my business needs?

• Do I really need to pay for a tool to extract data from one place to another?

• Should I ask pricy engineers to write the same low-value Python code over and over?

• What's my codebase vision? Does my codebase need a vision? Or can I trash some code easily?

• Is maintaining custom abstraction on top of abstraction a good idea (we often see custom Airflow codes in companies, but we have to remember that Airflow is already a layer of abstraction)?

• Do my high-level execs understand what the game we are playing for?

• Do my data analysts focus on business intelligence while understanding the need for good software engineering practices?

• Do they consider themselves as "coders" even if their tools tell them to drag and click?1

Again, no straight answers. Those questions are only here to suggest a way. To bring forward that idea, that small trick we are looking for while solving our technologies and organizational issues.

Like a city, a data system is not something neat. It's growing and changing constantly.

Our orchestrator is like the city's traffic controller, responsible for coordinating the movement of data through the city's streets and ensuring that everything is running smoothly. But we have to be at the city council level. Where we create a vision for our city's future and design the infrastructure to support it.

And like any city council, we have to be lazy. Don’t take me wrong here, it’s being lazy in the right way: we have to be bored in advance to deal with angry citizens, solve traffic jams, find resources from the government, etc.

There will always be unforeseen challenges, changes in the environment, and new technologies to integrate. And while the traffic controller has a major role to play here, it's our role to build solid architecture and a clear vision to guide our entire city's development.

This is essay #3 in the Symposium on Is the Orchestrator Dead or Alive? You can read more posts from Louise on Medium.

The world of data management is evolving rapidly, and one of the latest trends gaining traction is active metadata.

Active metadata refers to a dynamic approach to managing and using metadata, allowing it to flow across the different parts of the data stack, fetching people in the tools they are already using. Active metadata is actively updated processed, and integrated into various tools and workflows. This notion is often opposed to the idea of static metadata, or the traditional process of storing metadata in a static data catalog.

Now, I’ve heard the rumor that active metadata is a play by data catalogs to replace the orchestrator.

Where is this coming from?

Well, a data orchestrator is a software solution or platform responsible for automating and managing the flow of data across different systems, applications, and storage locations. Sounds familiar to the active metadata description? Indeed. Is active metadata trying to steal the fame?

I think not. Why is that? Active Metadata has better things to do. It’s hungry for something else.

In this piece, I’ll explore the extent to which Active Metadata could eat the orchestrator. Turns out, Metadata can eat half of the orchestrator, but it cannot eat it entirely. But more importantly, Active Metadata should not want to eat the orchestrator. In fact, it’s hungry for business users.

What Exactly is Active Metadata Eating?

Data orchestration involves managing complex workflows and ensuring that the right data is available, transformed, and integrated as needed.

This process can be broken down into two interconnected components: the “triggering mechanism” and the "optimization intelligence."

The triggering mechanism is in charge of starting data orchestration workflows based on factors like time, events, or changes in the data.

The optimization intelligence, on the other hand, is responsible for optimizing the triggering mechanism. This includes deciding which workflows to execute, the order of execution, and how to transform and route the data by analyzing the metadata.

Active metadata, with its real-time updates and integration into various tools and workflows, has the potential to replace the triggering mechanism aspect of data orchestrators.

It can automate the initiation of workflows, provide real-time updates for the efficient triggering of data processes, and create context-aware workflows that dynamically adapt to changes in the data landscape.

By replacing the triggering mechanism component of data orchestrators, active metadata can offer enhanced automation, better integration with existing tools, and more efficient collaboration between teams. Isn’t this lovely?

However, active metadata cannot completely replace the second component of the orchestrator: optimization intelligence.

This is because data orchestrators still play a vital role in handling complex data processing, transformation, and integration tasks that require their expertise and capabilities.

Optimization intelligence is essential in determining how to process and move data between different systems and applications, ensuring that the right data is available in the right format at the right time.

Active metadata, while powerful, cannot fully replicate the sophisticated decision-making and processing abilities of the orchestrator.

The latter remains an integral part of the orchestrator's role, as it handles the more complex and nuanced tasks of data processing, transformation, and integration.

This means that while active metadata can streamline certain aspects of data orchestration, the orchestrator continues to be crucial in managing the overall data workflows and transformations.

So, active metadata cannot replace the orchestrator. But the real question is: should it want to? The answer is no.

Active Metadata is hungry for something else

Active metadata is a technology that can tackle different use cases. And we do not think its true value lies in replacing the orchestrator.

At Castor, we believe that active metadata should pursue a much more rewarding goal: empowering business users. And it can do so by providing them with data context right where they are working.

Active metadata shouldn’t seek to eat the orchestrator. Instead, it should strive to become a personal data assistant that enhances collaboration, decision-making, and overall data literacy within the organization.

Active metadata can help business users make more informed decisions by offering necessary data context within their existing tools and workflows.

This context includes data definitions, lineage, relationships, and quality indicators, enabling users to better understand the data and make decisions based on accurate and up-to-date information.

Furthermore, active metadata can foster better collaboration and communication across the organization by integrating metadata into the tools and platforms that stakeholders already use as part of their workflow.

This seamless integration can bridge the gap between technical and non-technical team members, creating a more efficient and cohesive data-driven culture where everyone has access to the information they need to work effectively.

Finally, active metadata can play a crucial role in supporting data governance and compliance. By automating tasks such as data lineage tracking and data quality monitoring, active metadata can help organizations maintain control over their data and reduce the risk of non-compliance with regulatory requirements.

For these reasons, we sustain that the true value of active metadata lies in fostering innovation and growth within organizations.

By making it easier for employees to access, understand, and analyze data, active metadata empowers them to identify new opportunities, optimize processes, and develop innovative solutions to business challenges.

With a better understanding of data context and more efficient collaboration, organizations can leverage active Metadata to drive innovation, stay competitive, and achieve their strategic objectives.

In summary, even though active metadata can potentially influence the triggering mechanism aspect of data orchestrators, its real value is in its ability to act as a personal data assistant that enhances data literacy, collaboration, and decision-making within the organization.

By focusing on these more valuable goals, active metadata can bring tangible business value by empowering users to make the most of their data assets. We think the end game of active metadata is far more powerful than simply replacing orchestration, as it helps organizations unlock the full potential of their data and drive meaningful results.

Conclusion

In the ever-evolving landscape of data management, active metadata has emerged as a powerful force, offering new possibilities for streamlining data processes and empowering business users. While it may appear that active metadata has the potential to replace certain aspects of the orchestrator, particularly the triggering mechanism component, it should be put at the service of a different use case.

Active metadata's ultimate goal should be to serve as a personal data assistant, enhancing data literacy, collaboration, and decision-making across the organization. By focusing on these valuable objectives, active metadata not only complements the role of the orchestrator but also brings tangible business value by enabling users to maximize the potential of their data assets.

By embracing the possibilities offered by active metadata and leveraging its unique strengths, businesses can stay ahead of the curve, foster a data-driven culture, and achieve their strategic objectives in an increasingly competitive landscape.

So, while active metadata might have an appetite for some aspects of data orchestration, its true hunger lies in empowering business users to make better use of data.

This is essay #2 in the Symposium on Is the Orchestrator Dead or Alive? You can read more posts from Vinnie on his Substack, .

We’ve already accepted that engineers shouldn’t write ETL. “It’s a hot potato,” we said, so let’s “give people end-to-end ownership of the work they produce.” Let Data Scientists write ETL, let Data Analysts ingest their own data. Engineers don’t have time for that, they’re busy writing vendor glue and fixing the SaaSpool known as “modern data stack” after dbt published a breaking change in their most recent release, breaking all 37 connectors.1

I say this is a good step, but it’s a band-aid solution. All of the problems inherent with ETL workflows are still there, just someone else’s responsibility. Why don’t we just do away with it altogether?

Maybe I am taking the argument in the article too literally after the wine I was contractually obligated to drink for this symposium (thanks for fuelling my alcoholism, Stephen), but let’s run with it for a second. Let’s assume that the platform team is just keeping packages updated and writing more decorators we can all import into our scripts. We can not only @time, our functions, we can also @count how many times they’re called, automatically @send_telemetry_to_infra_team and beg them to @accept_this_pr. I promise this time the code is @tested and @stable.

We have our connectors. We have our Fivetrans, Stitchs,2 Airbytes, and that one pesky SDK some team wrote to access that one internal API.3

The Data Analytics team is happy, they can click here and there and get metrics from Snowplow into Snowflake. They don’t even have to model the data themselves, it’s already done.

And let the Data Science team do the same. End-to-end ownership means end-to-end silos. Those analysts could never begin to understand the sheer magnitude of import tensorflow as tf. Why use their data, it’s probably badly modeled anyway. We found our own, better package and can hook it up ourselves. Now if only the Airflow instance doesn’t crash when we push our new DAGs to prod.

Ask Vinnie

Musings on tech, data, philosophy, and the absence thereof.

By Vinnie Dalpiccol

Is it just me, or does this seem terribly inefficient? End-to-end ownership sounds great and the autonomy feels great, but even the author of the original piece says that “[they] are sacrificing technical efficiency for velocity and autonomy. It is important to recognize this as a deliberate trade-off.” It makes no mention of unnecessary duplication.

I guess I’ve been in one too many organizations where multiple teams were building the exact same thing without knowledge of one another (well, it’s only happened once, but I think it’s once too many) to accept this as a good solution. I may be too lost in abstractionland, but at a meta-level, most data work really isn’t that unique (for the sake of the argument, we’re not talking real-time or streaming processes), and as a matter of fact, it’s only the shape of the final exposure (yes, this could be another dbt pun) that really matters. Everything else going into it can, and should, be reused.

But if engineers aren’t writing ETL, and nobody really likes to do it, do we just accept this duplication? Do we just keep filling Snowflake’s pockets?

I think there’s a better way, and orchestration can be the answer. But not any orchestration, asset-aware orchestration.

Let’s all just stop writing ETL altogether. Let’s declare the outputs of our pipelines and let our systems figure out the rest. If humans don’t wanna deal with it, let the machines do it, at least while we can still command them.4

We declare our data assets in code. Our data platform knows how to create and maintain them. The engineers already wrote fancy abstracted vendor glue anyway, they might as well write a YAML config for it. The consumers, be it the data scientists or analysts, can stop importing the @fancy_new_decorator and instead import the source asset into their project. That’s materialized once and always kept up to date. They tell the system their downstream use case depends on it, and then go to sleep. Or to a bar, or to the beach, and everyone lives happily ever after.

Maybe the very notion of ETL is wrong. In 2023, our data warehouse is not the final destination, it’s just another step in the journey of the data. ETL, and its cousin Reverse ETL (AKA “ETL”[::-1]) were helpful as concepts a few years ago, but it’s time we start thinking about processes differently.

The world I wanna live in is one in which some team, be it the engineers or the data producers themselves, declare source assets, including metadata and contracts about its shape, and consumers can hook up to them, the system being able to intelligently tell where they’re saved and how often they’re updated, and taking care of keeping all downstream dependencies up to date. Maybe this whole article is just about solving the technical part of “data mesh”,5 but when we're still trying to hold our Data Scientist's hand (they might understand import tensorflow as tf, but they would never understand the Liskov Substitution Principle),6 we might as well go all the way and take back ownership for the sources. We all know they weren’t writing unit tests for them anyway.

And yes, this whole article is just another way for me to shill for Dagster’s software-defined assets and declarative scheduling. Guilty as charged.

I remember how very green I was when I joined my first data team. Excited to be given the opportunity because… well it was boring and no one else wanted that job. These were the days when the scope of data teams was minimal, before data scientists were called the sexiest job of the 21st century. I remember reading that headline at the time and feeling confused. My day to day responsibilities were decidedly not sexy. Our main responsibility was to ensure everything ran in a predictable fashion with as little excitement as possible. But that was over a decade ago! Jumping forward to the end of 2022 I had almost forgotten those unsexy times.

The last few months have been tough for tech. Luckily data teams have not been hit too hard (based on my entirely anecdotal evidence). But I can’t say the same about the data stack. Teams are abandoning many of the tools they had added. We have all found out the complexity of the modern data stack does not mirror the maturity of the business or the number or people on the team, as much as it reflects the economy.

From my conversations, it is interesting to see what tools have been making the cut. High level it seems it has been a lot of last in, first out. Not many people are giving up on their data warehouse, orchestration or ETL (turns out databases are recession-proof). But some of the new tools have been sidelined or put on pause.

There are a few reasons some tools have not been able to differentiate themselves from critical components. First of all, the closer you get to the datastore, the harder it is to pivot. Once companies get their data in one place, they generally don’t like to move it somewhere else unless they have a very good reason. Even moving to similar technologies risks unintended changes and cascading issues that can add months of development time.

Databases are still the bread and butter of the data stack which everything else revolves around. It follows that there is a lot less need for data metadata analysis tooling without an actual database to point it at.

Also, data teams, not surprisingly, have a better understanding of the systems they have been dealing with for most of their careers. dbt’s biggest contribution to the community was not an overly elegant framework but a common language for people on small teams to communicate with other data folks working on similar projects. Everyone working in the industry before dbt had built their own version of it or tried to build one. They were rarely that successful but analysts at least understood what they were trying to accomplish.

As the original responsibilities of data teams became “solve problems”, the community moved onto other challenges especially as it became more insular. But here it became a little bit tricker. What should come next? Do you build a feature store? (I don’t mean to pick on this one example) There are many legitimate use cases for a feature store but not as many smaller data teams have experience building one or needing one. Luckily there were a growing number of offerings allowing you to jumpstart that journey. But jumping into the deep end of a complex project with a complex solution does not solve much of anything. Yes it makes things easier to get started but it quickly showcases the deficiencies of data teams.

A recent trait of the data community is to put a new name to an existing software practice and proclaim something new has just been discovered. As the goals of data applications became more ambitious and their scope expanded, there were more needs to support. For example, you now need someone or some tool to help with safe and consistent deployment. You might be thinking DevOps would fill that role but that would be wrong. The correct answer is you need DataOps… or MLOps… or AIOps… the point is it never existed before and it was up to the data team to develop this competency.

The increase in specialization in data teams while ignoring expertise from existing, non-data, teams led to more and more pressure to layer in new increasingly specific tooling. At its worst, data teams would just build parallel engineering orgs. This could help the agility of pure data applications but started to betray the goal of more complicated applications.

In a lot of ways, the new tools added to the data stack were an excuse to not communicate outside of data. Data teams have always had an inconsistent home. Is it in Product, Eng, DevOps? But ambiguity is not a reason to start doubling up on responsibilities and tools. The truth is if you treat data applications as entirely separate from engineering applications on the critical path, those data applications will never be trusted outside of data.

Building any level of trust means going beyond yourself. It may mean holding the data team to some of the standards of engineering and not just doing everything that makes the data team feel the most comfortable. Not every member of the team needs to become a software engineer but it would be good to try and learn some SWE habits. One of which is the importance of working within parameters. You may not always make the perfect decision but you need to know enough to not paint yourself into a corner. Part of maintaining a system means you can’t rewrite it whenever you hit a wall.

So what will the next year look like in data tools? I think we are going to go back to behaving like boring practitioners who work with confines. We may have to use tools not entirely catered to us. There will still be analytics tools but I think they will behave in ways that are recognizable to both data and engineering teams. This will allow a number of applications to be built on top of them and help break down silos.

We should encourage looking across teams to see what we can leverage. We can also try to return the favor. On data teams, I have often found the most successful tools draw interest from outside of data. When rolling out orchestration it was not over when analytics jobs had been migrated but when engineers started to move their own jobs onto it. When data applications start to resemble any other application, you can build some really interesting things and some things you didn’t intend at the beginning. You may also find that when something has the approval of the entire organization, it is much harder to cut when times get tough.

This is essay #1 in the DPE Symposium on Is the Orchestrator Dead or Alive?

Essay Index

Symposium posts were released between March and April 2023.

• Data in a Downturn by

• Nobody should write ETL by

• Will active metadata eat the orchestrator by

• Good data engineers are lazy by

• Limits of the Event-Driven Orchestrator by

• Life After Orchestrators by

• Data Materialization is a Convergence Problem by

• 28 Dags Later by

• Orchestration isn’t going anywhere by

The Symposium

For my own part, indeed, I take an immense delight in philosophic discourses, whether I speak them myself or hear them from others: whereas in the case of other sorts of talk -- especially that of your wealthy, money-bag friends -- I am not only annoyed myself but sorry for dear intimates like you, who think you are doing a great deal when you really do nothing at all.

From your point of view, I daresay, I seem a hapless creature, and I think your thought is true. I, however, do not think it of you: I know it for sure.

— Apollodorus, in Plato’s Symposium

Too many people approach writing with undue reverence. True, you must learn the craft and treat the audience (and their time) seriously. But I’ve found a lot of joy this past year in rebalancing between constructive and expressive modes of writing, and I’d like to create a space for others to try that out.

Enter the symposium: the drinking party. What’s not to love? If a convention is a coming-together of people for a certain purpose, a symposium is a coming-apart over the same. The drinking frames the discourse, the discourse drives the drinking. Cheers to that!

So, this year, I plan to experiment with hosting a few symposia — an open invitation to submit essays on a particular prompt. What do you know that others don’t? What irritations do you need to get off your chest? What’s worth putting your glass down and moving the conversation towards?

This is writing as basic philosophical art: bullshitting, seriously. Whoever you are, consider grabbing a pen, paper, and beverage, and writing in on the topic of:

Is the orchestrator dead or alive?

Complete your Intention To Submit by March 3, 2023

To “unbundle” is to destroy something essential. The unbundled country is scarred by civil war. The unbundled house is empty from looting. The unbundled person is, well, luckier to be dead than undead.

So what is the unbundled orchestrator?

The orchestrator certainly has a claim to prominence in the data platform. It is the left-most tool, apart from the cloud provider. It knows your secrets. It has its tentacles everywhere. By the time it’s set up, it becomes wedged in a way comparable only to the database.

Yet, it’s squeezed on the left by simpler schedulers: Kubernetes cron jobs, Snowflake pipes, Amazon step functions. It’s pressed on the right by good-enough SaaS features: an unknowable wave of cron jobs.

Is there enough room for the orchestrator? Is it solving the right problems in 2023? Or should it wither and die, ceding its territory to the unbundled wilds.

I want to hear from others who are as obsessed with this question as I am. You don’t need authority to join a symposium — only spirit (and spirits).

And if Dead or Alive doesn’t summon your muse, remember it’s just the symposium’s opening salvo. Try out any of these other orchestrator-adjacent prompts:

• What sins did you commit to land in orchestration hell?

• Have you built your own orchestrator? Why? Are you okay?

• Is “active metadata” a play by the data catalog to eat the orchestrator?

• Is the “data platform OS” just a data-aware orchestrator?

• Is ML orchestration yet-another-pipeline, or is it sufficiently different to deserve its own niche?

• Are data engineers condemned to forever be ritually sacrificed to Jeff Bezos’ pet minotaur in the labyrinth of network policies, Terraform, Python, Docker, and managed services? Is the orchestrator the hero they need?

• Are cron schedules real? Is time an illusion?

Whatever your thoughts, I want to read them. And so do others, probably. And if they don’t, they at least want to drink while they act like they’re reading them.

So follow the instructions below, so you can tell your grandchildren you were a part of the first Data People Etc. symposium.

How To Submit

The authors behind Data People Etc. value doing as little work as possible. That means every submission will be reviewed, but not deeply edited for content and quality. It will be accepted if it meets the bar of a professional making a good-faith effort to write an interesting essay.

There are two routes to publication:

Published directly on Data People Etc.

1. You fill out the Intent to Submit.

2. You complete your essay by the due date.

3. You coordinate with Stephen to create a guest author profile and a draft post.

4. Stephen publishes the essay on the arranged date.

Cross-posted to Data People Etc.

1. You fill out the Intent to Submit.

2. You write your essay on your Substack.

3. You publish your essay on the arranged date.

4. Stephen cross-posts your essay to Data People Etc.

What to Submit

The literary symposium resembles the newspaper’s Op-Ed section. Articles should contain both argument and spirit — crankiness is an asset here. Other analogs: the folksy article on the last page of special interest magazines; the newspaper funnies in its golden age, circa 1992; scathing internal memos from early-stage hires that have finally had enough.

All you need is one good idea. It doesn’t have to be a great idea, and it doesn’t have to be a big idea. If you can breathe life into one good idea — and I know that you already have one, somewhere in you — then you can drive an essay worth reading.

Symposium essays can be as long as you want. Some DPE articles are long-form, registering 2500 words or more. They can also be short. Personally, I’d love to feature some truly weird writing — fiction, poetry, or songs. Whatever serves the idea.

Anyone can submit, including people affiliated with vendors. In that case, though, the author should respect the venue — present the ideas and the arguments, not the implementation details.

I’m also open to pseudonymous submissions. Is the orchestrator dead, but you are the CEO of a company that sells an orchestrator? Tell us more, Mr. Harry Flow.

The symposium is an opportunity to highlight divergent thinking. Be contrarian. Be elusive. Be metaphorical.

Or don’t. I don’t care, c'est la vie. I’ll be enjoying the wine.