Symposium: Is the orchestrator dead or alive?
Longform techno-philosophical jabberwocky? Hold my wine.
This page features running commentary and introductions of Symposium posts, and it will be continuously updated throughout the symposium.
Essay Index
Symposium posts were released between March and April 2023.
Will active metadata eat the orchestrator by
Good data engineers are lazy by
Limits of the Event-Driven Orchestrator by
Data Materialization is a Convergence Problem by
Orchestration isn’t going anywhere by
The Symposium
For my own part, indeed, I take an immense delight in philosophic discourses, whether I speak them myself or hear them from others: whereas in the case of other sorts of talk -- especially that of your wealthy, money-bag friends -- I am not only annoyed myself but sorry for dear intimates like you, who think you are doing a great deal when you really do nothing at all.
From your point of view, I daresay, I seem a hapless creature, and I think your thought is true. I, however, do not think it of you: I know it for sure.
— Apollodorus, in Plato’s Symposium
Too many people approach writing with undue reverence. True, you must learn the craft and treat the audience (and their time) seriously. But I’ve found a lot of joy this past year in rebalancing between constructive and expressive modes of writing, and I’d like to create a space for others to try that out.
Enter the symposium: the drinking party. What’s not to love? If a convention is a coming-together of people for a certain purpose, a symposium is a coming-apart over the same. The drinking frames the discourse, the discourse drives the drinking. Cheers to that!
So, this year, I plan to experiment with hosting a few symposia — an open invitation to submit essays on a particular prompt. What do you know that others don’t? What irritations do you need to get off your chest? What’s worth putting your glass down and moving the conversation towards?
This is writing as basic philosophical art: bullshitting, seriously. Whoever you are, consider grabbing a pen, paper, and beverage, and writing in on the topic of:
Is the orchestrator dead or alive?
Complete your Intention To Submit by March 3, 2023
To “unbundle” is to destroy something essential. The unbundled country is scarred by civil war. The unbundled house is empty from looting. The unbundled person is, well, luckier to be dead than undead.
So what is the unbundled orchestrator?
The orchestrator certainly has a claim to prominence in the data platform. It is the left-most tool, apart from the cloud provider. It knows your secrets. It has its tentacles everywhere. By the time it’s set up, it becomes wedged in a way comparable only to the database.
Yet, it’s squeezed on the left by simpler schedulers: Kubernetes cron jobs, Snowflake pipes, Amazon step functions. It’s pressed on the right by good-enough SaaS features: an unknowable wave of cron jobs.
Is there enough room for the orchestrator? Is it solving the right problems in 2023? Or should it wither and die, ceding its territory to the unbundled wilds.
I want to hear from others who are as obsessed with this question as I am. You don’t need authority to join a symposium — only spirit (and spirits).
And if Dead or Alive doesn’t summon your muse, remember it’s just the symposium’s opening salvo. Try out any of these other orchestrator-adjacent prompts:
What sins did you commit to land in orchestration hell?
Have you built your own orchestrator? Why? Are you okay?
Is “active metadata” a play by the data catalog to eat the orchestrator?
Is the “data platform OS” just a data-aware orchestrator?
Is ML orchestration yet-another-pipeline, or is it sufficiently different to deserve its own niche?
Are data engineers condemned to forever be ritually sacrificed to Jeff Bezos’ pet minotaur in the labyrinth of network policies, Terraform, Python, Docker, and managed services? Is the orchestrator the hero they need?
Are cron schedules real? Is time an illusion?
Whatever your thoughts, I want to read them. And so do others, probably. And if they don’t, they at least want to drink while they act like they’re reading them.
So follow the instructions below, so you can tell your grandchildren you were a part of the first Data People Etc. symposium.
How To Submit
The authors behind Data People Etc. value doing as little work as possible. That means every submission will be reviewed, but not deeply edited for content and quality. It will be accepted if it meets the bar of a professional making a good-faith effort to write an interesting essay.
There are two routes to publication:
Published directly on Data People Etc.
You fill out the Intent to Submit.
You complete your essay by the due date.
You coordinate with Stephen to create a guest author profile and a draft post.
Stephen publishes the essay on the arranged date.
Cross-posted to Data People Etc.
You fill out the Intent to Submit.
You write your essay on your Substack.
You publish your essay on the arranged date.
Stephen cross-posts your essay to Data People Etc.
What to Submit
The literary symposium resembles the newspaper’s Op-Ed section. Articles should contain both argument and spirit — crankiness is an asset here. Other analogs: the folksy article on the last page of special interest magazines; the newspaper funnies in its golden age, circa 1992; scathing internal memos from early-stage hires that have finally had enough.
All you need is one good idea. It doesn’t have to be a great idea, and it doesn’t have to be a big idea. If you can breathe life into one good idea — and I know that you already have one, somewhere in you — then you can drive an essay worth reading.
Symposium essays can be as long as you want. Some DPE articles are long-form, registering 2500 words or more. They can also be short. Personally, I’d love to feature some truly weird writing — fiction, poetry, or songs. Whatever serves the idea.
Anyone can submit, including people affiliated with vendors. In that case, though, the author should respect the venue — present the ideas and the arguments, not the implementation details.
I’m also open to pseudonymous submissions. Is the orchestrator dead, but you are the CEO of a company that sells an orchestrator? Tell us more, Mr. Harry Flow.
The symposium is an opportunity to highlight divergent thinking. Be contrarian. Be elusive. Be metaphorical.
Or don’t. I don’t care, c'est la vie. I’ll be enjoying the wine.
Love this idea!