What, exactly, isn't dbt?

A rational analysis of the popular tool's design principles

dbt is not an orchestrator.

No one said it was an orchestrator. I’m just clarifying.

The official word is that it’s a compiler and a code runner. It takes code, makes it runnable, and runs it.

Orchestrators also do this. They take code, and run it. They do a whole lot more, too. So I’m just saying it’s not a bad question, even if no one asked.

But dbt is not an orchestrator.

You know dbt is not an orchestrator because no one cares about orchestrators. You couldn’t pay someone to fly to New Orleans in October for a conference on orchestrators. Orchestrators are schedules. Partitions. Infrastructure. IAM roles. Kubernetes. Boo. Yawn.

dbt is its own box in the modern data stack ecosystem. It has no competitors. dbt is not a data prep tool. I’ve heard it’s not even “data build tool” anymore. dbt is dbt. dbt is letters. dbt is inside information. dbt is magic.

I’m talking about the command line tool here. The PyPi package. dbt Labs is a company. dbt Slack is people, I discovered. dbt Cloud is a SaaS product.

dbt Cloud is, in part, an orchestrator. People complain about dbt Cloud.

dbt is not an orchestrator. People do not complain about dbt.

You might think dbt would want to be an orchestrator. The left-most tool in the data stack. The conductor, the integrator, the organizer. Orchestrators direct. Orchestrators execute. Orchestrators lead.

But no, dbt eschews the label, because dbt killed the orchestrator.

The orchestrator, as a category, is lifeless. Aeons ago, Airflow was the hero that ran your data pipelines. Then — yoink! — dbt took ownership of the “data pipelines” part. Now Airflow is a scheduler. Its chief purpose is to run a 50 character python command once a day.

dbt harpooned Airflow, gutted it, and now lives inside it. dbt is primal. dbt is homo erectus. The orchestrator lays there, bloating, while dbt runs in and out of its propped up mouth-hole decorating the inside with colorful shells.

dbt needs an orchestrator. But it is not one.

Every other application, besides dbt, is an orchestrator. Every application pays the price for it. Every person hates every application as soon as it starts orchestrating. To orchestrate is to grapple with reality, to embrace mortality, to welcome death.

dbt is not an orchestrator. It is the anti-orchestrator.

To execute “dbt run” is to become a god. You are YHWH separating light from dark, land from water, life from unlife. You are Dr. Manhattan, raising a crystalline palace on the surface of Mars. You are Aragorn, son of Arathorn, heir of Elendil, summoning an army of ghost warriors to reclaim your nation and usher in a new age.

To orchestrate ”dbt run” is to lose the best years of your life. pip install; pip uninstall; python -m venv .venv; pip install; docker build; docker run; git commit; git push; git revert; terraform plan; terraform apply; ctrl+c; exit; :q; sudo rm -rf /; whoami; why; why; why;

To orchestrate is to never see your children grow up, to become estranged from your wife, to miss your fantasy football draft. To map cron expressions to your local timezone and watch an odometer spinning up through your most productive years. Tick tock, tick tock.

dbt cannot be an orchestrator. dbt is immediate. dbt is here, now.

Orchestrators do smart things, like saving data in databases when something important happens. dbt makes smart things sound like dumb things.

If you tell dbt to refresh your warehouse, it runs a dizzy bat race for 90 minutes and barfs out a massive JSON file right in front of you, right there in the house, right there where you are working. dbt wipes its mouth. Looks you in the eye. Again?

Again. Again, you say. Again!

dbt is not an orchestrator. You’re an orchestrator. Back off. Jeez.

Orchestrators are pedants. They compulsively explain themselves. Everyone turn to page 174 and repeat after me: Die-rekt-ed a-sick-lick graff. But the classroom is empty. Everyone has left. No one cares about graphs, where’s the business value in graphs. Quit trying to make graph databases happen. It’s not going to happen.

dbt doesn’t seek understanding. dbt is the graph. dbt turns you into a graph. Your company is the graph. dbt stole the fire from Mt. Graphlympus and brought it down here, for you. Selflessly, courageously. One day, dbt will be chained to a rock and pecked at by vultures for eternity. The least you can do is place the fire at the center of your city, today, right now. It’s an eternal fire, it will burn forever. You are not a believer. Yet, you know all other graphs are false graphs, and you will have no quarry with them.

dbt is not an orchestrator. Orchestrators tolerate false gods. dbt does not.

Orchestrators have bodies. They are tentacular. They are hybrid creatures. They cannot exist on their own. They need you, they wrap you, squeeze you, gorge themselves on you. They demand compute. They demand containers. They demand job definitions and schedules. Up front, no advances.

dbt is pure, simple, generous. dbt is you and your thoughts. dbt is your brain in a jar. dbt is your colleague’s brain in a jar. dbt is your company’s brain in a jar. dbt waits, patiently, for a body to fall in the jar, so it can take control, possess it, crawl out of the jar and start asking humans for sugar, more sugar.

dbt is a siren. “This SQL is nice but wouldn’t you like to give me data? Just a little bit of data? Good, didn’t that feel nice? What’s a little more data, then?”

dbt is that first prick of blood fed to a man-eating plant. Feed me, Seymour. dbt is a slit of the wrist to power a dashboard. Good, Seymour. dbt is your left hand for a team. Yes, Seymour, that’s right, we’re a good team, aren’t we. dbt is your eldest child in exchange for a seat at the table. FEED ME, SEYMOUR.

Orchestrators are stale process. dbt is living knowledge.

Orchestrators are requirements. dbt is appetite.

dbt is alpha and omega. There is no modern data stack without dbt. When dbt dies, so too will all data stacks. A black hole sun that pulls in everything, all of our work, all of our dreams, all of our data products.

Orchestrators are eternal. They were here before us and will be here after us. Skynet will be an orchestrator. The apocalypse will not be televised, but it will be scheduled. Idempotent. Retryable on network failure.

Orchestration is inevitable. But dbt is not an orchestator.

dbt. Is. Iron Man.