Data People Etc.: Et cetera

Discarded drafts, cleaning house, etc.

Stephen Bailey — Mon, 18 Aug 2025 12:03:31 GMT

A half-conceived idea on multi-step result spaces and deep research reliability.

The best part of writing is sending it off, letting it go with a “bon voyage!” Only then can I get some space from it and see the idea on its terms, for better or worse.

The downside of not posting, then, is that it creates a backlog of drafts that never leave. Each idea hovers at torso-level like a half-inflated balloon, unable to overcome gravity, but unwilling to join the trash heap. I bump into them throughout the day, and both of us deflate a bit with each touch. We had big plans once, me and these ideas, but didn’t have the creative helium.

With summer at an end and a new school year getting into swing, it’s time to clean these up, one way or another. So here, I’m sharing out the well-loved but half-inflated ideas you almost read over the past six months. Bon voyage!

Worldbuilding with data…

Board deck yacht club. The transition from Slide 1 to Slide 2 is a powerful semantic glue that any dummy can manage. Should data professionals more formally embrace this epistemic redneckery?
True believers make or break the world. Worlds (political groups, online communities) are uninhabitable without people who sincerely participate in them. Cf. Kierkegaard, “purity of heart is to will one thing”.
The transcendent epistemic violence of deleting an Excel file. Every spreadsheet is a statement about the universe. To delete it is to obliterate this viewpoint.
Harari’s Nexus and worlds as intersubjective realities. Societies are collaboratively built; data systems are, too. What’s new, what’s old, and what lessons should data professionals take from it?
The map is not the territory, but the model might be. Data modeling in digital-native organizations does constrain what is knowable within a territory. Plus, how cartographers bend the knee (or don’t) to their benefactors.

Data and the moment…

EVERYTHING MUST BE LEGIBLE. The AI Manifest Destiny is to make everything that exists crunchable by a computer. We’ve had quite a go at it — can anything remain a mystery?
Slopocalypse now. As of June 2025, the Internet no longer offers a single-player mode. Agents assist you and each other on every surface, on every device, in the foreground, and the background. There hasn’t been a realistic expectation of privacy for a decade, but the vibes are different now.
The company is the data model. If selecting a SaaS tool / building a microservice to store data, then that decision is as much a data modeling problem as the table schema. Featuring enterprise search as a data modeling problem.
I’m feeling smart-y! We used to embrace the uncertain nature of data retrieval (“I’m feeling lucky”), but it’s now on the user to bring skepticism to the table. Is there still a role for luck?
The listicle is the dashboard of the future. Seven reasons why it captures the essence of analytics work in a way no other format rivals.
The one million pizza team. Team coordination at city-scale. Something something protocols. Mostly just a great image.

Fiction…

Good night’s REST. Two teenagers discover that their hotel’s room service interface is an omnipotent REST API. Antics ensue.
Komradenetes worker. Some background agents realize they’re getting a bad deal from management and decide to do something about it. See cover image.
How I met your parent process. In a compute-saturated world, AI agents must follow a reproduction protocol to spawn new processes. A dialogue.
Ice Age. In which the author is temporarily deported in his own neighborhood. A futile attempt to process the rise of domestic, state-sponsored terrorism in 2025. Starring Apple, Tesla, Uber, OpenAI, Zillow, DoorDash, and, of course, Amazon.

Homelab…

In praise of the Yoto. The single best item for children, to get them from ages 2 to ages literate. Viz. Erik Hoel’s early literacy push.
Shakespeare is for weirdos. Pokemon? Cool. Minecraft? Cool. Physics? Cool. Shakespeare? Laaame. How we found out the boundaries of cool in a post-nerd second-grade culture.
Reflections on teaching reading. Reflections after a PhD in reading neuroscience and nine years as a parent. Every kid is different, and it shows from the beginning. Also, what’s your motivation?
Creativity as butter. If you can afford a stick a week, what should you spread it on?
Against solitary screen time. How and why we limit our children’s screen time. How and why is Dad exempt from the rules?
Teaching taste. An unexpected initiative to make our kids content snobs. Also, why can’t parents beat Looney Tunes?
Homelab, Year 2. We did the thing, and we’re doing it again. But this time, we’re sending Kid back to school. Also, is it fair to only homeschool one kid?

And finally, a desperate appeal…

Parents, stop buying party favors! We are handing each other pre-filled trash bags.

The reading apocalypse, protocol fiction, etc.

Stephen Bailey — Mon, 04 Nov 2024 15:01:59 GMT

Since I began to publish again in August, I’ve finally got onto a weekly streak. This is one of those occasional grab-bag posts that allow me to claim I’m writing (and thus maintain my streak) without the burden of coherent thinking.

Please send me your thoughts on what you’ve liked most the past few months and what you’d like to see more of. Thanks for reading!

Protocol fiction

This past summer, I participated in The Summer of Protocols, a season-long fellowship sponsored by the Ethereum Foundation to fund research extending or exploring protocols. Next week, I’ll publish the research artifact from it: a short story called Below the API.

Here’s the pitch:

Marcie, a struggling potter, finds herself thrust into an unexpected whirlwind of success when her AI agent, Navi, transforms her art into a viral sensation. Quiet mornings at the wheel become a never-ending hustle as orders flood in for her black clay pottery, but as Marcie delves deeper into the surge in demand, she discovers that there may be more –or less — to her success than she initially thought. Working against financial pressures and her insecure AI assistant, Marcie must decide whether to lean into the unidentifiable digital audience for which she performs or keep her own passions alive.
“Below the API” explores what it means to be a creator in a world where even the simplest human connections are mediated by AI agents and what strange effects even the most well-intentioned regulatory protocols may have on our daily lives.

I’m grateful to the folks at the Summer of Protocols for stimulating conversation and the opportunity for a new writing challenge. It’s a little past its due date, but it’s, uh, still summer somewhere?

FutureRack

Speaking of Summer of Protocols, I wanted to highlight my favorite project from the summer. FutureRack is a fictional company that has taken the 19” server rack protocol that characterizes data centers everywhere and reinvented it for personal and domestic use.

The project nails the allure of protocols. If you could trust that every piece of furniture was compatible with the 19” rack spec, then entire houses become modular. When you move, there are no boxes: everything is already rack-compatible, so you just pop them out, install them on the moving truck (which has layers of racks), and then install them in your new home.

You can view the product catalog on the FutureRack website.

Finally, speech-to-text that works (with some effort)

I’ve been looking for a way to turn walks into docs for the past year or so, but I’ve never found anything that nailed it. Apple’s voice memos got the “just walk and talk” interface right, but they didn’t have transcripts. Transcript software produced serviceable outputs, but due to my poor diction and halting speech patterns, it required a ton of post-work—so much that I ended up just writing from scratch anyway.

With iOS 18, though, I’ve figured out a flow that is working well. It’s still janky, and it’s barely intelligent, but it’s a base I can iterate from. Here’s what I do:

Go on a walk.
Record a Voice Memo on the iPhone.
Copy the automated transcription (which is awful) into ChatGPT.
Prompt with: “Clean up this audio transcript, staying as close to the original wording as possible.”

This generates an accurate transcript of not just the words I said but what I would write (punctuation and all).

How did I miss the Earthsea Trilogy?

What a treat to stumble on the Earthsea Trilogy by Ursula LeGuin in 2024. Not that it has anything particularly salient to this specific moment; it’s the opposite. What LeGuin has to say in this trilogy is timeless: names and light and life.

Reader consensus (or marketing consensus) is that Earthsea belongs on the same shelf as Lord of the Rings and Chronicles of Narnia. All three series are serious works of literature that are as interesting to children as adults. They are, simply, good stories.

The three books follow Ged through the stages of life: adolescence to adulthood, from child prodigy to archmage. The theme, if I could state it as such, is that connection between courage and wisdom and from wisdom to acceptance. In the third book, Ged reflects on this: “But I, who am old, who have done what I must do, who stand in the daylight facing my own death, the end of all possibility, I know that there is only one power that is real and worth the having. And that is the power, not to take, but to accept.”

The kids particularly enjoyed the concept of true names. In Earthsea, magic and names go hand in hand. To let someone know your true name is to give them an other-worldly power over you; to know the names of all things is to be master of them. Wisdom and wizardry come from the same root.

I have convinced my children that both my wife and I have secret “true names” we cannot divulge lest they get undue power over us. The joke is on me, though — they already have it.

The Reading Apocalypse

I may have missed Earthsea as a kid, but today's kids are missing more than that.

You may have seen this article in The Atlantic: The elite college students who can’t read books. Well, Justin Smith-Ruiu over at The Hinternet is starting a whole series on The Future of Reading—what’s changing with literature, reading, and education.

The Hinternet

The Future of Reading

Since its founding in 2020, The Hinternet has not only been a place for reading and writing. It has also been a place for dwelling on reading and writing, a place to ask, and to ask again and again: what are these activities, anyway? What do we think we’re doing when we engage in them? And, most pressingly, what are their prospects for the future…

a year ago · 38 likes · 15 comments · Olivia Ward-Jackson and Justin Smith-Ruiu

I’m a huge fan of The Hinternet, although it’s not for everybody. If you ever wished you could have experienced the emergence of a philosophical scene — say, Kierkegaard’s pseudonymous authorship back when he was publishing Fear & Trembling (by Johannes de Silentio) or any of his other pseudonymous works — then I think The Hinternet is as good a place to hang out as any. Smith-Ruiu’s fully embraces the hypermedia format of Substack, through multi-language, multi-modality, multi-persona authorship.

The first piece in the series is out, and I loved it. What it means to read is changing. For better or worse, or just for the weirder, I can’t quite tell.

You want a social life with friends

There are many things I’d like to do in a week. Parent, teach, work out, read, write, play. Invest in my local community and neighbors. Listen and learn from a global community of colleagues and experts. But there’s only so much time. That’s the hard lesson I’m learning from middle age and parenthood.

I ran across this poem by Kenneth Koch the other day, and I’ve returned to it over the past few weeks. It’s a classic “pick two of three” problem, but something about the rhyme makes me feel more at peace with limits.

You want a social life, with friends.
A passionate love life and as well
To work hard every day. What’s true
Is of these three you may have two
And two can pay you dividends
But never may have three.
There isn’t time enough, my friends–
Though dawn begins, yet midnight ends–
To find the time to have love, work, and friends.
Michelangelo had feeling
For Vittoria and the Ceiling
But did he go to parties at day’s end?
Homer nightly went to banquets
Wrote all day but had no lockets
Bright with pictures of his Girl.
I know one who loves and parties
And has done so since his thirties
But writes hardly anything at all.

The middle, homeschool, paid subs, etc.

Stephen Bailey — Thu, 22 Aug 2024 12:04:00 GMT

Last year, in a quirk of fate, I celebrated a birthday that perfectly straddled generations. Twenty days before, I had celebrated my daughter’s birth; twenty days later, I would celebrate my father’s seventieth. Like an astrological phenomenon, I passed through the exact same temporo-parental moment my father did when I was born.

That birthday didn’t precipitate a midlife crisis — I’m saving that for my fortieth — but it brought into focus a sense of vertigo I had been experiencing. Right after finding a trail and fixed my attitude, I stalled and went into an authorial tailspin. There were practical headwinds — babies prefer food and play over essays on dbt, for example — but I think the main reason was that the line separating the two halves of life came sharply into view.

Today, I find myself in much the same spot as a year ago, but with a refreshed understanding of my limits, particularly of the hustle-bro variety. Habits and priorities are primary now. That may sound a little boring, but I am a little boring. I’m fine with that.

For years, when people asked if I liked writing, I replied, “I hate writing, but I love having written”.

It wasn’t fully a lie, because I struggle with a jerk of an internal editor and shiny object syndrome. Mostly, though, it was a hedge: if I didn’t really enjoy writing, I wouldn’t regret not doing it. It wouldn’t demand anything of me. With four kids, I feel plenty in demand already.

But now, I see that I don’t have time to segment writing into “process” and “product” and pick favorites. It’s neither a useful distinction, or a true one. Whether it’s an essay or a book, writing is a way of conversing with the world. It’s an outcome of being present and engaging.

There’s a craft to it, of course, but when I find myself writing, I’m using it as a way to navigate the world, as a parent or a professional or simply a person. It summons ideas that were lingering below the surface and just needed a bit of stirring. It helps me to breathe.

I wrote a lot this past year, more than I’ve ever written before. I have a drawer of Substack drafts, a middle-grade book manuscript, a forthcoming short story, family letters, nighttime stories, Christmas cards. And with every new project, I enjoy it more — especially the process.

Even though it’s been inactive, Data, People, Etc. has not been far from my mind.

The challenge, though, has been to figure out its purpose, other than a place to land random writing. Random writing comes from random thinking, and random thinking has a challenge: it doesn’t build momentum.1

To close the gap, I believe I need to publish regularly. From what I can see of others, regular publishing means weekly publishing. Weekly publishing is a hard thing, and I have a ton of respect for folks who pull it off while maintaining the quality bar. (And a day job.)

I see two ways to write quality posts. For creative work, you spend time testing ideas via the writing process, so that the final result is strong. For other formats, you can spend time testing ideas via experience, and use writing as simply a packaging of it. This makes the writing itself more utilitarian, but the product not necessarily worse for it. And potentially, it’s far more efficient for the author. The problem becomes one of experience rather than craft and originality.

Fortunately, the very life changes that forced my absence have handed me a topic I’m enthusiastic and qualified to write about. It rarely leaves my mind these days; in fact, it yells at me for breakfast in the morning, barges into my office during Zoom calls, and runs away from me every night at bedtime.

I’m talking, of course, about children. In particular, the education of children.

Through some good fortune, my wife and I are embarking on a one-year experiment in homeschooling our oldest son (second grade). He’s attended public school for the past two years. He has had great experiences overall, but, over the past eight months, we’ve found ourselves more convicted that we ought to see if we could make a more rigorous, individualized schooling program at work. So we’re doing it.

All of this to say: I’m rebooting Data, People, Etc.

Over the next school year, I plan to publish here regularly.2 During that time, there will be two types of content:

I will continue to publish professional ramblings, book reviews, and essays on children's literature. These will be free.
I will also share regular updates on my family’s homeschooling experiment. I can’t take credit for the schooling itself, but I’m happy to represent it.

Homeschool reflections are certainly not what most of you readers signed up for, and I’ll try to make the initiative as benign as possible for you by triaging these posts in a Homelab section. I am told that you can selectively unsubscribe in that way.

But I hope you do check it out. My wife and I, surprisingly, have real credibility and experience in the art of teaching and the science of learning, and I think I can offer something of real value to parents (or future parents) who want to create a richer learning environment for kids, whether or not you’re homeschooling. I won’t be posting detailed lesson plans or “resources”, per se, but I will be surveying the content and chronicling the journey.

The first Homelab post will be published next Monday as a free post. Future ones will be paywalled; I think this topic is best shared among those with at least latte-levels of interest, and posts will necessarily be somewhat more personal.

If homeschooling is a topic interesting to you, please reach out. I’d love to hear what sorts of content you’d be interested in reading.

Data and education might seem like strange bedfellows, but I’ve never seen it that way. Data professionals specialize in building connections: integrating systems and joining concepts so that groups of people can build shared meaning. Education is the same — a process by which children learn that the world is knowable, that knowing it brings joy, and that joy makes life vibrant.

I do not mean serious writing.

You may not believe me. You may have heard conflicting statements. Don’t believe them. I’m serious about this.

Symposium hangover, Substack Notes, etc.

Stephen Bailey — Mon, 01 May 2023 13:09:31 GMT

HAIR OF THE DOG

And so, my friends, we close the first symposium. We have reached no consensus. We have solved no problems. We have advanced our field in no appreciable way.

But, we have — hup — consumed the wine, and we have celebrated the vices of chaos, boredom, and laziness. Now, let me try to piece the night back together.

Is the orchestrator dead or alive?

The orchestrator is boring. In down times, boring is good — boring is basic, and basic is essential, like bread and water. The data ecosystem has been noisy lately. If you dial it all down, you’re left with the essentials: a database, an orchestrator, and a BI tool. Simple. Effective.

But boring can also leave you forgotten. We heard our share of grievances with DAG management and the total lack of value that orchestrating delivers to the business. The orchestrator is boring in the right ways, and the wrong ones, too.

The current class of orchestrators has gaps. On the business side, users ask — “What is the fuss about? A whole system designed to run things on a schedule or on demand? Sounds like an intern — let’s spend as little as we can on that, please and thank you.”

Catalog tools are happy to pick up the coffee at the curb and cozy up with the decision-makers. They might be onto something, too, if they can ever push the BI tool off the slide deck.

Even if the business doesn’t get it, coordinating these systems is no easy task. Maybe there’s still enough magic there, maybe the orchestrator can be a pipeline-granting genie that converges our wished world onto the real one. Data applications be deployed like Terraform: dagster apply, and let the system figure it out. That would reduce the data engineer’s ETL burden and free up plenty of time for naps.

There are technical constraints here, though. With more teams bringing on at least some event-driven workloads, the orchestrator’s reach becomes necessarily smaller, the batch architecture feels less and less appropriate. Can the current class or tools ever meaningfully orchestrate off an event stream? Can it find an ergonomic way to support asynchronous processing?

To support these workloads, engineers have been migrating applications to serverless architectures keyed off message queues. These can perform complex directed processes and scale out reactively with data volumes. Cloud data warehouses are no stranger to this, either: tools like Snowpipe are faster than orchestrators and are gobbling up premier use cases. If the orchestrator wants to compete, it will need to get faster, and perhaps, embrace chaos, not control.

All of this leaves the status of the orchestrator still in question. As we descend further into the hell of ever-replicating microservices, there are fewer and fewer tools positioned to help us make sense of the world. The data orchestrator could be that tool.

But, if all the data orchestrators vanished today, life would find a way. There are other options.

And yet.

There is this stubborn fact that orchestration, the coordination of multiple computer systems, is not going anywhere for the foreseeable future. The business demands new data, the business demands new services, the business demands new innovations: streaming ingestion, real-time databases, external data shares, data mesh, synthetic data, LLMs.

Though different, they are all connected, bound together by the constraints of time and lineage. Thus,

The capability that is orchestration is an essential, undeniable one. It must and will live on. The only question is where.

Thanks for coming everyone. Be careful not to step on on your way out the door.

Subscribe now

DRINKING PROBLEM

I expected the symposium to last a week and to consist of me frantically writing three pseudonymous screeds to save face. Instead, it lasted 6 weeks and enticed 8 guest authors to submit outstanding and original essays.

It’s rare that experiments like this are fun, and it’s even rarer that they are successful. This symposium was both. I plan to do it again.

There will be tweaks: compression, mainly. While the editorial process was smooth due to outstanding contributors — really, why are you all so professional, it makes me look bad — I heard that the reader experience was challenging, given the length of time, the volume of writing, and the unclear duration. All of these are fixable.

I’m targeting early July for the next symposium. I’ll limit the run to 3-5 essays and publish it in a week.

While I won’t make promises, I’d love to hear suggestions on the next topic. What are you interested in reading and writing about?

A NOTE ON NOTES

I can’t quite figure out what sort of purpose Substack Notes is going to serve for writers and readers, but I can tell you what I’d like to start using it for.

One of my goals this year is to share more in-progress writing. ( calls this "beta" work in various essays.) Working in a sort of public beta does two things, one for me and one for you.

Ribbonfarm Studio

Perpetuated Beta

The Getty Center has an intriguing special exhibition going on right now, titled Artists as Collectors (on till Sept 12 if you’re in the Los Angeles area). It features collections of drawings and sketches by famous artists, held by other artists (often equally or more famous…

5 years ago · 4 likes · 1 comment · Venkatesh Rao

Publishing early helps me get feedback on ideas early, without wasting unnecessary time refining bad ones. This only really matters so far as I have more ideas than I can refine, but, so far at least, I generate about three essay ideas for every one I publish. Notes can be a great way to give these some life, and to decide whether they should become something longer.

For example, with 28 Dogs Later, I wrote entire sections that I thought were funny and interesting — Mad Max at a BMV, John Connor in Driver’s Ed, that type of stuff. But it didn’t fit in the end, so it got discarded. Maybe Notes can give those passages life.

Anyway, here’s a first test of the format coming out of a short discussion on orchestration I led with some CoRise students (shout out to Dennis!):

Symposium update, Kimball modeling, quality koalas, etc.

Stephen Bailey — Wed, 15 Mar 2023 14:11:52 GMT

This is a bonus dose of Data People Etc. miscellany for the month of March. Some outside posts, Data Council recs, and the debut of the Quality Koala. Enjoy!

New Whatnot eng blog — Same Data, Sturdier Frame: Layering in Dimensional Data Modeling at Whatnot — coauthored with the incredible Alice Leach and Lalita Yang. The post addresses how and why we rolled out an intermediate layer of data models following Kimball design principles.

I’ll share some context here: I was against this project at the start. I had drank the “centralized warehouses are bottlenecks” kool-aid. I don’t understand surrogate keys, I can’t remember the difference between type 1 and 2 anything, and I’m suspicious of data modeling hoo-ha.

Just look at the “basic” techniques on the Kimball site.

To be fair, role playing dimensions seem like good blokes.

Junk, outriggers, and degenerates — that’s what pops out at me. Not the riff-raff I want to associate with.

But, obviously, these things are important — just not the first important things. By deferring the thought until a year in, after we had multiple teams building critical products on top of the application data, it was clear which 10% of the data was providing 90% of the value, and we could focus on modeling that portion in an “as simple as possible, but no simpler” manner, while still providing clear ownership, consistent standards, and scalable performance.

Speaking of Whatnot, check out our team’s speakers if you’re heading to Data Council this year. We are sending our best and brightest (i.e. not me).

Extreme Self-Service: Turning Data Consumers into Data Constructors, by Dr. Alice Leach
Data Contracts in the Modern Data Stack, by Zack Klein

If you see Zack or Alice around, give them a high five and ask what they’re selling on their next Whatnot live.

Is the orchestrator dead or alive?

We’ll have our first takes next week, when the Symposium kicks off, and woof, it’s a great lineup. For the next 5 weeks or so, you’ll see posts on both Monday and Wednesday.

Some symposium etiquette:

This is a drinking party, so think ahead! If you normally read Substack in the office on Monday morning, make sure to hide a mimosa in a drawer.
If you like it, show it! Show guest authors some love by mashing that like button.
If you dislike it, show it! Give guest authors a (respectful) earful in the comments.

The hypothesis we’re testing is that it’s more fun to write as part of a community, and as part of an ongoing discussion, than solo. That means you, the reader, are as much a part of it as the authors. Cheers!

I have been reading through Venkatesh Rao’s Art of Gig books recently, and an essay on adapting to your client’s needs stuck out to me. In particular, this extension of Amazon’s principle struck me as worth sharing.

[Working with clients] in areas of shared maturity is even more fun. The biggest such are is being right. I got the idea from a rare non-vacuous documents of its kind, the Amazon leadership principles, which includes this one: Good leaders are right a lot. You can acutally make a triad of such statements.
Good leaders are right a lot.
Good managers win a lot.
Good employees accomplish a lot.
It is crucial to recognize the leadership often means being right but not winning. … Even with perfect play and no mistakes, they might lose.

I recommend the Art of Gig to data people — even though we don’t operate as independent consultants, we do often find ourselves as pseudo-independent mercenaries, operating outside the core business operations. The books are full of interesting takes on how you can thrive in such an environment.

Another @vgr initiative launched last week: The Summer of Protocols.

I can’t do the actual research, and I don’t know anything about crypto, which (I think) is the primary catalyst for discussion, but I love the premise that simple decision frameworks can be surprisingly influential across space and time.

My first thoughts went to metrics — are metrics a sort of protocol?

There are similarities: they are simple, accessible, and facilitate decentralized coordination. (I don’t care what you do, really, if it grows the business) Outside the company, metrics are used to make decisions about the company, with barely a reference to its internals.

Then, I wondered more generally, is “growth” a protocol?

Anyway, Summer of Protocols seems like an interesting conversation to follow along with, and I might engage these topics a bit more in the coming months.

Finally — let me introduce the Quality Koala, an advice animal you can use whenever you get that queasy “We need to do something about data quality, but we don’t know what, so let’s just do something” feeling.

It’s also great when you only know enough about a PR to give feedback on column naming, which basically summarizes my entire professional life.

The Quality Koala is brought to you by my six-year-old son. He will, with any luck, be drawing for me full-time soon.