Clutter is the cost of cheap
100% of data catalog initiatives fail if you live long enough
Imagine a therapist’s office: a mahogany desk, a ficus in the corner, a packed bookshelf, a polished desk lamp presides over a rarely opened folio. A pleated leather sofa faces the desk, anchoring the space. The room is clean, cozy, and ordered. Its purpose is clear. The room’s goal is a conversation between patient and professional, and everything works towards that end.
Now add fifteen folding chairs scattered around the office. Perhaps the therapist’s students were observing a case, or a disgruntled assistant pulled a prank. Possibly, the therapist was “just testing a few things out real quick.”
Regardless, the office is now cluttered. Nothing has been taken from it, but the room now creates a sense of unease and overwhelm for the patient. It doesn’t prevent conversation exactly, but it does makes it uncomfortable, especially at the start, especially for new patients.
A question hangs in the air: What is really going on here?
In the data world, there’s a common failure mode where data “stuff” accumulates until it becomes a comically unnavigable mess. Pick your favorite metaphor: data swamps, dashboard graveyards, spaghetti DAGs. Last week, I thought through this a bit by comparing tables to furniture: intrinsically purposeful objects that can nonetheless lead to purposeless spaces.
While the problem is exaggerated in the data landscape, it’s not unique to it. Google Docs, Slack channels, JIRA boards, wikis, security groups, and code repositories are as full of once-useful artifacts as the downstairs storage closet. The problem is endemic to the digital artifacts — “content” — we eat and breathe now.
This is the problem of the Internet, in fact. Everything accessible via the network exists until it’s destroyed, forming layers and layers of content sediment. And we’ve only had thirty years to inhabit the Internet; what will it look like in a hundred?
For the most part, we’ve adapted to it—or rather, it has adapted to us. Companies spend massive resources on hiding content, so that we can feel comfortable, so we don’t think of all the things we’re missing. Unlike in, say, a library, on the Internet, we must browse and snack off the top because we can’t navigate to anything deeper. There’s just too much too much-ness out there.
This is the problem of clutter.
At the organizational level, clutter is costly. Over-stuffed digital spaces take up disk space and consume compute, both of which are cheap but not free. The biggest problem, though, is confusion. Uncertain employees are unproductive employees.
An analogy may help here.
In startups, process is seen as a necessary evil. Leaders want as much as necessary but no more. Too much process leads to bureaucracy, a nightmare that’s only acceptable when accompanied by the dream of enterprise scale. Individually, processes are not evil, but when formalized, they create systematic friction to acting for no discernible reason than the process itself. Bureaucratic processes become ends unto themselves.
Objects, assets, furniture — stuff — are also necessary byproducts of doing work. These, too, accumulate over time. These, too, can become self-perpetuating ends unto themselves. In the same way a bureaucracy stifles agility, clutter clouds clarity.
Bureaucracy is a builder-side problem, while clutter affects the consumer side. Writing documentation, for example, is full of optionality for the author. They can write in a wiki, in Slack, in code comments, or in a document. There’s no intrinsically wrong answer. The reader, though, has only one right answer: wherever the author wrote the docs. Go to the wrong place, and they waste time. Worse, they may read the wrong docs.
In a cluttered environment, employees can never be quite sure which document someone is referencing, what Slack channel to join, or which JIRA ticket is the actual ticket. People feel lost—like they know things are happening but only see one piece of the picture.
Humans want to do the right thing, to be informed. They enjoy jeering things that are clearly wrong, celebrating things that are clearly great, and jabbing about things that are clearly interesting. All speakers are given the same advice: tell ‘em what you’re going to say, what you’re saying, and what you told ‘em. People want a clear message. Organizations should want to send one.
Newspapers mastered this centuries ago when technical constraints made prioritization critical. That pressure has been lost that in an age of cheap storage and immediate delivery, but the human need for clarity hasn’t.
There are certainly ways of combatting clutter, ranging from the executive, to the technical, to the personal. No single one is necessary or sufficient, but together they can make headway against the problem.
The first is transparency. Leaders can send clear and regular communication from the top which cuts through the clutter. Everyone can see exactly what conversations, information, and decks the leadership team is using and can update their priorities accordingly.
While this is a great way to focus attention globally, it runs into issues at local levels. Transparent communication strongly emphasizes the core responsibilities of the business (say, board metrics) while neglecting those areas that are “below the fold” (to reuse the newspaper analogy).
A second approach is expiration, allowing items to expire over time. This leverages a manual verification process to prevent deletion or archival. While this is scalable in theory, it’s heavily reliant on good citizens being good curators, which is not, in my experience, a safe assumption. There’s too much to do to prioritize cleaning your room, unless dad is threatening to ground you.
The techno-optimist’s approach is to push clutter below the API, and treat it like a search problem. In essence, treating the Intranet like the Internet. If you can reliably get the needle, the thinking goes, the haystack isn’t a problem. And with LLMs and RAG, there might be a future here.
Today, though, this has issues in practice. At Internet scales, enough user data is generated to reveal a useful information architecture. Organization scale is smaller and changes more abruptly. If a team updates a reference — “We’re using ‘q2_metrics_v2_final.xlsx’ now.” — this needs to propagate immediately and permanently, but it doesn’t. Artifacts are continually forked and merged in a way that no catalog can capture effectively — yet.
In some cases, all of these approaches come together. Amazon’s “S Team” meetings, for example, are a verification process that revolves around an artifact—the metrics deck. By focusing attention on this one artifact, anyone who views it walks away with clarity of what’s discussed and can trace back from there the information streams that drive the business. The artifact isn’t relevant after the meeting, but it becomes a part of the decision-making history of the business. It’s still a trail of digital objects, but they’re organized and sequenced rather than littered about haphazardly.
Ultimately, though, there’s no solution, because artifacts are not a problem so much as a consequence of digital life. Like process, clutter can only be managed and mitigated where and when it becomes most harmful.
It can, though, be helpful to organize it into rooms. Big monolithic applications are full of clutter, but this doesn’t reach out and infect other systems or other spaces. It’s contained. Other services can interface with it cleanly.
Domain-driven design is the most powerful aspect of the data mesh philosophy, in my opinion. It’s powerful, this idea that digital baggage can be stuffed into a “domain” and kept from spilling out into the rest of the org. If an architect draws the right lines, she can clarify purpose. If the purpose is clear, she can create accountability. If accountability is in place, she can enforce standards.
The challenge is that, unlike in physical spaces, the walls aren’t static. Services get entangled. Leaders leave. Reorgs happen. Someone unfolds fifteen chairs and leaves the room. Lines get blurry. What you thought was a cafeteria turns into a medical shelter. Clutter accumulates, and the only thing left to do is kick open the door into space and let it all be sucked into the vast, empty, unindexed tar.gz files orbiting Earth.