The Data Platform Hangover

More data in the hands of users is a competitive advantage to grow faster and increase revenue. — Every Data Tool Startup CEO (yes, I have said this many times 👀)

More data in the hands of users is a competitive advantage to grow faster and increase revenue. — Every Data Tool Startup CEO (yes, I have said this many times 👀)

This is the story perpetuated by data professionals and startup founders, especially in recent years. Every new tool or product claims that there are HUGE bottlenecks in data, and these bottlenecks MUST be lifted (with their product, of course) because when you lift them, the floodgates of holy water from Mt Olympus will open, and you will be bathing in insights that will allow you to steamroll your competition and achieve exponential growth.

Yes, we have all heard it, and while there are certain aspects I agree with, as I have spent more time in this space, I realize that narrative is not entirely true. Yes, in the end, the goal is to have more data in the hands of stakeholders. Yes, products do break down those bottlenecks. However, if teams are not disciplined and do not apply the right governance and structure to how that data is delivered, no one is happy.

The reality is that data and the infrastructure that runs it are insanely messy and hard to work with. The data stack is extremely fragmented, with hyper-niche tools for specific workloads. For each use case, there is a best-in-breed tool that warrants integrations, more complexity, and another layer of logic being stored across various platforms, all of which want to own the entire stack one day.

What usually happens is what I call ‘accidental complexity.’ As teams scale and evolve, they add tools, logic gets written in multiple places, fragility increases with more failures across workloads, and costs rise. Before you know it, data platforms are 3x over budget, no one knows where the actual logic is stored, a data engineer leaves the company, and life gets complicated fast.

Most data people reading will think this ain't my story. “I document every model I write; I ensure all data is governed and cataloged, and I only share accurate and high-quality data with teams.” Well, I am not talking to you. I am talking to the other 99% of data teams. Those with three dashboards in Looker, two internal tools, four dbt models, and a bunch of Snowflake queries all compute the same logic and workload for slightly different reasons.

We repeatedly hear this message from teams. They admit they let the Genie out of the bottle, and getting it back in is hard or feels next to impossible.

So what are the options? The first is to purchase a fully integrated, opinionated solution to manage your data warehouse or lakehouse. The second is to do a manual audit and dedicate a team of engineers to do this manually over the next 6-9 months. The third option is a solution like Artemis that helps manage the fragmentation and reduce toil for data engineers. The nice thing about the latter approach is that it meets organizations where they are. You don’t need to move off your data warehouse, you don’t need to move off of dbt, and you don’t need to adopt some vertically integrated solution.

We built our platform to meet teams where they are. The platform surfaces proactive insights to your team and auto-resolves them. This means your engineers can focus on delivering value and not maintaining your stack. Artemis tells you what's wrong and then fixes it for you.

So, even though I gave a shoutout to our product, how has my perspective on the initial quote changed? The modern data stack promised value, and while it made life easier to work with data, it meant data platforms got out of control. More data will be a competitive advantage. But now, data platforms and systems are too mangled to get the desired value. How companies build and maintain data platforms needs to change so teams can work with efficient, lean, and effective data platforms that are not bloated. Once data platforms are simplified, teams can deliver the promised value, allowing companies to reach Mt Olympus.