Fragmentation Hell

I had a call with a data engineer who talked about how the fragmentation in the data stack is crushing his team.

On Tuesday, I had a call with a data engineer who talked about how the fragmentation in the data stack is crushing his team. His team of six engineers use 13 different tools; the majority of them are point solutions with overlapping feature sets. Over the years, there has been a lack of consistency in how work is done (e.g., scheduling some jobs in Airflow and others with in-tool scheduling).

The platform's chaos is caused by the entirety of the platform, not by individual models or tools. So why is this a problem for teams, and what are the ripple effects?

Overhead

For starters, there's the obvious operational overhead. When different parts of your platform operate independently, you're essentially maintaining multiple mini-platforms rather than one cohesive system. Each requires monitoring, maintenance, and expertise. The expectation is to become experts in Snowflake, dbt, Airflow, Looker, and many other tools to execute your stack perfectly. This doesn’t happen, so over the years, little decisions have added up to create larger issues and cracks. This constant context switching means its overall tasks take longer.

Cross Workload Optimization

Since most aren’t experts, there's a hidden cost of missed optimization opportunities. When workloads run in isolation, you could have a metric in dbt already calculated elsewhere or run transformations on data that hasn't been updated since the last run. These issues multiply across your platform, which leads to massive increases in compute costs and inefficient processing times. The other problem is that error logs and alerts are siloed. An airflow job might fail which will trigger a dashboard to break, with every tool providing vague error logs, it can be difficult to find the real source of the problem.

How to fix it?

The solution isn't necessarily to force everything into a single, monolithic platform. Rather, as a data industry, we need to adopt a mentality of ‘doing more with less’. Since AI came along, data teams have been moved from R&D with large budgets to a cost center needing to bring ROI to the business. The benefit is that data platforms are the most important tool for companies to win in AI.

So, what is the solution? There are several. One key is to develop a platform-wide understanding of how your various tools interact and affect each other. The goal isn’t eliminating all complexity—it's making it manageable through better visibility and understanding.

In a world where data platforms are becoming increasingly complex and distributed, this holistic understanding isn't just nice to have—it's essential. For 2025, we are seeing a trend toward building systems that can scale efficiently and cost-effectively. The alternative is continuing to optimize locally while missing the global picture.

Artemis Comes To You

We work with teams on various stacks; the best part is that it meets organizations where they are. We consolidate all your metadata and logs into one place and then surface insights into what is broken or improved. This allows you to focus more on delivering value rather than fixing your stack.

“That’s what I like about Artemis. You don’t need to move off your data warehouse, you don’t need to move off of dbt, and you don’t need to adopt some vertically integrated solution. We’ve dug ourselves into a pretty big hole with the MDS tech stack, and tools like Artemis will play a role in helping us wrangle all this stuff.”
— Chris Riccomini — Contributor of Apache Samza and Airflow

Our users, on average, resolve 120 insights a week, merge 60+ PRs and save 20 hours! If you want to simplify your stack and get dbt under control, reach out!