The Evolution of Data

How Modular Systems Unlocked Success & A Whole New Wave of Issues

When it comes to data engineering, we’ve all seen the growing complexity of the data stack—whether it's new data sources, expanding analytics requirements, or keeping up with new tools. The modern data stack brought modular data stack they’ve been a game-changer for data teams, but not without it's rues.

Quick Recap: What Are Modular Systems?

The modern data stack turned end to end monoliths from Oracle and other providers into lego blocks. Instead of being stuck with one fixed pipeline, the modular stack let you break processes into smaller, manageable pieces (or modules). You can then mix and match these to create a workflow that works best for you and your data needs.

Over the past 10 years or so tools like dbt have popularized this approach, especially when it comes to managing data transformations. Instead of being locked into a complex, monolithic pipeline with scrappy SQL dbt’s modular techniques allow data engineers to work with more freedom, swapping in new models without breaking the whole thing. It’s like having a toolbox where you can just grab the right tool at the right time—no overhauling your entire system just because of a little change.

The Initial Promise

Modular data stacks were saving teams a ton of time. For example, in 2017, Siemens reported that using dbt reduced their dashboard maintenance time by 90%. Picture this—your team is handling a new data source, and instead of ripping apart your entire pipeline to fit it in, you just pop in a new model. Done. This kind of agility means your team can react quickly to new business requirements, new data sources, or just… you know, change without sinking weeks into reworking the whole system.

According to Gartner, and based on personal conversations with data professionals, this trend isn't slowing down. Gartner predicts that by 2025, over 75% of enterprise data engineering organizations will adopt these modern architectures. It's easy to understand why. The initial ROI is compelling, and the flexibility seems perfect for scaling operations. What could go wrong?

The Dark Side: Costs and Complexity Creep Up

But let’s not get too carried away with the "modularity is magic" idea of the mid 2010s—there are now obvious downsides that have become more prevalent as companies commit to modular tools.

On the surface, modularity seems perfect for growth—just keep adding more pieces, right? But as systems expanded, managing those individual modules have become a nightmare. Each module might need its own versioning, monitoring, and optimization, and suddenly data teams are spending just as much time maintaining the system as they are building it. According to Wakefield, data engineers spend up to 40% of their time on maintenance. It’s ironic, but modular systems actually introduced more technical debt.

Secondly, modular systems can drive up costs fast. Each new piece or service you add to your stack often comes with its own set of dependencies, resource requirements, and sometimes licensing or operational costs. It's easy to fall into the trap of adding one more tool here or another module there until you realize your bill is way higher than it would’ve been if you’d stuck to a more streamlined, less modular setup.

When you have multiple services or modules interacting, dependencies grow, and things can break in ways you wouldn’t expect. You might also find that tracking down issues gets harder

What This Means

In essence, while modularity offers flexibility and agility, it has also led to fragmentation, where different parts of the system are updated at different paces, causing friction. Managing the complexity of all these independent modules can be just as challenging as the limitations of a traditional, monolithic system—especially as your company grows.

What We Do At Artemis

Artemis gives data teams 500 hours back a year by maintaining and optimizing your stack for you. Our platform optimizes and maintains dbt models automatically, helping teams manage and improve documentation, warehouse performance, and overall system efficiency. By streamlining processes and reducing manual tasks, Artemis empowers data teams to focus on innovation, cut costs, and scale effectively without being bogged down by technical inefficiencies.