I believe the answer to that question is no.

Will AI fundamentally change data engineering work, and how will it be done? Yes.

The Context

I spoke with the CTO of a unicorn data startup, who said, “We are really good at gathering data, but we are not the best at knowing what to do with it.” Benn Stancil hints at this weekly in his blog, which is both sarcastically realistic and pessimistic. Based on his posts, he would agree that we are great at gathering data, not creating insights.

We all saw the promise of Moneyball and wanted to re-enact the concept of “winning the game” through data within our businesses to reap massive rewards. I started my data journey analyzing alternative data in finance, where my job was to find Moneyball-type insights for alpha-generating investment decisions. I was naive to think every data organization operated like this.

The first step in that journey was to capture every data point. This thinking aligned with "what you can't measure, you can't manage" (fun fact: this famous quote is misattributed and was part of a longer quote which states the opposite). The belief was that small teams could quantify everything through sheer force of will to find that needle in the haystack. Data was supposed to be the deciding factor between becoming the next Netflix or Blockbuster.

Data tools profited immensely from automating the nuts-and-bolts process of finding alpha, eliminating the complex technical skills of working with and manipulating data points and turning it into a seamless and automated system. Redshift may have accelerated this in the cloud, but billions of dollars have been created around this idea. I mean, Databricks just raised $10 billion based on this concept.

However, working with data is still incredibly manual, even these years later. Data platforms are fragile and nuanced and are only becoming more vital to teams (check out why I think the data platform is the most important internal tool).

What is the ultimate destination for data practitioners if we aim to enable everyone in our organizations to work with data effortlessly through automation and AI? And in the future, what role will we data engineers play?

We Are Becoming Pilots

Data engineers are trending to have a similar role to commercial airline pilots. Like pilots, data engineers will manage complex systems (i.e. data platforms) that automatically handle hundreds of tasks. As the pilot of this data platform, you’ll be required to have the technical chops to deal with any issues that pop up while also being responsible for ensuring the platform and insights get to their destination.

We already see this transition with orchestrators taking on more complex work; however, AI will enable tools to go two or three layers deeper. It will fix pipelines on the fly, optimize workloads, and even provide analysis for end users. Your job is to manage the work and ensure the work is moving smoothly, in the correct order, and to the right people.

On top of this, once these systems are set up, you won’t need a team of 20 engineers. You’ll need two or three. They will all understand the platform and ensure that it is still performing at its optimal rate.

How do we get there?

Tools like Artemis help you experience this future by isolating issues within your stack, prioritizing what needs to be solved, and surfacing insights. Once you approve an issue, our AI agents will resolve it. Artemis is the automated mechanic in your plane, ensuring you move as necessary.

A ton of innovation is still needed around the data stack. Automating the flow and cohesion of these systems is still tricky, but there are many opportunities to solve this.

The exciting part about this future is that with autopilot, you will have time to work on the Moneyball questions of the world. You have collected a lot of data. It's time to see what it says.

Where will you and your platform fly?

A familiar picture is drawn whenever I talk with data engineers or managers. They talk about how they have multiple BI tools, years of accumulated dbt models, and warehouse costs steadily climbing to reflect the bloat. What strikes me is how this happens so predictably across organizations. As data teams grow and evolve, they often end up with:

• Multiple BI tools serving different purposes.

• Years of dbt models written by different team members, often solving similar problems differently.

• A steady stream of ad-hoc queries and analysis, some of which could be consolidated into proper data models.

• Significant compute costs which are not necessary

I posted about this about a month ago, and the typical response was that this could be solved with the proper foundation and process.

While there is some truth in those statements, why does this still happen to almost every team that has scaled a data platform? How do data platforms become so messy in the first place?

After thinking about this for a while, the answer, while annoyingly so, could be that it is human nature. Looking beyond data engineering to other fields like software development and project management—or even our own homes—we see how clutter, mess, and disorganization naturally accumulate over time.

A large portion is because it takes focus and energy to stay on top of the mess. It’s also not fun work. It is not fun taking the trash out, just as it’s not fun combing through a model someone else wrote to find bugs and fixes. The benefit (and curse) is unlike in real life; the trash can continue to pile up on our data platforms without many people noticing.

As people, we almost always take the path of least resistance. We don’t finish the documentation because it's annoying. We don’t spend the time to see if a model already accomplishes what we need, so instead, we build a new one. This message has also been broadcast across the entire modern data stack. The answer for the longest time has been to add. Add another model, buy another solution, hire more analytics engineers.

So, sure, having a good model foundation, principles, and processes helps reduce or alleviate the chaos within a data platform. However, tech debt and bloat are going to happen.

Now that we know this, how can data teams stay on top of these issues and reduce them over time? This is a question a lot of data teams are asking themselves. From the teams we have spoken to, there are a few outcomes.

1. You can do nothing and hope it’ll figure itself out one day. (Spoiler: it won’t).

2. You commission a team of 1 or 2 engineers to focus on this project. They spend 6-9 months on it and are pulled from other projects.

3. They do the opposite and commission 1-2 to focus on new work while the rest of the team is set on fixing their foundation. This is a costly outcome.

4. They use Artemis.

Teams need a maniacal focus on maintaining a simple yet effective data platform. They must be diligent in their work to ensure the platform scales appropriately. This is the hard part of the modern data stack, and with the new trends around cost cutting, budget realignment, and vendors being cu, this complexity issue is brought to the forefront.

However, the issue for many teams is that they are being asked to trim a platform they did not build themselves while fulfilling all the work they had to do regardless. This lack of context makes fixing issues and cutting tech debt extremely challenging. You need either a person to do this or a tool to monitor your platform and surface issues and resolve them.

The reality is that this isn't about disorganization—it's about the natural evolution of data teams and platforms. As teams scale, "accidental complexity" appears, where solving immediate business needs gradually creates a more complicated system than anyone intended.

If your platform is bloated, reach out!

About Artemis

Artemis monitors your data stack, finds issues, and automatically resolves them. Our users approve over 120 insights, merge 60+ PRs, and save over 20 hours a week. There is no need to migrate; our platform integrates with your data stack within 15 minutes!

At the end of last year, I asked one of our customers, “What was your aha moment when using Artemis?”

Without skipping a beat, he said, “The moment I combed through the insights and realized I no longer needed to keep all this context in the back of my mind, Artemis does the research and then surfaces issues for me.”

This was not what I was expecting. I thought he would highlight all the cool AI agent tech we had built that takes the insights we surface and then resolves the tasks for him. Instead, the value that got him was that we researched for him.

In this context, research related to the work needed to understand what problems were occurring in his data platform, why they were happening, and what a suitable fix was to resolve those issues.

This pain of research is deep, and the more we dig into it, it is a huge cost driver for data teams. A dbt labs survey in 2024 found that data teams spend 26% of their week fixing and maintaining their data platform. This is in addition to the fact that engineers spend 55% of their time maintaining or organizing data sets; a combined 80% of their time is spent on some form of maintenance.

If teams spend 80% of their week on maintenance, a considerable amount of that is spent on research. Why are teams spending all this time on research?

A Few Possible Reasons for So Much Research

• You didn’t write it: Most practitioners do not build the platform they hold the keys to. Data engineers spend their time understanding, reverse engineering, and dissecting the work of others to fix, maintain, and improve data platforms.

• Fragmentation means no one is a master: When building and maintaining a platform of 8-12 tools, all of which have quirks and ways of becoming efficient, there is much to learn. As an engineer, you won’t know every tool and won’t have the time to understand the intricacies of each. So, you learn enough to get by and take little refresher courses when you need to solve a specific problem.

This convoluted web of work leads to tech debt, but it's also the workspace most data practitioners work in daily.

The other side of this is the mental drain on you. We like the path of least resistance, and it’s discouraging when we see a ticket that should take 15 minutes to resolve to take three or more hours due to the added complexity and research needed.

The emotional side of this problem is deep-rooted. It can crush momentum and motivation for a team performing at a high level and seriously derail progress.

Research is necessary for solving problems, and data is no different. However, in the long run, will we continue to do the research ourselves as practitioners, or will AI do it for us?

I spoke with the CTO of a unicorn data startup, who said, “We are really good at gathering data, but we are not the best at knowing what to do with it.”

This observation rings true for data platforms. After speaking with over 200 data teams, it became clear that scaling platforms face a common challenge: cutting through the noise to prioritize work that ensures stability and optimal costs is incredibly difficult, whether it's untangling complex dbt models to fix a schema change or determining which of the 15 Airflow pipelines needs immediate attention.

When it comes to enterprise, it's more like thousands of Airflow pipelines a day. This brittleness is only further driven by the fact that teams are stuck in a reactive state. It is really hard to future-proof your data stack.

The first step teams take to solve this problem is to purchase a data observability tool. At least you know what is happening in your stack. At first, you pour time into setting up all the alerts, funnels and tables to monitor, and it's fantastic. However, what typically happens as you scale your dbt models and warehouse spending?

1. Notification Overload: Your Slack channel gets spammed, and you stop responding or caring as false positives appear.

2. Lack of Context: You notice the alerts are coming in without context and provide no information on why a problem has occurred, how to fix it, or how it impacts other parts of your stack.

3. Capacity Shortage: As you get more alerts, your backlog expands, your stress heightens, and you feel that you can't solve all the problems you have. Each alert means you will get distracted and spend 3 to 4 hours figuring out what went wrong and solving the issue, taking you away from delivering new objectives.

The current rigid structure of observability tools doesn't allow for custom context, flexibility, and real value to be generated. Because teams have either struggled to implement them or implemented and experienced the pain points above, we hear this a lot:

"dbt is out of control."

"We don’t know why our Airflow jobs that keep failing”

"We are swamped with new objectives, so we can't dedicate time to refactoring our current setup."

These are all reactive responses to a challenging problem for teams. How do you know if something will break until it breaks? Often times in in the architecture, or the fact you have years of different engineers doing different things.

This is why we are building Artemis. We are building the next generation of observability, context-relevant insights which drive automated resolution. We help you turn your dbt mess into a manageable model layer. We accomplish this by turning your platform from reactive to proactive.

1. Surface Tasks, not alerts: The insights we surface leverage context from your stack to give you a task to solve, leading you to a solution faster, not creating more work.

2. Personalized Context: Our personalized knowledge graph provides deep context and root cause analysis to explain the issues, what caused them, how to fix them, and their impacts on downstream or upstream models and tables.

3. Auto-Resolve: Our action engine takes your tasks, makes the changes, updates pipelines, and refactors your models for you. All you have to do is review the PR and push it to production! You can be as little or as much involved as you want. Interested in setting and forgetting? You can do that! Want to be in the nitty gritty? That's also not a problem. You are always in control.

We give you total control, context, and solutions, so your Slack channel isn't annoying, and Jira doesn’t get bloated; it shows your team and managers how much you are getting done.

On Tuesday, I had a call with a data engineer who talked about how the fragmentation in the data stack is crushing his team. His team of six engineers use 13 different tools; the majority of them are point solutions with overlapping feature sets. Over the years, there has been a lack of consistency in how work is done (e.g., scheduling some jobs in Airflow and others with in-tool scheduling).

The platform's chaos is caused by the entirety of the platform, not by individual models or tools. So why is this a problem for teams, and what are the ripple effects?

Overhead

For starters, there's the obvious operational overhead. When different parts of your platform operate independently, you're essentially maintaining multiple mini-platforms rather than one cohesive system. Each requires monitoring, maintenance, and expertise. The expectation is to become experts in Snowflake, dbt, Airflow, Looker, and many other tools to execute your stack perfectly. This doesn’t happen, so over the years, little decisions have added up to create larger issues and cracks. This constant context switching means its overall tasks take longer.

Cross Workload Optimization

Since most aren’t experts, there's a hidden cost of missed optimization opportunities. When workloads run in isolation, you could have a metric in dbt already calculated elsewhere or run transformations on data that hasn't been updated since the last run. These issues multiply across your platform, which leads to massive increases in compute costs and inefficient processing times. The other problem is that error logs and alerts are siloed. An airflow job might fail which will trigger a dashboard to break, with every tool providing vague error logs, it can be difficult to find the real source of the problem.

How to fix it?

The solution isn't necessarily to force everything into a single, monolithic platform. Rather, as a data industry, we need to adopt a mentality of ‘doing more with less’. Since AI came along, data teams have been moved from R&D with large budgets to a cost center needing to bring ROI to the business. The benefit is that data platforms are the most important tool for companies to win in AI.

So, what is the solution? There are several. One key is to develop a platform-wide understanding of how your various tools interact and affect each other. The goal isn’t eliminating all complexity—it's making it manageable through better visibility and understanding.

In a world where data platforms are becoming increasingly complex and distributed, this holistic understanding isn't just nice to have—it's essential. For 2025, we are seeing a trend toward building systems that can scale efficiently and cost-effectively. The alternative is continuing to optimize locally while missing the global picture.

Artemis Comes To You

We work with teams on various stacks; the best part is that it meets organizations where they are. We consolidate all your metadata and logs into one place and then surface insights into what is broken or improved. This allows you to focus more on delivering value rather than fixing your stack.

“That’s what I like about Artemis. You don’t need to move off your data warehouse, you don’t need to move off of dbt, and you don’t need to adopt some vertically integrated solution. We’ve dug ourselves into a pretty big hole with the MDS tech stack, and tools like Artemis will play a role in helping us wrangle all this stuff.”
— Chris Riccomini — Contributor of Apache Samza and Airflow

Our users, on average, resolve 120 insights a week, merge 60+ PRs and save 20 hours! If you want to simplify your stack and get dbt under control, reach out!

I didn’t realize the amount of customer data was so high. Additionally, the statistic that 90% of 300 companies have built internal CRM tools makes a lot of sense. The first project a lot of data teams have is financial reporting, which creates more insight into sales and CRM. Use Fivetran and dump all 800 of those Salesforce tables into Snowflake to get an accurate picture of what is truly going on in your business. The question I then ask myself is, does this mean the Modern Data Stack won? I thought it was dead. While it was a marketing term which inspired a ‘movement,’ a great test to determine if it is valuable is if the system of record is moving to the data warehouse. This shift makes the case that the data platform will be the most critical tool for companies to win in AI.

As teams go to build AI products and internal tools, they want and need a centralized data source. The glorified one source of truth that we are all still trying to find. The issue with most AI tools that we purchase today is that they are siloed. They can only automate or increase value on the small point solution workflow they own. This is why SaaS will struggle; it fought hard to get your data, but with only a small sliver of the workflow, that data can only help so much. A Hubspot copilot is locked to Hubspot, a dbt copilot is locked to dbt and a text-to-SQL tool in Snowflake is locked to that ecosystem. With every data platform wanting to own the entire stack, those tools will stay that way.

Where does this lead us? It means that teams need to centralize their tools into the warehouse. The incentive to build a well-governed, holistic data platform to leverage AI has never been higher. Centralize, clean, and use all that data to empower the tools and people to move mountains. This is where data teams can bring huge ROI to businesses. This is what makes the data platform the most important internal tool. Those who can deliver on this vision will have a competitive advantage. Mixing external data with clean and trustworthy internal datasets will create billions of dollars in opportunities for teams.

This is also true for the tools across your data stack, which are fragmented. A large tailwind we see with Artemis is that data teams are hungry for tools that work across the stack. With Artemis, you centralize the metadata and error logs siloed across individual tools and combine them into one tool to automate workflows across the stack. We don’t look at your tools in isolation. We take a platform approach.

When teams were built with the MDS, no guard rails were built. The amount of bloat is staggering. We speak with teams with 500+ data models; in reality, they only need 60. The average team we speak with spends 1.5- 3 times their warehouse budget. This bloat not only makes platforms more expensive but also brings complexity. This complexity leads to teams not understanding what their data is doing and how it impacts the organization. It means no one trusts the data. Our motto is simple is best. The less you can do, the better. Clearing out the junk and getting data platforms in shape should be step 1 in using AI.

We work with a lot of data teams that use BigQuery, and we've noticed that most teams only use 40-60% of the compute they pay for due to misusing BigQuery slots. Sometimes, there is a business use case for overspending, but teams are often unaware of what they are spending.

What is a Slot?

BigQuery slots are virtual CPUs that power your SQL queries by providing compute resources. When you run a query, BigQuery calculates how many slots you need based on its size and complexity.

So what happens? When you run a query in BigQuery, the system first analyzes your query and creates a plan. Then, it assigns slots based on the query's complexity, available resources, and priorities. These slots work together to process your query efficiently.

Contention

There are a few edge cases when you can have slot contention, which occurs when more queries are waiting to be executed than available slots. This typically happens when too many queries run simultaneously, a particularly heavy query needs extra horsepower, or everyone's trying to run reports simultaneously. BigQuery will automatically react to this and change the number of slots based on fair scheduling. What’s fair scheduling?

Fair scheduling

Fair scheduling is the mechanism BigQuery uses to distribute slots among multiple queries and projects. Essentially, it ensures that all queries have equal access to available slots, regardless of the project or user. When a query is run, it is placed in a queue. The scheduler then fairly allocates slots to the queries in the queue. It ensures no queries are hogging resources. While this means queries are treated the same, if the system is overloaded, some queries can get paused or prioritize the larger queries in the queue, impacting performance, a tradeoff teams must wrestle with.

What goes wrong?

So now you have a sense of how they work, how do teams misuse them? In most cases, teams are unaware of over-provisioning slots and utilize more than they need. Slots also go hand in hand with poor query design. If a query has a lot of joins, doesn’t filter the data early in the query, or has a whole host of other minor issues, the query will take more resources and require more slots.

So, to start, teams first need to examine their queries. Whether it be to split your tables into smaller chunks, filter early to reduce unnecessary work, or keep your joins efficient, all of this impacts your query performance, which affects your slot utilization. Once you can fix and improve your queries, you can start to work on better scheduling. Syncing larger jobs at times of the day when teams use BI tools, dedicated resources for specific teams or workloads, or keeping an eye on when and what is using the most resources will help you use less while moving faster.

Final thoughts

While you might think that adding more slots will mean more queries get executed faster, it can cover significant performance issues. Like I said at the beginning, most teams we work with who use BigQuery don’t utilize 1/3rd of the compute they pay for. That is a ton of money!

Getting the most out of BigQuery means understanding how to use your slots wisely while balancing performance and cost.

Don't want to deal with the hassle? Our platform has a simulation model that can accurately predict the most optimal slot allocation for you. Let Artemis handle the heavy lifting - we'll optimize your slots automatically so you can focus on what matters.

Not too long ago, the Modern Data Stack (MDS) was hailed as a game-changer in the data engineering world. Its modular approach—featuring tools like Fivetran for ingestion, dbt for transformations, Snowflake for warehousing, and reverse ETL solutions such as Hightouch—promised a revolution. It offered flexibility, scalability, and a chance for organizations to democratize data. But today, the buzz has sorta died down. So, what happened?

From Game-Changer to Table Stakes

The MDS quickly became the go-to architecture for many companies, and in doing so, it transitioned from being modern to simply expected. What was once revolutionary became routine. Tools like Snowflake and dbt are now common place, and many of the stack’s principles have been internalized across the industry.

Yet, alongside its success came its (many) pitfalls, here are just some:

1. Complexity: The "six-vendor problem" emerged, where organizations needed to integrate have a dozen tools from separate vendors. (We will talk about my bet that the pendulum will start swinging in a bit)

• Cost: At the beginning it was a bit cool to brag about your fancy tool stack, but now people know that they each came with a fat price tag as well.

• Operational Overhead: Managing multiple contracts, SLAs, and integrations introduced friction that undercut the promised efficiency of the MDS.

The Buzzword Problem

Tristan Handy and Benn Stancil, in their dbt Roundup episode on the MDS, argue that the term "Modern Data Stack" was always more of a marketing construct than a clearly defined standard. It promised the allure of best-in-class modularity but often left organizations struggling to piece together a fragmented ecosystem.

Stancil succinctly summarized the problem: "The Modern Data Stack wasn’t a product—it was a collection of vendors selling different pieces of the puzzle." And while modularity was a strength, it became a liability when too many tools needed to be orchestrated together.

The Pendulum is Swinging: The Shift Towards Consolidation

A key development in the MDS’s evolution is platform consolidation. Organizations began moving away from assembling their stacks tool-by-tool, opting instead for single platforms that could manage multiple layers of the pipeline:

• Snowflake: Once primarily a warehouse, Snowflake now offers tools for app-building and more integrated transformation capabilities.

• Databricks: Initially focused on machine learning, it has expanded into data warehousing with Databricks SQL, offering an all-in-one lakehouse solution.

These shifts address the core weaknesses of the MDS—cost, complexity, and interoperability—by simplifying vendor relationships and unifying tooling.

The AI Effect

The rise of AI has also stolen the spotlight. Companies are now focused on LLMs, GenAI, and AI-native data architectures that promise insights far beyond traditional analytics. While the MDS once captured the zeitgeist of innovation, it has been overshadowed by this next wave of excitement.

In Handy’s view, "The conversation has shifted." The energy that once fuelled the MDS is now powering AI-driven architectures.

Before we get too deep into AI let’s take a look at some lessons learned from the MDS…

1. "Modern" Is a Moving Target: The term modern dates pretty poorly. Tools that feel cutting-edge today will probably be legacy tomorrow.

2. Integration Fatigue Is Real: Too many vendors and tools create operational friction, even if each tool excels individually.

3. Adaptability Wins: Platforms that evolve to simplify workflows—offering consolidated, efficiency and end-to-end solutions—tend to thrive in the long term.

4. Culture Matters: No matter how modern your stack is, you still need modern data culture.

So, Is the Modern Data Stack Dead?

Not quite—but it’s definitely evolved. The Modern Data Stack isn’t a failure; it’s a stepping stone. It’s taught the industry valuable lessons about modularity, scalability, and the importance of flexibility. Today, many organizations still use MDS-inspired architectures, albeit with fewer tools and more reliance on integrated platforms.

The data landscape will continue to change, whether through the rise of AI-native stacks, further platform consolidation, or entirely new paradigms. But the spirit of the Modern Data Stack—solving hard problems with creative, scalable solutions—remains alive and well.

So, what happened to the MDS? It matured, and in doing so, it paved the way for whatever comes next. Whether that’s called the "Next-Gen Data Stack" or something else, one thing is clear: data engineering isn’t slowing down anytime soon.

If you've ever found yourself drowning in urgent requests, constantly jumping from one crisis to another, you're experiencing the all-too-familiar world of reactive data engineering. But what if there's a better way?

The Reactive Reality: Trapped in the Moment

Let me paint a picture that might sound uncomfortably familiar. Your typical day as a data engineer looks like a never-ending game of whack-a-mole. Urgent requests flood your inbox. Dashboards break mysteriously. Stakeholders demand immediate fixes and you're responsive – maybe too responsive.

A reactive data team is characterized by constant firefighting:

• Responding to problems after they've already occurred

• Spending more time fixing than preventing

• Feeling like you're always one step behind

• Struggling to focus on strategic initiatives

Why Reactivity Becomes a Trap

Why do we find ourselves perpetually reacting instead of proactively planning? The root causes are both systemic and cultural:

• Lack of comprehensive data literacy

• Insufficient strategic planning

• Organizational structures that prioritize immediate fixes over long-term solutions

• A culture that celebrates heroic problem-solving rather than preventative thinking

The most insidious consequence? Your team is no longer seen as a strategic partner. Instead, you're viewed as a support function – always available, always fixing, but never truly driving innovation.

The Proactive Paradigm: A New Approach

Proactive data management is about anticipation, not just reaction. It's the difference between treating symptoms and preventing the disease.

A Concrete Example: Imagine two scenarios for a user experience tracking project:

Reactive Approach:

• Discover a usability issue after customer complaints

• Spend weeks investigating the problem

• Develop a retroactive fix

• Apologize to affected users

Proactive Approach:

• Continuously monitor user interaction patterns

• Develop predictive models that identify potential friction points

• Create preemptive improvements

• Enhance user experience before problems arise

The proactive method doesn't just solve problems – it prevents them from happening in the first place.

Changing the Status Quo: A Roadmap to Proactivity

Becoming a proactive data team requires a fundamental business shift:

1. Become Outcome-Focused

   • Align data initiatives directly with business objectives

   • Create measurable, forward-looking goals

2. Embrace Data-Informed Decision Making

   • Develop predictive analytics capabilities

   • Look beyond historical data to future possibilities

3. Increase Data Literacy

   • Educate stakeholders about data's strategic potential

   • Create clear, understandable data narratives

4. Document and Share

   • Maintain comprehensive project documentation

   • Create knowledge repositories that enable continuous learning

5. Be a Strategic Collaborator

   • Proactively suggest improvements

   • Position your team as innovation partners, not just service providers

The Time Constraint Dilemma

But who has time for this?

We get it, the reason the firefighting happens in the first place is because teams are stretched thin. But, this is where solutions like Artemis become a game-changer. By automating maintenance and optimization tasks, Artemis gives data teams a critical gift: time.

Artemis: Transforming Time into Innovation

Imagine reclaiming 500 hours annually. That's what Artemis offers by:

• Automatically maintaining and optimizing dbt models

• Streamlining documentation processes

• Improving warehouse performance

• Reducing manual technical inefficiencies

The result? Your team can shift from surviving to thriving, focusing on innovation rather than endless troubleshooting.

The Future of Data Engineering

The most valuable data engineers won't be those who are fastest at putting out fires, but those who prevent fires from starting.

By embracing proactive strategies and leveraging intelligent automation, you're not just improving efficiency – you're reimagining the entire role of data engineering.

Are you ready to step out of the emergency room and into the innovation lab?

The future is proactive. And it starts with you.

Costs of transformation workloads do add up quickly. Data teams consistently tell us that dbt-orchestrated workloads drive the vast majority of their data warehouse spend.

This is a quote from an article I wrote with Tino. There are a lot of reasons why Snowflake, Fivetran, and dbt are worth billions, but the explosion of SaaS is a big one. The amount of data with different schemas and tools used within organizations forces teams to transform hundreds of datasets. As we shifted from ETL to ELT, transformation workloads have become the main driver of data warehouse costs and Fivetran has the data to back it up. So how does this happen?

The Model Bloat

When teams rolled out dbt core in their organizations, it was fantastic. They finally had a central place to write transformations, version control and more. As Chris Riccomini referenced in his interview, a lot of large organizations have built a simple version of dbt internally. So when dbt labs came knocking, it made sense to adopt. However, slowly but surely, over the years, as teams added contributors, wrote more models and used their data platform, the dbt model sprawl became unmanageable. It's a familiar story that we hear all the time. “We started with 30 dbt models, and now we have over 400.” For what? Sometimes, there are little mistakes, configurations set incorrectly, or dated logic that has yet to be updated. Sometimes, it’s because the original architect of the models left and the new one has a new way of building.

It's important to remember this is happening across the stack. This post from Ryan Janssen, while funny, is quite accurate. The MDS was incredible at lowering the barrier to working with data; however, with that, we lost a lot of fundamentals, and teams were not disciplined in their approach.

The Alert Crisis

Alongside the bloat issue, data observability tools constantly ping you when things break down. The more junk in the system, the more likely it is to fail. Most of these alerts aren't helpful for two key reasons. First, they lack context—you only get the issue without information about why it occurred, how to fix it, or how it affects other parts of your stack. Second, as alerts pile up, your backlog grows, and stress mounts until you feel overwhelmed by unsolvable problems. Each alert disrupts your work, forcing you to spend 3 to 4 hours diagnosing and fixing issues instead of focusing on your primary tasks. While monitoring stack issues is important, it's time for tools that detect problems and automatically resolve them, allowing teams to focus on strategic initiatives rather than constant optimization and maintenance.

Insights & Resolution

Engineers should focus on building solutions that help businesses grow and scale—not on optimizing tools like Snowflake, dbt, or Airflow. Yet data teams today fall into two categories: those spending over a quarter of their week on maintenance tasks with limited context and those avoiding maintenance altogether. The latter group burns through their budget, accumulates technical debt, and moves at a snail’s pace.

Teams crave insight and visibility into what's going wrong in their data platforms. The exciting part is that uncovering these issues gives us the fantastic opportunity to automate the work in one experience. Insights lead to tasks, which lead to tasks resolved automatically, which leads to a merged PR. These automated workflows aren't built on rigid rules—they're driven by customized insights from your environment! This means less time on maintenance and more time spent on work, which makes an impact.

This is the world we are building. Come join us on the ride!

About Artemis

Artemis monitors your data stack, finds issues, and automatically resolves them. Our users approve over 120 insights, merge 60+ PRs, and save over 20 hours a week. There is no need to migrate; our platform integrates with your data stack within 15 minutes!

In 2006 when British mathematician Clive Humby declared that "data is the new oil", he illuminated a fundamental truth that many organizations are still grappling with: raw data, like crude oil, requires sophisticated (meaning costly and expensive) refinement to deliver rule value. Fast forward to today: the global “datasphere” is projected to grow from 33 zettabytes in 2018 to 175 zettabytes by 2025, according to IDC. Yet despite this explosive growth, 68% of data goes unused in most organizations.

One zettabyte is equal to a trillion gigabytes.

This underutilization isn’t for lack of effort. Organizations have poured resources into building sprawling data lakes, warehouses and pipelines, but without the context to make sense of it all, most data remains an untapped resource, languishing in obscurity.

The Challenge: Contextless Data in the Real World

For data engineers, the sheer volume is less daunting than the complexity of working with contextless data. Imagine constructing a skyscraper without a blueprint—just piles of bricks and steel. That’s the daily reality of working with disorganized datasets lacking proper documentation, lineage, or structure.

This lack of context manifests in several ways:

1. Query Optimization Challenges

• Query performance suffers when relationships between tables are unclear.

• Data engineers spend an average of 40% of their time on non-engineering work like deciphering poorly documented datasets.

• Complex JOIN operations often fail or run inefficiently due to misunderstood table relationships.

2. Schema Evolution Problems

• As business needs evolve, so do data schemas—but 77% of data teams report struggling with schema drift.

• Resolving data incidents takes an average of 40% longer when documentation is sparse.

• The result? Technical debt accumulates faster, bogging down engineering teams with avoidable problems.

A Blueprint for Data Context

Modern data engineering success hinges on robust contextualization frameworks that transform raw data into actionable insights. Think of this as building a detailed architectural plan for your skyscraper before laying the foundation.

1. Metadata Management Infrastructure

To create a sustainable framework, data teams need to connect the dots between raw data and its meaning:

Raw Data → Technical Metadata → Business Metadata → Semantic Layer ↓ ↓ ↓ ↓ Schema Definitions Business Glossary Knowledge Graph

2. Key Components of Context

• Lineage Tracking

  Understand how data flows through your systems:

  • Source-to-target mappings

  • Documentation of transformation logic

  • Impact analysis and version control

• Business Process Integration

  Align data context with operational needs:

  • APIs and service mappings

  • SLAs and quality thresholds

  • Clear ownership and accountability

From Pain Points to Performance Gains

Organizations that prioritize contextualization consistently outperform their peers. Let’s contrast the before and after of implementing robust data context management.

The Before:

• Poor data quality costs companies $12.9 million annually.

• Data teams spend about 35% of their time firefighting quality issues instead of building innovative solutions.

• Discovery processes are painfully slow, with teams taking 30% more time to locate relevant datasets.

The After:

• Implementing data quality frameworks reduces incident resolution times by 26%.

• Data lineage documentation makes teams 23% more likely to make data-driven decisions.

• Automated data catalogs cut discovery time by 40%.

The Road Ahead: Evolving Context

The future of data engineering is context-first. Advancements like AI and real-time processing are pushing the boundaries of what’s possible:

1. AI-Assisted Context Generation

• Algorithms that uncover hidden relationships.

• Natural language processing for automated documentation.

• Predictive models to assess the impact of changes.

2. Real-Time Context Updates

• Stream processing for evolving schemas.

• Automated lineage updates to reflect changes instantly.

3. Context-Aware Governance

• Proactive compliance monitoring.

• Automated privacy controls and secure context sharing.

Conclusion: Context as a Competitive Advantage

In today’s data-driven economy, context is everything. Organizations must treat context not as a nice-to-have but as a critical currency for modern data operations. It’s not the size of your data lake that matters—it’s how well you understand and utilize the data within.

The most successful teams:

✓ Automate metadata collection.

✓ Build clear mappings between technical and business contexts.

✓ Monitor and enhance context quality continuously.

In the words of a modern data architect:

"Context turns data from a liability into an asset."

By investing in context, you’re not just managing data—you’re unlocking its full potential.

The Hidden Cost Centres

Your data warehouse might be silently draining your budget in ways you haven't noticed. That "simple" dashboard query that runs for 20 minutes isn't just frustrating—it's expensive. Long-running queries consume unnecessary compute resources, and yes, even failed queries cost money.

Storage bloat is another silent budget killer. Duplicate test tables, outdated data that should have been archived months ago, and unnecessarily wide tables with unused columns—they're all taking up expensive storage space. Think of it like paying rent for a storage unit full of items you'll never use again.

Perhaps the most wasteful practice is inefficient resource allocation. Imagine running your air conditioning at full blast in an empty house—that's what an over-provisioned warehouse looks like. Multiple users running heavy transformations simultaneously without any resource prioritization only compounds the problem.

Why Manual Optimization Falls Short

Traditional approaches to optimization are like trying to manually balance your chequebook in the age of digital banking—it's time-consuming, error-prone, and ultimately inefficient. Data teams spend countless hours on manual query analysis and performance tuning, only to fall behind as query patterns change and data volumes grow.

According to Gartner, poor data quality alone costs organizations an average of $12.9M annually. This figure becomes even more concerning when you consider that much of this cost comes from inefficient data processing and storage.

The Power of Automated Optimization

This is where automated optimization comes into play. Modern solutions can continuously monitor your warehouse's performance, automatically identifying and optimizing problematic queries, managing resources dynamically, and cleaning up unused assets. It's like having a dedicated team of database performance experts working 24/7, just without the overhead.

Think of automated optimization as your warehouse's financial advisor—constantly looking for ways to save money while improving performance. It can identify when queries are consuming excessive resources, automatically adjust warehouse sizing based on actual usage patterns, and alert you to cost anomalies before they become budget disasters.

Beyond Cost Savings

While cost reduction is important, the benefits of optimization extend far beyond just saving money. When your warehouse runs efficiently, queries run faster, data teams become more productive, and your entire data infrastructure becomes more reliable. This means better insights, faster decision-making, and more time for innovation.

At Artemis, we've seen firsthand how automated optimization transforms data operations. Our platform gives data teams back an estimated 500 hours per year—time that can be invested in strategic initiatives rather than maintenance and troubleshooting.

Taking Action

Don't wait for your next shocking cloud bill to start thinking about optimization. Begin by understanding your current warehouse usage patterns and biggest cost drivers. Implement automated optimization tools to handle the heavy lifting of performance tuning and resource management. Most importantly, establish clear cost governance policies to prevent future waste.

Remember: Every dollar saved on warehouse operations is a dollar that can be invested in actual data innovation. In today's competitive landscape, can you afford not to optimize?

Imagine this: You're a data engineer. It's 3pm on a Friday, and your Slack pings with an ad hoc request from a business analyst for a highly specific, rarely-used metric. This requires diving into a dataset you've never touched before, scouring documentation (if it exists), and wrangling a pipeline to produce something coherent—all for a piece of information that will likely be used once. The response? A quick “thanks,” if you’re lucky.

Data engineers are the backbone of modern businesses, yet their work often goes unrecognized and they are often eclipsed by their flashy cousin, the software engineer. DE’s are the ones transforming raw data into actionable insights, ensuring systems run smoothly, and enabling teams to leverage the power of analytics. But where’s the love for these unsung heroes?

The Current State of Data Engineering

The data landscape in 2024 is, to put it lightly, a mess. With studies showing 97% of data engineers reporting burnout and being unsatisfied with their roles, it's no surprised they're overworked. They're often juggling multiple priorities and are rarely acknowledged for their technical abilities.

At the same time, executive teams are clamouring for AI-powered solutions, maybe without fully understanding the resource investment required. With global data creation projected to reach 175 zettabytes by 2025, implementing AI systems isn't just about flashy models; it's about building robust pipelines, maintaining clean and accessible data, and ensuring everything scales seamlessly. Without proper resourcing, this demand further exacerbates the strain on data teams (If that’s even possible).

Example: A C-suite decides they want real-time AI powered analytics to power customer dashboards. The expectation? Deliver it in two months. The reality? Data engineers scrambling to retrofit legacy systems, integrating machine learning models with incomplete training data, and building fragile pipelines that barely hold under the strain—all while racing against the clock.

What This Means for New Projects

When data engineers are perpetually stuck in firefighting mode—fixing broken pipelines, addressing tech debt, and fielding endless requests—it leaves little room for innovation. New projects, which should be exciting opportunities to leverage cutting-edge tools and techniques, instead can feel like nothing more than a headache.

This constant grind isn’t sustainable. Without better systems and recognition, businesses risk burning out their most critical technical teams.

Building a Better Future for Data Teams

It’s not all doom and gloom though, there’s hope. By addressing the root causes of these challenges, we can create a more sustainable, rewarding environment for data engineers. Here’s how I would do it:

1. Optimization: Evaluate and streamline your current data stack to reduce inefficiencies.

2. Cleaning Up Stacks: Invest in modernizing outdated systems and addressing tech debt.

3. Automation: Offload repetitive, manual tasks like pipeline maintenance and documentation generation.

4. Empowering Teams: Give data engineers the bandwidth to focus on meaningful, innovative projects rather than just staying afloat.

How Artemis Makes a Difference

At @Artemis, we believe data engineers deserve better. Our platform is designed to give data teams their time back—an estimated 500 hours per year—by automating and optimizing their stack. Here’s how:

• Automated dbt Model Optimization: We fine-tune and maintain your models to ensure peak performance.

• Streamlined Documentation: Our tools improve and manage documentation effortlessly, so your teams don’t have to.

• Warehouse Efficiency: By continuously monitoring and optimizing your warehouse, we ensure cost savings and enhanced performance.

By reducing the grunt work and improving overall efficiency, Artemis empowers data teams to innovate, save money, and scale seamlessly—all without drowning in the day-to-day grind.

Conclusion

Data engineers are the backbone of every data-driven decision—quietly building, maintaining, and optimizing the systems that power insights.

It’s time to recognize not only their impact but the deep technical expertise they bring. With the right support and tools, data engineers can move beyond the firefighting of day-to-day maintenance and focus on what they do best: building scalable, reliable, and efficient data systems.

Most data teams build a custom racecar for their analytics—not because they want to, but because that's just how the modern data stack works. You pick Snowflake, BigQuery, or Databricks as your engine and build the rest of the car around it. Whether you're using Fivetran or Airbyte, dbt or Coalesce, you've got to do a ton of custom work to get this car running. But hey, this part of the job is a blast! Teams love getting the budget to design and build a data platform from scratch.

The issue is that as the company scales, the car gets more complex. You start adding more features to handle all the new requests and edge cases. Before you know it, you've got new dashboards, a reverse ETL tool thrown into the mix, and even a catalogue to tie it all together. You and your team get busy building foundational data models that the analysts then run with. Even though these models pull data from standard data sources such as Quickbooks, Stripe, ERPs, etc., the models are customized to your organization and often overfitted with business logic.

All these tools are beneficial and serve a purpose, but the complexity grows with them. To make matters worse, while your core team does most of the work occasionally, you let a product team member create a few dashboards and write some queries in the name "self-service." As a result, context is added, or assumptions are made that don't fit your team's standard model. This mess lives in dbt, Airlow, as the core logic is stored there. However, this can expand into BI tools like Looker or upstream integration tools. This customization and maintenance is a huge drag on teams. A dbt labs survey suggests teams spend 26% of their week fixing and maintaining their infrastructure. The point is it gets complicated quickly!

Before you know it, the models start to break down, and the custom racecar you and the team built is in the shop and takes forever to fix. Why? It's for a multitude of reasons. A collection of people (some not at your company anymore) have written hundreds of models; you have ten workloads calculating similar things in slightly different ways hooked up to ‘core sources of truth.’ A lot of times, dbt models are added without rules or assumptions. You start to see duplications, models not referencing the correct upstream table, and models running hourly where the data is only updated every 6 hours.

The outcome? It's been 2-5 years with the MDS, and your costs are way higher than you predicted; your pipelines are fragile and constantly breaking, and logic is siloed across many tools. This is a reality for data teams.

So how do you fix it? This is a question a lot of data teams are asking themselves. From the teams we have spoken to, there are a few outcomes.

1. They do nothing and hope it’ll figure itself out one day. (Spoiler: it won’t).

2. They commission a team of 1 or 2 engineers to focus on this project. They spend 6-9 months on it and are pulled from projects.

3. They do the opposite and commission 1-2 to focus on new work while the rest of the team is set on fixing their foundation. This is a costly outcome.

4. They use Artemis.

Each option is time-consuming, expensive, and has a huge opportunity cost. When trying to solve these issues manually, you discover how hard it is to search for issues across the stack while holding the entire platform's design in one or two people's heads. It’s a large reason why this work doesn’t get done.

Data platforms are held together with duct tape, which is the nature of the beast. Still, teams can invest in tooling that helps maintain those systems so they can focus on delivering value for the business instead of constantly repairing what’s broken.

What is Artemis?

Artemis is an end-to-end data platform mechanic. Our platform monitors your stack, finds issues within your warehouse, dbt models, and BI tools, and then auto-resolves them, so all you need to do is approve insights and merge PRs. The platform does all the research and highlights where you can cut bloat, improve performance and save warehouse costs.

We tune your data platform to be a lean, fast, and smooth machine so your engine is humming, and you fly by your OKRs. The average team approves 120 insights, merges 60 PRs and saves 25 hours a week!

If you want to optimize costs or cut dbt bloat, reach out!

I recently had the chance to interview Chris Riccomini. Chris currently manages Materialized View Capital. He spent over 15 years working on infrastructure at major tech companies such as PayPal, LinkedIn, and WePay. He was involved in open-source projects like Apache Samza and Apache Airflow.

JG: How do you see the relationship between data engineers and analytics engineers (I am curious how your position has shifted since you tweeted in Nov 2022)?

CR: Good question. I realize that these definitions are quite fluid.

To me, a data engineer is responsible for the E and L in the data pipeline—extracting and loading data. Analytics engineers are responsible for the T—transformation. A data engineer’s job is to build a data integration layer to get data into the system where it’s needed reliably. One of those systems is a data warehouse. An analytics engineer’s job is to transform the raw (loaded) data into something more usable for product managers, sales ops, business analysts, and others. Note here that I did not mention deriving value and insight from the data; that’s a role I would describe as a business analyst.

In practice, I find that most people actually do more than one of these roles in an organization. For example, someone with an “analytics engineer” title might end up doing business analyst stuff, too: creating reports, data visualization, building business review dashboards, fielding questions from the executive team, and so on.

JG: We’ve spoken about how data warehouse spending became an issue at your last company. What happened, and how much work did it take to get it under control?

CR: The short answer is that we allowed the data warehouse to grow organically for several years with a focus on convenience instead of cost. At first, the data engineering team was just me. For many years, it was just two or three people. With the number of pipelines we had, we simply didn’t have time to optimize for cost.

As time went on, though, we grew—not just in team size but also in data and query size. We had more pipelines and more people running queries. Cost slowly crept up. It wasn’t overnight, but over a year or two. Then, ZIRP ended, and the CFO came knocking.

Fortunately for us, there was a lot of low hanging fruit. For example, we exposed views rather than tables to users. When a user queried a view, the view would de-duplicate rows to make sure that users only saw the latest version of each row in a table (we were using Debezium and CDC to load data, so a new table record was inserted for each update to a row). For frequently queried tables, it was cheaper to simply materialize those views rather than keep de-duplicating the entire table each time a user queried the data. Another more basic example was that we loaded all tables from each database into our data warehouse by default. Many of the tables were rarely used. We could remove such tables to save some money. There are many such examples, but you get the idea.

JG: What are your thoughts on dbt? What is it missing? How can it be better?

CR: dbt is a very useful tool. We built something internally at my previous job that looked similar, but was much more basic. Nearly every company I interacted with had some version of a Python script that could be used to auto-generate tables and views. And nearly every company had some kind of data workflow orchestrator. The use case for something like dbt was self-evident. Moreover, there was real value in standardizing such a tool. It benefited from network effects as developers contributed integrations and packages. It also, rather sneakily, snuck in SDLC best practices so analytics engineers could build better transformations.

As for how dbt could be improved, I think the folks at SQLMesh, sdf, Coalesce, and Dagster would be better positioned to answer that. What I can say is that—going back to your very first question—what I’ve never been able to understand is why we split the transform tool (dbt) out from the extract and load tools (workflow orchestrators, CDC, and such). These two things seem very much related (so much so, we keep them in a single acroynm—ELT). I think Dagster was early in recognizing this. Another signal of this oddity is that we now have dlt as well as dbt; they’re the same idea but one is for extract/load and the other is for transform. Unifying extract, load, and transform tools back together is something that needs to happen, though we’re in the infancy of this transition.

JG: What challenges have you experienced or seen data teams experience when optimizing or maintaining their data platform?

CR: Oh gosh, it’s really endless. Data quality issues, schema compatibility, data loss, cost, managing Python runtime environments, keeping pipelines within latency SLAs, difficulty getting access to observability data, dealing with an explosion of tools (the so called MDS), and so on. And that’s just on the data pipeline side.

Straddling the production and non-production domains presents some unique challenges, too. Many data teams have to act as a support team, security team, and ops team as well. They end up having to help users fix queries, are on-call when the platforms break, and often need to gatekeep access, monitor for sensitive information, and so on. While production engineers often have a single (usually standard) process to manage such things, data teams often get left behind for various reasons.

JG: In your opinion, why does dbt increase spending on data warehouses?

CR: dbt decreases the friction of running transformations in the data warehouse. This is a good thing, provided these new transformations are providing business value. However, given the way cloud data warehouses bill, this necessarily means you’re spending more money. dbt has attempted to help with this by providing incremental models, and the community has tried to do some transformation outside of the cloud data warehouse (in DuckDB, for example). Still, the default is that things run in an expensive way.

JG: What are your thoughts on increasing the focus on optimizing data warehouse costs, especially in relation to dbt workloads?

CR: I think the experience I discussed above is relevant to this question. We had to choose whether to optimize our data architecture for simplicity or cost when rolling out our data pipeline. There was no simple, cost-optimal way to build the pipeline at the time.

The ideal scenario is one in which we can build both simple and cost-effective data pipelines. We need to lower the cost of lowering the cost, so to speak. To do so, I think we need optimal-by-default tooling. Incremental ETL pipelines, integrated EL and T tools that can figure out where transformations should occur (i.e. not just in the data warehouse), cost observability tools, and so on.

JG: How do you see tools like Artemis changing how data engineers work?

CR: I think Artemis has an opportunity to address the stuff I just mentioned: make it easy for me to build a simple, cost-optimal data pipeline. Leveraging AI to figure out how to clean up waste dbt pipelines is a great first step. I expect a lot more from you guys, though. As I said, I can imagine query monitoring and optimization with AI, query rewriting to move transformations outside of the CDWH and into local (cheaper) engines like DuckDB, and a lot more.

JG: How do you think tools like Artemis will impact how data teams perform maintenance on their data stack?

CR: I see two approaches to solving the fragmentation in the modern data stack (MDS) space. The first is to provide a fully integrated, opinionated solution about how to manage your data warehouse or data lakehouse. The second is to provide AI-based solutions that help manage the MDS fragmentation and reduce toil for data engineers. The nice thing about the latter approach is that it meets organizations where they are. That’s what I like about Artemis. You don’t need to move off your data warehouse, you don’t need to move off of dbt, and you don’t need to adopt some vertically integrated solution. I’m excited to see more tools like this pop up. We’ve dug ourselves into a pretty big hole with the MDS tech stack, and tools like Artemis will play a role in helping us wrangle all this stuff.

JG: What advice would you give to organizations looking to implement cost optimization strategies for their data warehouses without sacrificing agility and innovation?

CR: Every organization has 7±2 tables that are really business critical and 7±2 tables that are really expensive. Most organizations know which tables are business critical, but it might not be obvious which tables (and query patterns) are most expensive. So job #1 is to set up some observability to figure out what is costing you the most.

Once you’ve got a clear picture of business critical and expensive tables. Take the union of these two table sets and apply the 80-20 rule: figure out how to optimize each of them enough to reap 80% of the savings with 20% of the effort.

Next, you need to set up some monitoring to alert you when future tables and queries go astray. Once you’ve established this beachhead, go back and ask yourself what (if anything) needs to be done longer term to manage costs. Do you need to fundamentally re-architect your pipeline? Do you need to change certain tools or eliminate query patterns? You don’t need to be perfect with this stuff, so it’s quite possible that the playbook I just outlined will get you where you need to be. Don’t go overboard.

In 2010, researchers in Japan made headlines with an astonishing demonstration: they used a species of slime mold to design an optimal railway network for Tokyo. The slime mold replicated the city's complex rail system with remarkable efficiency, using its natural ability to find the shortest paths between nutrient sources. The Tokyo train system is the host of nearly 40 million daily riders and just a fraction of decreased route efficiency equates to millions of additional load hours on the system a year.

This example of biomimicry – where solutions from nature inspire human innovation – highlights how studying biological systems can lead to groundbreaking advancements. Now, imagine applying these principles to the realm of data science and artificial intelligence (AI). Enter the world of swarm intelligence.

Understanding Biomimicry

Biomimicry is the practice of drawing inspiration from nature to solve human problems. Famous examples include Velcro, inspired by burrs that stick to animal fur, and the design of wind turbine blades which are modelled after whale fins. In the tech world, biomimicry is revolutionizing how we approach data processing and artificial intelligence.

The Concept of Swarms

Swarm intelligence refers to the collective behaviour of decentralized, self-organized systems, particularly natural ones like ant colonies, bird flocks, and bee hives. Each individual, following simple rules and local interactions, contributes to the complex, intelligent behaviour of the group.

Imagine the way a school of fish moves in harmony to distract and confuse predators.

Key characteristics of swarm intelligence include:

1. Decentralization: No single leader; individuals operate based on local information.

2. Self-Organization: Order emerges from simple interactions among individuals.

3. Robustness: The system can adapt to changes and recover from disruptions.

Biomimicry in Data Science

In data science, biomimicry has lead to innovative algorithms that solve complex problems efficiently. For example, ant colony optimization algorithms simulate the pheromone trails of ants to find the shortest paths in network routing and logistics.

Other Examples

• Particle Swarm Optimization: Inspired by bird flocking behaviour, used for optimization problems.

• Genetic Algorithms: Mimic natural selection to solve optimization and search problems. (See below)

The Power of Swarms and The Implications for Agents

For AI agents to be effective, they need well-defined roles, much like bees in a hive. Each bee has a specific task, from foraging to caring for the queen, these defined roles ensures the hive's success.

In nature, specialization allows individuals to excel, contributing to the overall efficiency and success of the group. This principle applies to AI agents as well. When agents are given very specific jobs with clear constraints, they perform these tasks more effectively than generalist. Specialization allows agents to:

• Optimize Skills: Focus on mastering a particular task, leading to higher proficiency.

• Reduce Complexity: Simplify problem-solving by narrowing the scope of their responsibilities.

• Enhance Coordination: Work seamlessly with other specialized agents, each contributing their expertise to achieve a common goal.

Examples of Role-Specific AI Agents

• Foraging Agents: In logistics, specialized agents could be tasked with finding the best routes for delivery, much like foraging bees.

• Security Agents: In cybersecurity, dedicated agents can focus solely on detecting and mitigating specific types of threats.

• Data Processing Agents: In data science, agents could be designed to handle specific types of data processing tasks, such as cleaning, transforming, or analyzing data sets.

In a bee hive, communication occurs through dances, pheromones, and direct interactions, guiding the collective behaviour. Similarly, AI agents can use environmental data, peer communication, and hierarchical directives to perform specialized roles. The efficiency and success of natural swarms hinge on the clear division of labor and specialized functions of each member, which can be mirrored in AI systems.

Adopting swarm principles leads to AI systems that are more efficient, adaptable, and resilient. By giving AI agents specific roles and leveraging their interactions, we can create intelligent systems capable of tackling complex challenges. This approach ensures that each agent contributes optimally to the system’s objectives, enhancing overall performance.

Future of AI agents

Swarm-based AI agents will revolutionize how tasks are managed in dynamic environments, enhancing collaboration and problem-solving. Without a doubt, specialized agents working together can address complex issues more effectively than a single, generalist agent.

Each agent’s expertise in a particular domain ensures that the collective system can handle a wide range of tasks with greater proficiency.

Potential Challenges

Implementing swarm intelligence in AI systems presents significant challenges, primarily centred around coordination and communication, complexity management, and robustness. As the number of agents increases, ensuring effective coordination and communication becomes more complex, introducing latency and requiring significant bandwidth. Achieving consensus among decentralized agents can be particularly difficult in dynamic or large-scale environments. Additionally, designing systems with multiple specialized agents necessitates careful planning to manage the complexity of their interactions and predict the emergent behaviour of the system.

Robustness and fault tolerance are also critical concerns. The system must handle individual agent failures without compromising overall performance, and prevent errors from propagating through the network.

Ensuring security and privacy in decentralized systems is challenging, as they can be more vulnerable to attacks and ensuring data privacy can be difficult. Adapting to rapidly changing environments and unexpected conditions requires sophisticated algorithms that can quickly reconfigure the swarm’s behavior, balancing the trade-offs between exploration and exploitation to optimize performance. Addressing these challenges is essential for harnessing the full potential of swarm intelligence in AI.

To Wrap It Up

Biomimicry, offers powerful insights for the future of AI. By mimicking natural systems, we can develop AI agents that are efficient, resilient, and capable of complex problem-solving. At Artemis, we are leveraging these methods to build AI agents specifically designed for data engineers. These agents, inspired by the collective behaviour of swarms, are crafted to create robust and optimal systems that enhance data processing and analysis.

Looking ahead, the principles of biomimicry and swarm intelligence will continue to shape the evolution of AI, driving innovation and unlocking new possibilities and at Artemis, we are committed to harnessing the power of nature’s code to revolutionize the field of data engineering.

At Artemis, we’ve seen firsthand how technical debt impacts companies of all sizes. As a data quality company, our mission is to help businesses optimize and alleviate their tech debt, allowing them to innovate without being held back by the limitations of the past. But what exactly is technical debt, how does it accumulate, and what can companies do to mitigate it?

What is Technical Debt?

Technical debt refers to the implied cost of additional rework caused by choosing an easier, faster solution now instead of a better, more sustainable one that would take longer to implement. Just as with financial debt, technical debt accrues interest over time—eventually, it must be repaid, and the longer you wait, the more costly that repayment becomes. This could manifest in several ways, such as not maintaining adequate test coverage, failing to document new code, or implementing quick fixes that sacrifice long-term maintainability for short-term reliability. These shortcuts may create robust but hard-to-read code, making it difficult for future developers to troubleshoot or extend, leading to increased costs down the road as systems become harder to manage and optimize

“With borrowed money, you can do something sooner than you might otherwise, but then until you pay back that money you'll be paying interest.” - Ward Cunningham

At its core, technical debt is the result of short-term thinking. When developers make trade-offs to meet immediate needs, they often leave behind incomplete or suboptimal code, patches, or infrastructure. These quick fixes pile up and can hinder future development efforts by increasing complexity, reducing code quality, and making systems harder to maintain or extend.

The Hidden Costs of Technical Debt

While the metaphor of debt captures the essence of the problem, the impact of technical debt is more than just a matter of technical inefficiency—it’s an anchor that weighs down innovation. As technical debt accumulates, development slows, bugs become more frequent, and teams find themselves spending more time fixing old problems than building new features. This creates a vicious cycle where innovation grinds to a halt, and competitive advantage dwindles.

Here are a few specific ways that tech debt can sabotage innovation:

1. Slower Development Cycles: Technical debt creates friction in the development process. As codebases become more complex, it takes longer for developers to implement new features or changes. Every new line of code is an opportunity to introduce new bugs, and older code is prone to breaking under new conditions.s

2. Increased Maintenance Costs: Maintenance of poorly written or hastily implemented systems requires more resources—both in terms of time and money. More time spent fixing problems means less time for developing innovative features. As the backlog grows, the weight of unresolved issues can feel like an insurmountable barrier.

3. Lower Morale: No one enjoys working with buggy, unmanageable systems. Tech debt can drain team morale, causing frustration and burnout. When engineers are consistently forced to work around broken code and inefficient systems, it’s hard to stay motivated and focused on innovation.

4. Stagnation of Ideas: When all available resources are used to manage technical debt, there’s little bandwidth left for creativity or experimentation. Innovation often requires risk-taking and flexibility, but technical debt restricts both, leading to stagnation.

5. Security Vulnerabilities: Technical debt can also introduce security risks. Outdated and unsupported code undoubtedly exposes the organization to vulnerabilities, making it harder to protect against threats and increasing the risk of a security breach.

But Kirsten, we all understand best practices and know how to write good code, this would never happen to us…

Nobody, goes into a new venture or project with the plan to create tech debt. But like all things in life, sh*t happens and it starts to accumulate for a variety of reasons, it also turns into a bit of a snowball effect.

Here’s some of what I’ve seen happen to the most well-meaning development teams. As they work under tight deadlines and pressure to deliver, they often find themselves balancing competing priorities, such as maintainability, scalability, and reliability—what can be thought of as the triangle of software development.

Really what that means for the business case is:

• Time-to-Market Pressures: In the race to get products or features to market quickly, corners are cut, and best practices are ignored.

• Inconsistent Processes: Lack of standardized coding practices across teams can lead to inconsistent codebases that are difficult to maintain.

• Legacy Systems: Older systems may not integrate well with newer technologies, creating a patchwork of temporary fixes that accumulate over time.

• Limited Resources: Teams often face constraints in terms of time, budget, and personnel, leading them to prioritize speed over long-term stability.

So let’s talk about the companies that handle it well

TLDR: Do: modernize early, build scalable systems, and stay agile to foster continuous innovation. Don’t: rely on outdated systems, this will kill you 10/10 times.

Some companies have shown remarkable success in managing their tech debt, turning what could have been a long-term liability into an opportunity for growth and innovation.

Netflix: A Tech Debt Success Story

Netflix is a great example of a company that managed tech debt wisely. In its early days as a DVD rental service, Netflix made the strategic decision to embrace microservices and cloud-native architecture as it transitioned to streaming.

Today, Netflix continues to manage technical debt by giving developers the freedom to fix it as part of their everyday work, through initiatives like "Freedom & Responsibility" (F&R) time. This allows engineers to focus on refactoring code and improving systems without needing formal approval.

They also use a "paved road" approach, providing developers with easy-to-use, well-supported tools to avoid bad practices that could lead to debt. Additionally Netflix's chaos engineering identifies weak spots early, preventing future problems.

By breaking technical debt into categories like Code, Design, and Documentation, Netflix is able to prioritize and tackle it regularly, ensuring it doesn’t pile up and slow down innovation. If you are interested in learning more about Netflix’s technical debt tackling techniques I suggest checking out this article.

How can my team fix this problem?

Assuming limited resources and the inability to turn back time*

While the challenges posed by legacy systems and inefficiencies are significant, they are far from insurmountable. In an ideal world, companies would proactively balance fast delivery with long-term sustainability. But for many, accumulated system issues are already weighing them down.

This is where AI agents, like the ones we develop at Artemis, step in to help. Our AI agents assist teams in tackling existing inefficiencies by optimizing codebases, automating documentation, and improving system performance. They handle the routine tasks that often create bottlenecks, giving your engineering teams the space to focus on innovation and high-impact work.

This combination helps clear the way for data engineers to focus on innovation by automating documentation, optimizing data warehouses, and ensuring that as new features roll out, projects remains up-to-date and systems run smoothly. By providing both visibility and automated resolution, Artemis allows teams to streamline their operations, scale efficiently, and stay ahead without being bogged down by past inefficiencies. Ultimately, empowering data teams of all sizes to focus on what truly matters—innovation—by removing operational obstacles and enabling them to continuously improve.

With automation becoming more and more accessible the role of human data engineers remains simply irreplaceable. The question to ask isn’t whether will AI replace data engineers—it’s how can AI empower data engineers to work smarter and more efficiently?

AI as an Ally, Not a Replacement

We've already proven that AI can automate tasks like schema generation, query writing, warehouse optimization and even basic data analysis. These advances certainly change the day-to-day tasks of data engineers, but they don’t replace the core skills that engineers bring to the table. Instead, AI becomes an ally—handling repetitive or tedious tasks, freeing engineers to focus on higher-level problem-solving and innovation.

As noted in previous Block Bulletin blogs , AI thrives in predefined environments with clear parameters, but when systems grow in complexity, ambiguity, and scale, humans have to step in. AI can suggest solutions based on algorithms, but it takes a data engineer’s experience and understanding of business context to determine the right approach.

AI’s Role: Automation of Tedious Tasks, but Not Strategy

AI lacks the strategic mindset needed to design resilient, future-proof data architectures. It can’t sit in a meeting and understand business requirements, nor can it predict how data models need to evolve with a company’s growth.

This is where the human element shines. When it comes to understanding the bigger picture—how different data systems interact, how to optimize for performance over time, or when to prioritize certain datasets—human engineers are irreplaceable. They bring a level of creativity and adaptability that AI just can’t match.

Preserving the Human Element

While there’s no denying that data engineers provide a level of strategic thinking and problem-solving that AI can’t replicate, if we are being honest with ourselves, AI is closing the gap. Emerging technologies, such as AI-powered anomaly detection and real-time monitoring, are enabling more autonomous data systems. AI tools can already alert engineers to potential issues before they escalate, but as of now they still rely on humans to fix more complex problems and make high-level decisions.

Today’s leading companies are focusing on creating a balance between AI automation and human oversight. For example, our agent Diana is designed to handle routine tasks like optimizing data warehouses, but data engineers remain at the helm, guiding overall strategy and ensuring data integrity.

The Human Edge: Problem-Solving and Creativity

It’s important to remember what GenAI actually stands for and means, you would think that generative and artificial makes it pretty self explanatory. But the whole intelligence thing, still seems to be throwing people for a loop. **AI may complete tasks that are traditionally reserved for human levels of intelligence, but that definitely doesn’t equate it to actually possessing it. It may seem obvious, but to clarify, the term "artificial" indicates that the intelligence is simply simulated, not genuine. Humans are essential in interpreting nuanced requirements, making decisions in complex scenarios, and understanding the intricate details of the data ecosystem that AI hasn’t yet mastered. Problem-solving in real-time, handling unexpected issues, and making strategic adjustments based on business changes remain firmly within the domain of human engineers.

Balancing AI and Human Expertise

The best data engineering teams of today are those that combine the efficiency of AI with the strategic, unmatched problem-solving abilities of human engineers. In this new landscape, it’s not about AI vs. humans—it’s about collaboration. AI takes on the heavy lifting and grunt work, like automating routine tasks and running performance optimizations, allowing data engineers to focus on higher-value activities like designing systems, ensuring data governance, and driving business insights.

In the shifting landscape of data engineering, the question isn’t whether AI is better than humans or vice versa. The future holds a two-pronged approach, where AI handles the menial tasks of data management, freeing human engineers to focus on higher-order responsibilities like innovation, architecture, and solving complex problems.

A Collaborative Future

The rise of AI in data engineering is transformative, but it isn’t an existential threat to data engineers. All though extremist fear mongering around AI taking over humans exists on the fringe, The vast majority, including myself believe, AI is unlikely to replace data engineers but will instead augment their work, taking over tedious, repetitive tasks, allowing us to focus on what matters most.

So, will AI replace data engineers? No—and the future looks even brighter because of it.

Quick Recap: What Are Modular Systems?

The modern data stack turned end to end monoliths from Oracle and other providers into lego blocks. Instead of being stuck with one fixed pipeline, the modular stack let you break processes into smaller, manageable pieces (or modules). You can then mix and match these to create a workflow that works best for you and your data needs.

Over the past 10 years or so tools like dbt have popularized this approach, especially when it comes to managing data transformations. Instead of being locked into a complex, monolithic pipeline with scrappy SQL dbt’s modular techniques allow data engineers to work with more freedom, swapping in new models without breaking the whole thing. It’s like having a toolbox where you can just grab the right tool at the right time—no overhauling your entire system just because of a little change.

The Initial Promise

Modular data stacks were saving teams a ton of time. For example, in 2017, Siemens reported that using dbt reduced their dashboard maintenance time by 90%. Picture this—your team is handling a new data source, and instead of ripping apart your entire pipeline to fit it in, you just pop in a new model. Done. This kind of agility means your team can react quickly to new business requirements, new data sources, or just… you know, change without sinking weeks into reworking the whole system.

According to Gartner, and based on personal conversations with data professionals, this trend isn't slowing down. Gartner predicts that by 2025, over 75% of enterprise data engineering organizations will adopt these modern architectures. It's easy to understand why. The initial ROI is compelling, and the flexibility seems perfect for scaling operations. What could go wrong?

The Dark Side: Costs and Complexity Creep Up

But let’s not get too carried away with the "modularity is magic" idea of the mid 2010s—there are now obvious downsides that have become more prevalent as companies commit to modular tools.

On the surface, modularity seems perfect for growth—just keep adding more pieces, right? But as systems expanded, managing those individual modules have become a nightmare. Each module might need its own versioning, monitoring, and optimization, and suddenly data teams are spending just as much time maintaining the system as they are building it. According to Wakefield, data engineers spend up to 40% of their time on maintenance. It’s ironic, but modular systems actually introduced more technical debt.

Secondly, modular systems can drive up costs fast. Each new piece or service you add to your stack often comes with its own set of dependencies, resource requirements, and sometimes licensing or operational costs. It's easy to fall into the trap of adding one more tool here or another module there until you realize your bill is way higher than it would’ve been if you’d stuck to a more streamlined, less modular setup.

When you have multiple services or modules interacting, dependencies grow, and things can break in ways you wouldn’t expect. You might also find that tracking down issues gets harder

What This Means

In essence, while modularity offers flexibility and agility, it has also led to fragmentation, where different parts of the system are updated at different paces, causing friction. Managing the complexity of all these independent modules can be just as challenging as the limitations of a traditional, monolithic system—especially as your company grows.

What We Do At Artemis

Artemis gives data teams 500 hours back a year by maintaining and optimizing your stack for you. Our platform optimizes and maintains dbt models automatically, helping teams manage and improve documentation, warehouse performance, and overall system efficiency. By streamlining processes and reducing manual tasks, Artemis empowers data teams to focus on innovation, cut costs, and scale effectively without being bogged down by technical inefficiencies.

1.Starbucks’ Data-Driven Success: Boosting Revenue Through Analytics

According to New Vantage, 79% of executives report that their organizations' investments in data initiatives are accelerating, and their revenue is growing in tandem.

Case Study: Starbucks ☕ Starbucks developed their app and Star Rewards program and used these new data points to analyze customer purchase history, app usage, and even Wi-Fi connection data to understand customer preferences and behaviours.

This approach allows Starbucks to personalize offerings and create targeted marketing campaigns. The results have been remarkable: As of 2023, mobile orders accounted for 27% of all U.S. Starbucks transactions, and the Stars Rewards program has grown to 32.6 million active members.

Going one step further, the addition of AI-powered personalization has led to a threefold increase in customer response rates to marketing offers, showing the power personal data can have.

2.Zara: Making Timely Decisions with Real-Time Data

According to a Deloitte survey, companies with strong data-driven decision-making are twice as likely to exceed business goals.

Case Study: Zara 👗 Zara's design-to-store time is the industry's best, at just three weeks, while others take an average of 6 months. They did this using in-store customer data. Zara equipped its store managers with handheld devices to report daily on customer reactions, popular items, and emerging trends. This data is then distributed to centralized design teams so they can make quick decisions about new products and adjustments to existing lines.

This rapid response to customer preferences has led to an impressive 85% full-price sales rate compared to the industry average of 60-70%.

3.Netflix’s Predictive Insights Create The Movies of Tomorrow

The predictive analytics market continues to rise, with it’s value expected to reach $28.1 billion by 2026, growing at a compound annual growth rate (CAGR) of 21.7%.

Case Study: Netflix 📺 Netflix spends a billion dollars a year on cloud computing costs. They do this to perfect your recommended watch list and drive content creation. The streaming giant analyzes everything from viewing patterns and completion rates to pause and rewind behaviours.

Netflix understands what people like to watch and how they watch it. For its hit show Stranger Things, Netflix used viewing data that showed a significant overlap between viewers who enjoyed supernatural themes and those who responded positively to nostalgic 80s content. Stranger Things, a combination of both, was 2022’s most streamed show on Netflix, pulling in more than 52 billion minutes of views. It had a 36% margin over the second most streamed show.

4.Spotify Knows Us Personally

Personalization increases average order value by 40% and customer satisfaction by 44%.

Case Study: Spotify 🎧 Spotify's algorithms that use billions of data points daily have transformed music discovery by creating a personalization engine that analyzes listening habits and preferences. Their insights help them understand what music people like, when they like it and how they prefer to listen to it, and create their Discover Weekly feature.

The result? Users have discovered over 2.3 billion hours of music through these personalized features, and Discover Weekly users stream 24% more music than non-users. By thoughtfully applying data to enhance the listening experience, Spotify has improved user satisfaction and increased the number of daily active users. 95% of Discover Weekly users return for more personalized recommendations the following week.

5.UPS is Finding Efficiencies Through Data

Data-driven operational improvements have led to significant cost savings across industries. McKinsey reports that data-driven organizations are 23 times more likely to acquire customers and six times more likely to retain them.

Case Study: UPS 📦 UPS revolutionized delivery logistics by developing ORION (On-Road Integrated Optimization and Navigation), a sophisticated system that uses data to optimize delivery routes. The company installed telematic sensors in its delivery vehicles to collect real-time data on everything from route progress to vehicle performance.

ORION processes this data and information about package destinations, promised delivery times, and weather conditions to calculate the most efficient routes.

The results have been remarkable: UPS now saves 100 million miles driven annually, reduces fuel consumption by 10 million gallons, and cuts 100,000 metric tons of CO2 emissions each year. The system has also generated significant cost savings, with an estimated $300-400 million reduced in operational expenses annually.

Embracing Data-Informed Decision Making

While data is known to create a competitive edge, it can sometimes be hard to find use cases and examples of massive results for organizations. Companies that have embedded data into their decision-making consistently outperform competitors by exceeding revenue targets, accelerating time to market, and improving customer retention. The best way to predict the future is to create it.