The Future of Data Engineering
Will AI replace data engineers? No. Will AI fundamentally change data engineering work, yes.
A lot of LinkedIn chatter is about whether AI will replace data engineers.
I believe the answer to that question is no.
Will AI fundamentally change data engineering work, and how will it be done? Yes.
The Context
I spoke with the CTO of a unicorn data startup, who said, “We are really good at gathering data, but we are not the best at knowing what to do with it.” Benn Stancil hints at this weekly in his blog, which is both sarcastically realistic and pessimistic. Based on his posts, he would agree that we are great at gathering data, not creating insights.
We all saw the promise of Moneyball and wanted to re-enact the concept of “winning the game” through data within our businesses to reap massive rewards. I started my data journey analyzing alternative data in finance, where my job was to find Moneyball-type insights for alpha-generating investment decisions. I was naive to think every data organization operated like this.
The first step in that journey was to capture every data point. This thinking aligned with "what you can't measure, you can't manage" (fun fact: this famous quote is misattributed and was part of a longer quote which states the opposite). The belief was that small teams could quantify everything through sheer force of will to find that needle in the haystack. Data was supposed to be the deciding factor between becoming the next Netflix or Blockbuster.
Data tools profited immensely from automating the nuts-and-bolts process of finding alpha, eliminating the complex technical skills of working with and manipulating data points and turning it into a seamless and automated system. Redshift may have accelerated this in the cloud, but billions of dollars have been created around this idea. I mean, Databricks just raised $10 billion based on this concept.
However, working with data is still incredibly manual, even these years later. Data platforms are fragile and nuanced and are only becoming more vital to teams (check out why I think the data platform is the most important internal tool).
What is the ultimate destination for data practitioners if we aim to enable everyone in our organizations to work with data effortlessly through automation and AI? And in the future, what role will we data engineers play?
We Are Becoming Pilots
Data engineers are trending to have a similar role to commercial airline pilots. Like pilots, data engineers will manage complex systems (i.e. data platforms) that automatically handle hundreds of tasks. As the pilot of this data platform, you’ll be required to have the technical chops to deal with any issues that pop up while also being responsible for ensuring the platform and insights get to their destination.
We already see this transition with orchestrators taking on more complex work; however, AI will enable tools to go two or three layers deeper. It will fix pipelines on the fly, optimize workloads, and even provide analysis for end users. Your job is to manage the work and ensure the work is moving smoothly, in the correct order, and to the right people.
On top of this, once these systems are set up, you won’t need a team of 20 engineers. You’ll need two or three. They will all understand the platform and ensure that it is still performing at its optimal rate.
How do we get there?
Tools like Artemis help you experience this future by isolating issues within your stack, prioritizing what needs to be solved, and surfacing insights. Once you approve an issue, our AI agents will resolve it. Artemis is the automated mechanic in your plane, ensuring you move as necessary.
A ton of innovation is still needed around the data stack. Automating the flow and cohesion of these systems is still tricky, but there are many opportunities to solve this.
The exciting part about this future is that with autopilot, you will have time to work on the Moneyball questions of the world. You have collected a lot of data. It's time to see what it says.
Where will you and your platform fly?