Too Many Cooks In The Kitchen
Bobby from product analytics wants more direct access to the models for his new cohort analysis.
New Hope
Two years ago, you set up your new data stack—Fivetran, Snowflake, dbt, and maybe even a new BI tool like Metabase.
You built your foundational models. You even migrated from a legacy stack in AWS to this beautiful new machine you built. At first, you kept it pristine and allowed only your team of engineers to access it.
The Empire Strikes Back
But quarters come and go, and next thing you know, Bobby from product analytics wants more direct access to the models for his new cohort analysis. He doesn't know dbt or fully understand the foundation on which you built. But he's been asking, so you give him access.
Soon, this sort of exception starts happening more often. At first, it doesn't seem like a big deal, until one day, you look at your Snowflake console, and costs have risen quickly and drastically. Your dbt repo becomes more and more polluted with spaghetti code and models. Models are being copied and pasted, not referenced. There is a massive duplication across your model layer and your BI tool.
Your stomach churns as you realize you've let too many cooks in the kitchen.
We see this all the time: Users within your organizations contribute to the data platform in the name of self-service, but they have no right to be there and lack proper oversight. At first, this makes things faster, and users resolve their tickets, but before you know it, your task list gets longer and longer, and the water becomes murkier.
How do you fix this? How do you get back to the platform you spent so many hours building?
We suggest a few key steps.
Start with a complete platform audit.
Remove access for everyone not on your data teams, review each person with access, and examine whether they genuinely need to contribute to the platform.
Start back at first principles and rebuild your model foundation.
Isolate the models that have skyrocketed your costs and refactor them to get quick wins.
Start to declutter.
Honestly, this sounds like a lot of work—because it is. There is no way around that. Cleaning up this kitchen and making it look brand new will take a few months of gruelling work. You will need to take your engineers off of critical projects to sort this mess out.
Return of the Jedi
What if we told you there was another option? What if I told you that you can clean your kitchen in hours, not months, and have it cleaned automatically from then on?
That's what we do at Artemis! When we work with a new team, we provide them with a full audit of their data stack. We highlight inefficient queries, models, and more. We then take these insights and resolve the tasks, so all you need to do is review the PR and push it to production.
No need to use valuable engineering time.
No need to refactor models manually.
Watch your costs go down, and your dbt models clear up.
After the audit, our platform constantly listens for pipelines to break and optimization opportunities to appear. Then, we auto-resolve them, so you only need to review, approve, and deliver kick-ass value.
Our customers experience awesome downstream impacts.
Centralized oversight and quality control: Artemis provides a single platform for reviewing all data models, automatically flagging inefficiencies and enforcing best practices across the organization. This allows team leads to easily monitor and manage the quality of all models, regardless of who created them.
Performance visibility and accountability: We offer clear metrics on model performance and cost, creating accountability among team members. Inefficient models can be quickly identified, so we encourage better and more efficient model design.
Standardization and education: You can upload your business processes to ensure work is done according to your standard and to ensure best practices throughout the organization. Some users leverage this as an educational tool, helping less experienced team members learn to create better models.
If you have too many cooks in the kitchen, reach out and let's get it cleaned up before the end of the year so you can start 2025 on the right foot.