Ensuring Human Oversight with Diana: The Human-in-the-Loop Approach

At Artemis, we see AI, specifically LLMs, as a tool for users to scale their work and operate faster. However, with plenty of examples to show, this technology, like any other, needs a high level of human oversight to ensure reliability and security.

Our AI agent, Diana, supports data operations teams by automating routine and time-consuming tasks while implementing and requiring a human-in-the-loop approach.

Understanding the Human-in-the-Loop Approach

The human-in-the-loop (HITL) approach involves a human decision-maker in AI operations. It merges AI's efficiency with human oversight. We believe AI should empower, not replace, humans, and HITL should be seen as teamwork.

With human interaction, AI models are given constant feedback to help them adjust their “view of the world.” This allows for context building and better decision-making.

@Levity has done a great job explaining the concept of prediction confidence.

Why Human Oversight Matters

While AI is improving in performing complex tasks, it is not infallible. AI models’ “understanding” is largely based on statistics. Because of this, there are scenarios where AI might make errors, misinterpret data, or require context that it hasn't been trained on. In such cases, human judgment is needed. By incorporating human oversight into Diana's workflows, we ensure that every action and decision made by Diana is validated and approved by a human before implementation.

This approach is particularly crucial in data operations, where the accuracy and integrity of data is paramount. As mentioned in this blog, incorrect data handling can lead to significant business consequences, including incorrect analysis, misguided strategies, and compliance issues.

You’ll make worse decisions with bad data than you will with no data.

Human oversight safeguards against these risks, ensuring AI-generated actions align with an organization's goals and standards.

How Diana Incorporates Human Oversight

Diana seamlessly integrates into your existing data infrastructure and workflows. It handles tasks such as creating documentation, identifying pipeline issues, and gathering information from business users. Diana works effortlessly with tools like @jira, @git, @Slack, and @Teams, functioning as a teammate rather than just another tool. Diana can take initiative and collaborate like any other colleague.

While Diana does the work for each task, it still needs approval. Functioning under the supervision of human team members who review and approve its actions, similar to anyone on your team currently.