A post from the founder of ZenHub, Aaron Upright, got me thinking.
In 2022, chatbots were the hottest new thing. Every new YC startup seemed to have launched a copilot or chatbot for some industry or use case—copilots for legal, enterprise sales, robotic process automation, and more!
These copilots merely functioned like chatbots, allowing users to ask questions in the same format you’d use when interacting with ChatGPT. There are several reasons for this. First and foremost, when this new tech was being released, we didn't know any better, so we took the safe bet, using the format people associated with AI.
@ economyapp
The world was conditioned to believe that LLMs were best introduced through chat interfaces.
However, as we've moved towards AI agents, we're starting to see some unique user experiences. Teams like Figma have launched “feeling boards.” I don't know what they are called, but they are a novel and different way to work with AI.
Still, these experiences are rare now, and despite this shift to agentic tools, users will encounter the same tired design patterns. Of course, hindsight is 20/20. In early 2023 Artemis proudly launched a chatbot app. We thought this was the user experience customers wanted, but since launching, we have learned a few things.
We all know that LLMs are not deterministic, meaning we can’t control them, the same way we can with a typical SaaS tool. This means your app will make mistakes for the user. You can try and prompt engineer to make the tool hallucinate less, but it will still produce various outputs. Users need a way to verify the output, make changes or edits, and rerun tasks from specific points.
A direct symptom of point #1 is that because total control of the tool is impossible, you need complete transparency instead. Avoid black boxes—you must show both the input and output, along with the steps used to arrive at the final number, answer, etc.
This is more nuanced. We experienced the “Oh shit; this just made me understand Nuclear fusion” moment when we used ChatGPT for the first time. I thought that magical moment came from the LLM, and therefore, we built products with the LLM at the center, hoping users would experience that exact moment in our product. This is met with depressive reactions since it will rarely replicate.