Created with AI.
What should we measure for an AI agent?
Agents are often evaluated based on the number of conversations. That number grows on its own and says nothing about their usefulness.
The key metric. What proportion of conversations were concluded without human intervention and without the customer getting back in touch shortly afterward? The latter is important—a conversation the customer abandoned appears in the statistics as a resolved conversation.
This is the most valuable list you’ll get. It tells you exactly which knowledge articles are missing, in your customers’ own words. Review it regularly.
The overall figure is one thing; where it happens is another. If the handoffs are concentrated around a single topic, that is where the work needs to be done.
A simple question after the conversation captures what the numbers miss. An agent may have a high resolution rate and still be perceived as an obstacle.
Ask a question or share what helped you.
Share a question or reflection.
Be the first to contribute.