Should Machines Be Quality-Scoring Machines?

I’ll often get asked about the topics that are top of mind for Forrester’s clients. And sure, there are some topics that are what I’d call “evergreen”—like how to improve agent efficiency with artificial intelligence. Classic.

But there are topics that shift in and out depending on the season. And right now? We are in a season of automated quality scoring (or “quality AI”).

I’m never quite sure what drives some topics in and out of vogue. But if I was a betting lady (I’m not, so don’t come collecting) I’d say that the renewed interest has two sources:

Spillover from the “generative AI halo effect.” All the buzz around genAI has raised awareness for anything and everything related to AI. Executives are generally more open to exploring AI-powered capabilities, and the contact center continues to be in the crosshairs of AI-ification.

The pressure to optimize human agent performance in the age of AI agents. For most organizations, a meaningful transition to AI-led customer service has yet to fully materialize. But man, oh man…people are very interested in getting ready for that eventuality. Once AI takes over a bigger share of the volume, humans will need to perform at a higher level and therefore we will need to equip them with feedback and coaching. The most effective way to do this is by leveraging AI scoring.

But there’s something a little different this time around.

I’ve started noticing an emerging narrative for using quality AI to…score the AI. Interesting.

Now, a cynical person might say, “Hmm, that sounds an awful lot like vendors trying to maintain relevance in a world of reduced agent seats.”

And that person might have a point. But we’re not cynical people, you and me. So let’s say there is something there. What would quality AI scoring for AI-led conversations need to look like?

I’ll tell you what it shouldn’t look like: traditional human agent quality scoring (automated or otherwise).

I simply do not believe that you can (or should) airdrop AI-led conversations into your current automated quality-scoring workflow and expect the outcome to make much sense.

I’ll give you three reasons why they should be treated differently:

1. Because you’re focusing on systemic design, not individualized performance. The typical quality program for human agents aims to evaluate an individual agent’s skills and adherence to policies. When quality scoring an AI model, we need to focus on identifying undesirable outputs that indicate issues with the AI’s training data, prompting strategies, or safety guardrails. We’re looking to improve the system, not coach the individual.

2. Because the goal is real-time optimization, not delayed feedback loops. AI systems can be monitored and adjusted in near real time, and a quality scoring system should leverage this to trigger immediate adjustments to the AI’s parameters, routing, knowledge base, and so on. This level of rapid iteration is a fundamentally different model than queuing up feedback for periodic coaching sessions with human agents. Neither option is inherently superior—but both must be designed for the task at hand.

3. Because you’re evaluating the outcome and the rationale. Traditional quality scoring looks at the outcome of the interaction. Did the agent follow the process? Did the customer’s issue get resolved? With AI, we need to consider how the AI arrived at a particular response. The other day I was speaking with a vendor whose AI solution allows users to examine not just whether the AI model was correct, but whether it was correct for the right reason.

There is something that holds true when evaluating interaction quality for both human- and AI-led interactions, though. The ultimate goal of the contact center remains to provide a positive and effective experience that leads to the desired outcome—issue resolution, a purchase, whatever. So you’ll still want to maintain metrics that measure customer success, regardless of who (or what) is helping to achieve it.

Christina McAllister is senior analyst, Forrester Research, covering customer service and contact center technology, strategy, and operations.

Related Articles

Who Are the Agents of Tomorrow?

In an automated contact center, humans will have a role to play. Time to figure out what that is.

A Backstage Pass to Technology Evaluations

New year, new…set of enterprise initiatives.

Some (Near) Year-End Customer Service Reflections

'Tis the season to not leave money on the table.