Opus, Haptik Advance Intelligence Satisfaction Score as an IVA Metric
Chatbots and virtual assistants have profoundly changed how companies measure their customer care efforts.
While core metrics have proven useful both to justify procurement and implementation of intelligent virtual assistants (IVA), according to an Opus Research webinar that featured insights from a new report from Haptik, a new metric, called the Intelligence Satisfaction Score, might be more effective in delving into the emerging frameworks to measure the effectiveness of intelligent virtual assistants
"This is something we should address early and often," said Dan Miller, lead analyst and founder of Opus Research, pointing to the need to determine how well conversational user interface, speech analytics, and text analytics work together to provide conversational intelligence.
"There are a lot of projects out there into what makes a good personal assistant, but a lot of the work is being done in silos," Miller said, adding that IVAs themselves are moving from the early adopters to the early majority.
"Spending has been growing exponentially and is in the billions," Miller said.
Among geographies, North America is dominating, with applications in banking, finance, and ecommerce. IVAs are popular because they help business units get their work done.
But not all IVAs are the same. To determine just how effective they are for the companies that employ them, Haptik developed the intelligent satisfaction score, said Aakrit Vaish, the company’s co-founder and CEO.
Vaish pointed out that there have been several metrics that have historically been used to measure IVA effectiveness, including time to resolution (TTR), customer effort in finding product information (CES), customer satisfaction (CSAT) and net promoter score (NPS). While they all are proven, essential metrics, they are solely focused on measuring customer experience (CX), which is influenced by a number of other factors. As a result, these metrics neglect what should be the core performance indicator of an IVA – its intelligence.
To measure the intelligence of an IVA, a company will want to look at several items that are important to the different business units, including time to resolution, time spent on website/store ratings, customer effort score (for overall CX), and lead conversion, to name a few.
Yet, the metrics that are most widely used by brands are CSAT and NPS, while sentiment analysis is also emerging as a powerful measurement tool.
The idea behind sentiment analysis is to link key phrases with specific emotions, which is largely an automated process, according to Vaish. However, it is still limited, as the usage of language and comprehension varies based on geography, audience, and other factors. This sometimes results in the sentiments being captured inappropriately. For instance, sentiment analysis might not capture certain nuances of language,such as sarcasm and jokes, leading to an inaccurate reflection of an IVA’s performance.
For an intelligent satisfaction score (ISat), companies need to measure how well their IVA understands the message of the user (natural language understanding), recognizes the intent of the user (intent detection), and processes its responses appropriately (natural language processing). The score is based on how successfully each of these elements works to enable the user to accurately determine the effectiveness of IVAs, according to Vaish.