In 2019, I worked at an early-stage startup that focused on customer support quality. We used machine learning (it was ML at the time and not GenAI) to analyze text customer support tickets of our customers (that included a $1B fitness startup) and evaluate them on quality criteria such as empathy, friendliness, helpfulness.
Before our solution, this entire process was carried out manually, if at all. Reviewers could review only 1% of tickets, leading to sampling error and bias.
In addition to the evaluation on quality criteria, we also calculated process metrics like First Contact Resolution, Time to First Response, and its impact on end quality metrics like CSAT, NPS, and Customer Effort Score and found a correlation.
The analysis led to better and more frequent feedback to the customer support team as a result improved the customer support quality.
For one of our customers (the $1B fitness startup) we were able to analyze 30K monthly support tickets increasing quality by 15% and resulting in $400K cost savings.
The Role of LLMs in Customer Support
Fast forward to 2024. In the age of LLMs that are real-time, cost-efficient, and cheap, I believe most customer support should be handled almost automatically using LLMs.
The characteristics of LLMs that include large context windows, function calling, and retrieval augmented generation (RAG) together can tackle a lot of use cases. Let’s take a look at each.
Large context windows can ensure that conversation history is maintained and the customer doesn’t have to repeat themselves across sessions.
Function calling can look at the customer's ID and check the systems to determine if they have a pending order and reliably retrieve the information from the database.
Retrieval augmented generation can pass through support docs and documentation to provide an answer with citations, ensuring authenticity and limiting hallucinations.
Using a prompt-engineered version of an LLM that caters specifically to the support use case is crucial. The prompt can include things a support agent should do, such as asking clarifying questions, empathizing with the problem, being polite, and providing a resolution.
I believe we have reached a stage where a large percentage of requests can be handled by a bot. For instance, when you ask Swiggy (a food ordering app in India) where your order is, it calls the delivery database, checks the expected delivery time, and provides the result along with an option to call the rider. This can be improved by making it more empathetic and human, checking order status, and offering to ping the rider if needed.
There are many benefits to using LLMs in customer support, like they can run 24/7 without requiring breaks or payment. But let’s look at some of the challenges.
Practical Implementation and Challenges
A sufficient trial period is needed to identify jailbreaks, edge cases, and determine when the system provides reliable solutions. For example, an LLM should be prohibited from writing code or going off-topic and to focus solely on the customer support use case, which can be achieved through prompt engineering - chain of thought reasoning, and few shot examples.
Another implementation is not offering an unbounded chat interface, but preset buttons for the most common support queries. While offering some flexibility, in case their question is not in the list.
Customer Support as a Driver of Product Improvement
From a product manager's perspective, customer support is also a source of product features. It helps identify problems people are facing with the product. Customer support is not just about solving requests; it's about addressing the root causes.
The AI can also take these requests, tag them, and write crisp problem statements, write specs or acceptance criteria for the devs to work on with the PM's blessing, and execute it that way.
Build vs Buy
Depending on the size of the company, they might decide to buy or build. A large large company might want create their own versions of this, especially if they’re offering some kind of service with a large customer support volume. Think DoorDash or Swiggy will decide to build this in house.
But startups might opt for off-the-shelf products since customer support is not core to their offering. Intercom has aligned itself with AI with their AI chatbot offering ‘Fin’ which solves up to 50% of support questions instantly while legacy platforms like Zendesk have also shipped AI features. There is space for a new platform that can take support requests and perform all the mentioned tasks.
Additional Considerations
Additional tasks include collecting objective and subjective feedback from the user in the form of CSAT, NPS, or other metrics and surveys. The text component can then be summarized and tagged and put into a system for it to make sense.
Correlation should be drawn between customer support quality and end business metrics like revenue, profitability, and retention. Especially for a subscription business, where churn is an issue, customer support becomes a profit driver rather than just a cost center.
Personal Reflections and Learnings
In my previous startup experience, as a product lead, I stayed close to the user by handling customer support on Twitter and Discord. Promptness was key and handling dissatisfied users involved being empathetic to their requests, understanding the root cause, and providing them with a resolution. If it was something that we previously encountered, I shared the existing support docs. If it was something new and happened a few times, I created new support doc so that we could share.
But if it were a feature request, then I tried to understand the motivation behind the request first and then put it into the queue for consideration. Not all requests were implemented of course, but the feedback is always useful.
Final Thoughts
The adoption of Large Language Models in customer support represents a significant leap toward more efficient and empathetic customer interactions. As we have explored, LLMs offer the potential to handle a substantial portion of routine inquiries autonomously, allowing human agents to address more complex and nuanced issues. While the transition incorporates challenges, particularly in accuracy and reliability, the benefits in cost savings and improved customer satisfaction are compelling. For businesses, staying ahead in customer support technology is no longer just an option but a necessity to thrive in a competitive landscape.