The Chatbot checklist.

Things to evaluate when evaluating a chatbot provider

The Chatbot evaluation checklist

With the growing popularity of automation and AI chatbots, more and more businesses are interested in developing a chatbot of their own.


Searching for a reliable software vendor can be challenging. Thus, we have listed key aspects that you should consider when choosing a chatbot provider.


Implementing a conversational assistance solution is as much a business decision as a tech decision. This decision impacts key business metrics such as conversions, sales, customer experience, productivity and revenue through AI and smart automations. 


The main objective of automating simple and repetitive requests is to scale the business without adding costs, free up agents' time to respond to more complex tasks. This allows 24/7 availability allowing for extended support outside of call centre hours, an almost instantaneous response without the need to wait for the availability of an agent, and create a delightful experience for existing and prospective customers.


Constructing a decision matrix is the best method to help select a software program. It consists of criteria weighted by relevancy and a list of solutions to be studied. Each criterion is scored for each solution. The sum of all scores of the weighted criteria provides an evaluation of the solution compared to the others.


What are the different types of chatbot companies?

Businesses need to understand the different types of chatbot softwares and providers and assess their requirements to select the correct partner.


A chatbot development platform is a platform that allows businesses to build a chatbot without the need for developers. When companies subscribe to a chatbot development platform, they can access a DIY platforms that enables them to create the user interface, a knowledge base, and dialogues and allow them to manage integrations with their third-party tech stack. 


At the same time, Natural Language Understanding platforms are linguistic technology platforms that power a chatbot and enable it to answer user queries. They are stand-alone platforms and must be combined with the framework to get a viable product. 


Criteria to select a chatbot


Once businesses identify the different options available, they should consider the following criteria when selecting chatbot providers - 


1) Ease of use of the developing tool


Ease of use is essential when building the chatbot and when managing it or developing new capabilities. A tool that is not user-friendly and difficult to use will lead to difficulties in maintaining and improving the chatbot. The platform's ease of use is an essential factor to consider while selecting a conversational AI chatbot provider. Businesses prefer a UI that is easy enough for anyone in the company to use.  


2) Artificial Intelligence


Chatbots are of two types: rule-based and AI-powered. Even though both conversational AI and regular chatbots aim to engage customers and respond to their queries, the manner of engagement differs significantly. 


Rule-based chatbots work on a bound system and are defined by commands and keywords to interact with customers. If the user doesn’t use one of the specified keywords, the chatbot will not understand the question and therefore not respond accurately. Additionally, they are not self-learning, and hence businesses will need to add new keywords manually. However, they are a good option if the user experience is not the top priority and the chatbots are required to handle only a limited number of questions and connect the user to a live agent. 


On the other hand, the AI-powered chatbot uses AI, ML, Predictive Analysis, and Neural Networks to understand users and respond in a natural language giving the conversation a human feel. Conversational chatbots are powered by AI that understands customer intent, uses past data to understand the context and provides a personalized answer. The conversational chatbot can continue the conversation by interpreting the context and shifting from one channel to another, regardless of where the user starts a conversation. Selecting a conversational AI over rule-based chatbots has another advantage. 


The AI chatbot can hand over the chat to an agent to answer the question based on the context. It ensures that the customer does not repeat their question and the agents get all the information they need. 


3) Security, Speed, and Robustness


Businesses cannot compromise on security as the customers share a lot of information with the company while conversing on the chatbot. Security is vital for businesses across industries. Banking, e-commerce, real estate, ed-tech, travel, and insurance. Therefore, it is paramount for companies to use a platform that offers the best security to ensure the data collected is safe. It is necessary to ensure that information is encrypted and GDPR and other guidelines are maintained.


Speed is another crucial criterion as businesses require a platform that provides users get their answers instantaneously regardless of traffic.


4) Dashboards and Actionable Statistics


A platform that offers dashboards and statistics will enable businesses to monitor the chatbot KPIs and improve efficiency. Conversational AI is among the most potent behavioral analysis tools when extrapolated correctly. Simple decision-tree heatmaps tied to complex keyword mapping allow brands to have detailed insights into consumer behavior. This helps businesses identify general trends or dig deep to observe the finer details. When selecting a conversational AI chatbot provider, companies must ask prospective providers to send dashboards templates that can be used to monitor the KPIs.


5) Natural Language Efficiency


Natural language efficiency refers to how the system matches answers with questions.

Sentiment analysis is also another factor that affects the ability of the conversational AI system to understand conversations, dialects, slang and context.


For instance :

A user says “My experience was not bad”.

This is usually a positive or neutral sentiment.


A smart NLP engine will understand the context and sentiment around this sentence and classify it as a neutral or positive experience.


However, a subpar NLP engine or a decision tree bot would pick up the Keyword “bad” and classify this as a negative experience and take the conversation in a different direction.


6) Number of Languages Supported


This ensures that the chatbot can answer queries in a range of languages and can be implemented on multilingual websites. Maintaining a language is relatively expensive since a complete set of data has to be optimized for each language, such as the different questions in the conversation tree and the answers' translation. Therefore, businesses should identify the languages used by their clientele to calculate the percentage of use of each of these languages. In case of low volume, your brand can calculate the viability of the associated cost and whether to replace it with an alternative language or maintain the current support for the language. 


7) Response Time, Scalability & Personalization


Chatbot response time is paramount. It may be possible that initially, the chatbot may have to manage only a couple of hundreds or thousands of user requests per month. Still, businesses would want to be sure that the technology that powers the chatbot can handle a higher number of user queries if needed. The two types of volumetry to consider are - 

  • the volume of conversations, i.e., the number of potential users and the number of requests made per month,
  • the volume of managed intentions, i.e., the number of questions the chatbot will have to manage.


Many chatbots are easy to use for a few scenarios but cannot scale up. Scalability is another important criterion for companies. While most chatbots can handle multiple queries simultaneously, only a few can handle them without crashing the server or showing a lag in response. While scalability is vital to ensure the customer experience is not affected, it’s also crucial to add personalization to the customer interaction, such as having previous chat history, speaking in the customer’s preferred language, or sharing recommendations based on their past behavior. There’s no easy way to determine whether a chatbot provider can handle the scale. Thus, while selecting a conversational AI chatbot provider, businesses should inform the provider of the message load per hour and how many conversations the chatbot can handle in a specific time frame. Finally, companies should ask about scale-up plans to understand if the vendor can keep up with business growth.


8) Understanding Business Specificities & Requirements


Businesses would want their potential chatbot developer to have a clear understanding of the specific use of the chatbot. This will ensure that the developer knows the utility of the chatbot concerning user expectations and best practices in the industry.


9) Reliability and Uptime.


Businesses prefer a reliable provider who can deliver what has been promised on schedule.


10) Documentation and maintenance


Businesses should ensure that the chatbot developer has good documentation practices and a low response time for maintenance requests. Clear documentation will help businesses understand the different functionalities of the chatbot and how they work. Enterprises want the provider to be available to fix a bug without waiting for long periods after raising maintenance requests.


11) Agent Assist 


A rule-based chatbot only answers customer queries, but a conversational AI chatbot also assists the agents in answering questions improving their productivity. Agents are crucial for customer support. While a conversational chatbot can resolve frequently asked questions, agents resolve complex queries that require special attention. When a chatbot escalates a query to the agents, they need the context and relevant information to answer the customer's queries. 


Conversational chatbots may help agents in a variety of ways -

  • Previous chat history records
  • Quick answers to frequently asked questions
  • Ability to take notes, add tags, and more before closing the conversation
  • Smart-plugins to show customer profiles with location, language preferences, and purchase history
  • Context of the current chat to continue the conversation


With an agent assist feature, the customer experience is multiplied. It allows agents to resolve queries accurately and reduces the average ticket handling time. It saves their time and enables them to take up more tasks that increase company revenues. 


12) Available Integrations 


Businesses want a chatbot that can seamlessly integrate with existing tools and software ecosystems rather than having to adapt existing tools to the new chatbot. Therefore it is vital to check that the chatbot has existing integrations with the current apps being used by the business and can connect with third-party applications for:

  • Identification and authentication systems
  • Support related systems such as Livechat, messaging system, ITSM (IT Service Management)
  • Systems linked to the business contexts covered like ERP (Enterprise Resources Planning), DMS (Document Management System), CRM (Customer Relationship Management), CMS (Content Management System), HR (Human Resources) software, banking software, etc
  • RPA (Robotic Process Automation) systems execute manual actions on applications.


Integrations and delightful consumer experiences go hand in hand. A brand has multiple customer touchpoints and uses many software providers to track, engage and service its customers. If a conversational AI platform doesn’t operate in sync with these services, it will cost the brand time, money, and resources. Therefore brands must ensure that the chatbot integrates with an expansive suite of tools. Additionally, the brand must enquire about future integrations, timelines, and how receptive the provider is to custom requests.


13) Pricing


Pricing is among the most critical aspects of the decision-making process. Different vendors use different pricing strategies based on a variety of factors. Businesses must remember that chatbot pricing has two elements – a flat fare + surcharges for add-ons. There are three primary methodologies to calculate pricing for a conversational AI chatbot.

  • Cost per message - ideal for easy-to-close interactions. 


The cost per message is the easy one here. A message is a string of alphanumeric variables exchanged between the brand and the end-user. Brands are charged a certain amount for each message, and that figure is usually in cents or a similar nominal denomination.

  • Cost per chat/conversation - ideal for interactions with lots of back-and-forths.


A conversation is a collection of messages. It doesn't matter how many messages are in a conversation. The cost per conversation charges the business for the number of conversations they have in a specific timeframe. This type of pricing is usually tiered.

  • Cost per customer/contact/user - ideal for a small, dedicated user base.
  • Cost per customer is a bit tricky. Often used by enterprise-style chatbot providers, cost per customer means that the business pays for each unique customer. 


Although vendors may use other synonyms, they're probably using one of these umbrella terms. In addition to this, the service provider may charge the business on the number of agents using the Chabot. 


14) Testimonials and Reviews


Finally, it is advised that businesses check the reputation of the chatbot provider by researching their clientele, company reviews, product reviews, and whether the provider has a stable working relationship with its clients, and develops new products for the same client again. These elements will help businesses make informed decisions and avoid surprises after the deal is closed.


15) Area of Expertise


This criterion is among the most critical and challenging to value. A change in the conversational assistance solutions market has been taking place over the last few years with the appearance of solutions dedicated to a specific business domain such as banking, insurance, human resources, and e-commerce. These solutions come pre-configured with a corpus of intentions, entities, dialogues, and responses, enabling a reduction in the creative effort, thus reducing costs and accelerating the deployment of the chatbot. In highly regulated and legislated areas, the advantage brought by these solutions is undeniable and will save time. This advantage is much weaker in more open and varying domains and should not be decisive for the final choice.

Therefore, a generalist solution should be adapted to all configurations but will require additional configuration work.


16) Complexity of Dialogues


Defining how the chatbot will respond when designing response scenarios is essential. The chatbot can provide a simple answer or a complex dialogue to ask sequences of questions and search for information in third-party applications. A static or dynamic FAQ solution would be sufficient for the first case. In the second case, the solution to be selected should allow for complex dialogues and integration with third-party applications. 


17) User Interface


The chatbot's user interface is the channel through which the requester can interact with it. This interface can be :

  • An avatar that simulates an agent's face can express emotions related to the conversation. Communication is done through speech and visuals. 
  • A telephone line (call bot or phone bot). The communication is carried out through the voice channel and can be completed by sending elements through email or SMS.
  • A social network. The user uses the capabilities of the social network to communicate with the chatbot.
  • A dialogue window (or pop-up also called webchat). The exchanges take place via a small window included on the company's website.


18) Omnichannel Experience 


The customer experience is different between omnichannel and multichannel. While considering a conversational AI chatbot provider, it is essential to determine the customer experience provided by the chatbot.  

In a multichannel experience, the chatbot is present on multiple channels to engage with the customers, such as the website, mobile app, social media pages, or instant messaging apps. However, these numerous touchpoints do not fetch the context from other channels. 

In an omnichannel experience, the chatbot platform syncs data with other channels and understands the context from a previous interaction on a different medium. Thus, leading to the continuity of the conversation. 

Both systems work differently and serve different purposes. Hence, businesses need to define the chatbot's purpose before selecting a conversational AI chatbot provider.  


Now that we have understood the various elements to evaluate when considering a chatbot provider, let’s look at the various tools, techniques and metrics to consider when you are testing your Conversational AI solution or Chatbot platform.




Get A Demo
Loading...

Thank you! Our team will get in touch shortly..