In Smartly.AI, the task of predicting which of intent corresponds best to a user requests is made possible using a statistical classification algorithm, also called the Classifier.
Statistical Classification is the task of identifying to which of a set of categories (sub-populations) a new observation belongs, on the basis of a training set of data containing observations (or instances) whose category membership is known.
Classification has many applications such as computer vision, medical image analysis, speech analysis, handwriting recognition and of course natural language processing.
In the example below, the classifier gets many various user requests as inputs, and we can see that the input is either a match with an existing intent or no intent at all.
A simple four classes classifier
In our implementation witch use Rasa Supervised Embeddings, each sentence is seen as a vector with an indication of how often distinct words of your training data appear in a message. In the illustration below you can see how the count vectors would differ for the sentences My bot is the best bot and
My bot is great, e.g. bot appears twice in
My bot is the best bot.
Count vector featurizer (Credits Rasa)
Our Classifier needs more training data
As this classifier trains word embeddings from scratch, it needs a little more training data and more training time but will generally perform better than pretrained models.
During the training phase, the classifier learns separate embeddings for feature and intent vectors.
During the inference phase, the classifier measure the vector distance between the embedded features and the embedded intent labels using cosine similarity to define which is intent is the best candidate.
Cosine similarity between trained intents and a user request
Below is a simplified representation of a word vector space (it should have much more dimensions in reality). Anyway, what you can see is that by measuring the distance between vectors we can see how similar a user request is from the pre-trained intents, and by doing so, transforming a semantic problem to a mathematical one.
With each inference, the classifier will give you score that is a good indication of the probability of confidence of the result. This score ranges between 0% and 100%.
In order to let you filter the most certain inferences, our platform allows you to define a confidence threshold.
What happens if the threshold is too low?
If the threshold is too low, you will almost always have a result, but these result can be false. This kind of results is called
What happens if the threshold is too high?
If the threshold is too high, you will have less results, but they will be almost always true. This kind of results is called
Setting the right confidence threshold
By default, all new intents come with a confidence threshold of 70% which is a quite good compromise in our experience and a good value to start with. Once you have more data and better results, you can put the bar higher between 80 and 90%.
For more tips on how to design your intents, please check out the section below.