Below is a description of our AI pipeline.

1/ Tokenization

Tokenization is where we split an input string into different atomic parts, usually words.
One of the simplest form of tokenization can be defined as separation by spaces:
“This is simple.” -> [“This”, “is”, “simple.”].

This task can be quite complex with languages where back spaces are not here to help:

2/ Typo Correction

Typo Correction is simply detecting and providing suggestions for incorrectly spelled words in a text. Our model generates the candidates by splitting the word or deleting, replacing, inserting, transposing letters in the word. It also uses double variations for each word like interesting one letter and deleting another one. For more on this check the dedicated Typo Correction section.

3/ Entity Extraction

Entity extraction is using two different approaches:

a) Custom entity extraction: Extracts a relevant information from a text, this information being learned to the bot by a list of element and their synonyms that you have provided.

b) System entity extraction: Extracts time, numbers, ect. Full list available here. This information is provided and trained by Smartly.AI for all the users of the platform.

4/ Vectorization

Because Natural Language Processing is a lot of Maths, we use Vectorization to transform the user request to a mathematical object.
For example, if our dictionary contains the words {Smartly.AI, is, the, not, great}, and we want to vectorize the text “Smartly.AI is great”, we would have the following vector: (1, 1, 0, 0, 1).
It's a bit more complex, but you got the idea 😉

5/ Intent Detection

One of the key component of the platform!
Basically, it compares the input vector to a set of pre-trained vectors (intents) to find the one that is the most similar. The result of this operation is the best intent candidate and a confidence score.

6/ Sentiment Analysis

Extracts sentiment polarity (positive or negative) from user input.
For more on this, please visit the Sentiment Analysis section.

7/ Pick up the best bot (Master Bot)

If you are using a Master Bot, the dialogue engine will decide to witch bot the user request should be delegated and when to switch to another one.

8/ Contextualization

When we receive a user request, before processing it, we retrieve all the context.
Some elements of this context are:

  • last active state,
  • last user request / bot answer
  • short term memory,
  • long term memory,
  • user data ( if for example the user is coming from Messenger )

9/ Dialog flow processing

Once we have an understanding of what the user wants and some elements of context, the dialog engine takes all this and navigate in the dialog flow to find the next best dialog state to activate.

10/ Client code processing

If the answer requires some custom code execution, a docker is allocated in our infrastructure for this code to be executed. This can be executed in a time 90 seconds time slots

11/ Answer generation

The answer generation prepares the response of the bot according to what you have defined in your dialog flow, the context and your custom code.

12/ Logging

All this exchange is logged in the platform database so you can monitor and improve your bot later.