NLU pipelines

What is an NLU Pipeline?

An NLU (Natural Language Understanding) pipeline is a structured framework that processes natural language input through various stages, typically starting from text preprocessing and ending with intent detection and entity extraction. These pipelines are central to chatbots and other conversational agents for understanding user queries and generating appropriate responses.

Given the rapid advancements in NLU technologies, we frequently release updated and improved pipelines featuring different algorithms and methodologies. Recognizing that one size doesn't fit all, we've made it a priority to empower our customers with the choice to select a pipeline that aligns with their specific needs and their readiness to upgrade to newer technologies.

What NLU Pipelines Are Available in Smartly.AI?

Below, you'll find a comprehensive list of NLU models available on Smartly.AI, along with their features and current status.

Available pipelines

Our most robust and actively developped pipeline is SNP 1.06
Therefore, this is the model you should use for all your existing / new bot projects
Older pipelines will be deprecated soon.

How can I choose the NLU pipeline of my bots?

The process is super easy, first open the settings of your bot

Then, in the NLU Algorithm section you can choose the desired NLU model for your chatbot.
If your bot is supporting multi language, you have the option to pick a pipeline for each language

Last but not least, train your bot
Please do it as your bot won't be able to handle any new requests until you train it on the new pipeline.

Here you go! 🚀

What is the best the NLU pipeline / NLU settings for my bots?

As said before, the recommended pipeline for all your existing / new bots is SNP 1.06

The answer for the NLU settings is more nuanced and will depend on your bot.

To help you navigate through the options, we have set a a quick guide to help you select the best NLU pipeline for your bot.

Before diving in the guide, you should have an idea of what mean by bot size.
We consider the number of utterances (total number of sentences in your intents) as the bot size

• Small: Less than 100 utterances
• Medium: Less than 1000 utterances
• Large: Anything bigger than Medium

Here is the guide, we hope that it can be helpful to you configure your SNP 1.06 pipeline.

Language	Bot size	Classifier	Vectorizer	Lemmatization	Stop words filtering	Confidence Threshold
French	Small	Logisitc Regression	TF-IDF			>20%
French	Medium	XG Boost	TF-IDF			>20%
French	Large	XG Boost	Flaubert			>40%
Other	Small	Logisitic Regression	TF-IDF			>20%
Other	Medium	XG Boost	TF-IDF			>20%
Other	Large	XG Boost	FastText			>40%

👍

Note

You can have different configurations for each bot of your project, even within an orchestrator / mini bots architecture.

🚧

SNP 1.06 migration

Note that all bots should have migrated to SNP 1.06 before December 1st 2023.
Also If you where using system entities (number, phones, emails, ect..) in a custom code, you will have to adapt your code to the new system entity syntax. More on this here.