Apr 17, 2020
We live in an incredible world of multilingualism. More and more people are starting to travel and learn new languages or live in foreign countries. Also, in many working environments multilingualism plays a big role. The probability that your clients are on the other side of the globe could be the same as the probability of them being in the same city as you. Erasing language barriers brings many opportunities, but it can be difficult.
Fortunately, for a conversational AI, learning new languages does not imply as much hardship as it does for most people. You can easily make your AI understand more languages by providing enough data, which makes it very useful in any multilingual environment. It's always nice to communicate in your mother tongue, especially in situations where you don’t expect it to be an option. Alternatively, you could translate the user input and bot output in real time (if you can live with the quality of the translation).
However, the possibility of talking in many languages comes with the responsibility of detecting the right one. It may not feel like such a big issue, but imagine a conversational AI being fed with data in more than ten different languages. Some of them may be linguistically related, some may have borrowings from others. Unfortunately, there is no accent the AI may investigate for clues. The pure text is the only source of information it has and usually the text consists of only few words. Suddenly, language detection becomes a much more difficult task than it might have seemed at the beginning. Which is why many attempts to create the best language detector have been made.
Conveniently, Cognigy.AI provides a Custom Module for language detection. The original code was trained on texts from Wikipedia using naive Bayes with character n-gram. During the testing on news articles, it detected languages correctly with over 99% precision. The library supports 49 languages.
If you add the Language node to your flow, you will see in the Cognigy Context or Input Object which languages were identified by the model and with which probability.
With that information you can adapt your conversational AI to the preferred language or prepare answers in languages your AI cannot work with. There are lots of possibilities, so give it a try!
