Cognigy Named a Leader in January 2022 Gartner® Magic Quadrant ™ for Enterprise Conversational AI Platforms! Download the report
Easy-to-use platform even for non-techies
Understand what your customers say in any language
Connect to your digital ecosystem with free Cognigy.AI Extensions
Integrate with any CRMs, ERPs, RPA, and more
Enhance customer interactions with AI on the frontline
Connect customers to the right agent on the first try
Give your agents the confidence to solve complex requests
Boost your customers’ experience and agents’ efficiency
Improve customer satisfaction with detailed insights
Engage with your customers on the platform of their choice
Drive appropriate actions with intelligent processes
Integrate easily with existing enterprise infrastructure
Increase operational efficiency and reduce agent attrition
Dive deep into the world of conversational AI
Learn everything about voice bots and virtual agents
Become an expert in Conversational AI and automation
See Conversational AI in action through demos
Explore Cognigy.AI and get support from the community
Join our mission to make Conversational AI more accessible
Find us at live conferences or virtual meetups
Help us achieve our vision with your talents and skills
Be the first to know about all the latest news
Reach out to our experts and get your questions answered
See how we help your organization secure sensitive data and comply with applicable laws and regulations.
An in-depth guide into the trusted use of AI in customer service automation
Find out everything you need to know about establishing "Explainable AI" based on the AIC4 criteria catalogue.
Multi-lingual ability in humans is a familiar trait. More than half the world population speaks at least two languages. This allows humans to do amazing transformations, like reading this article in English and then reproducing the gist of it fluently in another language. Quite a surprising and amazing feat if you think about it.
How does this work? When you think of a "quick brown fox jumping over a fence" it's still the same brown fox in your head - no matter if you speak to you in German, Dutch or English. Such human-like multilingual fluency seems far-fetched for machines - but an effective technology for multi-lingual natural language understanding is readily available.
Using Cognigy’s Universal Language model one can create intents and example sentences in English, and then speak to the agent in German, Japanese or Russion and it will still map the input to the correct intent:
Cognigy NLU correctly maps inputs to intents in a variety of input languages (right pane) even when example sentences are only provided in English (middle pane).
How does this work? First, a big disclaimer is in order that theory is lagging behind the practice of modern machine learning. No one claims to fully understand how, why and what is exactly going on in a model architecture that has taken NLU by storm since 2017 called Transformer models. Without going into the details: What follows is a crude attempt at explaining what they do.
If you consider a sentence like the "quick brown fox jumping over a fence" then somehow the whole is more than the sum of its parts. In order to arrive at such a meaningful representation, you take every word in the sentence and put it into context with the other words in the sentence. You throw your words in a blender, that blender is called the “attention mechanism” inside a transformer where every word can "interact" with every other word. To make that blender produce a meaningful sentence soup and not a bunch of gibberish, you train it. For that you ask the machine learning transformer model to always predict the next word in a sentence, and let it run across large text corpus units like Wikipedia, and other meaningful collections that you can get your hands on.
Now with multi-lingual language models, you train on a very large corpus of text in different languages. The way that multi-lingual transformer language models transfer across languages is suggestive of the fact that they really do build up some cross-lingual, more abstract meaningful representations internally. The same intent meaning or concept maps to the same internal representation no matter which language you use. This is also why these models are so useful in helping us predict intents and other natural language tasks in the first place. The result of which is what you see in Cognigy.AI in the above screenshot.
Researchers have begun investigating the phenomenon. Chi, Hewitt, and Manning (2020) for example find evidence to suggest that multi-lingual LMs learn certain linguistic universals. As illustrated in one of their visualizations of clustering of syntactic dependency labels across languages:
t-SNE visualization of head-dependent dependency pairs belonging to selected dependencies in English and French, projected into a syntactic subspace of Multilingual BERT, as learned on English syntax trees. Colors correspond to gold UD dependency type labels.
Source: Ethan A. Chi, John Hewitt, and Christopher D. Mannin: Finding Universal Grammatical Relations in Multilingual BERT
Choenni and Shutova (2020) investigate how multi-lingual sentence representations cluster across layers of increasing depth in multilingual models. This following graphic shows clustering of embedding representations across neural network layers of increasing depth. They are initially well separated by language. In subsequent layers inside the network the representations coalesce into common representations detached from the original language. The separation into languages towards output layers in the second row is due to a different training regime.
t-SNE plots of representations from layers of LASER (top) and M-BERT (bottom), where layer 0 corresponds to the non-contextualized token embeddings.
Source: Rochelle Choenni and Ekaterina Shutova: What does it mean to be language-agnostic? Probing multilingual sentence encoders for typological properties
While multi-lingual models are great, we still recommend for the main languages the virtual agent to use the main language. First of all, some information might get lost in translation so to speak. Well-written German example sentences will generally still outperform intents written in another language for German inputs. While you can also write multi-lingual intents for the universal model there are still practical considerations that yield advantage to the monolingual approach. Knowing many languages comes with a trade-off and a significant overhead. They are more challenging to train, are as a result larger, slower, and may not always benefit from the latest innovations.
Generally, monolingual models will be more optimized for the task at hand, faster, and more performant. It is also easier dealing with intent editing, intent training, etc. in separate locales which are made very easy in Cognigy. For quick time to market and understanding the long tail of less frequently used languages, however, the Universal language model does indeed render localization obsolete.
Stay up to date with the latest Conversational AI tips and news.