Machines learn language better by using a deep understanding of words
Computer systems are getting quite good at understanding what people say, but they also have some major weak spots. Among them is the fact that they have trouble with words that have multiple or complex meanings. A new system called ELMo adds this critical context to words, producing better understanding across the board.
To illustrate the problem, think of the word “queen.” When you and I are talking and I say that word, you know from context whether I’m talking about Queen Elizabeth, or the chess piece, or the matriarch of a hive, or RuPaul’s Drag Race.
This ability of words to have multiple meanings is called polysemy. And really, it’s the rule rather than the exception. Which meaning it is can usually be reliably determined by the phrasing — “God save the queen!” versus “I saved my queen!” — and of course all this informs the topic, the structure of the sentence, whether you’re expected to respond, and so on.
Machine learning systems, however, don’t really have that level of flexibility. The way they tend to represent words is much simpler: it looks at all those different definitions of the word and comes up with a sort of average — a complex representation, to be sure, but not reflective of its true complexity. When it’s critical that the correct meaning of a word gets through, they can’t be relied on.
A new method called ELMo (“Embeddings from Language Models”), however, lets the system handle polysemy with ease; as evidence of its utility, it was awarded best paper honors at NAACL last week. At its heart it uses its training data (a huge collection of text) to determine whether a word has multiple meanings and how those different meanings are signaled in language.
For instance, you could probably tell in my example “queen” sentences above, despite their being very similar, that one was about royalty and the other about a game. That’s because the way they are written contain clues to your own context-detection engine to tell you which queen is which.
Informing a system of these differences can be done by manually annotating the text corpus from which it learns — but who wants to go through millions of words making a note on which queen is which?
“We were looking for a method that would significantly reduce the need for human annotation,” explained Mathew Peters, lead author of the paper. “The goal was to learn as much as we can from unlabeled data.
