Natural Language Processing: Turning Words Into Data

Natural Language Processing (NLP) has long been one of the holy grails of computer science. While we all know that computers are better than humans at making sense of highly structured information, there are still some important areas where humans are undeniably better than machines. Understanding language is one of those areas.

For humans, understanding language is so natural we usually don’t even have to make a conscious effort to do it. In reality, though, processing language and turning it into meaningful information is an extremely complex and difficult task. Without consciously thinking about it, we correct grammar mistakes, resolve ambiguities, and infer meaning that isn’t explicitly stated.

Teaching computers to perform these tasks (even imperfectly) has huge implications in many areas of our lives, from the way we design products to the way we research cures for diseases to the way we get directions. In this article, we’re going to explore what Natural Language Processing is, how it works, and how it’s being paired with Big Data to solve problems in a wide array of fields.

What Is NLP?
Many of us already encounter NLP in our daily lives. It’s the technology that allows us to ask our smartphones for directions or help recognizing the song playing on the radio. It’s also the technology that powers the automated call centers we often reach when calling customer service .

The key to Natural Language Processing is taking data as complex and context-dependent as human language and translating it into the kind of structure that a computer can understand and act upon. But how do you do that? The earliest efforts at teaching computers how to understand human language looked a lot like a language class: Scientists tried to teach computers how language worked by explicitly teaching it the rules of grammar and syntax. But the way people actually speak and use language often doesn’t follow the rules. Misspellings, idioms, slang, and common grammatical errors may not prevent a human from understanding the meaning of a text, but computers aren’t able to understand when the rules aren’t followed to the letter.

This has changed with the advent of machine learning . Machine learning refers to the use of a combination of real-world and human-supplied characteristics (called “features”) to train computers to identify patterns and make predictions. In the case of NLP, using a real-world data set lets the computer and machine learning expert create algorithms that better capture how language is actually used in the real world, rather than on how the rules of syntax and grammar say it should be used. This allows computers to devise more sophisticated—and more accurate—models than would be possible solely using a static set of instructions from human developers.

For example, a typical NLP task might involve identifying the names of people in Facebook posts. The first step of the process is feature extraction, which involves identifying meaningful characteristics of something that set it apart from something else. To do this, we’d start with a training set of real Facebook posts. We might say that a name usually begins with a capital letter and is likely to be found in a book of baby names. Using these feature vectors, we would train the computer to recognize first names, taking all of our different features into account. (For instance, if the name “jason” appears uncapitalized, the computer might still recognize it as a name because it appears in our book of baby names.) Then, using a different set of Facebook posts, we’d test our computer’s model. If it successfully distinguishes names from non-names, we’ve built a successful model.

NLP and Big Data
An important part of the Big Data revolution has been the rise in the use of unstructured data.