Artificial-intelligence research has been transformed by machine-learning systems called neural networks, which learn how to perform tasks by analyzing huge volumes of training data.
During training, a neural net continually readjusts thousands of internal parameters until it can reliably perform some task, such as identifying objects in digital images or translating text from one language to another. But on their own, the final values of those parameters say very little about how the neural net does what it does.
Understanding what neural networks are doing can help researchers improve their performance and transfer their insights to other applications, and computer scientists have recently developed some clever techniques for divining the computations of particular neural networks.
But, at the 2017 Conference on Empirical Methods on Natural Language Processing starting this week, researchers from MIT’s Computer Science and Artificial Intelligence Laboratory are presenting a new general-purpose technique for making sense of neural networks that are trained to perform natural-language-processing tasks, in which computers attempt to interpret freeform texts written in ordinary, or “natural,” language (as opposed to a structured language, such as a database-query language).
The technique applies to any system that takes text as input and produces strings of symbols as output, such as an automatic translator. And because its analysis results from varying inputs and examining the effects on outputs, it can work with online natural-language-processing services, without access to the underlying software.
In fact, the technique works with any black-box text-processing system, regardless of its internal machinery. In their experiments, the researchers show that the technique can identify idiosyncrasies in the work of human translators, too.
The technique is analogous to one that has been used to analyze neural networks trained to perform computer vision tasks, such as object recognition. Software that systematically perturbs — or varies — different parts of an image and resubmits the image to an object recognizer can identify which image features lead to which classifications. But adapting that approach to Natural Language Processing isn’t straightforward.
“What does it even mean to perturb a sentence semantically?” asks Tommi Jaakkola, the Thomas Siebel Professor of electrical engineering and Computer Science at MIT and one of the new paper’s two authors. “I can’t just do a simple randomization. And what you are predicting is now a more complex object, like a sentence, so what does it mean to give an explanation?”
Somewhat ironically, to generate test sentences to feed to black-box neural nets, Jaakkola and David Alvarez-Melis, an MIT graduate student in electrical engineering and computer science and first author on the new paper, use a black-box neural net.
They begin by training a network to both compress and decompress natural sentences — to create some intermediate, compact digital representation of the sentence and then try to re-expand it into its original form.
