Is Deep Learning Approaching a Wall?
Deep learning and related machine learning advances have played a central role in AI’s recent achievements, giving computers the ability to be trained by ingesting and analyzing large amounts of data instead of being explicitly programmed. In just the past two years, Google’s deep-learning-based AlphaGo defeated the world’s top Go players, surprising most AI experts who thought that it would take another 5 to 10 years to achieve such a milestone. Similarly, when Google switched to its new deep learning AI system in late 2016, it achieved an overnight improvement in the quality of its machine translations roughly equal to the total gains that the previous program had accrued over its 10 year lifetime.
As is typically the case with major technology achievements — e.g. the dot-com bubble — deep learning has quickly climbed to the top of Gartner’s hype cycle, where all the excitement and publicity accompanying new, promising technologies often leads to inflated expectations, followed by disillusionment if the technology fails to deliver. AI may be particularly prone to such hype cycles, as the notion of machines achieving or surpassing human levels of intelligence leads to feelings of wonder as well as fear. Over the past several decades, AI has gone through a few such hype cycles, including the so-called AI winter in the 1980s that nearly killed the field.
In a recent article, Deep Learning: A Critical Appraisal, author and NYU professor Gary Marcus offers a serious assessment of deep learning. He argues that, despite its considerable achievements over the past five years, deep learning may well be approaching a wall, an opinion apparently shared by University of Toronto professor Geoffrey Hinton, the so-called Godfather of Deep Learning.
Deep learning is a powerful statistical technique for classifying patterns using large training data sets and multi-layer AI neural networks. It’s essentially a method for machines to learn from data that’s loosely modeled on the way a biological brain learns to solve problems. Each artificial neural unit is connected to many other such units, and the links can be statistically strengthened or decreased based on the data used to train the system. Each successive layer in a multi-layer network uses the output from the previous layer as input.
“In principle, given infinite data, deep learning systems are powerful enough to represent any finite deterministic ‘mapping’ between any given set of inputs and a set of corresponding outputs, though in practice whether they can learn such a mapping depends on many factors…” writes Marcus. “The technique excels at solving closed-end classification problems, in which a wide range of potential signals must be mapped onto a limited number of categories, given that there is enough data available and the test set closely resembles the training set. But deviations from these assumptions can cause problems; deep learning is just a statistical technique, and all statistical techniques suffer with deviation from their assumptions… In practice, results with large data sets are often quite good, on a wide range of potential mappings.”
Marcus’ article discusses ten such challenges faced by current deep learning systems. I’ll focus my discussions on four in particular.
The data requirements for deep learning are substantially different from those of other analytic methods in a number of dimensions. The performance of traditional analytics tends to plateau as the data set size increases. However, the performance of properly trained deep learning techniques will significantly improve as the data sets get larger. Deep learning methods are particularly valuable in extracting patterns from complex, unstructured data, including audio, speech, images and video. To do so, they require thousands of data records for models to become good at classification tasks and millions for them to perform at the level of humans.
