Rebooting AI: Deep learning, meet knowledge graphs
“This is what we need to do. It’s not popular right now, but this is why the stuff that is popular isn’t working.” That’s a gross oversimplification of what scientist, best-selling author, and entrepreneur Gary Marcus has been saying for a number of years now, but at least it’s one made by himself.
The “popular stuff which is not working” part refers to deep learning, and the “what we need to do” part refers to a more holistic approach to AI. Marcus is not short of ambition; he is set on nothing else but rebooting AI. He is not short of qualifications either. He has been working on figuring out the nature of intelligence, artificial or otherwise, more or less since his childhood.
Questioning deep learning may sound controversial, considering deep learning is seen as the most successful sub-domain in AI at the moment. Marcus on his part has been consistent in his critique. He has published work that highlights how deep learning fails, exemplified by language models such as GPT-2, Meena, and GPT-3.
Marcus has recently published a 60-page long paper titled “The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence.” In this work, Marcus goes beyond critique, putting forward concrete proposals to move AI forward.
As a precursor to Marcus’ upcoming keynote on the future of AI in Knowledge Connexions, ZDNet engaged with him on a wide array of topics. Picking up from where we left off in the first part, today we expand on specific approaches and technologies.
Recently, Geoff Hinton, one of the forefathers of deep learning, claimed that deep learning is going to be able to do everything. Marcus thinks the only way to make progress is to put together building blocks that are there already, but no current AI system combines.
Building block No. 1: A connection to the world of classical AI. Marcus is not suggesting getting rid of deep learning, but using it in conjunction with some of the tools of classical AI. Classical AI is good at representing abstract knowledge, representing sentences or abstractions. The goal is to have hybrid systems that can use perceptual information.
No. 2: We need to have rich ways of specifying knowledge, and we need to have large scale knowledge. Our world is filled with lots of little pieces of knowledge. Deep learning systems mostly aren’t. They’re mostly just filled with correlations between particular things. So we need a lot of knowledge.
No. 3: We need to be able to reason about these things. Let’s say we know physical objects and their position in the world — a cup, for example. The cup contains pencils. Then AI systems need to be able to realize that if we cut a hole in the bottom of the cup, the pencils might fall out. Humans do this kind of reasoning all the time, but current AI systems don’t.
No. 4: We need cognitive models — things inside our brain or inside of computers that tell us about the relations between the entities that we see around us in the world. Marcus points to some systems that can do this some of the time, and why the inferences they can make are far more sophisticated than what deep learning alone is doing.
To us, this looks like a well-rounded proposal. But there has been some pushback, by the likes of Yoshua Bengio no less. Yoshua Bengio, Geoff Hinton, and Yan LeCun are considered the forefathers of deep learning and recently won the Turing Award for their work.
Bengio and Marcus have engaged in a debate, in which Bengio acknowledged some of Marcus’ arguments, while also choosing to draw a metaphorical line in the sand. Marcus mentioned he finds Bengio’s early work on deep learning to be “more on the hype side of the spectrum”:
“I think Bengio took the view that if we had enough data we would solve all the problems. And he now sees that’s not true. In fact, he softened his rhetoric quite a bit.
