How Google is making music with artificial intelligence

Can computers be creative? That’s a question bordering on the philosophical, but artificial intelligence (AI) can certainly make music and artwork that people find pleasing. Last year, Google launched Magenta, a research project aimed at pushing the limits of what AI can do in the arts. Science spoke with Douglas Eck, the team’s lead in San Francisco, California, about the past, present, and future of creative AI. This interview has been edited for brevity and clarity.

A: Learning is the key. We’re not spending any effort on classical AI approaches, which build intelligence using rules. We’ve tried lots of different machine learning techniques, including recurrent neural networks, convolutional neural networks, variational methods, adversarial training methods, and reinforcement learning. Explaining all of those buzzwords is too much for a short answer. What I can say is that they’re all different techniques for learning by example to generate something new. 

Q: What examples does Magenta learn from?

A: We trained the NSynth algorithm, which uses neural networks to synthesize new sounds, on notes generated by different instruments. The SketchRNN algorithm was trained on millions of drawings from our Quick, Draw! game. Our most recent music algorithm, Performance RNN was trained on classical piano performances captured on a modern player piano. I'd like musicians to be able to easily train models on their own musical creations, then have fun with the resulting music, further improving it.

Q: How has computer composition changed over the years?

A: Currently the focus is on algorithms which learn by example, i.e., machine learning, instead of using hard-coded rules. I also think there’s been increased focus on using computers as assistants for human creativity rather than as a replacement technology, such as our work and Sony’s “Daddy’s Car” [a computer-composed song inspired by The Beatles and fleshed out by a human producer].

Q: Do the results of computer-generated music ever surprise you?

A: Yeah. All the time. I was really surprised at how expressive the short compositions were from Ian Simon and Sageev Oore’s recent Performance RNN algorithm.