DayDreamer: An algorithm to quickly teach robots new behaviors in the real world
Training robots to complete tasks in the real-world can be a very time-consuming process, which involves building a fast and efficient simulator, performing numerous trials on it, and then transferring the behaviors learned during these trials to the real world. In many cases, however, the performance achieved in simulations does not match the one attained in the real-world, due to unpredictable changes in the environment or task.
Researchers at the University of California, Berkeley (UC Berkeley) have recently developed DayDreamer, a tool that could be used to train robots to complete real-world tasks more effectively. Their approach, introduced in a paper pre-published on arXiv, is based on learning models of the world that allow robots to predict the outcomes of their movements and actions, reducing the need for extensive trial and error training in the real-world.
“We wanted to build robots that continuously learn directly in the real world, without having to create a simulation environment,” Danijar Hafner, one of the researchers who carried out the study, told TechXplore. “We had only learned world models of video games before, so it was super exciting to see that the same algorithm allows robots to quickly learn in the real world, too!”
Using their approach, the researchers were able to efficiently and quickly teach robots to perform specific behaviors in the real world. For instance, they trained a robotic dog to roll off its back, stand up and walk in just one hour.
After it was trained, the team started pushing the robot and found that, within 10 minutes, it was also able to withstand pushes or quickly roll back on its feet. The team also tested their tool on robotic arms, training them to pick up objects and place them in specific places, without telling them where the objects were initially located.
“We saw the robots adapt to changes in lighting conditions, such as shadows moving with the sun over the course of a day,” Hafner said. “Besides learning quickly and continuously in the real world, the same algorithm without any changes worked well across the four different robots and tasks. Thus, we think that world models and online adaptation will play a big role in robotics going forward.”
Computational models based on reinforcement learning can teach robots behaviors over time, by giving them rewards for desirable behavior, such as good object grasping strategies or moving at a suitable velocity. Typically, these models are trained through a lengthy trial and error process, using both simulations that can be sped up and experiments in the real world.
On the other hand, Dreamer, the algorithm developed by Hafner and his colleagues, builds a world model based on its past “experiences.” This world model can then be used to teach robots new behaviors based on “imagined” interactions. This significantly reduces the need for trials in real-world environment, thus substantially speeding up the training process.
“Directly predicting future sensory inputs would be too slow and expensive, especially when large inputs like camera images are involved,” Hafner said. “The world model first learns to encode its sensory inputs at each time step (motor angles, accelerometer measurements, camera images, etc.) into a compact representation. Given a representation and a motor command, it then learns to predict the resulting representation at the next time step.”
The world model produced by Dreamer allows robots to “imagine” future representations instead of processing raw sensory inputs. This in turn allows the model to plan thousands of action sequences in parallel, using a single graphics processing unit (GPU). These “imagined” sequences help to quickly improve the robots’ performance on specific tasks.
