Though it sounds like something out of a science fiction novel, researchers have successfully trained an artificial intelligence system to recreate images people have looked at based on their brain scans. The A.I. generated pictures of objects including a teddy bear, clock tower and airplane after participants viewed similar images.
“The accuracy of this new method is impressive,” says Iris Groen, a neuroscientist at the University of Amsterdam who was not involved in the research, to Science’s Kamal Nahas.
While this brain-scan-to-image A.I. technology is far from ready for public use, researchers say it could someday prove useful for understanding what’s happening inside people’s minds. Once scientists refine the concept a bit more, doctors may eventually be able to use it to help people, such as those suffering from paralysis, to communicate. It might also help neuroscientists interpret dreams or even understand how other species perceive the world around them.
The researchers from Osaka University in Japan are among the ranks of scientists using A.I. to make sense of human brain scans. Their approach to this, however, is the first to use the text-to-image generator Stable Diffusion, which came on the fast-growing A.I. scene in August 2022. Their model is also much simpler, requiring only thousands, instead of millions, of parameters, or values learned during training.
The team shared more details in a new paper, which has not been peer-reviewed, published on the preprint server bioRxiv. They also plan to present their findings at an upcoming computer vision conference, according to Science.
So, how does it all work? Typically, a user inputs a word or phrase that Stable Diffusion—or other similar technologies, such as DALL-E 2 and Midjourney—transforms into an image. This process works because the A.I. technologies have studied lots of existing images and their accompanying text captions—over time, this training allows the technology to identify patterns, which it can then recreate based on a prompt.
The researchers took this training one step further, by teaching an A.I. model to link functional magnetic resonance imaging (fMRI) data with images. More specifically, the researchers used the fMRI scans of four participants who had looked at 10,000 different images of people, landscapes and objects as part of an earlier, unrelated study. They also trained a second A.I. model to link brain activity in fMRI data with text descriptions of the pictures the study participants looked at.
