Researchers at the Graduate School of Frontier Biosciences in Osaka University, Japan, were able to reconstruct high-resolution and highly accurate images from brain activity by using the popular Stable Diffusion model.paper that was published in DecembeThe scientists presented individuals with a set of images and took fMRI scans of their brains while they concentrated on the image.
The team used a combination of the fMRI image output and Semantic Decoder to create the resulting image. However, they found that the addition of Stable Difusion to the process enabled the final generated images to look more similar to the original images that the participants were shown. In the figures above, the images on the top red rows show the original pictures that were presented to the individuals in the study. Meanwhile, the images on the bottom gray rows are the images reconstructed from human brain activity using Stable Diffusion, Semantic Decoder, and fMRI image generation., the team of researchers says that they first predicted a latent representation, which is a model of the image’s data, from fMRI signals.
Then, the model was processed and “noise” was added to it through the diffusion process. Finally, the researchers decoded text representations from fMRI signals within the higher visual cortex and used them as input to produce a final constructed image. The study is the latest example of how scientists are attempting to discover how AI models can work with the human brain to recreate images.