Decoding visual stimuli from brain recordings aims to deepen our understanding of the human visual system and build a solid foundation for bridging human and computer vision through the Brain-Computer Interface. However, reconstructing high-quality images with correct semantics from brain recordings is a challenging problem due to the complex underlying representations of brain signals and the scarcity of data annotations. In this work, we present MinD-Vis: Sparse Masked Brain Modeling with Double-Conditioned Latent Diffusion Model for Human Vision Decoding. Firstly, we learn an effective self-supervised representation of fMRI data using mask modeling in a large latent space inspired by the sparse coding of information in the primary visual cortex. Then by augmenting a latent diffusion model with double-conditioning, we show that MinD-Vis can reconstruct highly plausible images with semantically matching details from brain recordings using very few paired annotations. We benchmarked our model qualitatively and quantitatively; the experimental results indicate that our method outperformed state-of-the-art in both semantic mapping (100-way semantic classification) and generation quality (FID) by 66% and 41% respectively. An exhaustive ablation study was also conducted to analyze our framework.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
https://mind-vis.github.io/
https://archive.ph/MgFCv#selection-857.0-857.102
https://www.ted.com/talks/mary_lou_jepsen_how_we_can_use_lig...
https://pubmed.ncbi.nlm.nih.gov/35834524/
https://neurosciencenews.com/imagination-brain-wave-20212/
https://pubmed.ncbi.nlm.nih.gov/30890926/
https://www.wired.com/story/neurograins-could-be-the-next-br...
https://www.researchgate.net/publication/303974201_Neuropriv...
https://www.wired.com/story/a-huge-new-data-set-pushes-the-l...
https://www.wired.com/story/new-brain-implant-translates-tho...
https://www.wired.com/story/a-radical-new-model-of-the-brain...
https://www.wired.com/story/ai-pioneer-explains-evolution-ne...
https://www.natlawreview.com/article/brain-interfaces-bring-...
https://www.psychologytoday.com/intl/blog/the-future-brain/2...
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7417394/
https://www.sciencedaily.com/news/mind_brain/brain-computer_...
https://www.verdict.co.uk/brain-machine-interfaces-bmis/
https://www.sciencedaily.com/releases/2021/04/210401112415.h...
https://www.nature.com/articles/s41598-022-20460-9
https://www.wired.com/story/deep-neural-networks-are-helping...
https://www.frontiersin.org/articles/10.3389/fnins.2021.6589...
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8748882/
https://neurosciencenews.com/ultrasound-neuroimaging-21118/
https://www.newscientist.com/article/2335961-meta-ai-can-tel...
https://singularityhub.com/2022/09/13/meta-built-an-ai-that-...
https://www.wired.com/story/nextmind-noninvasive-brain-compu...
https://blogs.scientificamerican.com/observations/mind-readi...
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8722107/
https://finance.yahoo.com/news/neurotech-devices-market-wort...
https://www.sciencedaily.com/releases/2021/03/210322143320.h...
https://neurosciencenews.com/imagination-brain-wave-20212/
https://arstechnica.com/science/2022/08/the-age-of-brain-com...
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5134471/
https://pubmed.ncbi.nlm.nih.gov/28444626/
https://interestingengineering.com/brain-controlled-computer...
https://www.sciencedaily.com/releases/2021/07/210721120657.h...
sorry if i posted some more than once.
this is just a few, btw.
i'm crazy but i ain't that crazy.
this is REAL.
gl, gg.
There's already been preprints released showing fMRI reconstructions that appear to do better than an implicit multi-class classifier [1] [2]. But also, even if the result is an implicit multi-class classifier, if the n is sufficiently high then that would still be quite impressive!
[1] https://openreview.net/pdf?id=pHdiaqgh_nf
[2] https://arxiv.org/pdf/2211.06956.pdf
> Firstly, we learn an effective self-supervised representation of fMRI data using mask modeling in a large latent space inspired by the sparse coding of information in the primary visual cortex. Then by augmenting a latent diffusion model with double-conditioning, we show that MinD-Vis can reconstruct highly plausible images with semantically matching details from brain recordings using very few paired annotations.
https://arxiv.org/pdf/2211.06956.pdf
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
In honor of Alan Turing's hundredth birthday, I unwisely set out some thoughts about one of Turing's obsessions throughout his life, the question of physics and free will.