Skip to content
The methodOpen data
Back to the collection

DeWave: Discrete EEG Waves Encoding for Brain Dynamics to Text Translation

Yiqun Duan, Jinzhao Zhou, Zhen Wang, Yu-Kai Wang, Chin-Teng Lin

2023 · arXiv

NeuroscienceLanguage models
Read the paper
HN score
1.5
Mentions
4
Readers
3
Months
4
What’s in a score?

From the abstract

The translation of brain dynamics into natural language is pivotal for brain-computer interfaces (BCIs). With the swift advancement of large language models, such as ChatGPT, the need to bridge the gap between the brain and languages becomes increasingly pressing. Current methods, however, require eye-tracking fixations or event markers to segment brain dynamics into word-level features, which can restrict the practical application of these systems. To tackle these issues, we introduce a novel framework, DeWave, that integrates discrete encoding sequences into open-vocabulary EEG-to-text translation tasks. DeWave uses a quantized variational encoder to derive discrete codex encoding and align it with pre-trained language models. This discrete codex representation brings forth two advantages: 1) it realizes translation on raw waves without marker by introducing text-EEG contrastive alignment training, and 2) it alleviates the interference caused by individual differences in EEG waves through an invariant discrete codex with or without markers. Our model surpasses the previous baseline (40.1 and 31.7) by 3.06% and 6.34%, respectively, achieving 41.35 BLEU-1 and 33.71 Rouge-F on the ZuCo Dataset. This work is the first to facilitate the translation of entire EEG signal periods without word-level order markers (e.g., eye fixations), scoring 20.5 BLEU-1 and 29.5 Rouge-1 on the ZuCo Dataset.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

4 selected excerpts · 4 total mentions

bbor
Strong recommendation · rule
This is a very interesting experiment, and props to the team involved! Exploring the frontiers of the possible is almost always worthwhile. That said, in my humble (amateur!) opinion the framing from IEEE leaves a little to be desired, for one simple reason: they don't mention that most of what we're looking for is in the cortex (outer layer) of the brain, anyway!† And it kind of has to be, AFAIK... Namely; fNIRS[1] is one of the four main brain imaging technologies (that I know of?): EEG, fMRI, fNIRS, and ultrasound. Like fMRI (& ultrasound?), fNIRS measures the oxygenation levels of different parts of the brain, which has been shown to be a close analogue for brain…
Read the full comment
activity (more activity => more respiration, just like muscles). In this context, it's not enough to simply receive the signal you sent through -- you want to infer which emitter the signal came from so that you can infer the oxygenation levels of the regions it passed through/reflected-off-of. All of that is a very amateur, high-level overview, but hopefully it clearly supports my underlying point/question: how could you possibly make this work with a cross-head emitter-detector setup?? It seems impossible to disentangle more than one emitter's signals, and I'm not sure how you'd map oxygenation levels without more than one. The diagram in the article seems to support this confusion, given how chaotic it is. Then again, fNIRS and EEG both already rely on some serious statistical wizardy to turn 16-128 1D time series into a 3D model of activity, so perhaps I'm underestimating our tools! For example, the addition of frequency modulation to the fNIRS setup is an ongoing area of frontier research, which seems insanely complex to me. P.S. In case any of the hackers here haven't heard yet: BCI (Brain-Computer interaction) is blowing up right now thanks to the unreasonable efficacy of LLMs for decoding brain activity[2][3][4], and it's a very hackable field! There's a healthy open-source community for both fNIRS[5] and EEG[6], and I can personally highly recommend the ~$1000 Unicorn EEG system[7] for hackers. [1] https://en.wikipedia.org/wiki/Functional_near-infrared_spect... [2] https://www.nature.com/articles/s42003-025-07731-7 [3] https://arxiv.org/abs/2309.14030v2 [4] https://arxiv.org/pdf/2401.03851 [5] https://openfnirs.org/2024/01/01/continuous-wave-spectroscop... [6] https://openbci.com/ [7] https://www.gtec.at/product-configurator/unicorn-brain-inter... †: As a human, you're not even a brain piloting a skeleton -- you're a 3mm wrap around the basic mammalian brain! https://en.wikipedia.org/wiki/Cerebral_cortex
Original comment Scientists shine a laser through a human head
ShamelessC
Neutral mention · rule
Yeah...The research here doesn't even make the claim that it has no hallucinations. It seems to largely be exciting _despite_ hallucinations because it clearly does occasionally guess the correct words. They mention lots of issues but so long as it passes peer review, seems like a massive step forward. https://arxiv.org/pdf/2309.14030v2.pdf
Original comment BrainGPT Turns Thoughts into Text
bbor
Neutral mention · rule
First, a disclaimer: this is a great article and an important moral issue, and reminded me of the fantastic museum the Germans set up at the Charité in Berlin, a longstanding medical institution that obviously had a significant role to play in murderous Nazi “science”, and regardless deals with those sacrificing their wellbeing to science via testing and tissue donation to this day. Definitely do not skip, if you’re ever in town. Ok now what I’m really curious about — I would ask that we take “AI is a big deal now that the frame problem is tractable, and LLMs are shockingly good at encoding human brain activity” for granted in this discussion, if possible: Is anyone else…
Read the full comment
way more scared than they’re ever been about (science) news, and feel unable to read stories that are objectively interesting and important to you because they remind you of the uncertainty to come? Even if we abstract away all the economic automation, aesthetic and cultural conflict, and ethics/safety concerns of the new tech… we are reading people’s minds. Sure it’s mostly in fMRIs for now-it’s clearly in its infant stages, they’ve only had since early 2023 to work on it in earnest—I think it’s moving at lightning speed for life sciences. For example, look into the explosion of consumer/prosumer/freelancer BCI tech, such as EEG headsets (g.tech Unicorn being my hacker fave, $1000 for a modular Bluetooth headset) and fNIRs visors (Muse being the big player in the consumer space, the prosumer space is still coming down in price to feasible levels). For another example, this paper is what first alerted me to this underreported situation (tho now that there’s a whole issue of Nature on it, I imagine it’ll gain prominence): https://arxiv.org/abs/2309.14030v2 So, anyone have advice? Again, other than “you’re wrong it’s not a big deal”, please :)
Original comment Human neuroscience mustn't forget its human dimension

Keep following the thread.

More in Neuroscience

More from the same topics.