FNet: Mixing Tokens with Fourier Transforms

James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, Santiago Ontanon

2021 · arXiv

HN score: 0
Mentions: 5
Readers: 4
Months: 3

From the abstract

We show that Transformer encoder architectures can be sped up, with limited accuracy costs, by replacing the self-attention sublayers with simple linear transformations that "mix" input tokens. These linear mixers, along with standard nonlinearities in feed-forward layers, prove competent at modeling semantic relationships in several text classification tasks. Most surprisingly, we find that replacing the self-attention sublayer in a Transformer encoder with a standard, unparameterized Fourier Transform achieves 92-97% of the accuracy of BERT counterparts on the GLUE benchmark, but trains 80% faster on GPUs and 70% faster on TPUs at standard 512 input lengths. At longer input lengths, our FNet model is significantly faster: when compared to the "efficient" Transformers on the Long Range Arena benchmark, FNet matches the accuracy of the most accurate models, while outpacing the fastest models across all sequence lengths on GPUs (and across relatively shorter lengths on TPUs). Finally, FNet has a light memory footprint and is particularly efficient at smaller model sizes; for a fixed speed and accuracy budget, small FNet models outperform Transformer counterparts.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

5 selected excerpts · 5 total mentions

Show

westurnerMar 3, 2026

Neutral mention · rule

How could this lend insight into why Fast Fourier Transform approximates self-attention? > Because self-attention can be replaced with FFT for a loss in accuracy and a reduction in kWh [1], I suspect that the Quantum Fourier Transform can also be substituted for attention in LLMs. [1] "Fnet: Mixing tokens with fourier transforms" (2021) https://arxiv.org/abs/2105.03824 .. "Google Replaces BERT Self-Attention with Fourier Transform: 92% Accuracy, 7 Times Faster on GPUs" https://syncedreview.com/2021/05/14/deepmind-podracer-tpu-ba... "Why formalize mathematics – more than catching errors" (2025) https://news.ycombinator.com/item?id=45695541 Can the QFT Quantum Fourier Transform (and IQFT Inverse Quantum Fourier Transform) also be substituted for self-attention in LLMs, and do Lean formalisms provide any insight into how or why?

Original comment TorchLean: Formalizing Neural Networks in Lean

cs702Feb 26, 2025

Neutral mention · rule

TL;DR: 1. Take FNet (https://arxiv.org/abs/2105.03824). 2. Replace the fixed (frequency-domain) convolution filter with one that is dynamically computed from the data. 3. Apply non-linear functions to both real and imaginary components, before mapping the convolved data back to the time domain.

Original comment The FFT Strikes Back: An Efficient Alternative to Self-Attention

yagizdegirmenciFeb 26, 2025

Neutral mention · rule

Google introduced this idea in 2022 with "FNet: Mixing Tokens with Fourier Transforms" [0]. Later they found out that, performance of their TPU(s) for matrix multiplication was faster than FFT in the most scenarios. [0]: https://arxiv.org/abs/2105.03824

Original comment The FFT Strikes Back: An Efficient Alternative to Self-Attention

CGamesPlayMay 30, 2024

Neutral mention · rule

This is the paper, which appears to be cited in hundreds of others, some of which appear to be about efficiency gains. https://arxiv.org/abs/2105.03824

Original comment New attention mechanisms that outperform standard multi-head attention

westurnerMay 30, 2024

Neutral mention · rule

"Fnet: Mixing tokens with fourier transforms" (2021) https://arxiv.org/abs/2105.03824 https://scholar.google.com/scholar?cites=1423699627588508486... Fourier Transform and convolution. Shouldn't a deconvolvable NN be more explainable? #XAI Deconvolution: https://en.wikipedia.org/wiki/Deconvolution

Original comment New attention mechanisms that outperform standard multi-head attention

Keep following the thread.

Sparks of Artificial General Intelligence: Early experiments with GPT-4

Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11

Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.

Language models Psychology Mathematics

60 mentions14 months

3.5HN score

2020arXiv

Language Models are Few-Shot Learners

Tom B. Brown, Benjamin Mann, Nick Ryder +28

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.

Language models Machine learning

83 mentions35 months

3HN score

2025arXiv

Design Patterns for Securing LLM Agents against Prompt Injections

Luca Beurer-Kellner, Beat Buesser, Ana-Maria Creţu +11

As AI agents powered by Large Language Models (LLMs) become increasingly versatile and capable of addressing a broad spectrum of tasks, ensuring their security has become a critical challenge.

Security & cryptography Language models

4 mentions4 months

2.5HN score

2023arXiv

Toolformer: Language Models Can Teach Themselves to Use Tools

Timo Schick, Jane Dwivedi-Yu, Roberto Dessì +5

Language models (LMs) exhibit remarkable abilities to solve new tasks from just a few examples or textual instructions, especially at scale.

Language models

24 mentions5 months

2HN score