The Hardware Lottery

> If you can see that these models empirically get better with scale, why would you swap the main architecture? Those events will be pretty rare c.f. hardware lotter https://arxiv.org/abs/2009.06489

Hi Jeremy & Eric, great to see your newest endeavor. I hope that Answer.AI builds on the success and impact that fast.ai has already enjoyed. Given new developments in hardware (by companies not named NVIDIA), I'm wondering if you are keen on exploring the next generation of model architectures and optimization procedures that might exploit newer hardware. In other words, will research directions pivot based on the hardware lottery?[1] Are you in conversations with companies developing these alternative chips? [1] https://arxiv.org/abs/2009.06489

In the current world, deep learning with homogeneous computation graphs, tuned with backprop, has won the Hardware Lottery [1]. This is unfortunate for research outside of that area, but just looking at the momentum of development it seems like a sure bet to keep investing in GPU-based training and inference for the next decade. There's just too much lock-in already to this paradigm. If a new algorithm appears from with a novel approach (analog compute, heterogeneous computation graphs from genetic algorithms, quantum, much more...), there will be a whole generation of R&D + tool + framework building, which gives the major players enough time to adapt. [1] https://arxiv.org/abs/2009.06489

Exactly. It's easy to see in retrospect, but hard in prospect: the original paper [1] on GPU acceleration of NNs reports a measly 20x speedup. Assuming a bit of cherry-picking on the author's side to make get the paper published, the 'real-world speedup' will have been assumed by the readership to be less. But this triggered a virtuos cycle of continuous improvements at all levels that has been dubbed "winning the hardware lottery" [2]. [1] K.-S. Oh, K. Jung, GPU implementation of neural networks. [2] S. Hooker, The Hardware Lottery. https://arxiv.org/abs/2009.06489

Related: The Hardware Lottery (https://arxiv.org/abs/2009.06489) It goes both ways: processors are built to accelerate existing applications, and applications are built to run with the constraints of the hardware. This holds back progress towards better systems where both new kind of hardware and software is required. This is part of the reason why research into neural networks was so slow for a long time, and took off only after GPUs for 3d graphics came around.

Because I used to work at Google on TPUs with all the teams (from ads to Physics), and have about 30 years experience in high performance computing. Neural nets and DL are just one of the reasons to use TPUs; in the future, they might not even be the preeminent approach. DL is just one part of the much larger world of HPC, it's been very effective, but I expect that as things evolve people will rediscover the value of deep precision. https://arxiv.org/abs/2009.06489