Michael Diskin, Alexey Bukhtiyarov, Max Ryabinin
Lucile Saulnier, Quentin Lhoest, Anton Sinitsin, Dmitry Popov, Dmitry Pyrkin, Maxim Kashirin, Alexander Borzunov, Albert Villanova del Moral, Denis Mazur, Ilia Kobelev, Yacine Jernite, Thomas Wolf, Gennady Pekhimenko
2021 · arXiv
Modern deep learning applications require increasingly more compute to train state-of-the-art models. To address this demand, large corporations and institutions use dedicated High-Performance Computing clusters, whose construction and maintenance are both environmentally costly and well beyond the budget of most organizations. As a result, some research directions become the exclusive domain of a few large industrial and even fewer academic actors. To alleviate this disparity, smaller groups may pool their computational resources and run collaborative experiments that benefit all participants. This paradigm, known as grid- or volunteer computing, has seen successful applications in numerous scientific areas. However, using this approach for machine learning is difficult due to high latency, asymmetric bandwidth, and several challenges unique to volunteer computing. In this work, we carefully analyze these constraints and propose a novel algorithmic framework designed specifically for collaborative training. We demonstrate the effectiveness of our approach for SwAV and ALBERT pretraining in realistic conditions and achieve performance comparable to traditional setups at a fraction of the cost. Finally, we provide a detailed report of successful collaborative language model pretraining with 40 participants.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
3 selected excerpts · 3 total mentions
I think you're looking for federated/distributed learning. From what I understand, nobody quite figured it out yet. Relevant 2021 NIPS paper: <https://arxiv.org/abs/2106.10207> On difference between federated and distributed: <https://en.wikipedia.org/w/index.php?title=Federated_learnin...>
"This paper presents a distributed model-parallel training framework that enables training large neural networks on small CPU clusters with low Internet bandwidth." Low bandwidth being <1Gbps. They've also tested it with GPUs as well. https://arxiv.org/pdf/2201.12667 Maybe there's the possibility of a completely new AI architecture that can still be efficiently trained when there are very low bandwidth connections between nodes? Specifically targeting this use case would make sense, given all the millions of underutilized GPUs out there in peoples' desktop computers. Also https://arxiv.org/pdf/2106.10207 ?
By "SETI" I assume you mean the SETI@Home distributed computing project. There's a two-way market where you can rent out your GPU here: https://vast.ai/ If you want to donate your GPU you could set your price to $0.00 I guess. Yandex and Hugging Face trained a couple nets using distributed GPUs. Their lessons learned paper is here: https://arxiv.org/abs/2106.10207 Takeaway: using consumer GPUs on the other end of consumer internet connections means you end up very I/O limited. 50 megabit upload on a cable modem doesn't compare well with 50000 megabit NVswitch links in a proper GPU cluster. As far as I can tell they are no longer doing distributed training. And it's widely assumed the big nets are trained on copyrighted material through libgen and scihub. So if your question is "is there an open source net that's trained on pirated books" the answer is "Yes, and it's called Llama/Mistral/Falcon etc etc"
More from the same topics.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski +10
We trained a convolutional neural network (CNN) to map raw pixels from a single front-facing camera directly to steering commands.