I present a new way to parallelize the training of convolutional neural networks across multiple GPUs. The method scales significantly better than all alternatives when applied to modern convolutional neural networks.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
A. Krizhevsky put out a pretty interesting paper recently on this same topic. Also, it has the best research paper title I have seen [1]
By changing the type of parallelism depending on where in the network you are (fully connected vs. convolutional layers), you can take advantage of layer properties to optimize the sharing between GPUs. Whether that reduction in data movement is enough to make it feasible for AWS/local LAN links is up for debate.
[1] http://arxiv.org/abs/1404.5997
1. I missed this paper https://www.microsoft.com/en-us/research/uploads/prod/2018/0... when doing my own feature map compression research: https://liuliu.me/eyes/reduce-another-70-memory-usage-for-de.... Thanks for point it out!
2. Otherwise, most of these are obvious optimizations. One widely popular optimization (that I consider non-obvious) is ZeRO-Offload, in particularly the gradient sharding scheme (although once learned, the sharding itself is pretty straightforward, just a bit chatty). One thing I think undervalued these years though is Alex's "One weird trick": https://arxiv.org/abs/1404.5997. This scheme is much convoluted but very effective when training large…
Read the full comment
MLP. It is not popular probably because a). the implementation is non-obvious; b). large MLP fall out of fashion quickly, and the computation shape for transformer looks probably very different from the MLP its originally trying to solve (with 4096 activations per layer).
It's been this way since day 1. NVLINK remains the only real Tesla differentiator (although mini NVLINK is available on the new Turing consumer GPUs so WTFever). But because none of the DL frameworks support intra-layer model parallelism, all of the networks we see tend to run efficiently in data parallel because doing anything else makes them communication-limited, which they aren't because data scientists end up building networks that aren't, chicken and the egg style.
I continue to be boggled that Alex Krizhevsky's One Weird Trick never made it to TensorFlow or anywhere else:
https://arxiv.org/abs/1404.5997
I also suspect that's why so many thought leaders consider…
Read the full comment
ImageNet to be solved, when what's really solved is ImageNet-1K. That leaves ~21K more outputs on the softmax of the output layer for ImageNet-22K, which to my knowledge, is still not solved. A 22,000-wide output sourced by a 4096-wide embedding is 90K+ parameters (which is almost 4x as many parameters in the entire ResNet-50 network).
All that said, while it will always be cheaper to buy your ML peeps $10K quad-GPU workstations and upgrade their consumer GPUs whenever a brand new shiny becomes available, be aware NVIDIA is very passive aggressive about this following some strange magical thinking that this is OK for academics, but not OK for business. My own biased take is it's the right solution for anyone doing research, and the cloud is the right solution for scaling it up for production. Silly me.
Oh, I agree 100%.
But the truth is that without the hardware vendors putting significant resources into OpenCL it just isn't competitive and won't be until that happens.
The truth is that most of the work in Deep Learning is developing new NN architectures and other algorithmic optimisations. If you are working in the field there is no reason to put up with second class support from non NVidia vendors - just build in TensorFlow, Torch or a couple of other frameworks and wait for the day (one day, we are promised!) when OpenCL is competitive. Then the framework backends get ported, your code keeps running the same, and it can run on all those other architectures.
Everyone…
Read the full comment
has been waiting for that day since One Weird Trick[1]. There isn't really anything to indicate it is getting closer, and AMD's dismissal of the NVidia "doing something in the car industry"[2] doesn't give me a lot of confidence.
Anyway, I hope I'm wrong. Maybe Intel will step-up.
[1] https://arxiv.org/abs/1404.5997
[2] http://arstechnica.co.uk/gadgets/2016/04/amd-focusing-on-vr-...
Fair enough, but when neither Intel nor AMD is bothering to put out competitive software/hardware solutions for deep learning, which is apparently going to be a ~$40B market by 2020, I predict continued dominance here. That said, since AMD is getting beaten by a factor of 2-3 in performance while Intel's Accelerator/FPGA solutions are getting beaten by a factor of 5-10, AMD has a real opportunity to be the competitor to NVIDIA here.
And despite questions about it, GTX 1080 will be a fine deep learning board, especially if the framework engineers get off their lazy butts and implement Alex Krizhevsky's one weird trick algorithm, an approach which allows an 8-GPU Big Sur with 12.5GB/s P2P bandwidth to go toe to toe with an 8-GPU DGX-1 on training AlexNet.
http://arxiv.org/abs/1404.5997
One weird trick for parallelizing ConvNets: http://arxiv.org/abs/1404.5997
Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks:
http://arxiv.org/abs/1511.06434
A Neural Network of Artistic Style:
http://arxiv.org/abs/1508.06576
Yes, there are two types of parallelism: model and data. Data parallelism is simply training the the model on multiple computers with different minibatches, and aggregating the gradients. Model parallelism is hosting different parts of the model on different computers. Of course, these two parallelisms can be combined. A good explanation is the One Weird Trick paper: http://arxiv.org/abs/1404.5997
How is this on the front page? This is a completely incoherent.
For anyone actually interested in some interesting techniques for multi-GPU DNN training, http://arxiv.org/pdf/1404.5997v2.pdf and references therein are probably a good start.
Hey Dave! (Craig here) nice, achieved 6x speed up on an 8 gpu box (http://arxiv.org/pdf/1404.5997v2.pdf). Thanks for the pointer and hope to see you soon.
Looks like what it does is similar to what Alex did a few years back with One Weird Trick paper: https://arxiv.org/abs/1404.5997
When attempting to train transformers, I do notice a lot of time spend on allreduce more than with CNN models, probably due the parameter sizes. OWT seems to be natural to exploit for this situation (a lot of GEMMs, lot time spent on allreduce).
Edit:
Read the paper. The implementation is much less tricky than OWT, but for a good reason probably. Language model's GEMMs are smaller, therefore, partition the model would have efficiency impact (smaller GEMM will be slower). This does require much better interconnects, which NVLink / infiniband conveniently provides, that is also not available on consumer grade hardware anywhere (2-way NVLink is not meaningful).
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.