Training of large-scale deep neural networks is often constrained by the available computational resources. We study the effect of limited precision data representation and computation on neural network training. Within the context of low-precision fixed-point computations, we observe the rounding scheme to play a crucial role in determining the network's behavior during training. Our results show that deep networks can be trained using only 16-bit wide fixed-point number representation when using stochastic rounding, and incur little to no degradation in the classification accuracy. We also demonstrate an energy-efficient hardware accelerator that implements low-precision fixed-point arithmetic with stochastic rounding.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
For deep learning or chips for deep learning?
With an EE/Chem/Math background you should be set to go :)
For deep learning, I highly recommend the cs231n course materials (available for free online) and the Deep Learning Book by Goodfellow et al.
For chip design for deep learning specifically, it's a fairly new field, and a lot of commercial interests, hence why a lot of it isn't available in the form of an "Intro to Deep Learning Chip Design" course. But, the basic point is that deep learning can both train and perform inference in astonishingly low precision.
For training: https://arxiv.org/abs/1502.02551
For inference, there are so many papers confirming this fact…
Read the full comment
that there really isn't one key paper to point to. This article is a good introduction however: https://petewarden.com/2015/05/23/why-are-eight-bits-enough-...
Other than that, stripping out the cache hierarchy, HBM memory, etc. are the obvious steps to take in targeting deep learning. Note that it's my opinion that deep learning chip startups (such as ours) need another "secret sauce" beyond simply lowering precision, we're not like Intel which can just say "okay, here's $100mil, make me an 8-bit GPU, Go."
There are hints that even less than 8 bits per weight might be usable (for certain cases and on custom hw). Not sure if it's practical but it is definitely interesting.
https://arxiv.org/pdf/1502.02551.pdf
https://arxiv.org/pdf/1610.00324.pdf
I wanted to have some basic idea about hardware so I did some "research" (googling) and ended up giving a short informal talk. My slides with some links are here:
http://ml.kjx.cz/hw4nn2017.pdf
There are a few people that seem to be working with integer-based deep learning and FPGAs, and even though they're at the beginning of research, things are looking pretty promising. Here's a paper that was released earlier this year that seems to show that at least some algorithms can see a big benefit from FPGAs http://arxiv.org/pdf/1502.02551v1.pdf .
The problem with using digital calculations is that they are deterministic. If a result is really small, it is just rounded down to zero. So if you add a bunch of small numbers, you get zero. Even if the result should be large.
Stochastic rounding can fix this. You round each step with the probability so it's expected value is the same. Usually it will round down to 0, but sometimes it will round up to 1.
Relevant paper, using stochastic rounding. Without it the results get worse and worse before you even get to 8 bits. With stochastic rounding, there is no performance degradation. You could probably even reduce the bits even further. I think it may even be possible to get it down to 1 or 2 bits: http://arxiv.org/abs/1502.02551
The relevant graph: https://i.imgur.com/cOZ4fn3.jpg
Indeed, 64 bits of accuracy is overkill for a lot of ML algorithms, where there is so much noise that the additional quantization noise due to low precision is negligible. Most deep neural nets are already being trained in single precision because that is what NVIDIA GPUs can do the fastest. There has been quite some research on reducing the bit depth further the last couple of years, see e.g. http://arxiv.org/abs/1502.02551 and http://arxiv.org/abs/1412.7024.
The current generation of GPUs already has limited support for half-precision operations, and the tools for using these operations in neural networks (dot product, convolution implementation) are starting to become available as well, which is awesome (see e.g. https://github.com/NervanaSystems/nervanagpu).
NVIDIA themselves have also noticed and better hardware support for low-precision arithmetic is coming in Pascal: http://techreport.com/news/27978/nvidia-pascal-to-feature-mi...
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
The Fermi paradox is the conflict between an expectation of a high {\em ex ante} probability of intelligent life elsewhere in the universe and the apparently lifeless universe we in fact observe.
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.