Skip to content
The methodOpen data
Back to the collection

XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks

Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, Ali Farhadi

2016 · arXiv

Computer visionMachine learning
Read the paper
HN score
0
Mentions
7
Readers
6
Months
5
What’s in a score?

From the abstract

We propose two efficient approximations to standard convolutional neural networks: Binary-Weight-Networks and XNOR-Networks. In Binary-Weight-Networks, the filters are approximated with binary values resulting in 32x memory saving. In XNOR-Networks, both the filters and the input to convolutional layers are binary. XNOR-Networks approximate convolutions using primarily binary operations. This results in 58x faster convolutional operations and 32x memory savings. XNOR-Nets offer the possibility of running state-of-the-art networks on CPUs (rather than GPUs) in real-time. Our binary networks are simple, accurate, efficient, and work on challenging visual tasks. We evaluate our approach on the ImageNet classification task. The classification accuracy with a Binary-Weight-Network version of AlexNet is only 2.9% less than the full-precision AlexNet (in top-1 measure). We compare our method with recent network binarization methods, BinaryConnect and BinaryNets, and outperform these methods by large margins on ImageNet, more than 16% in top-1 accuracy.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

7 selected excerpts · 7 total mentions

emcq
Neutral mention · rule
The reality is that TrueNorth can use CNNs and be trained with backprop to achieve good performance with networks like ImageNet despite LeCun's comments [0, 1] LeCun's primary critique is that binary wont work; "to get good results on a task like ImageNet you need about 8 bit of precision on the neuron states". There was no evidence for his claim then, and now this is clearly false [2, 3]. LeCun's post is based more in pride than reason; he spends most of the time talking about NeuFlow which at one point was a competitor to True North for funding. In the end, NeuFlow never became a chip, but True North did. [0] https://papers.nips.cc/paper/5862-backpropagation-for-energy... [1] https://arxiv.org/pdf/1603.08270.pdf [2] https://arxiv.org/abs/1603.05279 [3] https://arxiv.org/abs/1602.02830
Original comment Intel Gets Serious About Neuromorphic, Cognitive Computing Future
paulsutter
Neutral mention · rule
The article misses the core issue: neural network architectures are still in flux. "Neuromorphic" chips are hardwired to one architecture, which makes them power efficient but less flexible. When designs are more stable, such chips could be more practical. Meanwhile, upcoming (non-neuromorphic) AI processors are taking two directions: larger numbers of simplified GPU-type cores (such as NVIDIA Xavier and Intel's Lake Crest/Nervana chips), and FPGAs. Simplifying cores means lower precision, as fp32 and fp64 are overkill for neural networks and take up lots of silicon. The current NVIDIA Pascal added fp16 and byte operations such as the DP4A convolution instruction[1]. Even…
Read the full comment
smaller precision is practical (down to 1 bit with XNORnet[2], and the DoReFa paper[3] gives an excellent summary of the falloff in accuracy through 32-8-4-2-1 bits for weights, activations, and gradients). [1] https://devblogs.nvidia.com/parallelforall/mixed-precision-p... [2] XNORnet, https://arxiv.org/abs/1603.05279 [3] DoReFa, https://arxiv.org/abs/1606.06160
Original comment Intel Gets Serious About Neuromorphic, Cognitive Computing Future
sapphireblue
Neutral mention · rule
This is incorrect. With custom ASIC that implements ultra-wide low precision SIMD arithmetic (f16, i16, i8) it is possible to squeeze an order of magnitude or more speedup over conventional GPGPUs using same area and power. GPGPUs have to have big f32 (and even f64!) FPUs, and then some additional 3d-specific hardware overhead, that's why they are still suboptimal for deep machine learning. Google's TPU and Nervana's similar ASIC illustrate this point. Also you don't need the management cores to be fast, it suffices to have lots of GHz+ cores controlling ultra-wide SIMD (as Intel's Xeon Phi has shown). Arguably, one could get another order of magnitude speedup by implementing XNOR-net (http://arxiv.org/abs/1603.05279) hardware accelerator, but it hasn't been done yet.
Original comment Notes from the fourth RISC-V workshop
emcq
Neutral mention · rule
XNOR-Net is not currently part of TensorFlow, although there is a feature request to support it. [0] The main difference is that XNOR-Net uses 1 bit quantized weights and XNOR+popcount to approximate the dot product of convolution, which can be implemented very efficiently by using 64-bit arithmetic instructions on a CPU to operate over 64 1-bit components in parallel. [1] TensorFlow's optimization is also using discretized weights, but AFAIK targeted at 8-bit quantized weights. [2] [0] https://github.com/tensorflow/tensorflow/issues/1592 [1] https://arxiv.org/pdf/1603.05279v4.pdf [2] https://petewarden.com/2015/05/23/why-are-eight-bits-enough-...
Original comment XNOR.ai frees AI from the prison of the supercomputer

Keep following the thread.

More in Computer vision

More from the same topics.