EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

There's a great motivation for small model work for big-model results: More efficient use of compute can be leveraged to make big models effectively bigger. Small-model architectural innovations are computational leverage. You can even see the convolution operation in this light; it's much more efficient than the 'giant dense matrix' approach. EfficientNet is an exemplar of this approach; they made much better small models, and wound up with much higher quality big models as a result of having better architecture overall: https://arxiv.org/pdf/1905.11946.pdf We're currently seeing some great results with more efficient attention layers, which will make the current 'big' models much more efficient... And unlock a next generation of higher quality big models.

It's not misleading assuming you know the field. If you don't you might not realize they are comparing ResNet (designed for ultimate performance) vs EffcientNet to show how close it gets in accuracy in the FLOPs budget. Note that the best accuracy for EffcientNet is roughly 1-2% better than ResNet/ResNeXT/SENet/etc but does have a much better FLOPs budget. But these other architectures were never optimised for FLOPs. It wasn't event a consideration when designing them. And EffcientNet is about a (manually designed) technique for scaling neural networks up in accuracy. Only EffcientNet-B0 is designed by AutoML, the others are scaled up. See the paper[1] for complete details. Like-for-like should be against MobileNet etc. EfficentNet is better, but the comparison is more reasonable in general. https://machinethink.net/blog/mobile-architectures/ is a really good overview. [1] https://arxiv.org/pdf/1905.11946.pdf

the whole idea of "evolving neural networks" didn't pan out To the contrary, evolving neural networks is all the rage these days ("Neural Architecture Search", e.g. SOTA on Imagenet [1]) [1] https://arxiv.org/abs/1905.11946

Actually this is still very good. Thanks for the links. I'll be timing some of these tricks tomorrow for my Imagenet experiments. By the way, I believe this is the current SOTA for Imagenet: https://arxiv.org/abs/1905.11946 (84/97%). CIFAR10 appears to be essentially solved (99%).