Visualizing and Understanding Convolutional Networks

The 2013 Zeiler and Fergus paper also explains this based on activations during training of the network resulting in feature detection across layers. Paper : https://arxiv.org/pdf/1311.2901.pdf

Some architectures are relatively well understood. Eg in CNNs, the first layers detect low level features like edges, gradients, etc. The next layer then combines these features to more complex structures like corners or circles. Next layer will combine these features to even higher level features and so on. [1] Typically, you can take a pre-trained model and retrain it on your new dataset by only changing the weights of the last layer(s). Some loss functions even measures the difference between the high-level features of two images, typically extracted from a pre-trained CNN (Perceptual Loss). [1]Matt Zeiler did an amazing work on these findings 10 years ago (https://arxiv.org/abs/1311.2901).

There’s no helping the incurious, but there are indeed reams of explorations unto weights and their explanations. One of my personal favourites is Zeiler and Fergus https://arxiv.org/abs/1311.2901 Tom Zahavy’s “Greying the black box” on deep RL comes to mind too https://arxiv.org/abs/1602.02658 OpenAI has their Microscope tool for investigating individual neurons https://openai.com/research/microscope; more recently OpenAI shared research and tooling leveraging a powerful language model to explain a smaller model’s neurons. https://openai.com/research/language-models-can-explain-neur... For a true toy box exploration playground.tensorflow.org is worth a play These may or may not satisfy your curiosities — your perfect tool may not exist…yet. I would argue that the sampling of extant work above are indicative that it and more are possible.

There's a pretty cool paper that visualises the different layers in a ConvNet trained on ImageNet: https://arxiv.org/pdf/1311.2901.pdf - page 4 has the layer imagery.

https://arxiv.org/abs/1311.2901 Check out figure 2. The network learns composition from fundamental shapes and gradients to compositional ones. It's kinda awe inspiring. Not NN specific, but some more work on explanations: https://homes.cs.washington.edu/~marcotcr/blog/lime/ Visualization of attention mechanisms is pretty cool for explanations: https://arxiv.org/pdf/1502.03044.pdf http://torch.ch/blog/2015/09/21/rmva.html

It's definitely possible to get insight into how a CNN would classify something like a stop sign. This paper does a good job of showing how CNNs learn a hierarchy of increasingly complex features to classify images: http://arxiv.org/abs/1311.2901

Seeing the face of a dog just means that the network has learned a good representation of dog faces. I don't think that has anything to do with the generalization ability of the network. I'm not clear on the details of this hallucination process, but this reminds me of earlier work by Zeiler & Fergus[1], where you can see the bottom layers of the network learning general low-level features and later layers learning higher level representations of the objects being classified. [1] http://arxiv.org/abs/1311.2901

A few other useful resources to play with convolutional kernels: * The linked demo really focuses only on layer 1. See Layers >=2 plotted here: http://arxiv.org/pdf/1311.2901v3.pdf * DeepViz is a nice tool: https://github.com/bruckner/deepViz * Another tool one can use to play with convolutional kernels is ShaderToy ( https://www.shadertoy.com/ ). Here's a Gaussian blur: https://www.shadertoy.com/view/XdfGDH * If you like playing with kernels in shaders, see also Brad Larson's GPUImage: ( https://github.com/BradLarson/GPUImage ) -- the demo app has a bunch of standard kernels. Just for fun, convolutional shader porn: https://www.shadertoy.com/view/4d2Xzc