Date: September 21, 2025

Topic: Visualizing Neural Networks

Recall

We can visualize weights, activations, gradients and test for robustness to understand what the network is learning.

Notes

Visualizing Neural Networks

For a trained model, we want to see what it learned

Weights

Convolutional parameters are kernel weights themselves, so we can visualize what the kernels look like
- Image patches have similar types of features as the kernel
We can see linear features (e.g., different oriented edges) learned

Activations

We can also plot activations from running the kernel across an image (activation map)
Activation maps represent spatial locations with high values that are highly correlated with the kernel
Let’s us know what kind of features the network has learned

Gradients

Can also plot gradients, which are the key bread-and-butter of DNNs
- These are used for optimizing neural networks
Gradient statistics let us know what the network is learning

Robustness

Can test for various aspects of robustness in the NN
Understand weaknesses or biases of neural networks

If the FC is connected to images, we can reshape the weights and visualize what the weights look like.

For CNNs, we can instead visualize the kernels. They learn edges, colors and textures. Comparing kernels against different architectures yields qualitative differences.

Visualizing Weights

Visualizing Fully Connected Layers

FC visualization

For fully-connected layers, if nodes are connected to image itself, can reshape the weights into images by scaling from 0-255
- For complicated CNNs, the FC layers at the end are not connected to pixels, but connected to feature maps of last conv. instead — harder to visualize
When visualizing the results, we see various rough templates for the objects

Visualizing Convolutional Layers (Kernels)

Conv visualization

For each kernel, scale values from 0-255 and visualize
Can see kernels with oriented edges, colors, textures, etc
Qualitative differences between different types of architectures (ResNet, AlexNet) also exist

By visualizing output maps, especially those early in the network, we can see what image patches tend to be highly activated.

<aside> 📌 SUMMARY: To understand NN, we can visualize weights, activations, gradients and the network’s robustness and biases From visualizing the output maps, we can see rough shapes of what the model is seeing, however if the output maps are too small then it is hard to visualize We can also do dimensionality reduction to understand the separation of classes, with good performance being expected on well-separated classes.

</aside>