[ { "path": ".gitignore", "content": "# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# C extensions\n*.so\n\n# Distribution / packaging\n.Python\nenv/\nbuild/\ndevelop-eggs/\ndist/\ndownloads/\neggs/\n.eggs/\nlib/\nlib64/\nparts/\nsdist/\nvar/\nwheels/\n*.egg-info/\n.installed.cfg\n*.egg\n\n# PyInstaller\n# Usually these files are written by a python script from a template\n# before PyInstaller builds the exe, so as to inject date/other infos into it.\n*.manifest\n*.spec\n\n# Installer logs\npip-log.txt\npip-delete-this-directory.txt\n\n# Unit test / coverage reports\nhtmlcov/\n.tox/\n.coverage\n.coverage.*\n.cache\nnosetests.xml\ncoverage.xml\n*.cover\n.hypothesis/\n\n# Translations\n*.mo\n*.pot\n\n# Django stuff:\n*.log\nlocal_settings.py\n\n# Flask stuff:\ninstance/\n.webassets-cache\n\n# Scrapy stuff:\n.scrapy\n\n# Sphinx documentation\ndocs/_build/\n\n# PyBuilder\ntarget/\n\n# Jupyter Notebook\n.ipynb_checkpoints\n\n# pyenv\n.python-version\n\n# celery beat schedule file\ncelerybeat-schedule\n\n# SageMath parsed files\n*.sage.py\n\n# dotenv\n.env\n\n# virtualenv\n.venv\nvenv/\nENV/\n\n# Spyder project settings\n.spyderproject\n.spyproject\n\n# Rope project settings\n.ropeproject\n\n# mkdocs documentation\n/site\n\n# mypy\n.mypy_cache/\n" }, { "path": "LICENSE", "content": "MIT License\n\nCopyright (c) 2025 Utku Ozbulak\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n" }, { "path": "README.md", "content": "# Convolutional Neural Network Visualizations \n\nThis repository contains a number of convolutional neural network visualization techniques implemented in PyTorch.\n\n**Note**: I removed cv2 dependencies and moved the repository towards PIL. A few things might be broken (although I tested all methods), I would appreciate if you could create an issue if something does not work.\n\n**Note**: The code in this repository was tested with torch version 0.4.1 and some of the functions may not work as intended in later versions. Although it shouldn't be too much of an effort to make it work, I have no plans at the moment to make the code in this repository compatible with the latest version because I'm still using 0.4.1.\n\n## Implemented Techniques\n\n* [Gradient visualization with vanilla backpropagation](#gradient-visualization)\n* [Gradient visualization with guided backpropagation](#gradient-visualization) [1]\n* [Gradient visualization with saliency maps](#gradient-visualization) [4]\n* [Gradient-weighted class activation mapping](#gradient-visualization) [3] (Generalization of [2]) \n* [Guided, gradient-weighted class activation mapping](#gradient-visualization) [3]\n* [Score-weighted class activation mapping](#gradient-visualization) [15] (Gradient-free generalization of [2])\n* [Element-wise gradient-weighted class activation mapping](#hierarchical-gradient-visualization) [16]\n* [Smooth grad](#smooth-grad) [8]\n* [CNN filter visualization](#convolutional-neural-network-filter-visualization) [9]\n* [Inverted image representations](#inverted-image-representations) [5]\n* [Deep dream](#deep-dream) [10]\n* [Class specific image generation](#class-specific-image-generation) [4] [14]\n* [Grad times image](#grad-times-image) [12]\n* [Integrated gradients](#gradient-visualization) [13]\n* [Layerwise relevance propagation](#gradient-visualization) [17]\n\n## General Information\n\nDepending on the technique, the code uses pretrained **AlexNet** or **VGG** from the model zoo. Some of the code also assumes that the layers in the model are separated into two sections; **features**, which contains the convolutional layers and **classifier**, that contains the fully connected layer (after flatting out convolutions). If you want to port this code to use it on your model that does not have such separation, you just need to do some editing on parts where it calls *model.features* and *model.classifier*.\n\nEvery technique has its own python file (e.g. *gradcam.py*) which I hope will make things easier to understand. *misc_functions.py* contains functions like image processing and image recreation which is shared by the implemented techniques.\n\nAll images are pre-processed with mean and std of the ImageNet dataset before being fed to the model. None of the code uses GPU as these operations are quite fast for a single image (except for deep dream because of the example image that is used for it is huge). You can make use of gpu with very little effort. The example pictures below include numbers in the brackets after the description, like *Mastiff (243)*, this number represents the class id in the ImageNet dataset.\n\nI tried to comment on the code as much as possible, if you have any issues understanding it or porting it, don't hesitate to send an email or create an issue.\n\nBelow, are some sample results for each operation.\n\n## Gradient Visualization\n\n\t\n \n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n \n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n \n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n \n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n \n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n \n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n \n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n \n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n \n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n \n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n \n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n \n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n \n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n \n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n \n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n \n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\n
Target class: King Snake (56) Target class: Mastiff (243) Target class: Spider (72)
Original Image
Colored Vanilla Backpropagation
Vanilla Backpropagation Saliency
Colored Guided Backpropagation

(GB)
Guided Backpropagation Saliency

(GB)
Guided Backpropagation Negative Saliency

(GB)
Guided Backpropagation Positive Saliency

(GB)
Gradient-weighted Class Activation Map

(Grad-CAM)
Gradient-weighted Class Activation Heatmap

(Grad-CAM)
Gradient-weighted Class Activation Heatmap on Image

(Grad-CAM)
Score-weighted Class Activation Map

(Score-CAM)
Score-weighted Class Activation Heatmap

(Score-CAM)
Score-weighted Class Activation Heatmap on Image

(Score-CAM)
Colored Guided Gradient-weighted Class Activation Map

(Guided-Grad-CAM)
Guided Gradient-weighted Class Activation Map Saliency

(Guided-Grad-CAM)
Integrated Gradients
(without image multiplication)
Layerwise Relevance
(LRP) - Layer 7
Layerwise Relevance
(LRP) - Layer 1
\n\n## Hierarchical Gradient Visualization\nLayerCAM [16] is a simple modification of Grad-CAM [3], which can generate reliable class activation maps from different layers. For the examples provided below, a pre-trained **VGG16** was used.\n\n\n\t \n \n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\n
Class Activation Map Class Activation HeatMap Class Activation HeatMap on Image
LayerCAM
(Layer 9)
LayerCAM
(Layer 16)
LayerCAM
(Layer 23)
LayerCAM
(Layer 30)
\n\n\n## Grad Times Image\nAnother technique that is proposed is simply multiplying the gradients with the image itself. Results obtained with the usage of multiple gradient techniques are below.\n\n\n\t \n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\n
Vanilla Grad
X
Image
Guided Grad
X
Image
Integrated Grad
X
Image
\n\n\n## Smooth Grad\nSmooth grad is adding some Gaussian noise to the original image and calculating gradients multiple times and averaging the results [8]. There are two examples at the bottom which use _vanilla_ and _guided_ backpropagation to calculate the gradients. Number of images (_n_) to average over is selected as 50. _σ_ is shown at the bottom of the images.\n\n\n\t \n \t\t\n\t\t\t\n\t\t\t\n\t\t\n\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\n
Vanilla Backprop
\n\n\n\n\t \n \t\t\n\t\t\t\n\t\t\t\n\t\t\n\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\n
Guided Backprop
\n\n## Convolutional Neural Network Filter Visualization\nCNN filters can be visualized when we optimize the input image with respect to output of the specific convolution operation. For this example I used a pre-trained **VGG16**. Visualizations of layers start with basic color and direction filters at lower levels. As we approach towards the final layer the complexity of the filters also increase. If you employ external techniques like blurring, gradient clipping etc. you will probably produce better images.\n\n\n\t \n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\n
Layer 2
(Conv 1-2)
Layer 10
(Conv 2-1)
Layer 17
(Conv 3-1)
Layer 24
(Conv 4-1)
\n\nAnother way to visualize CNN layers is to to visualize activations for a specific input on a specific layer and filter. This was done in [1] Figure 3. Below example is obtained from layers/filters of VGG16 for the first image using guided backpropagation. The code for this opeations is in *layer_activation_with_guided_backprop.py*. The method is quite similar to guided backpropagation but instead of guiding the signal from the last layer and a specific target, it guides the signal from a specific layer and filter. \n\n\n\t \n \t\t\n\t\t\t\n\t\t\t\n\t\t\n\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\n
Input Image Layer Vis. (Filter=0) Filter Vis. (Layer=29)
\n\n\n## Inverted Image Representations\nI think this technique is the most complex technique in this repository in terms of understanding what the code does. It is mainly because of complex regularization. If you truly want to understand how this is implemented I suggest you read the second and third page of the paper [5], specifically, the regularization part. Here, the aim is to generate original image after nth layer. The further we go into the model, the harder it becomes. The results in the paper are incredibly good (see Figure 6) but here, the result quickly becomes messy as we iterate through the layers. This is because the authors of the paper tuned the parameters for each layer individually. You can tune the parameters just like the to ones that are given in the paper to optimize results for each layer. The inverted examples from several layers of **AlexNet** with the previous *Snake* picture are below.\n\n\n\n\t \n \t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\n
Layer 0: Conv2d Layer 2: MaxPool2d Layer 4: ReLU
\n\n\t \n \t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\n
Layer 7: ReLU Layer 9: ReLU Layer 12: MaxPool2d
\n\n\n\n## Deep Dream\nDeep dream is technically the same operation as layer visualization the only difference is that you don't start with a random image but use a real picture. The samples below were created with **VGG19**, the produced result is entirely up to the filter so it is kind of hit or miss. The more complex models produce mode high level features. If you replace **VGG19** with an **Inception** variant you will get more noticable shapes when you target higher conv layers. Like layer visualization, if you employ additional techniques like gradient clipping, blurring etc. you might get better visualizations.\n\n\n\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\n
Original Image
VGG19
Layer: 34
(Final Conv. Layer) Filter: 94
VGG19
Layer: 34
(Final Conv. Layer) Filter: 103
\n\n\n## Class Specific Image Generation\nThis operation produces different outputs based on the model and the applied regularization method. Below, are some samples produced with **VGG19** incorporated with Gaussian blur every other iteration (see [14] for details). The quality of generated images also depend on the model, **AlexNet** generally has green(ish) artifacts but VGGs produce (kind of) better images. Note that these images are generated with regular CNNs with optimizing the input and **not with GANs**.\n\n\n\t\n \n\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\n
Target class: Worm Snake (52) - (VGG19) Target class: Spider (72) - (VGG19)
\n\nThe samples below show the produced image with no regularization, l1 and l2 regularizations on target class: **flamingo** (130) to show the differences between regularization methods. These images are generated with a pretrained AlexNet. \n\n\n\t \n \t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\n\t\n
No Regularization L1 Regularization L2 Regularization
\n\n\nProduced samples can further be optimized to resemble the desired target class, some of the operations you can incorporate to improve quality are; blurring, clipping gradients that are below a certain treshold, random color swaps on some parts, random cropping the image, forcing generated image to follow a path to force continuity.\n\nSome of these techniques are implemented in *generate_regularized_class_specific_samples.py* (courtesy of [alexstoken](https://github.com/alexstoken)).\n\n## Requirements:\n```\ntorch == 0.4.1\ntorchvision >= 0.1.9\nnumpy >= 1.13.0\nmatplotlib >= 1.5\nPIL >= 1.1.7\n```\n\n## Citation\n\nIf you find the code in this repository useful for your research consider citing it.\n\n\t@misc{uozbulak_pytorch_vis_2022,\n\t author = {Utku Ozbulak},\n\t title = {PyTorch CNN Visualizations},\n\t year = {2019},\n\t publisher = {GitHub},\n\t journal = {GitHub repository},\n\t howpublished = {\\url{https://github.com/utkuozbulak/pytorch-cnn-visualizations}},\n\t commit = {b7e60adaf64c9be97b480509285718603d1e9ba4}\n\t}\n\t\n## References:\n\n[1] J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller. *Striving for Simplicity: The All Convolutional Net*, https://arxiv.org/abs/1412.6806\n\n[2] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, A. Torralba. *Learning Deep Features for Discriminative Localization*, https://arxiv.org/abs/1512.04150\n\n[3] R. R. Selvaraju, A. Das, R. Vedantam, M. Cogswell, D. Parikh, and D. Batra. *Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization*, https://arxiv.org/abs/1610.02391\n\n[4] K. Simonyan, A. Vedaldi, A. Zisserman. *Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps*, https://arxiv.org/abs/1312.6034\n\n[5] A. Mahendran, A. Vedaldi. *Understanding Deep Image Representations by Inverting Them*, https://arxiv.org/abs/1412.0035\n\n[6] H. Noh, S. Hong, B. Han, *Learning Deconvolution Network for Semantic Segmentation* https://www.cv-foundation.org/openaccess/content_iccv_2015/papers/Noh_Learning_Deconvolution_Network_ICCV_2015_paper.pdf\n\n[7] A. Nguyen, J. Yosinski, J. Clune. *Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images* https://arxiv.org/abs/1412.1897\n\n[8] D. Smilkov, N. Thorat, N. Kim, F. Viégas, M. Wattenberg. *SmoothGrad: removing noise by adding noise* https://arxiv.org/abs/1706.03825\n\n[9] D. Erhan, Y. Bengio, A. Courville, P. *Vincent. Visualizing Higher-Layer Features of a Deep Network* https://www.researchgate.net/publication/265022827_Visualizing_Higher-Layer_Features_of_a_Deep_Network\n\n[10] A. Mordvintsev, C. Olah, M. Tyka. *Inceptionism: Going Deeper into Neural Networks* https://research.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html\n\n[11] I. J. Goodfellow, J. Shlens, C. Szegedy. *Explaining and Harnessing Adversarial Examples* https://arxiv.org/abs/1412.6572\n\n[12] A. Shrikumar, P. Greenside, A. Shcherbina, A. Kundaje. *Not Just a Black Box: Learning Important Features Through Propagating Activation Differences* https://arxiv.org/abs/1605.01713\n\n[13] M. Sundararajan, A. Taly, Q. Yan. *Axiomatic Attribution for Deep Networks* https://arxiv.org/abs/1703.01365\n\n[14] J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, Hod Lipson, *Understanding Neural Networks Through Deep Visualization* https://arxiv.org/abs/1506.06579\n\n[15] H. Wang, Z. Wang, M. Du, F. Yang, Z. Zhang, S. Ding, P. Mardziel, X. Hu. *Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks* https://arxiv.org/abs/1910.01279\n\n[16] P. Jiang, C. Zhang, Q. Hou, M. Cheng, Y. Wei. LayerCAM: *Exploring Hierarchical Class Activation Maps for Localization* http://mmcheng.net/mftp/Papers/21TIP_LayerCAM.pdf\n\n[17] G. Montavon1, A. Binder, S. Lapuschkin, W. Samek, and K. Muller. *Layer-Wise Relevance Propagation: An Overview* https://www.researchgate.net/publication/335708351_Layer-Wise_Relevance_Propagation_An_Overview\n\n" }, { "path": "src/LRP.py", "content": "# -*- coding: utf-8 -*-\n\"\"\"\nCreated on Mon Mar 14 13:32:09 2022\n\n@author: ut\n\"\"\"\nimport copy\nimport numpy as np\nfrom PIL import Image\nimport torch\nimport torch.nn as nn\n\nfrom misc_functions import apply_heatmap, get_example_params\n\n\nclass LRP():\n \"\"\"\n Layer-wise relevance propagation with gamma+epsilon rule\n\n This code is largely based on the code shared in: https://git.tu-berlin.de/gmontavon/lrp-tutorial\n Some stuff is removed, some stuff is cleaned, and some stuff is re-organized compared to that repository.\n \"\"\"\n def __init__(self, model):\n self.model = model\n\n def LRP_forward(self, layer, input_tensor, gamma=None, epsilon=None):\n # This implementation uses both gamma and epsilon rule for all layers\n # The original paper argues that it might be beneficial to sometimes use\n # or not use gamma/epsilon rule depending on the layer location\n # Have a look a the paper and adjust the code according to your needs\n\n # LRP-Gamma rule\n if gamma is None:\n gamma = lambda value: value + 0.05 * copy.deepcopy(value.data.detach()).clamp(min=0)\n # LRP-Epsilon rule\n if epsilon is None:\n eps = 1e-9\n epsilon = lambda value: value + eps\n\n # Copy the layer to prevent breaking the graph\n layer = copy.deepcopy(layer)\n\n # Modify weight and bias with the gamma rule\n try:\n layer.weight = nn.Parameter(gamma(layer.weight))\n except AttributeError:\n pass\n # print('This layer has no weight')\n try:\n layer.bias = nn.Parameter(gamma(layer.bias))\n except AttributeError:\n pass\n # print('This layer has no bias')\n # Forward with gamma + epsilon rule\n return epsilon(layer(input_tensor))\n\n def LRP_step(self, forward_output, layer, LRP_next_layer):\n # Enable the gradient flow\n forward_output = forward_output.requires_grad_(True)\n # Get LRP forward out based on the LRP rules\n lrp_rule_forward_out = self.LRP_forward(layer, forward_output, None, None)\n # Perform element-wise division\n ele_div = (LRP_next_layer / lrp_rule_forward_out).data\n # Propagate\n (lrp_rule_forward_out * ele_div).sum().backward()\n # Get the visualization\n LRP_this_layer = (forward_output * forward_output.grad).data\n\n return LRP_this_layer\n\n def generate(self, input_image, target_class):\n layers_in_model = list(self.model._modules['features']) + list(self.model._modules['classifier'])\n number_of_layers = len(layers_in_model)\n # Needed to know where flattening happens\n features_to_classifier_loc = len(self.model._modules['features'])\n\n # Forward outputs start with the input image\n forward_output = [input_image]\n # Then we do forward pass with each layer\n for conv_layer in list(self.model._modules['features']):\n forward_output.append(conv_layer.forward(forward_output[-1].detach()))\n\n # To know the change in the dimensions between features and classifier\n feature_to_class_shape = forward_output[-1].shape\n # Flatten so we can continue doing forward passes at classifier layers\n forward_output[-1] = torch.flatten(forward_output[-1], 1)\n for index, classifier_layer in enumerate(list(self.model._modules['classifier'])):\n forward_output.append(classifier_layer.forward(forward_output[-1].detach()))\n\n # Target for backprop\n target_class_one_hot = torch.FloatTensor(1, 1000).zero_()\n target_class_one_hot[0][target_class] = 1\n\n # This is where we accumulate the LRP results\n LRP_per_layer = [None] * number_of_layers + [(forward_output[-1] * target_class_one_hot).data]\n\n for layer_index in range(1, number_of_layers)[::-1]:\n # This is where features to classifier change happens\n # Have to flatten the lrp of the next layer to match the dimensions\n if layer_index == features_to_classifier_loc-1:\n LRP_per_layer[layer_index+1] = LRP_per_layer[layer_index+1].reshape(feature_to_class_shape)\n\n if isinstance(layers_in_model[layer_index], (torch.nn.Linear, torch.nn.Conv2d, torch.nn.MaxPool2d)):\n # In the paper implementation, they replace maxpool with avgpool because of certain properties\n # I didn't want to modify the model like the original implementation but\n # feel free to modify this part according to your need(s)\n lrp_this_layer = self.LRP_step(forward_output[layer_index], layers_in_model[layer_index], LRP_per_layer[layer_index+1])\n LRP_per_layer[layer_index] = lrp_this_layer\n else:\n LRP_per_layer[layer_index] = LRP_per_layer[layer_index+1]\n return LRP_per_layer\n\n\nif __name__ == '__main__':\n # Get params\n target_example = 2 # Spider\n (original_image, prep_img, target_class, file_name_to_export, pretrained_model) =\\\n get_example_params(target_example)\n\n # LRP\n layerwise_relevance = LRP(pretrained_model)\n\n # Generate visualization(s)\n LRP_per_layer = layerwise_relevance.generate(prep_img, target_class)\n\n # Convert the output nicely, selecting the first layer\n lrp_to_vis = np.array(LRP_per_layer[1][0]).sum(axis=0)\n lrp_to_vis = np.array(Image.fromarray(lrp_to_vis).resize((prep_img.shape[2],\n prep_img.shape[3]), Image.ANTIALIAS))\n\n # Apply heatmap and save\n heatmap = apply_heatmap(lrp_to_vis, 4, 4)\n heatmap.figure.savefig('../results/LRP_out.png')\n" }, { "path": "src/cnn_layer_visualization.py", "content": "\"\"\"\nCreated on Sat Nov 18 23:12:08 2017\n\n@author: Utku Ozbulak - github.com/utkuozbulak\n\"\"\"\nimport os\nimport numpy as np\n\nimport torch\nfrom torch.optim import Adam\nfrom torchvision import models\n\nfrom misc_functions import preprocess_image, recreate_image, save_image\n\n\nclass CNNLayerVisualization():\n \"\"\"\n Produces an image that minimizes the loss of a convolution\n operation for a specific layer and filter\n \"\"\"\n def __init__(self, model, selected_layer, selected_filter):\n self.model = model\n self.model.eval()\n self.selected_layer = selected_layer\n self.selected_filter = selected_filter\n self.conv_output = 0\n # Create the folder to export images if not exists\n if not os.path.exists('../generated'):\n os.makedirs('../generated')\n\n def hook_layer(self):\n def hook_function(module, grad_in, grad_out):\n # Gets the conv output of the selected filter (from selected layer)\n self.conv_output = grad_out[0, self.selected_filter]\n # Hook the selected layer\n self.model[self.selected_layer].register_forward_hook(hook_function)\n\n def visualise_layer_with_hooks(self):\n # Hook the selected layer\n self.hook_layer()\n # Generate a random image\n random_image = np.uint8(np.random.uniform(150, 180, (224, 224, 3)))\n # Process image and return variable\n processed_image = preprocess_image(random_image, False)\n # Define optimizer for the image\n optimizer = Adam([processed_image], lr=0.1, weight_decay=1e-6)\n for i in range(1, 31):\n optimizer.zero_grad()\n # Assign create image to a variable to move forward in the model\n x = processed_image\n for index, layer in enumerate(self.model):\n # Forward pass layer by layer\n # x is not used after this point because it is only needed to trigger\n # the forward hook function\n x = layer(x)\n # Only need to forward until the selected layer is reached\n if index == self.selected_layer:\n # (forward hook function triggered)\n break\n # Loss function is the mean of the output of the selected layer/filter\n # We try to minimize the mean of the output of that specific filter\n loss = -torch.mean(self.conv_output)\n print('Iteration:', str(i), 'Loss:', \"{0:.2f}\".format(loss.data.numpy()))\n # Backward\n loss.backward()\n # Update image\n optimizer.step()\n # Recreate image\n self.created_image = recreate_image(processed_image)\n # Save image\n if i % 5 == 0:\n im_path = '../generated/layer_vis_l' + str(self.selected_layer) + \\\n '_f' + str(self.selected_filter) + '_iter' + str(i) + '.jpg'\n save_image(self.created_image, im_path)\n\n def visualise_layer_without_hooks(self):\n # Process image and return variable\n # Generate a random image\n random_image = np.uint8(np.random.uniform(150, 180, (224, 224, 3)))\n # Process image and return variable\n processed_image = preprocess_image(random_image, False)\n # Define optimizer for the image\n optimizer = Adam([processed_image], lr=0.1, weight_decay=1e-6)\n for i in range(1, 31):\n optimizer.zero_grad()\n # Assign create image to a variable to move forward in the model\n x = processed_image\n for index, layer in enumerate(self.model):\n # Forward pass layer by layer\n x = layer(x)\n if index == self.selected_layer:\n # Only need to forward until the selected layer is reached\n # Now, x is the output of the selected layer\n break\n # Here, we get the specific filter from the output of the convolution operation\n # x is a tensor of shape 1x512x28x28.(For layer 17)\n # So there are 512 unique filter outputs\n # Following line selects a filter from 512 filters so self.conv_output will become\n # a tensor of shape 28x28\n self.conv_output = x[0, self.selected_filter]\n # Loss function is the mean of the output of the selected layer/filter\n # We try to minimize the mean of the output of that specific filter\n loss = -torch.mean(self.conv_output)\n print('Iteration:', str(i), 'Loss:', \"{0:.2f}\".format(loss.data.numpy()))\n # Backward\n loss.backward()\n # Update image\n optimizer.step()\n # Recreate image\n self.created_image = recreate_image(processed_image)\n # Save image\n if i % 5 == 0:\n im_path = '../generated/layer_vis_l' + str(self.selected_layer) + \\\n '_f' + str(self.selected_filter) + '_iter' + str(i) + '.jpg'\n save_image(self.created_image, im_path)\n\n\nif __name__ == '__main__':\n cnn_layer = 17\n filter_pos = 5\n # Fully connected layer is not needed\n pretrained_model = models.vgg16(pretrained=True).features\n layer_vis = CNNLayerVisualization(pretrained_model, cnn_layer, filter_pos)\n\n # Layer visualization with pytorch hooks\n layer_vis.visualise_layer_with_hooks()\n\n # Layer visualization without pytorch hooks\n # layer_vis.visualise_layer_without_hooks()\n" }, { "path": "src/deep_dream.py", "content": "\"\"\"\nCreated on Mon Nov 21 21:57:29 2017\n\n@author: Utku Ozbulak - github.com/utkuozbulak\n\"\"\"\nimport os\nfrom PIL import Image\n\nimport torch\nfrom torch.optim import SGD\nfrom torchvision import models\n\nfrom misc_functions import preprocess_image, recreate_image, save_image\n\n\nclass DeepDream():\n \"\"\"\n Produces an image that minimizes the loss of a convolution\n operation for a specific layer and filter\n \"\"\"\n def __init__(self, model, selected_layer, selected_filter, im_path):\n self.model = model\n self.model.eval()\n self.selected_layer = selected_layer\n self.selected_filter = selected_filter\n self.conv_output = 0\n # Generate a random image\n self.created_image = Image.open(im_path).convert('RGB')\n # Hook the layers to get result of the convolution\n self.hook_layer()\n # Create the folder to export images if not exists\n if not os.path.exists('../generated'):\n os.makedirs('../generated')\n\n def hook_layer(self):\n def hook_function(module, grad_in, grad_out):\n # Gets the conv output of the selected filter (from selected layer)\n self.conv_output = grad_out[0, self.selected_filter]\n\n # Hook the selected layer\n self.model[self.selected_layer].register_forward_hook(hook_function)\n\n def dream(self):\n # Process image and return variable\n self.processed_image = preprocess_image(self.created_image, True)\n # Define optimizer for the image\n # Earlier layers need higher learning rates to visualize whereas layer layers need less\n optimizer = SGD([self.processed_image], lr=12, weight_decay=1e-4)\n for i in range(1, 251):\n optimizer.zero_grad()\n # Assign create image to a variable to move forward in the model\n x = self.processed_image\n for index, layer in enumerate(self.model):\n # Forward\n x = layer(x)\n # Only need to forward until we the selected layer is reached\n if index == self.selected_layer:\n break\n # Loss function is the mean of the output of the selected layer/filter\n # We try to minimize the mean of the output of that specific filter\n loss = -torch.mean(self.conv_output)\n print('Iteration:', str(i), 'Loss:', \"{0:.2f}\".format(loss.data.numpy()))\n # Backward\n loss.backward()\n # Update image\n optimizer.step()\n # Recreate image\n self.created_image = recreate_image(self.processed_image)\n # Save image every 20 iteration\n if i % 10 == 0:\n print(self.created_image.shape)\n im_path = '../generated/ddream_l' + str(self.selected_layer) + \\\n '_f' + str(self.selected_filter) + '_iter' + str(i) + '.jpg'\n save_image(self.created_image, im_path)\n\n\nif __name__ == '__main__':\n # THIS OPERATION IS MEMORY HUNGRY! #\n # Because of the selected image is very large\n # If it gives out of memory error or locks the computer\n # Try it with a smaller image\n cnn_layer = 34\n filter_pos = 94\n\n im_path = '../input_images/dd_tree.png'\n # Fully connected layer is not needed\n pretrained_model = models.vgg19(pretrained=True).features\n dd = DeepDream(pretrained_model, cnn_layer, filter_pos, im_path)\n # This operation can also be done without Pytorch hooks\n # See layer visualisation for the implementation without hooks\n dd.dream()\n" }, { "path": "src/generate_class_specific_samples.py", "content": "\"\"\"\nCreated on Thu Oct 26 14:19:44 2017\n\n@author: Utku Ozbulak - github.com/utkuozbulak\n\"\"\"\nimport os\nimport numpy as np\n\nimport torch\nfrom torch.optim import SGD\nfrom torchvision import models\n\nfrom misc_functions import preprocess_image, recreate_image, save_image\n\n\nclass ClassSpecificImageGeneration():\n \"\"\"\n Produces an image that maximizes a certain class with gradient ascent\n \"\"\"\n def __init__(self, model, target_class):\n self.mean = [-0.485, -0.456, -0.406]\n self.std = [1/0.229, 1/0.224, 1/0.225]\n self.model = model\n self.model.eval()\n self.target_class = target_class\n # Generate a random image\n self.created_image = np.uint8(np.random.uniform(0, 255, (224, 224, 3)))\n # Create the folder to export images if not exists\n if not os.path.exists('../generated/class_'+str(self.target_class)):\n os.makedirs('../generated/class_'+str(self.target_class))\n\n def generate(self, iterations=150):\n \"\"\"Generates class specific image\n\n Keyword Arguments:\n iterations {int} -- Total iterations for gradient ascent (default: {150})\n\n Returns:\n np.ndarray -- Final maximally activated class image\n \"\"\"\n initial_learning_rate = 6\n for i in range(1, iterations):\n # Process image and return variable\n self.processed_image = preprocess_image(self.created_image, False)\n\n # Define optimizer for the image\n optimizer = SGD([self.processed_image], lr=initial_learning_rate)\n # Forward\n output = self.model(self.processed_image)\n # Target specific class\n class_loss = -output[0, self.target_class]\n\n if i % 10 == 0 or i == iterations-1:\n print('Iteration:', str(i), 'Loss',\n \"{0:.2f}\".format(class_loss.data.numpy()))\n # Zero grads\n self.model.zero_grad()\n # Backward\n class_loss.backward()\n # Update image\n optimizer.step()\n # Recreate image\n self.created_image = recreate_image(self.processed_image)\n if i % 10 == 0 or i == iterations-1:\n # Save image\n im_path = '../generated/class_'+str(self.target_class)+'/c_'+str(self.target_class)+'_'+'iter_'+str(i)+'.png'\n save_image(self.created_image, im_path)\n\n return self.processed_image\n\n\nif __name__ == '__main__':\n target_class = 130 # Flamingo\n pretrained_model = models.alexnet(pretrained=True)\n csig = ClassSpecificImageGeneration(pretrained_model, target_class)\n csig.generate()\n" }, { "path": "src/generate_regularized_class_specific_samples.py", "content": "\"\"\"\nCreated on Tues Mar 10 08:13:15 2020\n@author: Alex Stoken - https://github.com/alexstoken\n\nLast tested with torchvision 0.5.0 with image and model on cpu\n\"\"\"\nimport os\nimport numpy as np\nfrom PIL import Image, ImageFilter\n\nimport torch\nfrom torch.optim import SGD\nfrom torch.autograd import Variable\nfrom torchvision import models\n\nfrom misc_functions import recreate_image, save_image\n\nuse_cuda = torch.cuda.is_available()\n\nclass RegularizedClassSpecificImageGeneration():\n \"\"\"\n Produces an image that maximizes a certain class with gradient ascent. Uses Gaussian blur, weight decay, and clipping. \n \"\"\"\n\n def __init__(self, model, target_class):\n self.mean = [-0.485, -0.456, -0.406]\n self.std = [1/0.229, 1/0.224, 1/0.225]\n self.model = model.cuda() if use_cuda else model\n self.model.eval()\n self.target_class = target_class\n # Generate a random image\n self.created_image = np.uint8(np.random.uniform(0, 255, (224, 224, 3)))\n # Create the folder to export images if not exists\n if not os.path.exists(f'../generated/class_{self.target_class}'):\n os.makedirs(f'../generated/class_{self.target_class}')\n\n def generate(self, iterations=150, blur_freq=4, blur_rad=1, wd=0.0001, clipping_value=0.1):\n \"\"\"Generates class specific image with enhancements to improve image quality. \n See https://arxiv.org/abs/1506.06579 for details on each argument's effect on output quality. \n \n\n Play around with combinations of arguments. Besides the defaults, this combination has produced good images:\n blur_freq=6, blur_rad=0.8, wd = 0.05\n\n Keyword Arguments:\n iterations {int} -- Total iterations for gradient ascent (default: {150})\n blur_freq {int} -- Frequency of Gaussian blur effect, in iterations (default: {6})\n blur_rad {float} -- Radius for gaussian blur, passed to PIL.ImageFilter.GaussianBlur() (default: {0.8})\n wd {float} -- Weight decay value for Stochastic Gradient Ascent (default: {0.05})\n clipping_value {None or float} -- Value for gradient clipping (default: {0.1})\n \n Returns:\n np.ndarray -- Final maximally activated class image\n \"\"\"\n initial_learning_rate = 6\n for i in range(1, iterations):\n # Process image and return variable\n\n #implement gaussian blurring every ith iteration\n #to improve output\n if i % blur_freq == 0:\n self.processed_image = preprocess_and_blur_image(\n self.created_image, False, blur_rad)\n else:\n self.processed_image = preprocess_and_blur_image(\n self.created_image, False)\n\n if use_cuda:\n self.processed_image = self.processed_image.cuda()\n\n # Define optimizer for the image - use weight decay to add regularization\n # in SGD, wd = 2 * L2 regularization (https://bbabenko.github.io/weight-decay/)\n optimizer = SGD([self.processed_image],\n lr=initial_learning_rate, weight_decay=wd)\n # Forward\n output = self.model(self.processed_image)\n # Target specific class\n class_loss = -output[0, self.target_class]\n\n if i in np.linspace(0, iterations, 10, dtype=int):\n print('Iteration:', str(i), 'Loss',\n \"{0:.2f}\".format(class_loss.data.cpu().numpy()))\n # Zero grads\n self.model.zero_grad()\n # Backward\n class_loss.backward()\n\n if clipping_value:\n torch.nn.utils.clip_grad_norm(\n self.model.parameters(), clipping_value)\n # Update image\n optimizer.step()\n # Recreate image\n self.created_image = recreate_image(self.processed_image.cpu())\n\n if i in np.linspace(0, iterations, 10, dtype=int):\n # Save image\n im_path = f'../generated/class_{self.target_class}/c_{self.target_class}_iter_{i}_loss_{class_loss.data.cpu().numpy()}.jpg'\n save_image(self.created_image, im_path)\n\n #save final image\n im_path = f'../generated/class_{self.target_class}/c_{self.target_class}_iter_{i}_loss_{class_loss.data.cpu().numpy()}.jpg'\n save_image(self.created_image, im_path)\n\n #write file with regularization details\n with open(f'../generated/class_{self.target_class}/run_details.txt', 'w') as f:\n f.write(f'Iterations: {iterations}\\n')\n f.write(f'Blur freq: {blur_freq}\\n')\n f.write(f'Blur radius: {blur_rad}\\n')\n f.write(f'Weight decay: {wd}\\n')\n f.write(f'Clip value: {clipping_value}\\n')\n\n #rename folder path with regularization details for easy access\n os.rename(f'../generated/class_{self.target_class}',\n f'../generated/class_{self.target_class}_blurfreq_{blur_freq}_blurrad_{blur_rad}_wd{wd}')\n return self.processed_image\n\n\ndef preprocess_and_blur_image(pil_im, resize_im=True, blur_rad=None):\n \"\"\"\n Processes image with optional Gaussian blur for CNNs\n\n Args:\n PIL_img (PIL_img): PIL Image or numpy array to process\n resize_im (bool): Resize to 224 or not\n blur_rad (int): Pixel radius for Gaussian blurring (default = None)\n returns:\n im_as_var (torch variable): Variable that contains processed float tensor\n \"\"\"\n # mean and std list for channels (Imagenet)\n mean = [0.485, 0.456, 0.406]\n std = [0.229, 0.224, 0.225]\n\n #ensure or transform incoming image to PIL image\n if type(pil_im) != Image.Image:\n try:\n pil_im = Image.fromarray(pil_im)\n except Exception as e:\n print(\n \"could not transform PIL_img to a PIL Image object. Please check input.\")\n\n # Resize image\n if resize_im:\n pil_im.thumbnail((224, 224))\n\n #add gaussin blur to image\n if blur_rad:\n pil_im = pil_im.filter(ImageFilter.GaussianBlur(blur_rad))\n\n im_as_arr = np.float32(pil_im)\n im_as_arr = im_as_arr.transpose(2, 0, 1) # Convert array to D,W,H\n # Normalize the channels\n for channel, _ in enumerate(im_as_arr):\n im_as_arr[channel] /= 255\n im_as_arr[channel] -= mean[channel]\n im_as_arr[channel] /= std[channel]\n # Convert to float tensor\n im_as_ten = torch.from_numpy(im_as_arr).float()\n # Add one more channel to the beginning. Tensor shape = 1,3,224,224\n im_as_ten.unsqueeze_(0)\n # Convert to Pytorch variable\n if use_cuda:\n im_as_var = Variable(im_as_ten.cuda(), requires_grad=True)\n else:\n im_as_var = Variable(im_as_ten, requires_grad=True)\n return im_as_var\n\nif __name__ == '__main__':\n target_class = 130 # Flamingo\n pretrained_model = models.alexnet(pretrained=True)\n csig = RegularizedClassSpecificImageGeneration(pretrained_model, target_class)\n csig.generate()\n" }, { "path": "src/grad_times_image.py", "content": "\"\"\"\nCreated on Wed Jun 19 17:12:04 2019\n\n@author: Utku Ozbulak - github.com/utkuozbulak\n\"\"\"\nfrom misc_functions import (get_example_params,\n convert_to_grayscale,\n save_gradient_images)\nfrom vanilla_backprop import VanillaBackprop\n# from guided_backprop import GuidedBackprop # To use with guided backprop\n# from integrated_gradients import IntegratedGradients # To use with integrated grads\n\nif __name__ == '__main__':\n # Get params\n target_example = 0 # Snake\n (original_image, prep_img, target_class, file_name_to_export, pretrained_model) =\\\n get_example_params(target_example)\n # Vanilla backprop\n VBP = VanillaBackprop(pretrained_model)\n # Generate gradients\n vanilla_grads = VBP.generate_gradients(prep_img, target_class)\n\n # Make sure dimensions add up!\n grad_times_image = vanilla_grads * prep_img.detach().numpy()[0]\n # Convert to grayscale\n grayscale_vanilla_grads = convert_to_grayscale(grad_times_image)\n # Save grayscale gradients\n save_gradient_images(grayscale_vanilla_grads,\n file_name_to_export + '_Vanilla_grad_times_image_gray')\n print('Grad times image completed.')\n" }, { "path": "src/gradcam.py", "content": "\"\"\"\nCreated on Thu Oct 26 11:06:51 2017\n\n@author: Utku Ozbulak - github.com/utkuozbulak\n\"\"\"\nfrom PIL import Image\nimport numpy as np\nimport torch\n\nfrom misc_functions import get_example_params, save_class_activation_images\n\n\nclass CamExtractor():\n \"\"\"\n Extracts cam features from the model\n \"\"\"\n def __init__(self, model, target_layer):\n self.model = model\n self.target_layer = target_layer\n self.gradients = None\n\n def save_gradient(self, grad):\n self.gradients = grad\n\n def forward_pass_on_convolutions(self, x):\n \"\"\"\n Does a forward pass on convolutions, hooks the function at given layer\n \"\"\"\n conv_output = None\n for module_pos, module in self.model.features._modules.items():\n x = module(x) # Forward\n if int(module_pos) == self.target_layer:\n x.register_hook(self.save_gradient)\n conv_output = x # Save the convolution output on that layer\n return conv_output, x\n\n def forward_pass(self, x):\n \"\"\"\n Does a full forward pass on the model\n \"\"\"\n # Forward pass on the convolutions\n conv_output, x = self.forward_pass_on_convolutions(x)\n x = x.view(x.size(0), -1) # Flatten\n # Forward pass on the classifier\n x = self.model.classifier(x)\n return conv_output, x\n\n\nclass GradCam():\n \"\"\"\n Produces class activation map\n \"\"\"\n def __init__(self, model, target_layer):\n self.model = model\n self.model.eval()\n # Define extractor\n self.extractor = CamExtractor(self.model, target_layer)\n\n def generate_cam(self, input_image, target_class=None):\n # Full forward pass\n # conv_output is the output of convolutions at specified layer\n # model_output is the final output of the model (1, 1000)\n conv_output, model_output = self.extractor.forward_pass(input_image)\n if target_class is None:\n target_class = np.argmax(model_output.data.numpy())\n # Target for backprop\n one_hot_output = torch.FloatTensor(1, model_output.size()[-1]).zero_()\n one_hot_output[0][target_class] = 1\n # Zero grads\n self.model.features.zero_grad()\n self.model.classifier.zero_grad()\n # Backward pass with specified target\n model_output.backward(gradient=one_hot_output, retain_graph=True)\n # Get hooked gradients\n guided_gradients = self.extractor.gradients.data.numpy()[0]\n # Get convolution outputs\n target = conv_output.data.numpy()[0]\n # Get weights from gradients\n weights = np.mean(guided_gradients, axis=(1, 2)) # Take averages for each gradient\n # Create empty numpy array for cam\n cam = np.ones(target.shape[1:], dtype=np.float32)\n # Have a look at issue #11 to check why the above is np.ones and not np.zeros\n # Multiply each weight with its conv output and then, sum\n for i, w in enumerate(weights):\n cam += w * target[i, :, :]\n cam = np.maximum(cam, 0)\n cam = (cam - np.min(cam)) / (np.max(cam) - np.min(cam)) # Normalize between 0-1\n cam = np.uint8(cam * 255) # Scale between 0-255 to visualize\n cam = np.uint8(Image.fromarray(cam).resize((input_image.shape[2],\n input_image.shape[3]), Image.ANTIALIAS))/255\n # ^ I am extremely unhappy with this line. Originally resizing was done in cv2 which\n # supports resizing numpy matrices with antialiasing, however,\n # when I moved the repository to PIL, this option was out of the window.\n # So, in order to use resizing with ANTIALIAS feature of PIL,\n # I briefly convert matrix to PIL image and then back.\n # If there is a more beautiful way, do not hesitate to send a PR.\n\n # You can also use the code below instead of the code line above, suggested by @ ptschandl\n # from scipy.ndimage.interpolation import zoom\n # cam = zoom(cam, np.array(input_image[0].shape[1:])/np.array(cam.shape))\n return cam\n\n\nif __name__ == '__main__':\n # Get params\n target_example = 0 # Snake\n (original_image, prep_img, target_class, file_name_to_export, pretrained_model) =\\\n get_example_params(target_example)\n # Grad cam\n grad_cam = GradCam(pretrained_model, target_layer=11)\n # Generate cam mask\n cam = grad_cam.generate_cam(prep_img, target_class)\n # Save mask\n save_class_activation_images(original_image, cam, file_name_to_export)\n print('Grad cam completed')\n" }, { "path": "src/guided_backprop.py", "content": "\"\"\"\nCreated on Thu Oct 26 11:23:47 2017\n\n@author: Utku Ozbulak - github.com/utkuozbulak\n\"\"\"\nimport torch\nfrom torch.nn import ReLU\n\nfrom misc_functions import (get_example_params,\n convert_to_grayscale,\n save_gradient_images,\n get_positive_negative_saliency)\n\n\nclass GuidedBackprop():\n \"\"\"\n Produces gradients generated with guided back propagation from the given image\n \"\"\"\n def __init__(self, model):\n self.model = model\n self.gradients = None\n self.forward_relu_outputs = []\n # Put model in evaluation mode\n self.model.eval()\n self.update_relus()\n self.hook_layers()\n\n def hook_layers(self):\n def hook_function(module, grad_in, grad_out):\n self.gradients = grad_in[0]\n # Register hook to the first layer\n first_layer = list(self.model.features._modules.items())[0][1]\n first_layer.register_backward_hook(hook_function)\n\n def update_relus(self):\n \"\"\"\n Updates relu activation functions so that\n 1- stores output in forward pass\n 2- imputes zero for gradient values that are less than zero\n \"\"\"\n def relu_backward_hook_function(module, grad_in, grad_out):\n \"\"\"\n If there is a negative gradient, change it to zero\n \"\"\"\n # Get last forward output\n corresponding_forward_output = self.forward_relu_outputs[-1]\n corresponding_forward_output[corresponding_forward_output > 0] = 1\n modified_grad_out = corresponding_forward_output * torch.clamp(grad_in[0], min=0.0)\n del self.forward_relu_outputs[-1] # Remove last forward output\n return (modified_grad_out,)\n\n def relu_forward_hook_function(module, ten_in, ten_out):\n \"\"\"\n Store results of forward pass\n \"\"\"\n self.forward_relu_outputs.append(ten_out)\n\n # Loop through layers, hook up ReLUs\n for pos, module in self.model.features._modules.items():\n if isinstance(module, ReLU):\n module.register_backward_hook(relu_backward_hook_function)\n module.register_forward_hook(relu_forward_hook_function)\n\n def generate_gradients(self, input_image, target_class):\n # Forward pass\n model_output = self.model(input_image)\n # Zero gradients\n self.model.zero_grad()\n # Target for backprop\n one_hot_output = torch.FloatTensor(1, model_output.size()[-1]).zero_()\n one_hot_output[0][target_class] = 1\n # Backward pass\n model_output.backward(gradient=one_hot_output)\n # Convert Pytorch variable to numpy array\n # [0] to get rid of the first channel (1,3,224,224)\n gradients_as_arr = self.gradients.data.numpy()[0]\n return gradients_as_arr\n\n\nif __name__ == '__main__':\n target_example = 0 # Snake\n (original_image, prep_img, target_class, file_name_to_export, pretrained_model) =\\\n get_example_params(target_example)\n\n # Guided backprop\n GBP = GuidedBackprop(pretrained_model)\n # Get gradients\n guided_grads = GBP.generate_gradients(prep_img, target_class)\n # Save colored gradients\n save_gradient_images(guided_grads, file_name_to_export + '_Guided_BP_color')\n # Convert to grayscale\n grayscale_guided_grads = convert_to_grayscale(guided_grads)\n # Save grayscale gradients\n save_gradient_images(grayscale_guided_grads, file_name_to_export + '_Guided_BP_gray')\n # Positive and negative saliency maps\n pos_sal, neg_sal = get_positive_negative_saliency(guided_grads)\n save_gradient_images(pos_sal, file_name_to_export + '_pos_sal')\n save_gradient_images(neg_sal, file_name_to_export + '_neg_sal')\n print('Guided backprop completed')\n" }, { "path": "src/guided_gradcam.py", "content": "\"\"\"\nCreated on Thu Oct 23 11:27:15 2017\n\n@author: Utku Ozbulak - github.com/utkuozbulak\n\"\"\"\nimport numpy as np\n\nfrom misc_functions import (get_example_params,\n convert_to_grayscale,\n save_gradient_images)\nfrom gradcam import GradCam\nfrom guided_backprop import GuidedBackprop\n\n\ndef guided_grad_cam(grad_cam_mask, guided_backprop_mask):\n \"\"\"\n Guided grad cam is just pointwise multiplication of cam mask and\n guided backprop mask\n\n Args:\n grad_cam_mask (np_arr): Class activation map mask\n guided_backprop_mask (np_arr):Guided backprop mask\n \"\"\"\n cam_gb = np.multiply(grad_cam_mask, guided_backprop_mask)\n return cam_gb\n\n\nif __name__ == '__main__':\n # Get params\n target_example = 0 # Snake\n (original_image, prep_img, target_class, file_name_to_export, pretrained_model) =\\\n get_example_params(target_example)\n\n # Grad cam\n gcv2 = GradCam(pretrained_model, target_layer=11)\n # Generate cam mask\n cam = gcv2.generate_cam(prep_img, target_class)\n print('Grad cam completed')\n\n # Guided backprop\n GBP = GuidedBackprop(pretrained_model)\n # Get gradients\n guided_grads = GBP.generate_gradients(prep_img, target_class)\n print('Guided backpropagation completed')\n\n # Guided Grad cam\n cam_gb = guided_grad_cam(cam, guided_grads)\n save_gradient_images(cam_gb, file_name_to_export + '_GGrad_Cam')\n grayscale_cam_gb = convert_to_grayscale(cam_gb)\n save_gradient_images(grayscale_cam_gb, file_name_to_export + '_GGrad_Cam_gray')\n print('Guided grad cam completed')\n" }, { "path": "src/integrated_gradients.py", "content": "\"\"\"\nCreated on Wed Jun 19 17:06:48 2019\n\n@author: Utku Ozbulak - github.com/utkuozbulak\n\"\"\"\nimport torch\nimport numpy as np\n\nfrom misc_functions import get_example_params, convert_to_grayscale, save_gradient_images\n\n\nclass IntegratedGradients():\n \"\"\"\n Produces gradients generated with integrated gradients from the image\n \"\"\"\n def __init__(self, model):\n self.model = model\n self.gradients = None\n # Put model in evaluation mode\n self.model.eval()\n # Hook the first layer to get the gradient\n self.hook_layers()\n\n def hook_layers(self):\n def hook_function(module, grad_in, grad_out):\n self.gradients = grad_in[0]\n\n # Register hook to the first layer\n first_layer = list(self.model.features._modules.items())[0][1]\n first_layer.register_backward_hook(hook_function)\n\n def generate_images_on_linear_path(self, input_image, steps):\n # Generate uniform numbers between 0 and steps\n step_list = np.arange(steps+1)/steps\n # Generate scaled xbar images\n xbar_list = [input_image*step for step in step_list]\n return xbar_list\n\n def generate_gradients(self, input_image, target_class):\n # Forward\n model_output = self.model(input_image)\n # Zero grads\n self.model.zero_grad()\n # Target for backprop\n one_hot_output = torch.FloatTensor(1, model_output.size()[-1]).zero_()\n one_hot_output[0][target_class] = 1\n # Backward pass\n model_output.backward(gradient=one_hot_output)\n # Convert Pytorch variable to numpy array\n # [0] to get rid of the first channel (1,3,224,224)\n gradients_as_arr = self.gradients.data.numpy()[0]\n return gradients_as_arr\n\n def generate_integrated_gradients(self, input_image, target_class, steps):\n # Generate xbar images\n xbar_list = self.generate_images_on_linear_path(input_image, steps)\n # Initialize an iamge composed of zeros\n integrated_grads = np.zeros(input_image.size())\n for xbar_image in xbar_list:\n # Generate gradients from xbar images\n single_integrated_grad = self.generate_gradients(xbar_image, target_class)\n # Add rescaled grads from xbar images\n integrated_grads = integrated_grads + single_integrated_grad/steps\n # [0] to get rid of the first channel (1,3,224,224)\n return integrated_grads[0]\n\n\nif __name__ == '__main__':\n # Get params\n target_example = 0 # Snake\n (original_image, prep_img, target_class, file_name_to_export, pretrained_model) =\\\n get_example_params(target_example)\n # Vanilla backprop\n IG = IntegratedGradients(pretrained_model)\n # Generate gradients\n integrated_grads = IG.generate_integrated_gradients(prep_img, target_class, 100)\n # Convert to grayscale\n grayscale_integrated_grads = convert_to_grayscale(integrated_grads)\n # Save grayscale gradients\n save_gradient_images(grayscale_integrated_grads, file_name_to_export + '_Integrated_G_gray')\n print('Integrated gradients completed.')\n" }, { "path": "src/inverted_representation.py", "content": "\"\"\"\nCreated on Wed Jan 17 08:05:11 2018\n\n@author: Utku Ozbulak - github.com/utkuozbulak\n\"\"\"\nimport torch\nfrom torch.autograd import Variable\nfrom torch.optim import SGD\nimport os\n\nfrom misc_functions import get_example_params, recreate_image, save_image\n\n\nclass InvertedRepresentation():\n def __init__(self, model):\n self.model = model\n self.model.eval()\n if not os.path.exists('../generated'):\n os.makedirs('../generated')\n\n def alpha_norm(self, input_matrix, alpha):\n \"\"\"\n Converts matrix to vector then calculates the alpha norm\n \"\"\"\n alpha_norm = ((input_matrix.view(-1))**alpha).sum()\n return alpha_norm\n\n def total_variation_norm(self, input_matrix, beta):\n \"\"\"\n Total variation norm is the second norm in the paper\n represented as R_V(x)\n \"\"\"\n to_check = input_matrix[:, :-1, :-1] # Trimmed: right - bottom\n one_bottom = input_matrix[:, 1:, :-1] # Trimmed: top - right\n one_right = input_matrix[:, :-1, 1:] # Trimmed: top - right\n total_variation = (((to_check - one_bottom)**2 +\n (to_check - one_right)**2)**(beta/2)).sum()\n return total_variation\n\n def euclidian_loss(self, org_matrix, target_matrix):\n \"\"\"\n Euclidian loss is the main loss function in the paper\n ||fi(x) - fi(x_0)||_2^2& / ||fi(x_0)||_2^2\n \"\"\"\n distance_matrix = target_matrix - org_matrix\n euclidian_distance = self.alpha_norm(distance_matrix, 2)\n normalized_euclidian_distance = euclidian_distance / self.alpha_norm(org_matrix, 2)\n return normalized_euclidian_distance\n\n def get_output_from_specific_layer(self, x, layer_id):\n \"\"\"\n Saves the output after a forward pass until nth layer\n This operation could be done with a forward hook too\n but this one is simpler (I think)\n \"\"\"\n layer_output = None\n for index, layer in enumerate(self.model.features):\n x = layer(x)\n if str(index) == str(layer_id):\n layer_output = x[0]\n break\n return layer_output\n\n def generate_inverted_image_specific_layer(self, input_image, img_size, target_layer=3):\n # Generate a random image which we will optimize\n opt_img = Variable(1e-1 * torch.randn(1, 3, img_size, img_size), requires_grad=True)\n # Define optimizer for previously created image\n optimizer = SGD([opt_img], lr=1e4, momentum=0.9)\n # Get the output from the model after a forward pass until target_layer\n # with the input image (real image, NOT the randomly generated one)\n input_image_layer_output = \\\n self.get_output_from_specific_layer(input_image, target_layer)\n\n # Alpha regularization parametrs\n # Parameter alpha, which is actually sixth norm\n alpha_reg_alpha = 6\n # The multiplier, lambda alpha\n alpha_reg_lambda = 1e-7\n\n # Total variation regularization parameters\n # Parameter beta, which is actually second norm\n tv_reg_beta = 2\n # The multiplier, lambda beta\n tv_reg_lambda = 1e-8\n\n for i in range(201):\n optimizer.zero_grad()\n # Get the output from the model after a forward pass until target_layer\n # with the generated image (randomly generated one, NOT the real image)\n output = self.get_output_from_specific_layer(opt_img, target_layer)\n # Calculate euclidian loss\n euc_loss = 1e-1 * self.euclidian_loss(input_image_layer_output.detach(), output)\n # Calculate alpha regularization\n reg_alpha = alpha_reg_lambda * self.alpha_norm(opt_img, alpha_reg_alpha)\n # Calculate total variation regularization\n reg_total_variation = tv_reg_lambda * self.total_variation_norm(opt_img,\n tv_reg_beta)\n # Sum all to optimize\n loss = euc_loss + reg_alpha + reg_total_variation\n # Step\n loss.backward()\n optimizer.step()\n # Generate image every 5 iterations\n if i % 5 == 0:\n print('Iteration:', str(i), 'Loss:', loss.data.numpy())\n recreated_im = recreate_image(opt_img)\n im_path = '../generated/Inv_Image_Layer_' + str(target_layer) + \\\n '_Iteration_' + str(i) + '.jpg'\n save_image(recreated_im, im_path)\n\n # Reduce learning rate every 40 iterations\n if i % 40 == 0:\n for param_group in optimizer.param_groups:\n param_group['lr'] *= 1/10\n\n\nif __name__ == '__main__':\n # Get params\n target_example = 0 # Snake\n (original_image, prep_img, target_class, file_name_to_export, pretrained_model) =\\\n get_example_params(target_example)\n\n inverted_representation = InvertedRepresentation(pretrained_model)\n image_size = 224 # width & height\n target_layer = 4\n inverted_representation.generate_inverted_image_specific_layer(prep_img,\n image_size,\n target_layer)\n" }, { "path": "src/layer_activation_with_guided_backprop.py", "content": "\"\"\"\nCreated on Thu Oct 26 11:23:47 2017\n\n@author: Utku Ozbulak - github.com/utkuozbulak\n\"\"\"\nimport torch\nfrom torch.nn import ReLU\n\nfrom misc_functions import (get_example_params,\n convert_to_grayscale,\n save_gradient_images,\n get_positive_negative_saliency)\n\n\nclass GuidedBackprop():\n \"\"\"\n Produces gradients generated with guided back propagation from the given image\n \"\"\"\n def __init__(self, model):\n self.model = model\n self.gradients = None\n self.forward_relu_outputs = []\n # Put model in evaluation mode\n self.model.eval()\n self.update_relus()\n self.hook_layers()\n\n def hook_layers(self):\n def hook_function(module, grad_in, grad_out):\n self.gradients = grad_in[0]\n # Register hook to the first layer\n first_layer = list(self.model.features._modules.items())[0][1]\n first_layer.register_backward_hook(hook_function)\n\n def update_relus(self):\n \"\"\"\n Updates relu activation functions so that\n 1- stores output in forward pass\n 2- imputes zero for gradient values that are less than zero\n \"\"\"\n def relu_backward_hook_function(module, grad_in, grad_out):\n \"\"\"\n If there is a negative gradient, change it to zero\n \"\"\"\n # Get last forward output\n corresponding_forward_output = self.forward_relu_outputs[-1]\n corresponding_forward_output[corresponding_forward_output > 0] = 1\n modified_grad_out = corresponding_forward_output * torch.clamp(grad_in[0], min=0.0)\n del self.forward_relu_outputs[-1] # Remove last forward output\n return (modified_grad_out,)\n\n def relu_forward_hook_function(module, ten_in, ten_out):\n \"\"\"\n Store results of forward pass\n \"\"\"\n self.forward_relu_outputs.append(ten_out)\n\n # Loop through layers, hook up ReLUs\n for pos, module in self.model.features._modules.items():\n if isinstance(module, ReLU):\n module.register_backward_hook(relu_backward_hook_function)\n module.register_forward_hook(relu_forward_hook_function)\n\n def generate_gradients(self, input_image, target_class, cnn_layer, filter_pos):\n self.model.zero_grad()\n # Forward pass\n x = input_image\n for index, layer in enumerate(self.model.features):\n # Forward pass layer by layer\n # x is not used after this point because it is only needed to trigger\n # the forward hook function\n x = layer(x)\n # Only need to forward until the selected layer is reached\n if index == cnn_layer:\n # (forward hook function triggered)\n break\n conv_output = torch.sum(torch.abs(x[0, filter_pos]))\n # Backward pass\n conv_output.backward()\n # Convert Pytorch variable to numpy array\n # [0] to get rid of the first channel (1,3,224,224)\n gradients_as_arr = self.gradients.data.numpy()[0]\n return gradients_as_arr\n\n\nif __name__ == '__main__':\n cnn_layer = 10\n filter_pos = 5\n target_example = 2 # Spider\n (original_image, prep_img, target_class, file_name_to_export, pretrained_model) =\\\n get_example_params(target_example)\n\n # File export name\n file_name_to_export = file_name_to_export + '_layer' + str(cnn_layer) + '_filter' + str(filter_pos)\n # Guided backprop\n GBP = GuidedBackprop(pretrained_model)\n # Get gradients\n guided_grads = GBP.generate_gradients(prep_img, target_class, cnn_layer, filter_pos)\n # Save colored gradients\n save_gradient_images(guided_grads, file_name_to_export + '_Guided_BP_color')\n # Convert to grayscale\n grayscale_guided_grads = convert_to_grayscale(guided_grads)\n # Save grayscale gradients\n save_gradient_images(grayscale_guided_grads, file_name_to_export + '_Guided_BP_gray')\n # Positive and negative saliency maps\n pos_sal, neg_sal = get_positive_negative_saliency(guided_grads)\n save_gradient_images(pos_sal, file_name_to_export + '_pos_sal')\n save_gradient_images(neg_sal, file_name_to_export + '_neg_sal')\n print('Layer Guided backprop completed')\n" }, { "path": "src/layercam.py", "content": "\"\"\"\nCreated on Mon Jul 5 12:39:11 2021\n\n@author: Peng-Tao Jiang - github.com/PengtaoJiang\n\"\"\"\nfrom PIL import Image\nimport numpy as np\nimport torch\n\nfrom misc_functions import get_example_params, save_class_activation_images\n\n\nclass CamExtractor():\n \"\"\"\n Extracts cam features from the model\n \"\"\"\n def __init__(self, model, target_layer):\n self.model = model\n self.target_layer = target_layer\n self.gradients = None\n\n def save_gradient(self, grad):\n self.gradients = grad\n\n def forward_pass_on_convolutions(self, x):\n \"\"\"\n Does a forward pass on convolutions, hooks the function at given layer\n \"\"\"\n conv_output = None\n for module_pos, module in self.model.features._modules.items():\n x = module(x) # Forward\n if int(module_pos) == self.target_layer:\n x.register_hook(self.save_gradient)\n conv_output = x # Save the convolution output on that layer\n return conv_output, x\n\n def forward_pass(self, x):\n \"\"\"\n Does a full forward pass on the model\n \"\"\"\n # Forward pass on the convolutions\n conv_output, x = self.forward_pass_on_convolutions(x)\n x = x.view(x.size(0), -1) # Flatten\n # Forward pass on the classifier\n x = self.model.classifier(x)\n return conv_output, x\n\n\nclass LayerCam():\n \"\"\"\n Produces class activation map\n \"\"\"\n def __init__(self, model, target_layer):\n self.model = model\n self.model.eval()\n # Define extractor\n self.extractor = CamExtractor(self.model, target_layer)\n\n def generate_cam(self, input_image, target_class=None):\n # Full forward pass\n # conv_output is the output of convolutions at specified layer\n # model_output is the final output of the model (1, 1000)\n conv_output, model_output = self.extractor.forward_pass(input_image)\n if target_class is None:\n target_class = np.argmax(model_output.data.numpy())\n # Target for backprop\n one_hot_output = torch.FloatTensor(1, model_output.size()[-1]).zero_()\n one_hot_output[0][target_class] = 1\n # Zero grads\n self.model.features.zero_grad()\n self.model.classifier.zero_grad()\n # Backward pass with specified target\n model_output.backward(gradient=one_hot_output, retain_graph=True)\n # Get hooked gradients\n guided_gradients = self.extractor.gradients.data.numpy()[0]\n # Get convolution outputs\n target = conv_output.data.numpy()[0]\n # Get weights from gradients\n weights = guided_gradients\n weights[weights < 0] = 0 # discard negative gradients\n # Element-wise multiply the weight with its conv output and then, sum\n cam = np.sum(weights * target, axis=0)\n cam = (cam - np.min(cam)) / (np.max(cam) - np.min(cam)) # Normalize between 0-1\n cam = np.uint8(cam * 255) # Scale between 0-255 to visualize\n cam = np.uint8(Image.fromarray(cam).resize((input_image.shape[2],\n input_image.shape[3]), Image.ANTIALIAS))/255\n\n return cam\n\n\nif __name__ == '__main__':\n # Get params\n target_example = 0 # Snake\n (original_image, prep_img, target_class, file_name_to_export, pretrained_model) =\\\n get_example_params(target_example)\n # Layer cam\n layer_cam = LayerCam(pretrained_model, target_layer=9)\n # Generate cam mask\n cam = layer_cam.generate_cam(prep_img, target_class)\n # Save mask\n save_class_activation_images(original_image, cam, file_name_to_export)\n print('Layer cam completed')\n" }, { "path": "src/misc_functions.py", "content": "\"\"\"\nCreated on Thu Oct 21 11:09:09 2017\n\n@author: Utku Ozbulak - github.com/utkuozbulak\n\"\"\"\nimport os\nimport copy\nimport numpy as np\nfrom PIL import Image\nimport matplotlib.cm as mpl_color_map\nfrom matplotlib.colors import ListedColormap\nfrom matplotlib import pyplot as plt\n\nimport torch\nfrom torch.autograd import Variable\nfrom torchvision import models\n\n\ndef convert_to_grayscale(im_as_arr):\n \"\"\"\n Converts 3d image to grayscale\n\n Args:\n im_as_arr (numpy arr): RGB image with shape (D,W,H)\n\n returns:\n grayscale_im (numpy_arr): Grayscale image with shape (1,W,D)\n \"\"\"\n grayscale_im = np.sum(np.abs(im_as_arr), axis=0)\n im_max = np.percentile(grayscale_im, 99)\n im_min = np.min(grayscale_im)\n grayscale_im = (np.clip((grayscale_im - im_min) / (im_max - im_min), 0, 1))\n grayscale_im = np.expand_dims(grayscale_im, axis=0)\n return grayscale_im\n\n\ndef save_gradient_images(gradient, file_name):\n \"\"\"\n Exports the original gradient image\n\n Args:\n gradient (np arr): Numpy array of the gradient with shape (3, 224, 224)\n file_name (str): File name to be exported\n \"\"\"\n if not os.path.exists('../results'):\n os.makedirs('../results')\n # Normalize\n gradient = gradient - gradient.min()\n gradient /= gradient.max()\n # Save image\n path_to_file = os.path.join('../results', file_name + '.png')\n save_image(gradient, path_to_file)\n\n\ndef save_class_activation_images(org_img, activation_map, file_name):\n \"\"\"\n Saves cam activation map and activation map on the original image\n\n Args:\n org_img (PIL img): Original image\n activation_map (numpy arr): Activation map (grayscale) 0-255\n file_name (str): File name of the exported image\n \"\"\"\n if not os.path.exists('../results'):\n os.makedirs('../results')\n # Grayscale activation map\n heatmap, heatmap_on_image = apply_colormap_on_image(org_img, activation_map, 'hsv')\n # Save colored heatmap\n path_to_file = os.path.join('../results', file_name+'_Cam_Heatmap.png')\n save_image(heatmap, path_to_file)\n # Save heatmap on iamge\n path_to_file = os.path.join('../results', file_name+'_Cam_On_Image.png')\n save_image(heatmap_on_image, path_to_file)\n # SAve grayscale heatmap\n path_to_file = os.path.join('../results', file_name+'_Cam_Grayscale.png')\n save_image(activation_map, path_to_file)\n\n\ndef apply_colormap_on_image(org_im, activation, colormap_name):\n \"\"\"\n Apply heatmap on image\n Args:\n org_img (PIL img): Original image\n activation_map (numpy arr): Activation map (grayscale) 0-255\n colormap_name (str): Name of the colormap\n \"\"\"\n # Get colormap\n color_map = mpl_color_map.get_cmap(colormap_name)\n no_trans_heatmap = color_map(activation)\n # Change alpha channel in colormap to make sure original image is displayed\n heatmap = copy.copy(no_trans_heatmap)\n heatmap[:, :, 3] = 0.4\n heatmap = Image.fromarray((heatmap*255).astype(np.uint8))\n no_trans_heatmap = Image.fromarray((no_trans_heatmap*255).astype(np.uint8))\n\n # Apply heatmap on image\n heatmap_on_image = Image.new(\"RGBA\", org_im.size)\n heatmap_on_image = Image.alpha_composite(heatmap_on_image, org_im.convert('RGBA'))\n heatmap_on_image = Image.alpha_composite(heatmap_on_image, heatmap)\n return no_trans_heatmap, heatmap_on_image\n\n\ndef apply_heatmap(R, sx, sy):\n \"\"\"\n Heatmap code stolen from https://git.tu-berlin.de/gmontavon/lrp-tutorial\n\n This is (so far) only used for LRP\n \"\"\"\n b = 10*((np.abs(R)**3.0).mean()**(1.0/3))\n my_cmap = plt.cm.seismic(np.arange(plt.cm.seismic.N))\n my_cmap[:, 0:3] *= 0.85\n my_cmap = ListedColormap(my_cmap)\n plt.figure(figsize=(sx, sy))\n plt.subplots_adjust(left=0, right=1, bottom=0, top=1)\n plt.axis('off')\n heatmap = plt.imshow(R, cmap=my_cmap, vmin=-b, vmax=b, interpolation='nearest')\n return heatmap\n # plt.show()\n\n\ndef format_np_output(np_arr):\n \"\"\"\n This is a (kind of) bandaid fix to streamline saving procedure.\n It converts all the outputs to the same format which is 3xWxH\n with using sucecssive if clauses.\n Args:\n im_as_arr (Numpy array): Matrix of shape 1xWxH or WxH or 3xWxH\n \"\"\"\n # Phase/Case 1: The np arr only has 2 dimensions\n # Result: Add a dimension at the beginning\n if len(np_arr.shape) == 2:\n np_arr = np.expand_dims(np_arr, axis=0)\n # Phase/Case 2: Np arr has only 1 channel (assuming first dim is channel)\n # Result: Repeat first channel and convert 1xWxH to 3xWxH\n if np_arr.shape[0] == 1:\n np_arr = np.repeat(np_arr, 3, axis=0)\n # Phase/Case 3: Np arr is of shape 3xWxH\n # Result: Convert it to WxHx3 in order to make it saveable by PIL\n if np_arr.shape[0] == 3:\n np_arr = np_arr.transpose(1, 2, 0)\n # Phase/Case 4: NP arr is normalized between 0-1\n # Result: Multiply with 255 and change type to make it saveable by PIL\n if np.max(np_arr) <= 1:\n np_arr = (np_arr*255).astype(np.uint8)\n return np_arr\n\n\ndef save_image(im, path):\n \"\"\"\n Saves a numpy matrix or PIL image as an image\n Args:\n im_as_arr (Numpy array): Matrix of shape DxWxH\n path (str): Path to the image\n \"\"\"\n if isinstance(im, (np.ndarray, np.generic)):\n im = format_np_output(im)\n im = Image.fromarray(im)\n im.save(path)\n\n\ndef preprocess_image(pil_im, resize_im=True):\n \"\"\"\n Processes image for CNNs\n\n Args:\n PIL_img (PIL_img): PIL Image or numpy array to process\n resize_im (bool): Resize to 224 or not\n returns:\n im_as_var (torch variable): Variable that contains processed float tensor\n \"\"\"\n # Mean and std list for channels (Imagenet)\n mean = [0.485, 0.456, 0.406]\n std = [0.229, 0.224, 0.225]\n\n # Ensure or transform incoming image to PIL image\n if type(pil_im) != Image.Image:\n try:\n pil_im = Image.fromarray(pil_im)\n except Exception as e:\n print(\"could not transform PIL_img to a PIL Image object. Please check input.\")\n\n # Resize image\n if resize_im:\n pil_im = pil_im.resize((224, 224), Image.ANTIALIAS)\n\n im_as_arr = np.float32(pil_im)\n im_as_arr = im_as_arr.transpose(2, 0, 1) # Convert array to D,W,H\n # Normalize the channels\n for channel, _ in enumerate(im_as_arr):\n im_as_arr[channel] /= 255\n im_as_arr[channel] -= mean[channel]\n im_as_arr[channel] /= std[channel]\n # Convert to float tensor\n im_as_ten = torch.from_numpy(im_as_arr).float()\n # Add one more channel to the beginning. Tensor shape = 1,3,224,224\n im_as_ten.unsqueeze_(0)\n # Convert to Pytorch variable\n im_as_var = Variable(im_as_ten, requires_grad=True)\n return im_as_var\n\n\ndef recreate_image(im_as_var):\n \"\"\"\n Recreates images from a torch variable, sort of reverse preprocessing\n Args:\n im_as_var (torch variable): Image to recreate\n returns:\n recreated_im (numpy arr): Recreated image in array\n \"\"\"\n reverse_mean = [-0.485, -0.456, -0.406]\n reverse_std = [1/0.229, 1/0.224, 1/0.225]\n recreated_im = copy.copy(im_as_var.data.numpy()[0])\n for c in range(3):\n recreated_im[c] /= reverse_std[c]\n recreated_im[c] -= reverse_mean[c]\n recreated_im[recreated_im > 1] = 1\n recreated_im[recreated_im < 0] = 0\n recreated_im = np.round(recreated_im * 255)\n\n recreated_im = np.uint8(recreated_im).transpose(1, 2, 0)\n return recreated_im\n\n\ndef get_positive_negative_saliency(gradient):\n \"\"\"\n Generates positive and negative saliency maps based on the gradient\n Args:\n gradient (numpy arr): Gradient of the operation to visualize\n\n returns:\n pos_saliency ( )\n \"\"\"\n pos_saliency = (np.maximum(0, gradient) / gradient.max())\n neg_saliency = (np.maximum(0, -gradient) / -gradient.min())\n return pos_saliency, neg_saliency\n\n\ndef get_example_params(example_index):\n \"\"\"\n Gets used variables for almost all visualizations, like the image, model etc.\n\n Args:\n example_index (int): Image id to use from examples\n\n returns:\n original_image (numpy arr): Original image read from the file\n prep_img (numpy_arr): Processed image\n target_class (int): Target class for the image\n file_name_to_export (string): File name to export the visualizations\n pretrained_model(Pytorch model): Model to use for the operations\n \"\"\"\n # Pick one of the examples\n example_list = (('../input_images/snake.png', 56),\n ('../input_images/cat_dog.png', 243),\n ('../input_images/spider.png', 72))\n img_path = example_list[example_index][0]\n target_class = example_list[example_index][1]\n file_name_to_export = img_path[img_path.rfind('/')+1:img_path.rfind('.')]\n # Read image\n original_image = Image.open(img_path).convert('RGB')\n # Process image\n prep_img = preprocess_image(original_image)\n # Define model\n pretrained_model = models.alexnet(pretrained=True)\n return (original_image,\n prep_img,\n target_class,\n file_name_to_export,\n pretrained_model)\n" }, { "path": "src/scorecam.py", "content": "\"\"\"\nCreated on Wed Apr 29 16:11:20 2020\n\n@author: Haofan Wang - github.com/haofanwang\n\"\"\"\nfrom PIL import Image\nimport numpy as np\nimport torch\nimport torch.nn.functional as F\n\nfrom misc_functions import get_example_params, save_class_activation_images\n\n\nclass CamExtractor():\n \"\"\"\n Extracts cam features from the model\n \"\"\"\n def __init__(self, model, target_layer):\n self.model = model\n self.target_layer = target_layer\n\n def forward_pass_on_convolutions(self, x):\n \"\"\"\n Does a forward pass on convolutions, hooks the function at given layer\n \"\"\"\n conv_output = None\n for module_pos, module in self.model.features._modules.items():\n x = module(x) # Forward\n if int(module_pos) == self.target_layer:\n conv_output = x # Save the convolution output on that layer\n return conv_output, x\n\n def forward_pass(self, x):\n \"\"\"\n Does a full forward pass on the model\n \"\"\"\n # Forward pass on the convolutions\n conv_output, x = self.forward_pass_on_convolutions(x)\n x = x.view(x.size(0), -1) # Flatten\n # Forward pass on the classifier\n x = self.model.classifier(x)\n return conv_output, x\n\n\nclass ScoreCam():\n \"\"\"\n Produces class activation map\n \"\"\"\n def __init__(self, model, target_layer):\n self.model = model\n self.model.eval()\n # Define extractor\n self.extractor = CamExtractor(self.model, target_layer)\n\n def generate_cam(self, input_image, target_class=None):\n # Full forward pass\n # conv_output is the output of convolutions at specified layer\n # model_output is the final output of the model (1, 1000)\n conv_output, model_output = self.extractor.forward_pass(input_image)\n if target_class is None:\n target_class = np.argmax(model_output.data.numpy())\n # Get convolution outputs\n target = conv_output[0]\n # Create empty numpy array for cam\n cam = np.ones(target.shape[1:], dtype=np.float32)\n # Multiply each weight with its conv output and then, sum\n for i in range(len(target)):\n # Unsqueeze to 4D\n saliency_map = torch.unsqueeze(torch.unsqueeze(target[i, :, :],0),0)\n # Upsampling to input size\n saliency_map = F.interpolate(saliency_map, size=(224, 224), mode='bilinear', align_corners=False)\n if saliency_map.max() == saliency_map.min():\n continue\n # Scale between 0-1\n norm_saliency_map = (saliency_map - saliency_map.min()) / (saliency_map.max() - saliency_map.min())\n # Get the target score\n w = F.softmax(self.extractor.forward_pass(input_image*norm_saliency_map)[1],dim=1)[0][target_class]\n cam += w.data.numpy() * target[i, :, :].data.numpy()\n cam = np.maximum(cam, 0)\n cam = (cam - np.min(cam)) / (np.max(cam) - np.min(cam)) # Normalize between 0-1\n cam = np.uint8(cam * 255) # Scale between 0-255 to visualize\n cam = np.uint8(Image.fromarray(cam).resize((input_image.shape[2],\n input_image.shape[3]), Image.ANTIALIAS))/255\n return cam\n\n\nif __name__ == '__main__':\n # Get params\n target_example = 0 # Snake\n (original_image, prep_img, target_class, file_name_to_export, pretrained_model) =\\\n get_example_params(target_example)\n # Score cam\n score_cam = ScoreCam(pretrained_model, target_layer=11)\n # Generate cam mask\n cam = score_cam.generate_cam(prep_img, target_class)\n # Save mask\n save_class_activation_images(original_image, cam, file_name_to_export)\n print('Score cam completed')\n" }, { "path": "src/smooth_grad.py", "content": "\"\"\"\nCreated on Wed Mar 28 10:12:13 2018\n\n@author: Utku Ozbulak - github.com/utkuozbulak\n\"\"\"\nimport numpy as np\n\nfrom torch.autograd import Variable\nimport torch\n\nfrom misc_functions import (get_example_params,\n convert_to_grayscale,\n save_gradient_images)\nfrom vanilla_backprop import VanillaBackprop\n# from guided_backprop import GuidedBackprop # To use with guided backprop\n\n\ndef generate_smooth_grad(Backprop, prep_img, target_class, param_n, param_sigma_multiplier):\n \"\"\"\n Generates smooth gradients of given Backprop type. You can use this with both vanilla\n and guided backprop\n Args:\n Backprop (class): Backprop type\n prep_img (torch Variable): preprocessed image\n target_class (int): target class of imagenet\n param_n (int): Amount of images used to smooth gradient\n param_sigma_multiplier (int): Sigma multiplier when calculating std of noise\n \"\"\"\n # Generate an empty image/matrix\n smooth_grad = np.zeros(prep_img.size()[1:])\n\n mean = 0\n sigma = param_sigma_multiplier / (torch.max(prep_img) - torch.min(prep_img)).item()\n for x in range(param_n):\n # Generate noise\n noise = Variable(prep_img.data.new(prep_img.size()).normal_(mean, sigma**2))\n # Add noise to the image\n noisy_img = prep_img + noise\n # Calculate gradients\n vanilla_grads = Backprop.generate_gradients(noisy_img, target_class)\n # Add gradients to smooth_grad\n smooth_grad = smooth_grad + vanilla_grads\n # Average it out\n smooth_grad = smooth_grad / param_n\n return smooth_grad\n\n\nif __name__ == '__main__':\n # Get params\n target_example = 0 # Snake\n (original_image, prep_img, target_class, file_name_to_export, pretrained_model) =\\\n get_example_params(target_example)\n\n VBP = VanillaBackprop(pretrained_model)\n # GBP = GuidedBackprop(pretrained_model) # if you want to use GBP dont forget to\n # change the parametre in generate_smooth_grad\n\n param_n = 50\n param_sigma_multiplier = 4\n smooth_grad = generate_smooth_grad(VBP, # ^This parameter\n prep_img,\n target_class,\n param_n,\n param_sigma_multiplier)\n\n # Save colored gradients\n save_gradient_images(smooth_grad, file_name_to_export + '_SmoothGrad_color')\n # Convert to grayscale\n grayscale_smooth_grad = convert_to_grayscale(smooth_grad)\n # Save grayscale gradients\n save_gradient_images(grayscale_smooth_grad, file_name_to_export + '_SmoothGrad_gray')\n print('Smooth grad completed')\n" }, { "path": "src/vanilla_backprop.py", "content": "\"\"\"\nCreated on Thu Oct 26 11:19:58 2017\n\n@author: Utku Ozbulak - github.com/utkuozbulak\n\"\"\"\nimport torch\n\nfrom misc_functions import get_example_params, convert_to_grayscale, save_gradient_images\n\n\nclass VanillaBackprop():\n \"\"\"\n Produces gradients generated with vanilla back propagation from the image\n \"\"\"\n def __init__(self, model):\n self.model = model\n self.gradients = None\n # Put model in evaluation mode\n self.model.eval()\n # Hook the first layer to get the gradient\n self.hook_layers()\n\n def hook_layers(self):\n def hook_function(module, grad_in, grad_out):\n self.gradients = grad_in[0]\n\n # Register hook to the first layer\n first_layer = list(self.model.features._modules.items())[0][1]\n first_layer.register_backward_hook(hook_function)\n\n def generate_gradients(self, input_image, target_class):\n # Forward\n model_output = self.model(input_image)\n # Zero grads\n self.model.zero_grad()\n # Target for backprop\n one_hot_output = torch.FloatTensor(1, model_output.size()[-1]).zero_()\n one_hot_output[0][target_class] = 1\n # Backward pass\n model_output.backward(gradient=one_hot_output)\n # Convert Pytorch variable to numpy array\n # [0] to get rid of the first channel (1,3,224,224)\n gradients_as_arr = self.gradients.data.numpy()[0]\n return gradients_as_arr\n\n\nif __name__ == '__main__':\n # Get params\n target_example = 1 # Snake\n (original_image, prep_img, target_class, file_name_to_export, pretrained_model) =\\\n get_example_params(target_example)\n # Vanilla backprop\n VBP = VanillaBackprop(pretrained_model)\n # Generate gradients\n vanilla_grads = VBP.generate_gradients(prep_img, target_class)\n # Save colored gradients\n save_gradient_images(vanilla_grads, file_name_to_export + '_Vanilla_BP_color')\n # Convert to grayscale\n grayscale_vanilla_grads = convert_to_grayscale(vanilla_grads)\n # Save grayscale gradients\n save_gradient_images(grayscale_vanilla_grads, file_name_to_export + '_Vanilla_BP_gray')\n print('Vanilla backprop completed')\n" } ]